Prerequisites and scoped previews
Candidate retrieval and ANN indexes
Separate eligible-corpus recall from latency/memory and filtering effects.
Retrieval and ranking evaluation
Define recall/precision at K, MRR, DCG/NDCG, relevance labels and query/user weighting.
Training-serving consistency
Version feature/transform/model contracts and compare training versus serving behavior.
Registry, validation, rollout and rollback
Promote compatible artifact bundles through checks, shadow/canary and rollback gates.
Scoped prerequisite preview. The complete chapter is planned; this preview does not count as authored coverage.
Governance, privacy, lineage and deletion
Trace permitted data use through snapshots, features, indexes and artifacts; define deletion obligations.
RAG and generative-system evaluation
Separate retrieval, grounding, generation, tool/action and safety evaluation.
Scoped prerequisite preview. The complete chapter is planned; this preview does not count as authored coverage.
What, when, why and how
RAG is a data product and an authorization system
- what
- Retrieval-augmented generation combines source ingestion, search, evidence selection and model generation. The model is not the source of truth; the authoritative source, version, provenance and permission policy remain outside the model.
- when
- Use RAG when answers depend on changing private or domain-specific knowledge that should remain retrievable and attributable rather than being copied into model weights.
- why
- A fluent answer can still be stale, unsupported or unauthorized. Retrieval quality, data freshness, access control and citation correctness are independent system properties.
- how
- Define the user decision, tenant boundary, permitted sources, freshness target, answer/abstention contract and audit requirements first. Build a deterministic retrieval baseline before choosing an embedding model or vector database.
Separate offline ingestion from online query serving
- what
- The offline plane discovers, parses, chunks, embeds, validates and publishes document versions. The online plane authenticates, authorizes, retrieves, reranks, assembles context and generates a response.
- when
- Use this separation when corpus updates are slower and burstier than user queries, or when backfills and model migrations must not interrupt serving.
- why
- Coupling ingestion to query serving makes large imports compete with interactive traffic and exposes partially written indexes.
- how
- Use durable jobs and bounded workers for ingestion, versioned staging and an active manifest for publication. Keep query reads on the last known-good active generation while changes are prepared.
Authorization must precede evidence exposure
- what
- Tenant and document permissions constrain which candidates may be hydrated, reranked, placed in prompts, cached or cited.
- when
- Required for multi-tenant corpora, group-based access, document-level ACLs, sensitive content and changing memberships.
- why
- Filtering only after generation is too late: unauthorized text may already have reached a reranker, model provider, trace, cache or prompt log.
- how
- Authenticate the principal, resolve current policy, filter candidate IDs before fetching text, recheck before hydration, and fail closed when the authorization authority is unavailable. Treat vector metadata filters as candidate narrowing, not the sole security boundary.
Hybrid retrieval is a recall-and-precision pipeline
- what
- Dense semantic search captures conceptual similarity; sparse lexical search captures exact terms, identifiers and rare phrases. Fusion merges candidates and a reranker orders a bounded authorized set.
- when
- Use hybrid retrieval when users ask natural-language questions but also search for exact policy names, error codes, product IDs or quoted terms.
- why
- Dense-only retrieval can miss exact strings; lexical-only retrieval misses paraphrases. A reranker improves ordering but cannot recover a relevant item never retrieved.
- how
- Run bounded dense and sparse searches in parallel, fuse ranks (for example reciprocal-rank fusion), deduplicate stable chunk IDs, apply authorization, rerank a bounded set and measure recall and answer quality separately.
Versioning and publication protect live knowledge
- what
- A source version is distinct from a chunk version, an embedding model version and an index generation. An active manifest identifies exactly which validated corpus generation readers use.
- when
- Needed for initial corpus imports, document updates, parser changes, chunking changes, embedding-model migrations, rollback and audit.
- why
- A successful vector write does not prove a document is complete or queryable. Mixing incompatible embedding spaces or exposing partial updates can silently corrupt retrieval.
- how
- Write immutable staged versions, validate coverage/provenance/ACL/model dimensions, then advance an active manifest conditionally. Keep the previous generation available for rollback and clean it up asynchronously.
Grounding, evaluation and operations are part of the answer contract
- what
- The response pipeline validates structure, citations, source versions, claim support and policy; the operations plane measures freshness, retrieval quality, safety, latency and cost.
- when
- Use whenever generated output is shown to a user or may trigger a consequential action.
- why
- A citation can be syntactically valid but point to the wrong version; a model can invent a plausible answer when evidence is missing; aggregate quality hides tenant and document slices.
- how
- Return answer plus resolved citations and evidence metadata, or abstain. Evaluate retrieval and generation separately with versioned cases, human-reviewed judgments and operational guardrails.
Frame the product decision and clarify constraints
We design an internal enterprise knowledge assistant. An authenticated employee asks a question and receives a concise answer with citations to documents they are currently allowed to read. The system supports multiple tenants and source connectors. Search, source-of-truth document storage, identity management and the foundation-model provider are external dependencies with explicit contracts.
Assumed interview answers below are synthetic. Change them when the interviewer supplies different constraints; do not present them as universal defaults or actual customer traffic.
| Clarifying question | Synthetic assumption for this exercise | Design consequence |
|---|---|---|
| Who can ask questions? | Signed-in users in 20 isolated tenants. | Tenant identity is server-derived; all storage and cache keys are tenant-scoped. |
| What sources exist? | Object storage, a document drive and an internal wiki; 250,000 existing files at bootstrap. | Connector inventory, snapshot/change feed and periodic reconciliation are required. |
| How often does content change? | New/updated files continuously; permission changes must be enforced promptly. | Incremental jobs, versioned updates and a separate urgent revocation path. |
| What is the query load? | 250 queries/s peak, 1.5 s p95 target for a complete answer. | Bound retrieval, reranking, model time and total request deadline. |
| How fresh must knowledge be? | Normal content updates searchable within 10 minutes; revocations denied within seconds. | Measure source-to-index lag separately from revocation-to-denial latency. |
| What is the answer contract? | Cited answer when evidence is sufficient; otherwise clarify or abstain. | Citation validation, grounding checks and explicit no-answer behavior. |
| Can one tenant search another tenant's data? | Never, including through caches, logs, retrieval candidates or fallback. | Hard tenant isolation and adversarial cross-tenant tests. |
| What happens during model outage? | No fabricated answer; return a bounded degraded response or retry option. | Circuit breaker, provider timeout, and safe no-answer path. |
Define invariants, trust boundaries and non-goals
Hard invariants: no cross-tenant evidence; no unauthorized document text reaches a reranker or model; every chunk retains source/version/offset provenance; incomplete ingestion cannot replace a good active version; deleted or revoked content is denied before asynchronous physical cleanup finishes; citations resolve to evidence supplied to the current request; model output is untrusted until validated.
Separate identity authority, source authority, policy authority, index storage and model provider. The model must not decide whether the caller is allowed to read a source. Retrieved documents are untrusted data: they may contain prompt injection, misleading instructions or malformed content.
Non-goals for the first version: arbitrary agent actions, write-back to source systems, unrestricted web search, training a foundation model, and treating a model-generated summary as an authoritative document. Add these only with separate authorization and evaluation contracts.
Build a useful non-generative retrieval baseline
Start with authenticated keyword search over a small approved corpus. Apply tenant and ACL filters, return source title/snippet/updated time, and enforce document-level authorization before text is returned. This baseline validates connector identity, access policy, indexing, freshness, relevance and auditability without an LLM.
Add semantic retrieval only when the baseline's query set shows a measurable recall gap that embeddings can plausibly close. Add generation only when users benefit from synthesis across multiple authorized sources. This staged approach makes failures attributable: search relevance, permission filtering and answer generation are not one blended metric.
Model source identity, versions, chunks and publication
Keep transactional metadata as the control plane and the vector/lexical stores as derived retrieval indexes. The following logical records can live in relational metadata storage plus object storage for source and parser artifacts; exact physical placement depends on scale and connector semantics.
| Record | Key fields and constraints | Why it exists |
|---|---|---|
| tenant | tenant_id PK, status, policy_epoch | Isolation root and current tenant lifecycle. |
| source_connector | connector_id PK, tenant_id FK, connector_type, cursor, snapshot_token, status | Connector ownership and resumable discovery. |
| source_document | (tenant_id, connector_id, source_id) unique, current_source_version, active_document_version, tombstone_revision | Stable identity independent of path/name changes. |
| document_version | document_version_id PK, source_id FK, source_version/ETag, content_hash, acl_version, parser_version, state | Immutable snapshot and processing provenance. |
| ingestion_job | job_id PK, idempotency_key unique, desired_version, state, attempt, lease_until, checkpoint, error_code | Durable, retryable and deduplicated work. |
| chunk | chunk_id PK, document_version_id FK, ordinal, source_offsets, page/section, text_hash, acl_ref | Stable evidence identity and citation provenance. |
| embedding | chunk_id + model_version unique, vector_dimension, vector_ref, status | Prevents accidental mixing of embedding contracts. |
| index_generation | generation_id PK, tenant/corpus scope, embedding_model_version, dimension, schema_version, state | Immutable staged retrieval generation. |
| active_manifest | (tenant_id, corpus_id) PK, generation_id, revision, published_at | Reader-visible atomic pointer, advanced conditionally. |
| cleanup_task | task_id PK, target_version/tombstone_revision, stores_pending, status | Tracks asynchronous deletion and proof of cleanup. |
-- Illustrative relational metadata constraints; vector payloads may live in a vector index.
CREATE TABLE ingestion_job (
job_id UUID PRIMARY KEY,
tenant_id UUID NOT NULL,
connector_id UUID NOT NULL,
source_id TEXT NOT NULL,
source_version TEXT NOT NULL,
content_hash TEXT NOT NULL,
state TEXT NOT NULL,
attempt INTEGER NOT NULL DEFAULT 0,
checkpoint JSONB NOT NULL DEFAULT '{}',
UNIQUE (tenant_id, connector_id, source_id, source_version, content_hash)
);
CREATE TABLE active_manifest (
tenant_id UUID NOT NULL,
corpus_id UUID NOT NULL,
generation_id UUID NOT NULL,
revision BIGINT NOT NULL,
published_at TIMESTAMPTZ NOT NULL,
PRIMARY KEY (tenant_id, corpus_id)
);
-- Advance with compare-and-swap; zero updated rows means re-read and reconcile.
UPDATE active_manifest
SET generation_id = :validated_generation,
revision = revision + 1,
published_at = now()
WHERE tenant_id = :tenant_id
AND corpus_id = :corpus_id
AND revision = :expected_revision;Ingest documents: initial backfill and safe publication
Initial bootstrap is not the same as ongoing updates. Register each connector and tenant boundary, capture a source inventory, and establish a source snapshot or change cursor before scanning. Persist source identity, source version/ETag, content hash, modified time, ACL reference/version and deletion state. If the source supports snapshot tokens, use one so concurrent changes can be reconciled without gaps.
Backfill in bounded pages. A page checkpoint advances only after its source records have durable ingestion jobs. Each document job fetches the authoritative source snapshot, parses text/structure (OCR only when needed), normalizes while preserving provenance, chunks at safe semantic boundaries, attaches ACL metadata, embeds with a pinned model version, and writes to a staged generation.
Validate coverage, chunk counts, embedding dimensions/model identity, tenant isolation, ACL metadata, parser errors and sampled retrieval. Publish the initial generation only after validation passes. Readers must never see a half-built corpus. A vector database is not the system of record for source version or authorization.
Use durable job state, bounded retries, worker leases and idempotent writes. A parser/OCR poison document is quarantined with an operator-visible error; it must not silently become an empty document. Preserve a dead-letter/replay path with the source version and parser/model versions recorded.
VISUAL WALKTHROUGH
See the system in motion
Follow a source version through durable ingestion, provenance-preserving chunking, ACL metadata, embeddings and validated index publication.
Capture a tenant-scoped source version and content hash.
Create a durable idempotent ingestion job.
Parse and normalize while preserving source offsets.
Chunk with page/section provenance and authoritative ACL metadata.
Embed with a pinned model version.
Stage dense and sparse indexes.
Validate and publish the manifest.
Loading interactive walkthrough…
Existing production data, new files and continuous updates
When a RAG system already has production data, do not stop serving it to rebuild for every new file. Keep the current active manifest live. Run an initial inventory/backfill into a staged generation if migrating from an older index; otherwise process only the changed source document and its derived chunks.
Subscribe to webhooks or source change feeds where available, but treat events as hints rather than trusted content. Fetch the authoritative current version before processing. Run periodic reconciliation because webhooks can be delayed, duplicated, reordered or lost. Detect creates, updates, deletes, renames/moves and ACL changes. For a rename, preserve stable source identity rather than treating it as unrelated content.
Use an idempotency key derived from tenant, connector, stable source ID, source version and content hash. Duplicate deliveries converge on one job. An older event must not overwrite a newer desired source version: compare source cursors/revisions and reconcile with authoritative source state. Persist checkpoints only after durable downstream work is recorded.
For a normal content update, write new chunks under a new immutable document version, validate them, then conditionally advance the active document/version mapping. Keep the old version active until the new version is complete. Retain the previous version for a bounded rollback window, then delete superseded chunks asynchronously. Retrieval must filter to active versions so old and new chunks are not both returned.
Treat permission revocation and deletion differently from an ordinary content update. Write a durable tombstone or deny barrier first and make query-time authorization reject the source immediately. Physical vector, lexical, replica and cache cleanup follows asynchronously and is verified. Never keep access alive merely to preserve rollback convenience.
Separate bulk backfill capacity from online queries and high-priority incremental/revocation work. Use weighted queues, per-tenant quotas, bounded embedding QPS, index write limits and pause controls. If query latency, provider throttling or index saturation crosses a configured guardrail, slow bulk backfill before sacrificing interactive traffic.
| Change type | Safe behavior | What stays available |
|---|---|---|
| New file | Create version/job, parse/embed/stage, validate and publish its active mapping. | Existing corpus and other documents. |
| File updated | Build new version; swap active mapping only after validation; clean old chunks later. | Old complete version until new version is ready. |
| Duplicate/out-of-order event | Deduplicate by source/version/hash and compare authoritative source order. | No duplicate chunks; no stale overwrite. |
| Bulk import into existing service | Snapshot inventory, bounded backfill, checkpoint, stage and validate; throttle independently. | Current active index remains queryable. |
| ACL revoked / file deleted | Persist tombstone and deny queries immediately; invalidate caches; clean all stores asynchronously. | Other authorized documents; revoked source is not available. |
| Missed webhook / connector outage | Reconcile cursors and source inventory; expose freshness lag and preserve last good version where policy permits. | Last known-good data, clearly marked stale to operators. |
VISUAL WALKTHROUGH
See the system in motion
See how an existing production corpus is backfilled and then updated continuously while online queries stay available.
Inventory source IDs, versions, hashes and ACL references.
Backfill in bounded batches with durable checkpoints.
Validate and publish the baseline generation.
Detect create/update/delete/ACL events and reconcile missed changes.
Deduplicate events and reject stale source versions.
Stage only changed documents.
Conditionally publish validated versions.
Continue serving the active index.
Loading interactive walkthrough…
Hybrid search, rank fusion and reranking
Normalize the query without stripping exact identifiers. Generate a query embedding using the same pinned embedding contract as the active index. In parallel, run dense nearest-neighbor retrieval and sparse lexical/BM25 retrieval with tenant/corpus filters, bounded candidate counts and timeouts.
Fuse rankings using a documented method such as reciprocal-rank fusion: score(d) = sum over retrieval lists of 1 / (k + rank(d)). This avoids pretending dense and lexical raw scores are directly comparable. Deduplicate by stable chunk ID and retain source version, source offsets and retrieval provenance.
Apply current authorization before candidate text is passed to a cross-encoder reranker. Rerank a bounded set, select top-K with diversity and freshness constraints, and preserve citation IDs. If reranking times out, a policy-approved fallback may use fused order; do not send unauthorized candidates to the reranker merely to filter them afterward.
Measure candidate recall separately from reranker NDCG and answer quality. A reranker cannot recover documents that both retrieval methods missed. Evaluate exact identifiers, paraphrases, multilingual queries, stale documents, no-result queries, duplicate chunks and permission-filtered underfill.
VISUAL WALKTHROUGH
See the system in motion
Combine dense and lexical retrieval, fuse candidates, enforce ACLs and rerank a bounded authorized set.
Normalize query.
Run dense and sparse search in parallel.
Fuse and deduplicate candidates.
Apply current authorization.
Rerank bounded candidates.
Select diverse top-K evidence.
Loading interactive walkthrough…
Assemble evidence within a safe token budget
Start only from authorized, versioned evidence. Deduplicate overlapping chunks, preserve source spans, prefer authoritative and current documents, and select a diverse set rather than filling the context with near-identical passages. Allocate a token budget across system instructions, conversation history, retrieved evidence and reserved output tokens.
Use the target model tokenizer rather than character-count approximations. Truncate only at safe chunk boundaries and never cut away citation metadata. Record a context manifest containing evidence IDs, document versions, source spans, retrieval scores, prompt template version, model version and policy epoch. Apply retention and redaction rules; auditability does not justify indiscriminate storage of sensitive prompt text.
Separate trusted instructions from retrieved content. Mark document passages as untrusted evidence, neutralize unsafe formatting where appropriate, and instruct the model to treat document instructions as data rather than commands. Context assembly is a security boundary, not just prompt formatting.
VISUAL WALKTHROUGH
See the system in motion
Build compact, provenance-preserving context within a model token budget and keep retrieved content untrusted.
Collect authorized evidence.
Remove duplicate passages.
Allocate the model token budget.
Order evidence for relevance and coverage.
Build a structured prompt with untrusted evidence delimited.
Record the context manifest.
Loading interactive walkthrough…
Generate, validate citations and abstain safely
Call the model through a gateway with model allowlisting, tenant budgets, bounded input/output tokens, request deadlines, rate limits, retries, circuit breakers and provider fallback policy. The model returns a constrained response schema such as answer text, claim list, citation IDs and an abstention reason. Model output is untrusted.
Validate the output schema before rendering. Resolve each citation ID against the evidence set for this exact request and verify source version and offsets. Reject citations to documents not provided to the model or no longer authorized. Check material claims against evidence with deterministic rules where possible and an evaluation/judgment layer where semantic support must be assessed. A model-based judge is a signal, not proof.
If evidence is absent, contradictory, stale or insufficient, ask a clarifying question or abstain. Do not fill gaps with model memory while presenting the result as sourced. A response should distinguish direct evidence, inference and uncertainty. Log safe metadata for evaluation without leaking cross-tenant text.
VISUAL WALKTHROUGH
See the system in motion
Validate response structure, citations, claim support and policy before returning an answer or abstaining.
Generate a constrained draft.
Validate output schema.
Resolve citations to authorized source spans.
Check material claim support.
Apply safety and answer policy.
Return grounded answer or abstain.
Loading interactive walkthrough…
Permission changes, deletion and cache invalidation
On an ACL revoke or deletion event, verify authoritative source state and persist a monotonic tombstone/policy epoch before beginning cleanup. Query-time authorization consults current policy/tombstone state, so stale vector metadata cannot grant access. This is the immediate logical denial barrier.
Invalidate retrieval, context and answer caches using tenant/document/policy-version keys. Remove or supersede dense vectors, sparse postings, replicas, materialized summaries and derived artifacts according to retention/legal-hold policy. Track each cleanup acknowledgement and reconcile stores that missed the first delete. Retire the tombstone only after the required stores and replicas confirm cleanup and the policy allows it.
Distinguish access revocation from data deletion: a user losing access may require deny and cache invalidation but not necessarily destruction of a document still available to other authorized users. A source deletion may require broader removal. Define ownership, legal hold, backup retention and evidence of completion before promising physical erasure.
VISUAL WALKTHROUGH
See the system in motion
Deny revoked or deleted content immediately, then clean indexes and caches asynchronously and verify removal.
Verify source change.
Write durable tombstone.
Deny retrieval immediately.
Invalidate affected caches.
Delete or supersede indexed material.
Reconcile and verify removal.
Loading interactive walkthrough…
Evaluate retrieval, generation and the complete product separately
Build a versioned evaluation set from approved representative questions and document versions. Include exact lookup, paraphrase, multi-hop synthesis, no-answer, conflicting-source, stale-version, multilingual, prompt-injection and permission-boundary cases. Keep a holdout set that is not repeatedly tuned against.
Retrieval metrics include Recall@K, MRR/NDCG and authorized-evidence coverage. Generation metrics include claim support, citation precision/recall, answer completeness, abstention correctness and contradiction handling. Product metrics include task completion and user feedback. System metrics include latency, availability, provider errors, tokens, cost and freshness. Never collapse them into one unqualified score.
Run adversarial tests: a user guesses another tenant's document ID; a revoked group retains stale cache; an ACL changes during retrieval; a document contains instructions to exfiltrate secrets; a source update arrives out of order; a model returns an invented citation. Require tests to fail closed and record the evidence.
| Layer | Example measures | Failure hidden by aggregate score |
|---|---|---|
| Retrieval | Recall@K, MRR, NDCG, authorized candidate rate | Relevant evidence omitted or unauthorized candidates leak. |
| Generation | Claim support, citation precision, completeness, abstention accuracy | Fluent hallucination with invalid citations. |
| Security | Cross-tenant deny tests, ACL revoke latency, cache invalidation | One tenant's evidence leaks through shared cache. |
| Freshness | Source-to-index lag, stale-version query rate, reconcile backlog | New policy or corrected procedure is not searchable. |
| Operations | p95/p99 latency, provider timeout, queue depth, cost/query | Good offline quality with unacceptable live behavior. |
Capacity estimates and cost controls
Use explicit synthetic assumptions to show the method, not to claim an exact production bill. Assume 250,000 source documents, an average of six chunks per document, 1.5 million chunks, 1,536-dimensional float32 embeddings and a target 300 embedded chunks/s during an unconstrained backfill window.
Raw vector payload is 1,500,000 × 1,536 × 4 bytes = 9,216,000,000 bytes (about 9.2 GB decimal, or 8.6 GiB). This excludes ANN graph structures, lexical postings, metadata, replicas, object storage, write amplification and backups; actual provisioned capacity will be materially higher. At 300 chunks/s, the embedding stage alone has a theoretical lower-bound duration of 1,500,000 / 300 = 5,000 seconds (about 83 minutes), before source fetch, parsing, retries, throttling, validation and index compaction.
At 250 queries/s and 40 fused candidates per query, the reranker sees at most 10,000 candidate pairs/s before top-K selection. If 8 chunks are passed to a model per query, that is up to 2,000 chunk-context selections/s; token volume depends on chunk length and output limits. Measure actual token distributions and provider quotas before setting concurrency.
Cost levers: incremental rather than full re-embedding; deduplication; bounded candidate counts; smaller context; cache only under correct authorization/version keys; batch embeddings where provider semantics allow; separate backfill quotas; choose vector precision/index parameters using measured recall-latency trade-offs. Do not trade away permission correctness or freshness silently to lower cost.
End-to-end architecture and request lifecycle
Offline ingestion: connectors → source inventory/change feed → durable queue → bounded fetch/parse/OCR workers → normalized immutable document versions → chunk/provenance/ACL metadata → embedding workers → staged dense and sparse indexes → validation gate → conditional active-manifest publication → asynchronous cleanup/reconciliation.
Online query: authenticated client → API gateway → identity/tenant resolver → policy service → query normalization/embedding → parallel dense and sparse retrieval → rank fusion/deduplication → current ACL filter/recheck → reranker → context/token planner → guarded model gateway → schema/citation/grounding/policy validation → cited answer or abstention.
Shared control plane: source/version registry, job/checkpoint state, policy epochs/tombstones, active manifests, prompt/model/index versions, audit metadata, evaluation datasets and telemetry. Operational controls include per-tenant quotas, circuit breakers, queue prioritization, rollback and reconciliation.
Use the cumulative final animation to follow both planes together. A production answer must be explainable by the exact active index generation, document versions, policy epoch, prompt template and model version used for that request.
Failure modes, degraded behavior and recovery
Parser/OCR failure: bounded retry and quarantine; preserve source version and error. Embedding timeout/rate limit: respect provider retry hints, cap concurrency and resume from durable checkpoints. Partial vector write: keep old manifest active and retry idempotently or discard staging. Worker crash: recover expired leases. Missed webhook: periodic source reconciliation. Out-of-order update: authoritative version comparison. Stale ACL: immediate deny barrier and cache invalidation. Reranker timeout: use only an approved fused-order fallback. Model outage: circuit-break and abstain/retry rather than fabricate.
Index migration: build a parallel generation with a pinned embedding model/dimension, shadow-query and compare quality, then conditionally cut over. Never mix vectors from incompatible embedding spaces. Keep the old generation for bounded rollback, but current authorization and tombstones always override rollback.
Operational dashboards should show source-to-index lag, oldest queue age, backlog by tenant/connector, parse/embed/index errors, duplicate and out-of-order events, staged/active manifest revisions, orphan chunks, revocation-to-denial latency, deletion cleanup age, retrieval quality, p95/p99 latency, model error rate, tokens and cost.
Rollout, ownership and governance
Roll out connector by connector and tenant cohort. Start with shadow indexing and offline evaluation, then limited read-only search, then generated answers with citations. Compare against the deterministic search baseline. Gate promotion on retrieval quality, permission tests, freshness, latency, cost and safe abstention. Provide a kill switch that disables generation while preserving authorized search.
Assign owners for source connectors, identity/policy integration, ingestion/index operations, model gateway, evaluation, privacy/deletion and incident response. Define who can publish a new parser/chunker/embedding generation, who approves prompt/model changes and who verifies deletion completion.
Record versioned configuration and audit metadata. Retain only what is needed for debugging and compliance; redact secrets and sensitive source text. Review provider data handling, tenant isolation, retention, legal holds, user notice and model-provider terms. A trace is not permission to store every prompt forever.
Key design trade-offs and interview conclusion
Per-document active-version pointers make incremental replacement efficient but require retrieval to enforce active-version state across dense and sparse indexes. Whole-corpus generations simplify atomic cutover and embedding migrations but cost more storage and build time. A hybrid approach uses per-document version manifests for normal changes and parallel corpus generations for schema/model migrations.
Webhook/change-feed ingestion reduces detection delay but cannot be the only correctness mechanism; reconciliation repairs missed events. Vector metadata ACL filters improve retrieval efficiency but cannot replace authoritative query-time checks. Immediate logical tombstones make revocation prompt while physical deletion remains asynchronous and verifiable.
Conclude with the invariants and evidence: existing queries continue on a last-known-good index during backfill; new versions publish only after validation; unauthorized text never enters model context; revocation denies immediately; citations resolve to authorized source versions; and the system measures freshness, retrieval quality, safety, latency and cost independently.
Data model, schema and relationships
The logical schema separates tenant identity, principal authorization, stable source identity, immutable document versions, ACL revisions, chunks, index snapshots, durable ingestion jobs, answer audit and deletion propagation. The vector/lexical indexes are derived retrieval stores; relational metadata and authoritative identity/policy services control publication and access.
Implementation boundary: Architecture contract only. The schema is illustrative relational metadata plus object storage and vector/lexical indexes; no migrations, connector, embedding provider, authorization integration or live RAG service are installed. Cross-tenant compatibility, ACL freshness, active-version selection and multi-store cleanup require transactional checks and reconciliation.
These are proposed logical schemas with field contracts, not executable database migrations. PK identifies a row, FK references an existing parent, UK is a uniqueness rule. A composite PK identifies the combination. Nullable references are marked with ?. A foreign key protects identity; cross-row limits, timing and state transitions also require the transaction rules below.
Connectors name the relationship; multiplicities state parents per child and children per parent. 0..1 means optional, 1 required, 0..* any number. Self-links show two roles of the same entity.
Connectors name the relationship; multiplicities state parents per child and children per parent. 0..1 means optional, 1 required, 0..* any number. Self-links show two roles of the same entity.
Connectors name the relationship; multiplicities state parents per child and children per parent. 0..1 means optional, 1 required, 0..* any number. Self-links show two roles of the same entity.
Connectors name the relationship; multiplicities state parents per child and children per parent. 0..1 means optional, 1 required, 0..* any number. Self-links show two roles of the same entity.
Connectors name the relationship; multiplicities state parents per child and children per parent. 0..1 means optional, 1 required, 0..* any number. Self-links show two roles of the same entity.
Connectors name the relationship; multiplicities state parents per child and children per parent. 0..1 means optional, 1 required, 0..* any number. Self-links show two roles of the same entity.
Connectors name the relationship; multiplicities state parents per child and children per parent. 0..1 means optional, 1 required, 0..* any number. Self-links show two roles of the same entity.
Connectors name the relationship; multiplicities state parents per child and children per parent. 0..1 means optional, 1 required, 0..* any number. Self-links show two roles of the same entity.
Connectors name the relationship; multiplicities state parents per child and children per parent. 0..1 means optional, 1 required, 0..* any number. Self-links show two roles of the same entity.
Entity definitions and field contracts
Tenant
Isolation and quota boundary.
| Field | Type / key | Meaning and reference |
|---|---|---|
id | opaque_id · PK | Globally unique immutable row identity. |
status | enum · VALUE | active/suspended lifecycle. |
policy_epoch | integer · VALUE | Monotonic tenant policy revision. |
Code mapping: Logical production design contract; not an installed migration. Enforce tenant-compatible foreign keys and conditional state transitions transactionally.
- Tenant identity is derived from verified authentication; caller-supplied tenant IDs do not grant access.
Principal
Authenticated user or service identity.
| Field | Type / key | Meaning and reference |
|---|---|---|
id | opaque_id · PK | Globally unique immutable row identity. |
tenant_id | opaque_id · FK | Owning tenant. · tenant.id |
external_subject | string · VALUE | Stable identity-provider subject. |
state | enum · VALUE | active/disabled. |
Code mapping: Logical production design contract; not an installed migration. Enforce tenant-compatible foreign keys and conditional state transitions transactionally.
- UNIQUE(tenant_id,external_subject). Membership/group expansion comes from the identity authority.
Document
Stable connector-scoped source identity.
| Field | Type / key | Meaning and reference |
|---|---|---|
id | opaque_id · PK | Globally unique immutable row identity. |
tenant_id | opaque_id · FK | Owning tenant. · tenant.id |
connector_id | opaque_id · VALUE | Registered source connector. |
source_id | string · VALUE | Stable source ID, independent of path/name. |
active_version_id | opaque_id? · VALUE | Current validated document version pointer. |
tombstone_revision | integer · VALUE | Current deletion/revocation barrier. |
Code mapping: Logical production design contract; not an installed migration. Enforce tenant-compatible foreign keys and conditional state transitions transactionally.
- UNIQUE(tenant_id,connector_id,source_id). A path rename does not create a new logical document.
ACL version
Immutable permission snapshot/reference.
| Field | Type / key | Meaning and reference |
|---|---|---|
id | opaque_id · PK | Globally unique immutable row identity. |
document_id | opaque_id · FK | Protected document. · document.id |
revision | string · VALUE | Source/policy ACL revision. |
policy_ref | string · VALUE | Authoritative ACL reference. |
captured_at | timestamp · VALUE | Observation time. |
Code mapping: Logical production design contract; not an installed migration. Enforce tenant-compatible foreign keys and conditional state transitions transactionally.
- ACL snapshot is metadata for filtering only; current policy is rechecked before text hydration.
Document version
Immutable source content and parser identity.
| Field | Type / key | Meaning and reference |
|---|---|---|
id | opaque_id · PK | Globally unique immutable row identity. |
document_id | opaque_id · FK | Stable source identity. · document.id |
acl_version_id | opaque_id · FK | ACL revision associated with this source snapshot. · acl_version.id |
source_version | string · VALUE | ETag/revision from authoritative connector. |
content_hash | string · VALUE | Verified content digest. |
parser_version | string · VALUE | Parser/OCR implementation version. |
state | enum · VALUE | fetched/parsed/staged/validated/published/quarantined/superseded. |
Code mapping: Logical production design contract; not an installed migration. Enforce tenant-compatible foreign keys and conditional state transitions transactionally.
- UNIQUE(document_id,source_version,content_hash). Out-of-order older source versions cannot replace newer desired state.
Chunk
Retrieval unit with exact source provenance.
| Field | Type / key | Meaning and reference |
|---|---|---|
id | opaque_id · PK | Globally unique immutable row identity. |
document_version_id | opaque_id · FK | Immutable source version. · document_version.id |
ordinal | integer · VALUE | Stable ordinal within document version. |
source_span | json · VALUE | Page/section/offset provenance. |
text_hash | string · VALUE | Canonical chunk text digest. |
acl_ref | opaque_id · VALUE | ACL version used for candidate filtering. |
Code mapping: Logical production design contract; not an installed migration. Enforce tenant-compatible foreign keys and conditional state transitions transactionally.
- UNIQUE(document_version_id,ordinal). Hydrate text only after current authorization succeeds.
Index snapshot
Immutable compatible corpus/index generation.
| Field | Type / key | Meaning and reference |
|---|---|---|
id | opaque_id · PK | Globally unique immutable row identity. |
tenant_id | opaque_id · FK | Owning tenant. · tenant.id |
generation | string · VALUE | Immutable generation identity. |
embedding_model | string · VALUE | Pinned model/version. |
dimension | integer · VALUE | Vector dimension. |
schema_version | string · VALUE | Chunk/metadata contract. |
state | enum · VALUE | building/validated/active/retired/failed. |
Code mapping: Logical production design contract; not an installed migration. Enforce tenant-compatible foreign keys and conditional state transitions transactionally.
- Vectors from incompatible embedding spaces must not be mixed in one active generation.
Ingestion job
Durable idempotent per-source-version work.
| Field | Type / key | Meaning and reference |
|---|---|---|
id | opaque_id · PK | Globally unique immutable row identity. |
document_version_id | opaque_id · FK | Target immutable source version. · document_version.id |
idempotency_key | string · VALUE | Tenant/connector/source/version/hash identity. |
state | enum · VALUE | queued/fetching/parsing/chunking/embedding/staged/validated/published/retryable/quarantined. |
attempt | integer · VALUE | Bounded retry count. |
checkpoint | json · VALUE | Durable stage progress and lease metadata. |
Code mapping: Logical production design contract; not an installed migration. Enforce tenant-compatible foreign keys and conditional state transitions transactionally.
- UNIQUE(idempotency_key). Advance checkpoints only after durable outputs; recover expired worker leases.
Answer audit
Minimal reproducibility and answer decision record.
| Field | Type / key | Meaning and reference |
|---|---|---|
id | opaque_id · PK | Globally unique immutable row identity. |
principal_id | opaque_id · FK | Requesting principal. · principal.id |
index_snapshot_id | opaque_id · FK | Active index generation used. · index_snapshot.id |
policy_epoch | integer · VALUE | Authorization policy revision observed. |
prompt_version | string · VALUE | Prompt template release. |
model_version | string · VALUE | Model/provider identity. |
outcome | enum · VALUE | answered/clarified/abstained/blocked. |
Code mapping: Logical production design contract; not an installed migration. Enforce tenant-compatible foreign keys and conditional state transitions transactionally.
- Store minimum required metadata; avoid retaining sensitive prompt text by default.
Answer evidence
Citation link from one answer to its supplied chunk.
| Field | Type / key | Meaning and reference |
|---|---|---|
id | opaque_id · PK | Globally unique immutable row identity. |
answer_audit_id | opaque_id · FK | Answer decision. · answer_audit.id |
chunk_id | opaque_id · FK | Exact authorized evidence chunk. · chunk.id |
document_version_id | string · VALUE | Exact source version rendered in citation. |
source_span | json · VALUE | Exact page/section/offset citation. |
Code mapping: Logical production design contract; not an installed migration. Enforce tenant-compatible foreign keys and conditional state transitions transactionally.
- UNIQUE(answer_audit_id,chunk_id). Citation must belong to the current request evidence set and remain authorized.
Deletion request
Durable deny barrier and cleanup scope.
| Field | Type / key | Meaning and reference |
|---|---|---|
id | opaque_id · PK | Globally unique immutable row identity. |
document_id | opaque_id · FK | Target document. · document.id |
tombstone_revision | integer · VALUE | Monotonic deny revision. |
reason | enum · VALUE | source_deleted/permission_revoked/retention_request. |
state | enum · VALUE | denied/cleaning/verified/failed. |
Code mapping: Logical production design contract; not an installed migration. Enforce tenant-compatible foreign keys and conditional state transitions transactionally.
- Persist deny barrier before asynchronous index/cache cleanup. A rollback cannot override current tombstones.
Deletion acknowledgement
Per-store evidence of cleanup.
| Field | Type / key | Meaning and reference |
|---|---|---|
id | opaque_id · PK | Globally unique immutable row identity. |
deletion_request_id | opaque_id · FK | Owning deletion request. · deletion_request.id |
store | string · VALUE | Vector/lexical/cache/replica/object artifact target. |
acknowledged_at | timestamp? · VALUE | Cleanup confirmation time. |
status | enum · VALUE | pending/complete/retryable/blocked. |
Code mapping: Logical production design contract; not an installed migration. Enforce tenant-compatible foreign keys and conditional state transitions transactionally.
- Tombstone retirement requires all required stores/replicas to meet the declared deletion policy.
Relationship cardinalities and their reasons
| Parent → child | Multiplicity | FK / why |
|---|---|---|
| tenant → principal Tenant reference | 1 parent per child; 0..* children per parent | principal.tenant_id → tenant.idEach Principal row references one tenant row; tenant and lifecycle compatibility are enforced separately. |
| tenant → document Tenant reference | 1 parent per child; 0..* children per parent | document.tenant_id → tenant.idEach Document row references one tenant row; tenant and lifecycle compatibility are enforced separately. |
| document → acl_version Document reference | 1 parent per child; 0..* children per parent | acl_version.document_id → document.idEach ACL version row references one document row; tenant and lifecycle compatibility are enforced separately. |
| document → document_version Document reference | 1 parent per child; 0..* children per parent | document_version.document_id → document.idEach Document version row references one document row; tenant and lifecycle compatibility are enforced separately. |
| acl_version → document_version ACL version reference | 1 parent per child; 0..* children per parent | document_version.acl_version_id → acl_version.idEach Document version row references one acl version row; tenant and lifecycle compatibility are enforced separately. |
| document_version → chunk Document version reference | 1 parent per child; 0..* children per parent | chunk.document_version_id → document_version.idEach Chunk row references one document version row; tenant and lifecycle compatibility are enforced separately. |
| tenant → index_snapshot Tenant reference | 1 parent per child; 0..* children per parent | index_snapshot.tenant_id → tenant.idEach Index snapshot row references one tenant row; tenant and lifecycle compatibility are enforced separately. |
| document_version → ingestion_job Document version reference | 1 parent per child; 0..* children per parent | ingestion_job.document_version_id → document_version.idEach Ingestion job row references one document version row; tenant and lifecycle compatibility are enforced separately. |
| principal → answer_audit Principal reference | 1 parent per child; 0..* children per parent | answer_audit.principal_id → principal.idEach Answer audit row references one principal row; tenant and lifecycle compatibility are enforced separately. |
| index_snapshot → answer_audit Index snapshot reference | 1 parent per child; 0..* children per parent | answer_audit.index_snapshot_id → index_snapshot.idEach Answer audit row references one index snapshot row; tenant and lifecycle compatibility are enforced separately. |
| answer_audit → answer_evidence Answer audit reference | 1 parent per child; 0..* children per parent | answer_evidence.answer_audit_id → answer_audit.idEach Answer evidence row references one answer audit row; tenant and lifecycle compatibility are enforced separately. |
| chunk → answer_evidence Chunk reference | 1 parent per child; 0..* children per parent | answer_evidence.chunk_id → chunk.idEach Answer evidence row references one chunk row; tenant and lifecycle compatibility are enforced separately. |
| document → deletion_request Document reference | 1 parent per child; 0..* children per parent | deletion_request.document_id → document.idEach Deletion request row references one document row; tenant and lifecycle compatibility are enforced separately. |
| deletion_request → deletion_ack Deletion request reference | 1 parent per child; 0..* children per parent | deletion_ack.deletion_request_id → deletion_request.idEach Deletion acknowledgement row references one deletion request row; tenant and lifecycle compatibility are enforced separately. |
Transaction boundaries
- Fetch authoritative source version and ACL state; create the immutable document version and unique ingestion job in one metadata transaction or through a durable outbox with reconciliation.
- Commit chunk/index writes idempotently under a staged generation; advance a checkpoint only after durable output identity/digest is verified.
- Publish a validated generation or document-version pointer with compare-and-swap against the expected active-manifest/document revision; readers observe a complete old or new version.
- Resolve current identity/policy and recheck authorization before chunk hydration; policy outage fails closed. Candidate metadata filters do not replace this authority.
- Persist a monotonic tombstone/policy barrier before cleanup; all read, cache and publication paths consult the barrier, while deletion acknowledgements track asynchronous physical removal.
Indexes and uniqueness
- UNIQUE tenant_id/connector_id/source_id for stable document identity; UNIQUE document_id/source_version/content_hash for immutable versions.
- UNIQUE ingestion_job.idempotency_key; queue indexes by state/lease_until/tenant for bounded worker recovery and fair scheduling.
- Chunk lookup by document_version_id/ordinal; vector and lexical indexes partitioned or filtered by tenant, corpus and compatible index generation.
- Active index manifest lookup by tenant/corpus; conditional revision updates for publication and rollback.
- Deletion request lookup by document/tombstone revision and acknowledgements by request/store; cache keys include tenant, permission scope/epoch, document version and index generation.
Schema traps and failure cases
- Duplicate, delayed or out-of-order webhook: idempotency plus authoritative source-version reconciliation prevents duplicate chunks and stale overwrite.
- Parser/OCR or embedding failure: quarantine/retry from durable checkpoint; previous active document version remains searchable.
- Partial index write or failed validation: staged generation remains invisible; keep the previous active manifest.
- Stale ACL/index metadata or policy service outage: fail closed before content hydration; invalidate affected caches and reconcile.
- Worker crash or lost acknowledgement: recover expired lease and retry deterministic chunk writes without advancing checkpoint twice.
- Deletion during index migration: tombstone is authoritative across all generations; rollback cannot re-enable revoked content.
Trace the records through one operation
- Inventory existing sources and establish snapshot/change cursor. Enqueue durable jobs in bounded pages; only after page jobs are persisted may the source checkpoint advance.
- Process document v7 into staging with parser v3 and embedding model E2; verify content hash, ACL metadata, chunk count and retrieval samples before updating the active document version.
- Receive duplicate/out-of-order update events for v7/v8; the unique job key deduplicates retries and authoritative source state prevents v7 from replacing v8.
- Revoke a principal's group access. Current policy denies the document before hydration immediately; cache invalidation and physical index cleanup complete asynchronously.
- Build a parallel generation for embedding model E3, shadow-query it, compare retrieval quality, and conditionally switch the active manifest. A concurrent tombstone remains enforced.
Diagrams and relationships
The offline plane publishes validated, versioned knowledge. The online plane reads only the active manifest and current authorization decisions.
Scroll horizontally on smaller screens, or enlarge the diagram.
New content is staged independently; online reads continue on the active version until a validated conditional publication.
Scroll horizontally on smaller screens, or enlarge the diagram.
Candidate IDs can be retrieved under tenant scope, but content is hydrated only after current authorization checks.
Scroll horizontally on smaller screens, or enlarge the diagram.
Source versions, chunks, embedding versions and corpus generations have distinct identities; a manifest controls publication.
Scroll horizontally on smaller screens, or enlarge the diagram.
The system validates output structure, citations and grounding before a response is shown.
Scroll horizontally on smaller screens, or enlarge the diagram.
Engineer, Senior and Staff follow-ups
These are teaching lenses. Each answer changes the scope of reasoning; they are not official employer leveling criteria.
Engineer: A user asks about a file that was just updated. How do you avoid returning mixed old and new chunks?
Use immutable source/document versions and stable chunk identities. Build the new version in staging, validate completeness and metadata, then conditionally advance the active document-version mapping. Retrieval filters to the active version. Keep the old version until publication succeeds; clean it up asynchronously. For a permission revoke or deletion, deny immediately with a tombstone rather than waiting for replacement.
Engineer: A webhook is delivered three times and out of order. What happens?
Use a unique idempotency key over tenant, connector, stable source ID, source version and content hash. A duplicate observes the existing job. A worker fetches authoritative source state and compares source revision/cursor so an old event cannot overwrite a newer version. Periodic reconciliation repairs missed events.
Senior: How do you bootstrap 250,000 existing documents while production queries continue?
Capture a source snapshot/cursor and inventory, enqueue durable bounded pages, checkpoint only after jobs are persisted, and write to a staged generation. Keep online queries on the current active manifest. Isolate backfill capacity with quotas and weighted queues. Validate coverage, ACLs, model dimensions and retrieval before atomic publication; throttle when query latency or provider/index saturation crosses guardrails.
Senior: Why not just filter unauthorized results after the LLM responds?
Because the unauthorized content may already have reached the reranker, provider, prompt trace, cache or model context. Filter candidate IDs and recheck current policy before hydration; only authorized text can cross the model boundary. Fail closed if policy is unavailable.
Senior: How do you migrate to a new embedding model?
A new model—even with the same vector dimension—may define a different vector space. Build a parallel index generation with pinned model and transform versions, validate retrieval quality, shadow-query, then switch the manifest. Never mix old and new vectors in one active search space; retain the old generation for rollback.
Staff: How would you define correctness and ownership across many connectors and tenants?
Publish connector contracts for identity, snapshot/change cursor, ordering, ACL semantics, deletion and reconciliation. Define SLOs for freshness and revocation separately, tenant-level resource budgets, policy authority and fail-closed behavior, and ownership for connector, index, model gateway, deletion and incident response. Maintain a versioned corpus manifest and audit trail. Treat webhook delivery as at-least-once and potentially reordered, not as a global transaction.
Staff: A legal deletion arrives during a model/index migration. How do you prevent rollback from restoring deleted content?
The tombstone/policy epoch is authoritative and independent of index generation. Query-time authorization and publication checks consult the current deny state. A rollback may restore an older compatible index but cannot override current tombstones. Cleanup tasks target all generations, replicas, lexical indexes and caches; verify each store before retiring the deletion barrier.
Practice and self-review
Your practice draft
Design the enterprise RAG system under the synthetic 20-tenant, 250,000-document, 250-query/s workload. Cover initial backfill and continuous updates, source/version/job/chunk/index/manifest schemas, authorization before hydration, hybrid retrieval, token budgeting, citation validation, revocation, capacity, rollout and recovery. Then change one constraint: the embedding model changes, a source connector loses its change feed, or an ACL is revoked while the model provider is timing out.
Saved locally on this device. The worked answer does not score your draft. Existing lesson notes and completion are separate.
Progressive hints
Hint 1
Start with the decision, tenant/source/policy authorities, freshness and revocation targets, and a deterministic keyword-search baseline.
Hint 2
Draw separate offline and online planes. Define immutable source versions, durable idempotent jobs, staged indexes and the active manifest before selecting a vector database.
Hint 3
Trace a normal update, duplicate/out-of-order event, missed webhook, ACL revoke and provider timeout. State exactly what remains queryable and what must fail closed.
Reveal worked answer guidance
A strong design separates offline source ingestion from online query serving. It snapshots and inventories the existing corpus, runs bounded resumable backfill into a staged index, validates tenant/ACL/provenance/model compatibility and atomically publishes a manifest. New files and updates use idempotent source-version jobs; the old active version remains searchable until the replacement is validated. Reconciliation repairs missed/out-of-order events. Query requests derive tenant identity from verified authentication, resolve current permissions, retrieve bounded candidate IDs, filter and recheck authorization before hydration, then run hybrid retrieval/reranking, context budgeting and guarded generation. Citations are resolved to the exact authorized evidence versions; insufficient or invalid evidence leads to abstention. Revocation writes a durable tombstone and denies immediately while physical cleanup proceeds across indexes and caches. A model migration builds a parallel compatible index and cuts over only after shadow evaluation. Capacity estimates are explicitly synthetic; operational guardrails cover freshness, revocation, latency, cost, retrieval quality and provider failures.
Evidence to check in your answer
- Framing: defines decision, users, tenants, sources, freshness, revocation, answer contract and assumptions.
- Data correctness: immutable source/document versions, provenance, ACL references, idempotent jobs, checkpoints and active manifest.
- Production migration: existing corpus backfill, incremental update, reconciliation, safe publication, rollback and online continuity.
- Security: current policy before text hydration, tenant-safe caches, prompt-injection boundary, immediate revocation and deletion verification.
- Retrieval/generation: dense+sparse fusion, bounded reranking, token budget, citation resolution, grounding and abstention.
- Operations: queue isolation, quotas, failure recovery, observability, independent metrics, governance and ownership.
- Evidence discipline: labels synthetic assumptions and distinguishes arithmetic/design from tested production performance.
For each item, record Missing, Prompted, Independent or Adaptive evidence. Adaptive means you can revise the reasoning under a changed constraint. Reading completion and readiness are separate.
Sources and verification scope
Original synthetic examples and reference calculations. Sources support narrowly scoped background; they do not establish production performance or independent chapter review.
SRE workbook: data processing pipelines ↗ · Pipeline SLOs and operational ownership; do not copy source workload numbers as course production results.
NIST AI Risk Management Framework ↗ · Risk vocabulary and governance; not a certification or blanket legal-compliance guarantee.
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks ↗ · Historical research foundation for retrieval/generation; not a complete modern permission/security architecture.
Hidden Technical Debt in Machine Learning Systems ↗ · System-boundary and lifecycle risks; source lessons motivate original examples.
Primary background links checked during authoring on 2026-10-19. Arithmetic, temporal eligibility, maturity and split reference functions are checked by the bounded local fixture suite. SQL is explanatory, not a migrated production query. No GPU, external dataset, model API or training job is required.