Learning pathsPricingChecking sign-in…
← ML curriculum

MLSD-023 · WORKED SYSTEM DESIGN

Permission-aware enterprise RAG

Design a permission-aware enterprise RAG system that continuously ingests existing and changing knowledge sources, serves only currently authorized evidence, produces grounded answers with citations, and recovers safely from stale data, provider failures, revocation and index migrations. All workload numbers and examples in this chapter are synthetic interview assumptions, not production measurements.

Author self-review; independent technical review pending. Architecture and capacity calculations use declared synthetic assumptions; no live RAG service, external corpus, embedding provider, vector database benchmark or production authorization integration has been executed..

Practice this design before reading →

Prerequisites and scoped previews

Candidate retrieval and ANN indexes

Separate eligible-corpus recall from latency/memory and filtering effects.

Retrieval and ranking evaluation

Define recall/precision at K, MRR, DCG/NDCG, relevance labels and query/user weighting.

Training-serving consistency

Version feature/transform/model contracts and compare training versus serving behavior.

Registry, validation, rollout and rollback

Promote compatible artifact bundles through checks, shadow/canary and rollback gates.

Scoped prerequisite preview. The complete chapter is planned; this preview does not count as authored coverage.

Governance, privacy, lineage and deletion

Trace permitted data use through snapshots, features, indexes and artifacts; define deletion obligations.

RAG and generative-system evaluation

Separate retrieval, grounding, generation, tool/action and safety evaluation.

Scoped prerequisite preview. The complete chapter is planned; this preview does not count as authored coverage.

What, when, why and how

RAG is a data product and an authorization system

what
Retrieval-augmented generation combines source ingestion, search, evidence selection and model generation. The model is not the source of truth; the authoritative source, version, provenance and permission policy remain outside the model.
when
Use RAG when answers depend on changing private or domain-specific knowledge that should remain retrievable and attributable rather than being copied into model weights.
why
A fluent answer can still be stale, unsupported or unauthorized. Retrieval quality, data freshness, access control and citation correctness are independent system properties.
how
Define the user decision, tenant boundary, permitted sources, freshness target, answer/abstention contract and audit requirements first. Build a deterministic retrieval baseline before choosing an embedding model or vector database.

Separate offline ingestion from online query serving

what
The offline plane discovers, parses, chunks, embeds, validates and publishes document versions. The online plane authenticates, authorizes, retrieves, reranks, assembles context and generates a response.
when
Use this separation when corpus updates are slower and burstier than user queries, or when backfills and model migrations must not interrupt serving.
why
Coupling ingestion to query serving makes large imports compete with interactive traffic and exposes partially written indexes.
how
Use durable jobs and bounded workers for ingestion, versioned staging and an active manifest for publication. Keep query reads on the last known-good active generation while changes are prepared.

Authorization must precede evidence exposure

what
Tenant and document permissions constrain which candidates may be hydrated, reranked, placed in prompts, cached or cited.
when
Required for multi-tenant corpora, group-based access, document-level ACLs, sensitive content and changing memberships.
why
Filtering only after generation is too late: unauthorized text may already have reached a reranker, model provider, trace, cache or prompt log.
how
Authenticate the principal, resolve current policy, filter candidate IDs before fetching text, recheck before hydration, and fail closed when the authorization authority is unavailable. Treat vector metadata filters as candidate narrowing, not the sole security boundary.

Hybrid retrieval is a recall-and-precision pipeline

what
Dense semantic search captures conceptual similarity; sparse lexical search captures exact terms, identifiers and rare phrases. Fusion merges candidates and a reranker orders a bounded authorized set.
when
Use hybrid retrieval when users ask natural-language questions but also search for exact policy names, error codes, product IDs or quoted terms.
why
Dense-only retrieval can miss exact strings; lexical-only retrieval misses paraphrases. A reranker improves ordering but cannot recover a relevant item never retrieved.
how
Run bounded dense and sparse searches in parallel, fuse ranks (for example reciprocal-rank fusion), deduplicate stable chunk IDs, apply authorization, rerank a bounded set and measure recall and answer quality separately.

Versioning and publication protect live knowledge

what
A source version is distinct from a chunk version, an embedding model version and an index generation. An active manifest identifies exactly which validated corpus generation readers use.
when
Needed for initial corpus imports, document updates, parser changes, chunking changes, embedding-model migrations, rollback and audit.
why
A successful vector write does not prove a document is complete or queryable. Mixing incompatible embedding spaces or exposing partial updates can silently corrupt retrieval.
how
Write immutable staged versions, validate coverage/provenance/ACL/model dimensions, then advance an active manifest conditionally. Keep the previous generation available for rollback and clean it up asynchronously.

Grounding, evaluation and operations are part of the answer contract

what
The response pipeline validates structure, citations, source versions, claim support and policy; the operations plane measures freshness, retrieval quality, safety, latency and cost.
when
Use whenever generated output is shown to a user or may trigger a consequential action.
why
A citation can be syntactically valid but point to the wrong version; a model can invent a plausible answer when evidence is missing; aggregate quality hides tenant and document slices.
how
Return answer plus resolved citations and evidence metadata, or abstain. Evaluate retrieval and generation separately with versioned cases, human-reviewed judgments and operational guardrails.

Frame the product decision and clarify constraints

We design an internal enterprise knowledge assistant. An authenticated employee asks a question and receives a concise answer with citations to documents they are currently allowed to read. The system supports multiple tenants and source connectors. Search, source-of-truth document storage, identity management and the foundation-model provider are external dependencies with explicit contracts.

Assumed interview answers below are synthetic. Change them when the interviewer supplies different constraints; do not present them as universal defaults or actual customer traffic.

Clarifying questionSynthetic assumption for this exerciseDesign consequence
Who can ask questions?Signed-in users in 20 isolated tenants.Tenant identity is server-derived; all storage and cache keys are tenant-scoped.
What sources exist?Object storage, a document drive and an internal wiki; 250,000 existing files at bootstrap.Connector inventory, snapshot/change feed and periodic reconciliation are required.
How often does content change?New/updated files continuously; permission changes must be enforced promptly.Incremental jobs, versioned updates and a separate urgent revocation path.
What is the query load?250 queries/s peak, 1.5 s p95 target for a complete answer.Bound retrieval, reranking, model time and total request deadline.
How fresh must knowledge be?Normal content updates searchable within 10 minutes; revocations denied within seconds.Measure source-to-index lag separately from revocation-to-denial latency.
What is the answer contract?Cited answer when evidence is sufficient; otherwise clarify or abstain.Citation validation, grounding checks and explicit no-answer behavior.
Can one tenant search another tenant's data?Never, including through caches, logs, retrieval candidates or fallback.Hard tenant isolation and adversarial cross-tenant tests.
What happens during model outage?No fabricated answer; return a bounded degraded response or retry option.Circuit breaker, provider timeout, and safe no-answer path.

Define invariants, trust boundaries and non-goals

Hard invariants: no cross-tenant evidence; no unauthorized document text reaches a reranker or model; every chunk retains source/version/offset provenance; incomplete ingestion cannot replace a good active version; deleted or revoked content is denied before asynchronous physical cleanup finishes; citations resolve to evidence supplied to the current request; model output is untrusted until validated.

Separate identity authority, source authority, policy authority, index storage and model provider. The model must not decide whether the caller is allowed to read a source. Retrieved documents are untrusted data: they may contain prompt injection, misleading instructions or malformed content.

Non-goals for the first version: arbitrary agent actions, write-back to source systems, unrestricted web search, training a foundation model, and treating a model-generated summary as an authoritative document. Add these only with separate authorization and evaluation contracts.

Build a useful non-generative retrieval baseline

Start with authenticated keyword search over a small approved corpus. Apply tenant and ACL filters, return source title/snippet/updated time, and enforce document-level authorization before text is returned. This baseline validates connector identity, access policy, indexing, freshness, relevance and auditability without an LLM.

Add semantic retrieval only when the baseline's query set shows a measurable recall gap that embeddings can plausibly close. Add generation only when users benefit from synthesis across multiple authorized sources. This staged approach makes failures attributable: search relevance, permission filtering and answer generation are not one blended metric.

Model source identity, versions, chunks and publication

Keep transactional metadata as the control plane and the vector/lexical stores as derived retrieval indexes. The following logical records can live in relational metadata storage plus object storage for source and parser artifacts; exact physical placement depends on scale and connector semantics.

RecordKey fields and constraintsWhy it exists
tenanttenant_id PK, status, policy_epochIsolation root and current tenant lifecycle.
source_connectorconnector_id PK, tenant_id FK, connector_type, cursor, snapshot_token, statusConnector ownership and resumable discovery.
source_document(tenant_id, connector_id, source_id) unique, current_source_version, active_document_version, tombstone_revisionStable identity independent of path/name changes.
document_versiondocument_version_id PK, source_id FK, source_version/ETag, content_hash, acl_version, parser_version, stateImmutable snapshot and processing provenance.
ingestion_jobjob_id PK, idempotency_key unique, desired_version, state, attempt, lease_until, checkpoint, error_codeDurable, retryable and deduplicated work.
chunkchunk_id PK, document_version_id FK, ordinal, source_offsets, page/section, text_hash, acl_refStable evidence identity and citation provenance.
embeddingchunk_id + model_version unique, vector_dimension, vector_ref, statusPrevents accidental mixing of embedding contracts.
index_generationgeneration_id PK, tenant/corpus scope, embedding_model_version, dimension, schema_version, stateImmutable staged retrieval generation.
active_manifest(tenant_id, corpus_id) PK, generation_id, revision, published_atReader-visible atomic pointer, advanced conditionally.
cleanup_tasktask_id PK, target_version/tombstone_revision, stores_pending, statusTracks asynchronous deletion and proof of cleanup.
-- Illustrative relational metadata constraints; vector payloads may live in a vector index.
CREATE TABLE ingestion_job (
  job_id UUID PRIMARY KEY,
  tenant_id UUID NOT NULL,
  connector_id UUID NOT NULL,
  source_id TEXT NOT NULL,
  source_version TEXT NOT NULL,
  content_hash TEXT NOT NULL,
  state TEXT NOT NULL,
  attempt INTEGER NOT NULL DEFAULT 0,
  checkpoint JSONB NOT NULL DEFAULT '{}',
  UNIQUE (tenant_id, connector_id, source_id, source_version, content_hash)
);

CREATE TABLE active_manifest (
  tenant_id UUID NOT NULL,
  corpus_id UUID NOT NULL,
  generation_id UUID NOT NULL,
  revision BIGINT NOT NULL,
  published_at TIMESTAMPTZ NOT NULL,
  PRIMARY KEY (tenant_id, corpus_id)
);

-- Advance with compare-and-swap; zero updated rows means re-read and reconcile.
UPDATE active_manifest
SET generation_id = :validated_generation,
    revision = revision + 1,
    published_at = now()
WHERE tenant_id = :tenant_id
  AND corpus_id = :corpus_id
  AND revision = :expected_revision;

Ingest documents: initial backfill and safe publication

Initial bootstrap is not the same as ongoing updates. Register each connector and tenant boundary, capture a source inventory, and establish a source snapshot or change cursor before scanning. Persist source identity, source version/ETag, content hash, modified time, ACL reference/version and deletion state. If the source supports snapshot tokens, use one so concurrent changes can be reconciled without gaps.

Backfill in bounded pages. A page checkpoint advances only after its source records have durable ingestion jobs. Each document job fetches the authoritative source snapshot, parses text/structure (OCR only when needed), normalizes while preserving provenance, chunks at safe semantic boundaries, attaches ACL metadata, embeds with a pinned model version, and writes to a staged generation.

Validate coverage, chunk counts, embedding dimensions/model identity, tenant isolation, ACL metadata, parser errors and sampled retrieval. Publish the initial generation only after validation passes. Readers must never see a half-built corpus. A vector database is not the system of record for source version or authorization.

Use durable job state, bounded retries, worker leases and idempotent writes. A parser/OCR poison document is quarantined with an operator-visible error; it must not silently become an empty document. Preserve a dead-letter/replay path with the source version and parser/model versions recorded.

VISUAL WALKTHROUGH

See the system in motion

Follow a source version through durable ingestion, provenance-preserving chunking, ACL metadata, embeddings and validated index publication.

Process overviewAvailable even when interactive playback cannot load.
  1. Capture a tenant-scoped source version and content hash.

  2. Create a durable idempotent ingestion job.

  3. Parse and normalize while preserving source offsets.

  4. Chunk with page/section provenance and authoritative ACL metadata.

  5. Embed with a pinned model version.

  6. Stage dense and sparse indexes.

  7. Validate and publish the manifest.

Loading interactive walkthrough…

Existing production data, new files and continuous updates

When a RAG system already has production data, do not stop serving it to rebuild for every new file. Keep the current active manifest live. Run an initial inventory/backfill into a staged generation if migrating from an older index; otherwise process only the changed source document and its derived chunks.

Subscribe to webhooks or source change feeds where available, but treat events as hints rather than trusted content. Fetch the authoritative current version before processing. Run periodic reconciliation because webhooks can be delayed, duplicated, reordered or lost. Detect creates, updates, deletes, renames/moves and ACL changes. For a rename, preserve stable source identity rather than treating it as unrelated content.

Use an idempotency key derived from tenant, connector, stable source ID, source version and content hash. Duplicate deliveries converge on one job. An older event must not overwrite a newer desired source version: compare source cursors/revisions and reconcile with authoritative source state. Persist checkpoints only after durable downstream work is recorded.

For a normal content update, write new chunks under a new immutable document version, validate them, then conditionally advance the active document/version mapping. Keep the old version active until the new version is complete. Retain the previous version for a bounded rollback window, then delete superseded chunks asynchronously. Retrieval must filter to active versions so old and new chunks are not both returned.

Treat permission revocation and deletion differently from an ordinary content update. Write a durable tombstone or deny barrier first and make query-time authorization reject the source immediately. Physical vector, lexical, replica and cache cleanup follows asynchronously and is verified. Never keep access alive merely to preserve rollback convenience.

Separate bulk backfill capacity from online queries and high-priority incremental/revocation work. Use weighted queues, per-tenant quotas, bounded embedding QPS, index write limits and pause controls. If query latency, provider throttling or index saturation crosses a configured guardrail, slow bulk backfill before sacrificing interactive traffic.

Change typeSafe behaviorWhat stays available
New fileCreate version/job, parse/embed/stage, validate and publish its active mapping.Existing corpus and other documents.
File updatedBuild new version; swap active mapping only after validation; clean old chunks later.Old complete version until new version is ready.
Duplicate/out-of-order eventDeduplicate by source/version/hash and compare authoritative source order.No duplicate chunks; no stale overwrite.
Bulk import into existing serviceSnapshot inventory, bounded backfill, checkpoint, stage and validate; throttle independently.Current active index remains queryable.
ACL revoked / file deletedPersist tombstone and deny queries immediately; invalidate caches; clean all stores asynchronously.Other authorized documents; revoked source is not available.
Missed webhook / connector outageReconcile cursors and source inventory; expose freshness lag and preserve last good version where policy permits.Last known-good data, clearly marked stale to operators.

VISUAL WALKTHROUGH

See the system in motion

See how an existing production corpus is backfilled and then updated continuously while online queries stay available.

Process overviewAvailable even when interactive playback cannot load.
  1. Inventory source IDs, versions, hashes and ACL references.

  2. Backfill in bounded batches with durable checkpoints.

  3. Validate and publish the baseline generation.

  4. Detect create/update/delete/ACL events and reconcile missed changes.

  5. Deduplicate events and reject stale source versions.

  6. Stage only changed documents.

  7. Conditionally publish validated versions.

  8. Continue serving the active index.

Loading interactive walkthrough…

Authorize before content hydration and model exposure

Authenticate the caller and derive tenant identity from the verified session/token, never from a client-supplied tenant field alone. Resolve current groups, roles and effective permissions using authoritative identity and policy services. Record a policy epoch or decision version for observability.

Search dense and sparse indexes with tenant and ACL filters to narrow candidate IDs. Do not hydrate candidate text yet. Apply current policy checks before chunk bodies are fetched; recheck immediately before hydration to close the stale-index window. If policy cannot be evaluated, fail closed. Only authorized text and provenance may enter a reranker, prompt, model provider, cache or trace.

Cache keys must include tenant, principal or a proven equivalent permission scope, policy epoch, query normalization version and index generation. A cached answer must be invalidated or rejected when permissions or source versions change. Do not rely on a long TTL as the revocation mechanism.

Retrieved content is untrusted. Delimit it as evidence, do not execute its instructions, and prevent it from overriding system/developer policy. Keep tools and write actions out of the first version; if introduced later, each tool requires independent authorization, schema validation, allowlisting and confirmation policy.

VISUAL WALKTHROUGH

See the system in motion

Trace identity and current policy checks. Unauthorized text must never reach rerankers or model context.

Process overviewAvailable even when interactive playback cannot load.
  1. Authenticate principal and tenant.

  2. Resolve current effective permissions.

  3. Retrieve bounded candidate IDs.

  4. Filter before content hydration.

  5. Recheck authorization.

  6. Hydrate only authorized evidence.

Loading interactive walkthrough…

Hybrid search, rank fusion and reranking

Normalize the query without stripping exact identifiers. Generate a query embedding using the same pinned embedding contract as the active index. In parallel, run dense nearest-neighbor retrieval and sparse lexical/BM25 retrieval with tenant/corpus filters, bounded candidate counts and timeouts.

Fuse rankings using a documented method such as reciprocal-rank fusion: score(d) = sum over retrieval lists of 1 / (k + rank(d)). This avoids pretending dense and lexical raw scores are directly comparable. Deduplicate by stable chunk ID and retain source version, source offsets and retrieval provenance.

Apply current authorization before candidate text is passed to a cross-encoder reranker. Rerank a bounded set, select top-K with diversity and freshness constraints, and preserve citation IDs. If reranking times out, a policy-approved fallback may use fused order; do not send unauthorized candidates to the reranker merely to filter them afterward.

Measure candidate recall separately from reranker NDCG and answer quality. A reranker cannot recover documents that both retrieval methods missed. Evaluate exact identifiers, paraphrases, multilingual queries, stale documents, no-result queries, duplicate chunks and permission-filtered underfill.

VISUAL WALKTHROUGH

See the system in motion

Combine dense and lexical retrieval, fuse candidates, enforce ACLs and rerank a bounded authorized set.

Process overviewAvailable even when interactive playback cannot load.
  1. Normalize query.

  2. Run dense and sparse search in parallel.

  3. Fuse and deduplicate candidates.

  4. Apply current authorization.

  5. Rerank bounded candidates.

  6. Select diverse top-K evidence.

Loading interactive walkthrough…

Assemble evidence within a safe token budget

Start only from authorized, versioned evidence. Deduplicate overlapping chunks, preserve source spans, prefer authoritative and current documents, and select a diverse set rather than filling the context with near-identical passages. Allocate a token budget across system instructions, conversation history, retrieved evidence and reserved output tokens.

Use the target model tokenizer rather than character-count approximations. Truncate only at safe chunk boundaries and never cut away citation metadata. Record a context manifest containing evidence IDs, document versions, source spans, retrieval scores, prompt template version, model version and policy epoch. Apply retention and redaction rules; auditability does not justify indiscriminate storage of sensitive prompt text.

Separate trusted instructions from retrieved content. Mark document passages as untrusted evidence, neutralize unsafe formatting where appropriate, and instruct the model to treat document instructions as data rather than commands. Context assembly is a security boundary, not just prompt formatting.

VISUAL WALKTHROUGH

See the system in motion

Build compact, provenance-preserving context within a model token budget and keep retrieved content untrusted.

Process overviewAvailable even when interactive playback cannot load.
  1. Collect authorized evidence.

  2. Remove duplicate passages.

  3. Allocate the model token budget.

  4. Order evidence for relevance and coverage.

  5. Build a structured prompt with untrusted evidence delimited.

  6. Record the context manifest.

Loading interactive walkthrough…

Generate, validate citations and abstain safely

Call the model through a gateway with model allowlisting, tenant budgets, bounded input/output tokens, request deadlines, rate limits, retries, circuit breakers and provider fallback policy. The model returns a constrained response schema such as answer text, claim list, citation IDs and an abstention reason. Model output is untrusted.

Validate the output schema before rendering. Resolve each citation ID against the evidence set for this exact request and verify source version and offsets. Reject citations to documents not provided to the model or no longer authorized. Check material claims against evidence with deterministic rules where possible and an evaluation/judgment layer where semantic support must be assessed. A model-based judge is a signal, not proof.

If evidence is absent, contradictory, stale or insufficient, ask a clarifying question or abstain. Do not fill gaps with model memory while presenting the result as sourced. A response should distinguish direct evidence, inference and uncertainty. Log safe metadata for evaluation without leaking cross-tenant text.

VISUAL WALKTHROUGH

See the system in motion

Validate response structure, citations, claim support and policy before returning an answer or abstaining.

Process overviewAvailable even when interactive playback cannot load.
  1. Generate a constrained draft.

  2. Validate output schema.

  3. Resolve citations to authorized source spans.

  4. Check material claim support.

  5. Apply safety and answer policy.

  6. Return grounded answer or abstain.

Loading interactive walkthrough…

Permission changes, deletion and cache invalidation

On an ACL revoke or deletion event, verify authoritative source state and persist a monotonic tombstone/policy epoch before beginning cleanup. Query-time authorization consults current policy/tombstone state, so stale vector metadata cannot grant access. This is the immediate logical denial barrier.

Invalidate retrieval, context and answer caches using tenant/document/policy-version keys. Remove or supersede dense vectors, sparse postings, replicas, materialized summaries and derived artifacts according to retention/legal-hold policy. Track each cleanup acknowledgement and reconcile stores that missed the first delete. Retire the tombstone only after the required stores and replicas confirm cleanup and the policy allows it.

Distinguish access revocation from data deletion: a user losing access may require deny and cache invalidation but not necessarily destruction of a document still available to other authorized users. A source deletion may require broader removal. Define ownership, legal hold, backup retention and evidence of completion before promising physical erasure.

VISUAL WALKTHROUGH

See the system in motion

Deny revoked or deleted content immediately, then clean indexes and caches asynchronously and verify removal.

Process overviewAvailable even when interactive playback cannot load.
  1. Verify source change.

  2. Write durable tombstone.

  3. Deny retrieval immediately.

  4. Invalidate affected caches.

  5. Delete or supersede indexed material.

  6. Reconcile and verify removal.

Loading interactive walkthrough…

Evaluate retrieval, generation and the complete product separately

Build a versioned evaluation set from approved representative questions and document versions. Include exact lookup, paraphrase, multi-hop synthesis, no-answer, conflicting-source, stale-version, multilingual, prompt-injection and permission-boundary cases. Keep a holdout set that is not repeatedly tuned against.

Retrieval metrics include Recall@K, MRR/NDCG and authorized-evidence coverage. Generation metrics include claim support, citation precision/recall, answer completeness, abstention correctness and contradiction handling. Product metrics include task completion and user feedback. System metrics include latency, availability, provider errors, tokens, cost and freshness. Never collapse them into one unqualified score.

Run adversarial tests: a user guesses another tenant's document ID; a revoked group retains stale cache; an ACL changes during retrieval; a document contains instructions to exfiltrate secrets; a source update arrives out of order; a model returns an invented citation. Require tests to fail closed and record the evidence.

LayerExample measuresFailure hidden by aggregate score
RetrievalRecall@K, MRR, NDCG, authorized candidate rateRelevant evidence omitted or unauthorized candidates leak.
GenerationClaim support, citation precision, completeness, abstention accuracyFluent hallucination with invalid citations.
SecurityCross-tenant deny tests, ACL revoke latency, cache invalidationOne tenant's evidence leaks through shared cache.
FreshnessSource-to-index lag, stale-version query rate, reconcile backlogNew policy or corrected procedure is not searchable.
Operationsp95/p99 latency, provider timeout, queue depth, cost/queryGood offline quality with unacceptable live behavior.

Capacity estimates and cost controls

Use explicit synthetic assumptions to show the method, not to claim an exact production bill. Assume 250,000 source documents, an average of six chunks per document, 1.5 million chunks, 1,536-dimensional float32 embeddings and a target 300 embedded chunks/s during an unconstrained backfill window.

Raw vector payload is 1,500,000 × 1,536 × 4 bytes = 9,216,000,000 bytes (about 9.2 GB decimal, or 8.6 GiB). This excludes ANN graph structures, lexical postings, metadata, replicas, object storage, write amplification and backups; actual provisioned capacity will be materially higher. At 300 chunks/s, the embedding stage alone has a theoretical lower-bound duration of 1,500,000 / 300 = 5,000 seconds (about 83 minutes), before source fetch, parsing, retries, throttling, validation and index compaction.

At 250 queries/s and 40 fused candidates per query, the reranker sees at most 10,000 candidate pairs/s before top-K selection. If 8 chunks are passed to a model per query, that is up to 2,000 chunk-context selections/s; token volume depends on chunk length and output limits. Measure actual token distributions and provider quotas before setting concurrency.

Cost levers: incremental rather than full re-embedding; deduplication; bounded candidate counts; smaller context; cache only under correct authorization/version keys; batch embeddings where provider semantics allow; separate backfill quotas; choose vector precision/index parameters using measured recall-latency trade-offs. Do not trade away permission correctness or freshness silently to lower cost.

End-to-end architecture and request lifecycle

Offline ingestion: connectors → source inventory/change feed → durable queue → bounded fetch/parse/OCR workers → normalized immutable document versions → chunk/provenance/ACL metadata → embedding workers → staged dense and sparse indexes → validation gate → conditional active-manifest publication → asynchronous cleanup/reconciliation.

Online query: authenticated client → API gateway → identity/tenant resolver → policy service → query normalization/embedding → parallel dense and sparse retrieval → rank fusion/deduplication → current ACL filter/recheck → reranker → context/token planner → guarded model gateway → schema/citation/grounding/policy validation → cited answer or abstention.

Shared control plane: source/version registry, job/checkpoint state, policy epochs/tombstones, active manifests, prompt/model/index versions, audit metadata, evaluation datasets and telemetry. Operational controls include per-tenant quotas, circuit breakers, queue prioritization, rollback and reconciliation.

Use the cumulative final animation to follow both planes together. A production answer must be explainable by the exact active index generation, document versions, policy epoch, prompt template and model version used for that request.

Failure modes, degraded behavior and recovery

Parser/OCR failure: bounded retry and quarantine; preserve source version and error. Embedding timeout/rate limit: respect provider retry hints, cap concurrency and resume from durable checkpoints. Partial vector write: keep old manifest active and retry idempotently or discard staging. Worker crash: recover expired leases. Missed webhook: periodic source reconciliation. Out-of-order update: authoritative version comparison. Stale ACL: immediate deny barrier and cache invalidation. Reranker timeout: use only an approved fused-order fallback. Model outage: circuit-break and abstain/retry rather than fabricate.

Index migration: build a parallel generation with a pinned embedding model/dimension, shadow-query and compare quality, then conditionally cut over. Never mix vectors from incompatible embedding spaces. Keep the old generation for bounded rollback, but current authorization and tombstones always override rollback.

Operational dashboards should show source-to-index lag, oldest queue age, backlog by tenant/connector, parse/embed/index errors, duplicate and out-of-order events, staged/active manifest revisions, orphan chunks, revocation-to-denial latency, deletion cleanup age, retrieval quality, p95/p99 latency, model error rate, tokens and cost.

Rollout, ownership and governance

Roll out connector by connector and tenant cohort. Start with shadow indexing and offline evaluation, then limited read-only search, then generated answers with citations. Compare against the deterministic search baseline. Gate promotion on retrieval quality, permission tests, freshness, latency, cost and safe abstention. Provide a kill switch that disables generation while preserving authorized search.

Assign owners for source connectors, identity/policy integration, ingestion/index operations, model gateway, evaluation, privacy/deletion and incident response. Define who can publish a new parser/chunker/embedding generation, who approves prompt/model changes and who verifies deletion completion.

Record versioned configuration and audit metadata. Retain only what is needed for debugging and compliance; redact secrets and sensitive source text. Review provider data handling, tenant isolation, retention, legal holds, user notice and model-provider terms. A trace is not permission to store every prompt forever.

Key design trade-offs and interview conclusion

Per-document active-version pointers make incremental replacement efficient but require retrieval to enforce active-version state across dense and sparse indexes. Whole-corpus generations simplify atomic cutover and embedding migrations but cost more storage and build time. A hybrid approach uses per-document version manifests for normal changes and parallel corpus generations for schema/model migrations.

Webhook/change-feed ingestion reduces detection delay but cannot be the only correctness mechanism; reconciliation repairs missed events. Vector metadata ACL filters improve retrieval efficiency but cannot replace authoritative query-time checks. Immediate logical tombstones make revocation prompt while physical deletion remains asynchronous and verifiable.

Conclude with the invariants and evidence: existing queries continue on a last-known-good index during backfill; new versions publish only after validation; unauthorized text never enters model context; revocation denies immediately; citations resolve to authorized source versions; and the system measures freshness, retrieval quality, safety, latency and cost independently.

Data model, schema and relationships

The logical schema separates tenant identity, principal authorization, stable source identity, immutable document versions, ACL revisions, chunks, index snapshots, durable ingestion jobs, answer audit and deletion propagation. The vector/lexical indexes are derived retrieval stores; relational metadata and authoritative identity/policy services control publication and access.

Implementation boundary: Architecture contract only. The schema is illustrative relational metadata plus object storage and vector/lexical indexes; no migrations, connector, embedding provider, authorization integration or live RAG service are installed. Cross-tenant compatibility, ACL freshness, active-version selection and multi-store cleanup require transactional checks and reconciliation.

These are proposed logical schemas with field contracts, not executable database migrations. PK identifies a row, FK references an existing parent, UK is a uniqueness rule. A composite PK identifies the combination. Nullable references are marked with ?. A foreign key protects identity; cross-row limits, timing and state transitions also require the transaction rules below.

Tenant: relationship map

Connectors name the relationship; multiplicities state parents per child and children per parent. 0..1 means optional, 1 required, 0..* any number. Self-links show two roles of the same entity.

Tenant relationshipstenant 1 to principal 0..*: Tenant reference. principal.tenant_id → tenant.id. tenant 1 to document 0..*: Tenant reference. document.tenant_id → tenant.id. tenant 1 to index_snapshot 0..*: Tenant reference. index_snapshot.tenant_id → tenant.id.Tenant reference1 parent · 0..* childrenPrincipalid [PK]tenant_id [FK]external_subjectTenant reference1 parent · 0..* childrenDocumentid [PK]tenant_id [FK]connector_idTenant reference1 parent · 0..* childrenIndex snapshotid [PK]tenant_id [FK]generationTenantid [PK]statuspolicy_epoch
Principal: relationship map

Connectors name the relationship; multiplicities state parents per child and children per parent. 0..1 means optional, 1 required, 0..* any number. Self-links show two roles of the same entity.

Principal relationshipsprincipal 1 to answer_audit 0..*: Principal reference. answer_audit.principal_id → principal.id.Principal reference1 parent · 0..* childrenAnswer auditid [PK]principal_id [FK]index_snapshot_id [FK]Principalid [PK]tenant_id [FK]external_subject
Document: relationship map

Connectors name the relationship; multiplicities state parents per child and children per parent. 0..1 means optional, 1 required, 0..* any number. Self-links show two roles of the same entity.

Document relationshipsdocument 1 to acl_version 0..*: Document reference. acl_version.document_id → document.id. document 1 to document_version 0..*: Document reference. document_version.document_id → document.id. document 1 to deletion_request 0..*: Document reference. deletion_request.document_id → document.id.Document reference1 parent · 0..* childrenACL versionid [PK]document_id [FK]revisionDocument reference1 parent · 0..* childrenDocument versionid [PK]document_id [FK]acl_version_id [FK]Document reference1 parent · 0..* childrenDeletion requestid [PK]document_id [FK]tombstone_revisionDocumentid [PK]tenant_id [FK]connector_id
ACL version: relationship map

Connectors name the relationship; multiplicities state parents per child and children per parent. 0..1 means optional, 1 required, 0..* any number. Self-links show two roles of the same entity.

ACL version relationshipsacl_version 1 to document_version 0..*: ACL version reference. document_version.acl_version_id → acl_version.id.ACL version reference1 parent · 0..* childrenDocument versionid [PK]document_id [FK]acl_version_id [FK]ACL versionid [PK]document_id [FK]revision
Document version: relationship map

Connectors name the relationship; multiplicities state parents per child and children per parent. 0..1 means optional, 1 required, 0..* any number. Self-links show two roles of the same entity.

Document version relationshipsdocument_version 1 to chunk 0..*: Document version reference. chunk.document_version_id → document_version.id. document_version 1 to ingestion_job 0..*: Document version reference. ingestion_job.document_version_id → document_version.id.Document version reference1 parent · 0..* childrenChunkid [PK]document_version_id [FK]ordinalDocument version reference1 parent · 0..* childrenIngestion jobid [PK]document_version_id [FK]idempotency_keyDocument versionid [PK]document_id [FK]acl_version_id [FK]
Chunk: relationship map

Connectors name the relationship; multiplicities state parents per child and children per parent. 0..1 means optional, 1 required, 0..* any number. Self-links show two roles of the same entity.

Chunk relationshipschunk 1 to answer_evidence 0..*: Chunk reference. answer_evidence.chunk_id → chunk.id.Chunk reference1 parent · 0..* childrenAnswer evidenceid [PK]answer_audit_id [FK]chunk_id [FK]Chunkid [PK]document_version_id [FK]ordinal
Index snapshot: relationship map

Connectors name the relationship; multiplicities state parents per child and children per parent. 0..1 means optional, 1 required, 0..* any number. Self-links show two roles of the same entity.

Index snapshot relationshipsindex_snapshot 1 to answer_audit 0..*: Index snapshot reference. answer_audit.index_snapshot_id → index_snapshot.id.Index snapshot reference1 parent · 0..* childrenAnswer auditid [PK]principal_id [FK]index_snapshot_id [FK]Index snapshotid [PK]tenant_id [FK]generation
Answer audit: relationship map

Connectors name the relationship; multiplicities state parents per child and children per parent. 0..1 means optional, 1 required, 0..* any number. Self-links show two roles of the same entity.

Answer audit relationshipsanswer_audit 1 to answer_evidence 0..*: Answer audit reference. answer_evidence.answer_audit_id → answer_audit.id.Answer audit reference1 parent · 0..* childrenAnswer evidenceid [PK]answer_audit_id [FK]chunk_id [FK]Answer auditid [PK]principal_id [FK]index_snapshot_id [FK]
Deletion request: relationship map

Connectors name the relationship; multiplicities state parents per child and children per parent. 0..1 means optional, 1 required, 0..* any number. Self-links show two roles of the same entity.

Deletion request relationshipsdeletion_request 1 to deletion_ack 0..*: Deletion request reference. deletion_ack.deletion_request_id → deletion_request.id.Deletion request reference1 parent · 0..* childrenDeletion acknowledgementid [PK]deletion_request_id [FK]storeDeletion requestid [PK]document_id [FK]tombstone_revision

Entity definitions and field contracts

Tenant

Isolation and quota boundary.

FieldType / keyMeaning and reference
idopaque_id · PKGlobally unique immutable row identity.
statusenum · VALUEactive/suspended lifecycle.
policy_epochinteger · VALUEMonotonic tenant policy revision.

Code mapping: Logical production design contract; not an installed migration. Enforce tenant-compatible foreign keys and conditional state transitions transactionally.

  • Tenant identity is derived from verified authentication; caller-supplied tenant IDs do not grant access.

Principal

Authenticated user or service identity.

FieldType / keyMeaning and reference
idopaque_id · PKGlobally unique immutable row identity.
tenant_idopaque_id · FKOwning tenant. · tenant.id
external_subjectstring · VALUEStable identity-provider subject.
stateenum · VALUEactive/disabled.

Code mapping: Logical production design contract; not an installed migration. Enforce tenant-compatible foreign keys and conditional state transitions transactionally.

  • UNIQUE(tenant_id,external_subject). Membership/group expansion comes from the identity authority.

Document

Stable connector-scoped source identity.

FieldType / keyMeaning and reference
idopaque_id · PKGlobally unique immutable row identity.
tenant_idopaque_id · FKOwning tenant. · tenant.id
connector_idopaque_id · VALUERegistered source connector.
source_idstring · VALUEStable source ID, independent of path/name.
active_version_idopaque_id? · VALUECurrent validated document version pointer.
tombstone_revisioninteger · VALUECurrent deletion/revocation barrier.

Code mapping: Logical production design contract; not an installed migration. Enforce tenant-compatible foreign keys and conditional state transitions transactionally.

  • UNIQUE(tenant_id,connector_id,source_id). A path rename does not create a new logical document.

ACL version

Immutable permission snapshot/reference.

FieldType / keyMeaning and reference
idopaque_id · PKGlobally unique immutable row identity.
document_idopaque_id · FKProtected document. · document.id
revisionstring · VALUESource/policy ACL revision.
policy_refstring · VALUEAuthoritative ACL reference.
captured_attimestamp · VALUEObservation time.

Code mapping: Logical production design contract; not an installed migration. Enforce tenant-compatible foreign keys and conditional state transitions transactionally.

  • ACL snapshot is metadata for filtering only; current policy is rechecked before text hydration.

Document version

Immutable source content and parser identity.

FieldType / keyMeaning and reference
idopaque_id · PKGlobally unique immutable row identity.
document_idopaque_id · FKStable source identity. · document.id
acl_version_idopaque_id · FKACL revision associated with this source snapshot. · acl_version.id
source_versionstring · VALUEETag/revision from authoritative connector.
content_hashstring · VALUEVerified content digest.
parser_versionstring · VALUEParser/OCR implementation version.
stateenum · VALUEfetched/parsed/staged/validated/published/quarantined/superseded.

Code mapping: Logical production design contract; not an installed migration. Enforce tenant-compatible foreign keys and conditional state transitions transactionally.

  • UNIQUE(document_id,source_version,content_hash). Out-of-order older source versions cannot replace newer desired state.

Chunk

Retrieval unit with exact source provenance.

FieldType / keyMeaning and reference
idopaque_id · PKGlobally unique immutable row identity.
document_version_idopaque_id · FKImmutable source version. · document_version.id
ordinalinteger · VALUEStable ordinal within document version.
source_spanjson · VALUEPage/section/offset provenance.
text_hashstring · VALUECanonical chunk text digest.
acl_refopaque_id · VALUEACL version used for candidate filtering.

Code mapping: Logical production design contract; not an installed migration. Enforce tenant-compatible foreign keys and conditional state transitions transactionally.

  • UNIQUE(document_version_id,ordinal). Hydrate text only after current authorization succeeds.

Index snapshot

Immutable compatible corpus/index generation.

FieldType / keyMeaning and reference
idopaque_id · PKGlobally unique immutable row identity.
tenant_idopaque_id · FKOwning tenant. · tenant.id
generationstring · VALUEImmutable generation identity.
embedding_modelstring · VALUEPinned model/version.
dimensioninteger · VALUEVector dimension.
schema_versionstring · VALUEChunk/metadata contract.
stateenum · VALUEbuilding/validated/active/retired/failed.

Code mapping: Logical production design contract; not an installed migration. Enforce tenant-compatible foreign keys and conditional state transitions transactionally.

  • Vectors from incompatible embedding spaces must not be mixed in one active generation.

Ingestion job

Durable idempotent per-source-version work.

FieldType / keyMeaning and reference
idopaque_id · PKGlobally unique immutable row identity.
document_version_idopaque_id · FKTarget immutable source version. · document_version.id
idempotency_keystring · VALUETenant/connector/source/version/hash identity.
stateenum · VALUEqueued/fetching/parsing/chunking/embedding/staged/validated/published/retryable/quarantined.
attemptinteger · VALUEBounded retry count.
checkpointjson · VALUEDurable stage progress and lease metadata.

Code mapping: Logical production design contract; not an installed migration. Enforce tenant-compatible foreign keys and conditional state transitions transactionally.

  • UNIQUE(idempotency_key). Advance checkpoints only after durable outputs; recover expired worker leases.

Answer audit

Minimal reproducibility and answer decision record.

FieldType / keyMeaning and reference
idopaque_id · PKGlobally unique immutable row identity.
principal_idopaque_id · FKRequesting principal. · principal.id
index_snapshot_idopaque_id · FKActive index generation used. · index_snapshot.id
policy_epochinteger · VALUEAuthorization policy revision observed.
prompt_versionstring · VALUEPrompt template release.
model_versionstring · VALUEModel/provider identity.
outcomeenum · VALUEanswered/clarified/abstained/blocked.

Code mapping: Logical production design contract; not an installed migration. Enforce tenant-compatible foreign keys and conditional state transitions transactionally.

  • Store minimum required metadata; avoid retaining sensitive prompt text by default.

Answer evidence

Citation link from one answer to its supplied chunk.

FieldType / keyMeaning and reference
idopaque_id · PKGlobally unique immutable row identity.
answer_audit_idopaque_id · FKAnswer decision. · answer_audit.id
chunk_idopaque_id · FKExact authorized evidence chunk. · chunk.id
document_version_idstring · VALUEExact source version rendered in citation.
source_spanjson · VALUEExact page/section/offset citation.

Code mapping: Logical production design contract; not an installed migration. Enforce tenant-compatible foreign keys and conditional state transitions transactionally.

  • UNIQUE(answer_audit_id,chunk_id). Citation must belong to the current request evidence set and remain authorized.

Deletion request

Durable deny barrier and cleanup scope.

FieldType / keyMeaning and reference
idopaque_id · PKGlobally unique immutable row identity.
document_idopaque_id · FKTarget document. · document.id
tombstone_revisioninteger · VALUEMonotonic deny revision.
reasonenum · VALUEsource_deleted/permission_revoked/retention_request.
stateenum · VALUEdenied/cleaning/verified/failed.

Code mapping: Logical production design contract; not an installed migration. Enforce tenant-compatible foreign keys and conditional state transitions transactionally.

  • Persist deny barrier before asynchronous index/cache cleanup. A rollback cannot override current tombstones.

Deletion acknowledgement

Per-store evidence of cleanup.

FieldType / keyMeaning and reference
idopaque_id · PKGlobally unique immutable row identity.
deletion_request_idopaque_id · FKOwning deletion request. · deletion_request.id
storestring · VALUEVector/lexical/cache/replica/object artifact target.
acknowledged_attimestamp? · VALUECleanup confirmation time.
statusenum · VALUEpending/complete/retryable/blocked.

Code mapping: Logical production design contract; not an installed migration. Enforce tenant-compatible foreign keys and conditional state transitions transactionally.

  • Tombstone retirement requires all required stores/replicas to meet the declared deletion policy.

Relationship cardinalities and their reasons

Parent → childMultiplicityFK / why
tenant → principal

Tenant reference

1 parent per child; 0..* children per parentprincipal.tenant_id → tenant.id

Each Principal row references one tenant row; tenant and lifecycle compatibility are enforced separately.

tenant → document

Tenant reference

1 parent per child; 0..* children per parentdocument.tenant_id → tenant.id

Each Document row references one tenant row; tenant and lifecycle compatibility are enforced separately.

document → acl_version

Document reference

1 parent per child; 0..* children per parentacl_version.document_id → document.id

Each ACL version row references one document row; tenant and lifecycle compatibility are enforced separately.

document → document_version

Document reference

1 parent per child; 0..* children per parentdocument_version.document_id → document.id

Each Document version row references one document row; tenant and lifecycle compatibility are enforced separately.

acl_version → document_version

ACL version reference

1 parent per child; 0..* children per parentdocument_version.acl_version_id → acl_version.id

Each Document version row references one acl version row; tenant and lifecycle compatibility are enforced separately.

document_version → chunk

Document version reference

1 parent per child; 0..* children per parentchunk.document_version_id → document_version.id

Each Chunk row references one document version row; tenant and lifecycle compatibility are enforced separately.

tenant → index_snapshot

Tenant reference

1 parent per child; 0..* children per parentindex_snapshot.tenant_id → tenant.id

Each Index snapshot row references one tenant row; tenant and lifecycle compatibility are enforced separately.

document_version → ingestion_job

Document version reference

1 parent per child; 0..* children per parentingestion_job.document_version_id → document_version.id

Each Ingestion job row references one document version row; tenant and lifecycle compatibility are enforced separately.

principal → answer_audit

Principal reference

1 parent per child; 0..* children per parentanswer_audit.principal_id → principal.id

Each Answer audit row references one principal row; tenant and lifecycle compatibility are enforced separately.

index_snapshot → answer_audit

Index snapshot reference

1 parent per child; 0..* children per parentanswer_audit.index_snapshot_id → index_snapshot.id

Each Answer audit row references one index snapshot row; tenant and lifecycle compatibility are enforced separately.

answer_audit → answer_evidence

Answer audit reference

1 parent per child; 0..* children per parentanswer_evidence.answer_audit_id → answer_audit.id

Each Answer evidence row references one answer audit row; tenant and lifecycle compatibility are enforced separately.

chunk → answer_evidence

Chunk reference

1 parent per child; 0..* children per parentanswer_evidence.chunk_id → chunk.id

Each Answer evidence row references one chunk row; tenant and lifecycle compatibility are enforced separately.

document → deletion_request

Document reference

1 parent per child; 0..* children per parentdeletion_request.document_id → document.id

Each Deletion request row references one document row; tenant and lifecycle compatibility are enforced separately.

deletion_request → deletion_ack

Deletion request reference

1 parent per child; 0..* children per parentdeletion_ack.deletion_request_id → deletion_request.id

Each Deletion acknowledgement row references one deletion request row; tenant and lifecycle compatibility are enforced separately.

Transaction boundaries

  1. Fetch authoritative source version and ACL state; create the immutable document version and unique ingestion job in one metadata transaction or through a durable outbox with reconciliation.
  2. Commit chunk/index writes idempotently under a staged generation; advance a checkpoint only after durable output identity/digest is verified.
  3. Publish a validated generation or document-version pointer with compare-and-swap against the expected active-manifest/document revision; readers observe a complete old or new version.
  4. Resolve current identity/policy and recheck authorization before chunk hydration; policy outage fails closed. Candidate metadata filters do not replace this authority.
  5. Persist a monotonic tombstone/policy barrier before cleanup; all read, cache and publication paths consult the barrier, while deletion acknowledgements track asynchronous physical removal.

Indexes and uniqueness

  1. UNIQUE tenant_id/connector_id/source_id for stable document identity; UNIQUE document_id/source_version/content_hash for immutable versions.
  2. UNIQUE ingestion_job.idempotency_key; queue indexes by state/lease_until/tenant for bounded worker recovery and fair scheduling.
  3. Chunk lookup by document_version_id/ordinal; vector and lexical indexes partitioned or filtered by tenant, corpus and compatible index generation.
  4. Active index manifest lookup by tenant/corpus; conditional revision updates for publication and rollback.
  5. Deletion request lookup by document/tombstone revision and acknowledgements by request/store; cache keys include tenant, permission scope/epoch, document version and index generation.

Schema traps and failure cases

  1. Duplicate, delayed or out-of-order webhook: idempotency plus authoritative source-version reconciliation prevents duplicate chunks and stale overwrite.
  2. Parser/OCR or embedding failure: quarantine/retry from durable checkpoint; previous active document version remains searchable.
  3. Partial index write or failed validation: staged generation remains invisible; keep the previous active manifest.
  4. Stale ACL/index metadata or policy service outage: fail closed before content hydration; invalidate affected caches and reconcile.
  5. Worker crash or lost acknowledgement: recover expired lease and retry deterministic chunk writes without advancing checkpoint twice.
  6. Deletion during index migration: tombstone is authoritative across all generations; rollback cannot re-enable revoked content.

Trace the records through one operation

  1. Inventory existing sources and establish snapshot/change cursor. Enqueue durable jobs in bounded pages; only after page jobs are persisted may the source checkpoint advance.
  2. Process document v7 into staging with parser v3 and embedding model E2; verify content hash, ACL metadata, chunk count and retrieval samples before updating the active document version.
  3. Receive duplicate/out-of-order update events for v7/v8; the unique job key deduplicates retries and authoritative source state prevents v7 from replacing v8.
  4. Revoke a principal's group access. Current policy denies the document before hydration immediately; cache invalidation and physical index cleanup complete asynchronously.
  5. Build a parallel generation for embedding model E3, shadow-query it, compare retrieval quality, and conditionally switch the active manifest. A concurrent tombstone remains enforced.

Diagrams and relationships

Offline ingestion and online query planes

The offline plane publishes validated, versioned knowledge. The online plane reads only the active manifest and current authorization decisions.

Scroll horizontally on smaller screens, or enlarge the diagram.

Offline ingestion and online query planesThe offline plane publishes validated, versioned knowledge. The online plane reads only the active manifest and current authorization decisions. Enterprise sources to Queue + ingestion workers: versioned change. Queue + ingestion workers to Staged dense + sparse index: parse / chunk / embed. Staged dense + sparse index to Validated active manifest: validate + publish. Authenticated user to Identity + live ACL: verified identity. Identity + live ACL to Hybrid retrieval + rerank: authorized scope. Hybrid retrieval + rerank to Cited answer / abstain: validated evidence. Validated active manifest to Hybrid retrieval + rerank: active generation.versioned changeparse / chunk / embedvalidate + publishverified identityauthorized scopevalidated evidenceactive generationEnterprise sourcesQueue + ingestion workersStaged dense + sparse indexValidated active manifestAuthenticated userIdentity + live ACLHybrid retrieval + rerankCited answer / abstain
Offline ingestion and online query planesThe offline plane publishes validated, versioned knowledge. The online plane reads only the active manifest and current authorization decisions. Enterprise sources to Queue + ingestion workers: versioned change. Queue + ingestion workers to Staged dense + sparse index: parse / chunk / embed. Staged dense + sparse index to Validated active manifest: validate + publish. Authenticated user to Identity + live ACL: verified identity. Identity + live ACL to Hybrid retrieval + rerank: authorized scope. Hybrid retrieval + rerank to Cited answer / abstain: validated evidence. Validated active manifest to Hybrid retrieval + rerank: active generation.versioned changeparse / chunk / embedvalidate + publishverified identityauthorized scopevalidated evidenceactive generationEnterprise sourcesQueue + ingestion workersStaged dense + sparse indexValidated active manifestAuthenticated userIdentity + live ACLHybrid retrieval + rerankCited answer / abstain
Initial backfill and incremental document replacement

New content is staged independently; online reads continue on the active version until a validated conditional publication.

Scroll horizontally on smaller screens, or enlarge the diagram.

Initial backfill and incremental document replacementNew content is staged independently; online reads continue on the active version until a validated conditional publication. Inventory + source cursor to Bounded backfill / change feed: snapshot + cursor. Bounded backfill / change feed to Idempotent version job: durable jobs. Idempotent version job to New chunks in staging: parse / chunk / embed. New chunks in staging to Validate + compare-and-swap: complete candidate. Validate + compare-and-swap to Active version pointer: atomic publish. Active version pointer to Async old-version cleanup: superseded version. Queries continue on old version to Active version pointer: read active version.snapshot + cursordurable jobsparse / chunk / embedcomplete candidateatomic publishsuperseded versionread active versionInventory + source cursorBounded backfill / change feedIdempotent version jobNew chunks in stagingValidate + compare-and-swapActive version pointerQueries continue on old versionAsync old-version cleanup
Initial backfill and incremental document replacementNew content is staged independently; online reads continue on the active version until a validated conditional publication. Inventory + source cursor to Bounded backfill / change feed: snapshot + cursor. Bounded backfill / change feed to Idempotent version job: durable jobs. Idempotent version job to New chunks in staging: parse / chunk / embed. New chunks in staging to Validate + compare-and-swap: complete candidate. Validate + compare-and-swap to Active version pointer: atomic publish. Active version pointer to Async old-version cleanup: superseded version. Queries continue on old version to Active version pointer: read active version.snapshot + cursordurable jobsparse / chunk / embedcomplete candidateatomic publishsuperseded versionread active versionInventory + source cursorBounded backfill / change feedIdempotent version jobNew chunks in stagingValidate + compare-and-swapActive version pointerQueries continue on old versionAsync old-version cleanup
Authorization before model context

Candidate IDs can be retrieved under tenant scope, but content is hydrated only after current authorization checks.

Scroll horizontally on smaller screens, or enlarge the diagram.

Authorization before model contextCandidate IDs can be retrieved under tenant scope, but content is hydrated only after current authorization checks. Verified identity + tenant to Current policy resolver: principal. Current policy resolver to Dense + sparse candidate IDs: tenant + scope. Dense + sparse candidate IDs to ACL filter + recheck: IDs only. ACL filter + recheck to Authorized chunk hydration: allow only. Authorized chunk hydration to Rerank + prompt context: authorized text.principaltenant + scopeIDs onlyallow onlyauthorized textVerified identity + tenantCurrent policy resolverDense + sparse candidate IDsACL filter + recheckAuthorized chunk hydrationRerank + prompt context
Authorization before model contextCandidate IDs can be retrieved under tenant scope, but content is hydrated only after current authorization checks. Verified identity + tenant to Current policy resolver: principal. Current policy resolver to Dense + sparse candidate IDs: tenant + scope. Dense + sparse candidate IDs to ACL filter + recheck: IDs only. ACL filter + recheck to Authorized chunk hydration: allow only. Authorized chunk hydration to Rerank + prompt context: authorized text.principaltenant + scopeIDs onlyallow onlyauthorized textVerified identity + tenantCurrent policy resolverDense + sparse candidate IDsACL filter + recheckAuthorized chunk hydrationRerank + prompt context
Document and index version relationships

Source versions, chunks, embedding versions and corpus generations have distinct identities; a manifest controls publication.

Scroll horizontally on smaller screens, or enlarge the diagram.

Document and index version relationshipsSource versions, chunks, embedding versions and corpus generations have distinct identities; a manifest controls publication. Source document identity to Immutable document version: has versions. Immutable document version to Provenance-preserving chunks: parsed + chunked. Provenance-preserving chunks to Embedding model version: embedded by. Embedding model version to Index generation: indexed into. Index generation to Active manifest revision: published by. Immutable document version to Cleanup / tombstone task: superseded / deleted.has versionsparsed + chunkedembedded byindexed intopublished bysuperseded / deletedSource document identityImmutable document versionProvenance-preserving chunksEmbedding model versionIndex generationActive manifest revisionCleanup / tombstone task
Document and index version relationshipsSource versions, chunks, embedding versions and corpus generations have distinct identities; a manifest controls publication. Source document identity to Immutable document version: has versions. Immutable document version to Provenance-preserving chunks: parsed + chunked. Provenance-preserving chunks to Embedding model version: embedded by. Embedding model version to Index generation: indexed into. Index generation to Active manifest revision: published by. Immutable document version to Cleanup / tombstone task: superseded / deleted.has versionsparsed + chunkedembedded byindexed intopublished bysuperseded / deletedSource document identityImmutable document versionProvenance-preserving chunksEmbedding model versionIndex generationActive manifest revisionCleanup / tombstone task
Request lifecycle, validation and abstention

The system validates output structure, citations and grounding before a response is shown.

Scroll horizontally on smaller screens, or enlarge the diagram.

Request lifecycle, validation and abstentionThe system validates output structure, citations and grounding before a response is shown. Question + verified principal to Authorized evidence: authorize + search. Authorized evidence to Bounded evidence context: provenance. Bounded evidence context to LLM draft: bounded prompt. LLM draft to Schema + citation + grounding: untrusted output. Schema + citation + grounding to Cited answer: supported. Schema + citation + grounding to Clarify / abstain: insufficient / invalid.authorize + searchprovenancebounded promptuntrusted outputsupportedinsufficient / invalidQuestion + verified principalAuthorized evidenceBounded evidence contextLLM draftSchema + citation + groundingCited answerClarify / abstain
Request lifecycle, validation and abstentionThe system validates output structure, citations and grounding before a response is shown. Question + verified principal to Authorized evidence: authorize + search. Authorized evidence to Bounded evidence context: provenance. Bounded evidence context to LLM draft: bounded prompt. LLM draft to Schema + citation + grounding: untrusted output. Schema + citation + grounding to Cited answer: supported. Schema + citation + grounding to Clarify / abstain: insufficient / invalid.authorize + searchprovenancebounded promptuntrusted outputsupportedinsufficient / invalidQuestion + verified principalAuthorized evidenceBounded evidence contextLLM draftSchema + citation + groundingCited answerClarify / abstain

Engineer, Senior and Staff follow-ups

These are teaching lenses. Each answer changes the scope of reasoning; they are not official employer leveling criteria.

Engineer: A user asks about a file that was just updated. How do you avoid returning mixed old and new chunks?

Use immutable source/document versions and stable chunk identities. Build the new version in staging, validate completeness and metadata, then conditionally advance the active document-version mapping. Retrieval filters to the active version. Keep the old version until publication succeeds; clean it up asynchronously. For a permission revoke or deletion, deny immediately with a tombstone rather than waiting for replacement.

Engineer: A webhook is delivered three times and out of order. What happens?

Use a unique idempotency key over tenant, connector, stable source ID, source version and content hash. A duplicate observes the existing job. A worker fetches authoritative source state and compares source revision/cursor so an old event cannot overwrite a newer version. Periodic reconciliation repairs missed events.

Senior: How do you bootstrap 250,000 existing documents while production queries continue?

Capture a source snapshot/cursor and inventory, enqueue durable bounded pages, checkpoint only after jobs are persisted, and write to a staged generation. Keep online queries on the current active manifest. Isolate backfill capacity with quotas and weighted queues. Validate coverage, ACLs, model dimensions and retrieval before atomic publication; throttle when query latency or provider/index saturation crosses guardrails.

Senior: Why not just filter unauthorized results after the LLM responds?

Because the unauthorized content may already have reached the reranker, provider, prompt trace, cache or model context. Filter candidate IDs and recheck current policy before hydration; only authorized text can cross the model boundary. Fail closed if policy is unavailable.

Senior: How do you migrate to a new embedding model?

A new model—even with the same vector dimension—may define a different vector space. Build a parallel index generation with pinned model and transform versions, validate retrieval quality, shadow-query, then switch the manifest. Never mix old and new vectors in one active search space; retain the old generation for rollback.

Staff: How would you define correctness and ownership across many connectors and tenants?

Publish connector contracts for identity, snapshot/change cursor, ordering, ACL semantics, deletion and reconciliation. Define SLOs for freshness and revocation separately, tenant-level resource budgets, policy authority and fail-closed behavior, and ownership for connector, index, model gateway, deletion and incident response. Maintain a versioned corpus manifest and audit trail. Treat webhook delivery as at-least-once and potentially reordered, not as a global transaction.

Staff: A legal deletion arrives during a model/index migration. How do you prevent rollback from restoring deleted content?

The tombstone/policy epoch is authoritative and independent of index generation. Query-time authorization and publication checks consult the current deny state. A rollback may restore an older compatible index but cannot override current tombstones. Cleanup tasks target all generations, replicas, lexical indexes and caches; verify each store before retiring the deletion barrier.

Practice and self-review

Your practice draft

Design the enterprise RAG system under the synthetic 20-tenant, 250,000-document, 250-query/s workload. Cover initial backfill and continuous updates, source/version/job/chunk/index/manifest schemas, authorization before hydration, hybrid retrieval, token budgeting, citation validation, revocation, capacity, rollout and recovery. Then change one constraint: the embedding model changes, a source connector loses its change feed, or an ACL is revoked while the model provider is timing out.

Saved locally on this device. The worked answer does not score your draft. Existing lesson notes and completion are separate.

Progressive hints

Hint 1

Start with the decision, tenant/source/policy authorities, freshness and revocation targets, and a deterministic keyword-search baseline.

Hint 2

Draw separate offline and online planes. Define immutable source versions, durable idempotent jobs, staged indexes and the active manifest before selecting a vector database.

Hint 3

Trace a normal update, duplicate/out-of-order event, missed webhook, ACL revoke and provider timeout. State exactly what remains queryable and what must fail closed.

Reveal worked answer guidance

A strong design separates offline source ingestion from online query serving. It snapshots and inventories the existing corpus, runs bounded resumable backfill into a staged index, validates tenant/ACL/provenance/model compatibility and atomically publishes a manifest. New files and updates use idempotent source-version jobs; the old active version remains searchable until the replacement is validated. Reconciliation repairs missed/out-of-order events. Query requests derive tenant identity from verified authentication, resolve current permissions, retrieve bounded candidate IDs, filter and recheck authorization before hydration, then run hybrid retrieval/reranking, context budgeting and guarded generation. Citations are resolved to the exact authorized evidence versions; insufficient or invalid evidence leads to abstention. Revocation writes a durable tombstone and denies immediately while physical cleanup proceeds across indexes and caches. A model migration builds a parallel compatible index and cuts over only after shadow evaluation. Capacity estimates are explicitly synthetic; operational guardrails cover freshness, revocation, latency, cost, retrieval quality and provider failures.

Evidence to check in your answer

  • Framing: defines decision, users, tenants, sources, freshness, revocation, answer contract and assumptions.
  • Data correctness: immutable source/document versions, provenance, ACL references, idempotent jobs, checkpoints and active manifest.
  • Production migration: existing corpus backfill, incremental update, reconciliation, safe publication, rollback and online continuity.
  • Security: current policy before text hydration, tenant-safe caches, prompt-injection boundary, immediate revocation and deletion verification.
  • Retrieval/generation: dense+sparse fusion, bounded reranking, token budget, citation resolution, grounding and abstention.
  • Operations: queue isolation, quotas, failure recovery, observability, independent metrics, governance and ownership.
  • Evidence discipline: labels synthetic assumptions and distinguishes arithmetic/design from tested production performance.

For each item, record Missing, Prompted, Independent or Adaptive evidence. Adaptive means you can revise the reasoning under a changed constraint. Reading completion and readiness are separate.

Sources and verification scope

Original synthetic examples and reference calculations. Sources support narrowly scoped background; they do not establish production performance or independent chapter review.

SRE workbook: data processing pipelines ↗ · Pipeline SLOs and operational ownership; do not copy source workload numbers as course production results.

NIST AI Risk Management Framework ↗ · Risk vocabulary and governance; not a certification or blanket legal-compliance guarantee.

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks ↗ · Historical research foundation for retrieval/generation; not a complete modern permission/security architecture.

Hidden Technical Debt in Machine Learning Systems ↗ · System-boundary and lifecycle risks; source lessons motivate original examples.

Primary background links checked during authoring on 2026-10-19. Arithmetic, temporal eligibility, maturity and split reference functions are checked by the bounded local fixture suite. SQL is explanatory, not a migrated production query. No GPU, external dataset, model API or training job is required.