Learning pathsPricingChecking sign-in…
← ML curriculum

AIF-005 · FOUNDATION

RAG ingestion, chunking and index lifecycle

RAG ingestion converts changing source systems into versioned, searchable evidence. Chunking must preserve structure and provenance, while incremental ingestion, deletion handling and atomic index publication keep an existing production corpus correct during continuous updates.

Author self-review; independent technical review pending. Synthetic deterministic examples; no live embedding provider, vector database, benchmark or production deployment.

Prerequisites and scoped previews

Context windows and context engineering

Construct bounded, relevant and permission-safe context from instructions, conversation state, retrieved evidence and tool results.

Embedding spaces, similarity and model compatibility

Choose a task-appropriate embedding contract, understand vector similarity and migrate model/index generations without mixing incompatible spaces.

What, when, why and how

A chunk is a retrieval unit with provenance

what
A chunk is a bounded passage derived from a source revision with stable identity, offsets and metadata.
when
Use chunking when source documents exceed model/index limits or retrieval needs precise evidence spans.
why
Oversized chunks dilute relevance; tiny chunks lose context; arbitrary boundaries break tables, code and section meaning.
how
Parse structure first, split by semantic boundaries, apply token limits and overlap only where needed, and retain exact source spans.

Ingestion is a versioned state machine

what
A source revision moves through capture, parse, normalize, chunk, embed, stage, validate and publish states.
when
Use it for connectors, uploads, scheduled crawls and ongoing source changes.
why
A partial pipeline can expose mixed versions, duplicate chunks or stale permissions.
how
Use durable jobs, idempotency keys, checkpoints, source-version ordering and an atomic active-generation pointer.

Existing production data needs backfill plus change capture

what
A migration must establish a consistent baseline and then apply creates, updates, deletes and ACL changes without losing concurrent writes.
when
Use it when launching RAG over an existing corpus or changing parser, chunker, embedding model or index schema.
why
A one-time bulk load misses updates during the scan; a change stream alone has no complete baseline.
how
Take a source watermark/snapshot, backfill bounded partitions, replay changes after the watermark, reconcile counts/hashes and publish only a complete generation.

1. Chunk by structure before token count

Start with document structure: title, headings, paragraphs, lists, tables, code blocks and page boundaries. Split at semantic boundaries, then enforce a model-specific token ceiling. Keep table headers with rows, code with relevant signatures, and page/section metadata with each chunk. A parent-child strategy can retrieve a focused child while hydrating a bounded parent window.

Overlap can protect boundary context, but it duplicates tokens, storage and near-identical search results. Use a measured overlap policy, stable chunk IDs and deduplication. Evaluate chunk size/overlap on retrieval and answer-support outcomes; there is no universal best size.

Source typePreferAvoid
ProseHeading/paragraph boundariesBlind fixed character slicing
TablesHeader + row groupsSeparating values from headers
CodeSymbol/function/class boundariesSplitting signatures from bodies
PDFPage + section + bounding/source spanLosing page and figure references

2. Give every derived chunk a reproducible identity

A chunk record should include tenant ID, source ID, source revision/content hash, parser version, chunker version, chunk ID, source offsets/page/section, ACL reference/version, language, modality, embedding version and index generation. The chunk text is derived content; source system and policy remain authoritative.

Use deterministic identities such as hash(tenant, source, revision, parser, chunker, ordinal, span). If the same event is delivered twice, the same logical chunk is upserted rather than duplicated. If the source revision changes, obsolete chunks are tombstoned and removed from active retrieval.

3. Add RAG to an existing production corpus safely

Example migration: at T0 record a consistent source watermark W and create generation g2. Backfill the snapshot at W in bounded batches. Meanwhile, durably capture all source events after W. After backfill, replay those events in source-version order, deduplicate by event ID and reject stale revisions. Reconcile source counts, content hashes, ACL revisions and embedding coverage before marking g2 ready.

Deletes and permission revocations need immediate deny semantics. Write a durable tombstone or authoritative deny marker before asynchronous vector cleanup; query-time authorization must reject stale candidates. Publish a complete generation atomically; never expose half of a backfill. Keep g1 serving until g2 passes validation and canary checks.

PhaseRead pathWrite pathExit condition
Snapshotg1Capture watermark WBaseline fixed
Backfillg1Build g2 from WAll partitions checkpointed
Catch upg1Replay events after WNo event lag / stale revision
Validateg1Reconcile g2Counts, hashes, ACL and coverage pass
Cut overg2 by cohortContinue updatesCanary and rollback healthy

VISUAL WALKTHROUGH

See the system in motion

Backfill an existing corpus while replaying concurrent updates and publishing a complete index generation.

Process overviewAvailable even when interactive playback cannot load.
  1. Capture a source watermark

  2. Backfill bounded partitions

  3. Persist checkpoints and idempotency keys

  4. Replay updates and deletions after the watermark

  5. Reconcile source hashes and ACL revisions

  6. Publish the validated generation atomically

Loading interactive walkthrough…

4. Recover from partial ingestion and source drift

Persist job state and per-partition checkpoints. A retry resumes from the last committed stage; it does not create another active generation. Poison documents go to a visible quarantine with reason and replay path. Provider failures use bounded backoff and concurrency limits; never mark failed embeddings as successful empty vectors.

Reconciliation compares source revision/hash and ACL version against the indexed manifest. It repairs missed events, detects stale chunks and verifies deletion completion. Retention and legal deletion must cover source copies, extracted text, object storage, vector/sparse indexes, caches, backups and derived summaries according to the applicable policy.

Diagrams and relationships

Existing corpus to a new searchable generation

The old index stays live while a baseline backfill and ordered change replay build a validated replacement.

Scroll horizontally on smaller screens, or enlarge the diagram.

Existing corpus to a new searchable generationThe old index stays live while a baseline backfill and ordered change replay build a validated replacement. Source snapshot W to Backfill workers: baseline. Backfill workers to Staged generation g2: chunks + vectors. Change log after W to Replay + reconcile: ordered events. Replay + reconcile to Staged generation g2: catch up. Staged generation g2 to Validate + publish: complete generation.baselinechunks + vectorsordered eventscatch upcomplete generationSource snapshot WBackfill workersStaged generation g2Change log after WReplay + reconcileValidate + publish
Existing corpus to a new searchable generationThe old index stays live while a baseline backfill and ordered change replay build a validated replacement. Source snapshot W to Backfill workers: baseline. Backfill workers to Staged generation g2: chunks + vectors. Change log after W to Replay + reconcile: ordered events. Replay + reconcile to Staged generation g2: catch up. Staged generation g2 to Validate + publish: complete generation.baselinechunks + vectorsordered eventscatch upcomplete generationSource snapshot WBackfill workersStaged generation g2Change log after WReplay + reconcileValidate + publish

Engineer, Senior and Staff follow-ups

These are teaching lenses. Each answer changes the scope of reasoning; they are not official employer leveling criteria.

Engineer: Why is overlap not always beneficial?

It increases duplicate vectors and near-identical candidates, consumes storage and tokens, and can crowd out distinct evidence. Measure boundary misses against the added cost.

Engineer: What metadata must survive chunking?

At minimum tenant/source identity, source revision, chunk ID, source offsets or page/section, parser/chunker version and authoritative ACL reference.

Senior: How do you backfill while documents keep changing?

Record a consistent watermark, backfill the baseline, durably capture changes after it, replay in source-version order, reconcile and atomically publish only when caught up.

Senior: What happens when an ACL is revoked during a backfill?

Write an authoritative deny/tombstone first and enforce it at query time; asynchronous index cleanup follows. A stale vector must not expose content while cleanup is pending.

Staff: How do you guarantee no source events are lost?

Use a durable event log or connector cursor with a defined watermark, idempotent consumers, ordered version checks, replayable checkpoints and reconciliation against the source of truth.

Staff: How do you make reindexing reversible?

Use immutable generations and an atomic routing manifest, keep the prior generation available for a bounded rollback period, and bind query encoders and schemas to each generation.

Practice and self-review

Your practice draft

A company has 4 million existing files across SharePoint, Google Drive and S3. RAG must launch without downtime while users continue to edit documents and revoke access. Design ingestion and index migration.

Saved locally on this device. The worked answer does not score your draft. Existing lesson notes and completion are separate.

Progressive hints

Hint 1

Establish source-specific watermarks and stable source revision identities.

Hint 2

Backfill in bounded, checkpointed partitions while durably capturing updates/deletes/ACL changes.

Hint 3

Keep old generation serving until reconciliation and query-quality canary gates pass.

Reveal worked answer guidance

Build connector-specific source manifests with tenant/source identity, revision or ETag, content hash and ACL version. For each connector, establish a consistent baseline watermark and durable change cursor. Backfill a new immutable index generation with bounded workers and deterministic chunk IDs; persist per-file checkpoints and retry outcomes. Replay changes after each watermark in source-version order, treating deletes and ACL revocations as deny-first tombstones. Reconcile source inventory, hashes, revision lag, ACL state, chunk/embedding coverage and index readiness. Run representative query and permission tests against the staged generation, then atomically route a canary cohort to the matching query encoder/index. Roll back to the previous generation if quality, latency or authorization guardrails fail; retire it only after the rollback window and retention checks.

Evidence to check in your answer

  • Consistent snapshot and change-capture strategy.
  • Idempotent chunk identity and durable checkpoints.
  • Deny-first deletion and ACL revocation.
  • Atomic publication, validation and rollback.

For each item, record Missing, Prompted, Independent or Adaptive evidence. Adaptive means you can revise the reasoning under a changed constraint. Reading completion and readiness are separate.

Sources and verification scope

Original synthetic examples and reference calculations. Sources support narrowly scoped background; they do not establish production performance or independent chapter review.

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks ↗ · Historical research foundation for retrieval/generation; not a complete modern permission/security architecture.

Hidden Technical Debt in Machine Learning Systems ↗ · System-boundary and lifecycle risks; source lessons motivate original examples.

Chunk large documents for RAG and vector search ↗ · Chunking strategy, structure-aware splitting and overlap trade-offs; Azure examples are not universal defaults.

Define index projections ↗ · Parent/child chunk indexing and change-tracking behavior for supported Azure AI Search data sources; connector semantics remain provider-specific.

Primary background links checked during authoring on 2026-10-19. Arithmetic, temporal eligibility, maturity and split reference functions are checked by the bounded local fixture suite. SQL is explanatory, not a migrated production query. No GPU, external dataset, model API or training job is required.