Learning pathsPricingChecking sign-in…
← ML curriculum

AISD-004 · WORKED SYSTEM DESIGN

Multimodal video understanding platform

Design a production multimodal video understanding platform that ingests uploaded and live video, extracts synchronized visual/audio signals, indexes searchable moments, answers grounded questions and supports asynchronous processing with safe retries, tenant isolation and deletion.

independent technical review pending. source-contract checks pending.

Practice this design before reading →

Prerequisites and scoped previews

Events, identity and data contracts

Define keys, event/ingestion/availability times, schema evolution, dedupe and permitted use.

Calibration, uncertainty and decision thresholds

Distinguish score quality from calibrated probabilities, uncertainty and action capacity.

Multi-tenant LLM inference gateway

Tenant-safe model routing, atomic budget admission, KV-aware serving, resumable streams and evidence-based usage settlement.

What, when, why and how

Treat media as a timeline, not a file

what
A video is a synchronized set of timed streams: video frames, audio segments, captions, metadata and derived artifacts.
when
For search, summarization, clip generation and live understanding.
why
Flattening everything into one transcript or one embedding loses temporal alignment and modality-specific evidence.
how
Use a canonical media asset ID, rational timestamps, stream metadata and versioned derived segments.

Separate control plane from media data plane

what
The API owns identity, job state, policy and quotas; object storage and workers move large media bytes.
when
For large uploads, long-running jobs and bursty GPU workloads.
why
Proxying multi-GB media through application servers couples API availability to media throughput.
how
Use resumable direct-to-object-store uploads, signed scoped URLs, durable job records and event-driven workers.

Make preprocessing reproducible

what
Every frame, audio segment, transcript and embedding records source version, time range, extractor/model version and configuration.
when
When reprocessing, debugging, comparing model versions or answering with citations.
why
Unversioned derived data cannot be safely invalidated or reproduced.
how
Store immutable artifact manifests and publish an index generation only after validation.

Choose temporal sampling by task

what
Frame sampling, scene-boundary detection, audio windows and adaptive crops trade compute against missed events.
when
For long videos, dense action recognition, OCR or sparse semantic search.
why
Uniform low-rate sampling is cheap but can miss short events; dense sampling can overwhelm cost and context limits.
how
Use coarse scan then targeted refinement, preserve timestamps and measure recall on short-event slices.

Fuse modalities with evidence provenance

what
Text, image/frame, audio and metadata retrieval can be fused at segment level while retaining original modality and time span.
when
For cross-modal search and question answering.
why
A single fused vector hides why a result matched and can blur contradictory evidence.
how
Retrieve modality-specific candidates, normalize scores, fuse/rerank and return timestamped evidence references.

Use async jobs for expensive work

what
Upload acceptance, processing, indexing and result publication are distinct durable states.
when
For minutes-long video and GPU-intensive extraction or generation.
why
A synchronous HTTP request cannot safely represent long work, retries or partial completion.
how
Persist state transitions, checkpoint stages, use idempotent artifact keys and expose progress and cancellation.

Enforce rights, privacy and deletion across derivatives

what
Authorization and retention apply to source media, transcripts, frames, embeddings, caches and generated clips.
when
For user uploads, enterprise recordings and regulated media.
why
Deleting the original while retaining searchable derivatives is incomplete deletion.
how
Maintain lineage manifests, deny access immediately on revocation, tombstone descendants and reconcile physical cleanup.

Frame the multimodal video platform

Build a platform where a tenant uploads or streams video and later searches, summarizes or asks questions about specific moments. The system must preserve temporal grounding across visual and audio evidence, process long media asynchronously, and return citations that resolve to playable time ranges.

Synthetic exercise assumptions: 50,000 tenants, 500,000 uploaded videos/month, median duration 12 minutes, p95 duration 2 hours, 20% with audio, and 10% requiring high-resolution frame analysis. These are interview inputs, not measured production facts. Model storage and compute from duration distributions, not file count alone.

Define product requirements and SLOs

Functional scope: resumable upload, metadata validation, asynchronous processing, transcript and scene extraction, multimodal indexing, natural-language search, timestamped Q&A, clip export, progress, cancellation, retention and deletion.

Separate upload availability, job acceptance latency, processing throughput (media-minutes per wall-clock minute), indexing freshness, search recall, answer groundedness, timestamp accuracy and clip export latency. A 30-minute video processed in 30 minutes is 1x real-time; do not confuse worker utilization with user-visible completion.

Define a canonical media and timeline contract

Represent the asset with tenant-scoped assetId, immutable sourceVersion, content hash, container/codec metadata, duration, frame rate, dimensions, audio stream descriptors, language hints, rights/consent metadata and retention deadline. Normalize timestamps to a canonical timebase; preserve original stream time bases and offsets for round-trip accuracy.

A derived segment records [start,end), modality, sourceVersion, extractor version, model version, configuration hash, object URI, checksum and provenance. Use half-open intervals to avoid boundary duplication. Variable frame rate and audio/video offset must be explicit rather than assuming frameNumber / nominalFps is exact.

Build resumable upload and durable ingestion

Authenticate and authorize before issuing a short-lived multipart upload capability scoped to one tenant, asset and object prefix. The client uploads directly to object storage; the API stores upload session state and validates completion, size, checksum, content signature and declared metadata before creating a processing job.

Object-created notifications are at-least-once and may arrive out of order. Deduplicate on tenant, asset, sourceVersion and object checksum. Persist an outbox/job record before acknowledging acceptance; reconcile incomplete multipart uploads and orphaned objects. Never trust a client-supplied MIME type as proof of media format.

VISUAL WALKTHROUGH

See the system in motion

Follow a scoped resumable upload into validated immutable media and a durable processing job.

Process overviewAvailable even when interactive playback cannot load.
  1. Authorize upload

  2. Upload media parts

  3. Validate checksum and probe

  4. Persist idempotent job

Loading interactive walkthrough…

Orchestrate the processing DAG

A durable workflow branches into media probing, video scene/frame extraction, audio extraction, ASR/diarization, OCR and metadata enrichment. Each stage writes immutable artifacts and a manifest checkpoint. A join stage waits for required branches or marks optional branches degraded with an explicit reason.

Use stage-specific queues and worker pools because CPU decoding, OCR, ASR and GPU vision have different resource profiles. Apply bounded concurrency, per-tenant fairness, queue age limits, retries with jitter and dead-letter handling. Retry only idempotent stages; deterministic artifact keys prevent duplicate billing and output corruption.

Sample frames and detect scene boundaries

Start with a coarse temporal scan, scene-change detection and task-specific sampling. High-level summaries may use sparse keyframes; action recognition, small-object detection and brief events need denser targeted windows. OCR-heavy content may need crops and resolution-aware processing.

Record each frame timestamp, source frame identity, crop transform and extractor configuration. Adaptive refinement should be budgeted: use a cheap pass to find candidate intervals, then spend additional frame/GPU budget only where uncertainty or task value warrants it. Evaluate short-event recall separately from average video retrieval quality.

VISUAL WALKTHROUGH

See the system in motion

See source-version pinning, stream probing and timestamp-preserving frame extraction.

Process overviewAvailable even when interactive playback cannot load.
  1. Pin source version

  2. Read stream time bases

  3. Extract keyframes and scenes

  4. Commit immutable artifacts

Loading interactive walkthrough…

Process audio without losing alignment

Extract audio with its original time base, then run voice activity detection, ASR, optional diarization and audio-event classification. Store word or phrase timestamps, speaker IDs with confidence, language, channel and alignment quality. Keep raw transcript and normalized transcript as separate artifacts.

Audio and video clocks can drift or start at different offsets. Align using container timestamps and synchronization metadata; never infer alignment solely from segment order. For live streams, use overlapping windows and revise provisional transcripts while exposing a stable committed watermark.

VISUAL WALKTHROUGH

See the system in motion

Trace audio extraction, speech recognition and alignment to the canonical media timeline.

Process overviewAvailable even when interactive playback cannot load.
  1. Extract original audio stream

  2. Detect speech windows

  3. Transcribe and diarize

  4. Align words to canonical time

Loading interactive walkthrough…

Build a temporal multimodal index

Create modality-specific representations: text embeddings for transcript/captions, visual embeddings for keyframes or clips, and structured metadata for time, speaker, scene, entities and access policy. Keep a segment catalog that maps each vector back to the exact source interval and artifact version.

At query time, classify intent and retrieve from relevant modalities in parallel. Normalize and fuse candidates, deduplicate overlapping intervals, rerank with a cross-modal model where justified, and enforce authorization before content hydration. Do not treat cosine similarity across separately trained embedding spaces as directly comparable without calibration.

VISUAL WALKTHROUGH

See the system in motion

Build a versioned temporal index from modality-specific artifacts and embeddings.

Process overviewAvailable even when interactive playback cannot load.
  1. Load visual artifacts

  2. Load timestamped text spans

  3. Create modality-specific vectors

  4. Stage and validate index

Loading interactive walkthrough…

Answer questions with timestamped evidence

Assemble a bounded evidence pack containing transcript spans, frame descriptions, keyframe references, speaker/time metadata and source versions. Ask the model to answer only from supplied evidence and return claims linked to one or more intervals. Validate that every citation resolves to an authorized asset version and playable timestamp.

When modalities disagree, expose uncertainty or retrieve more evidence rather than silently blending. If no evidence supports a claim, abstain. A generated summary is not a source of truth; preserve provenance to the underlying media and extraction versions.

VISUAL WALKTHROUGH

See the system in motion

Authorize retrieval, fuse timestamped candidates and return playable citations.

Process overviewAvailable even when interactive playback cannot load.
  1. Authorize asset scope

  2. Retrieve modality candidates

  3. Fuse and rerank intervals

  4. Return grounded citations

Loading interactive walkthrough…

Generate clips as a separate derived workflow

Clip creation is an asynchronous derivative job: validate requested ranges, enforce rights and policy, resolve source version, render with bounded CPU/GPU resources, store the output as a new asset and publish only after checksum and playback validation.

A clip references its parent asset and exact source interval, plus transformations such as crop, captions, redaction and audio mix. Recheck authorization at download time. Do not mutate the original media or imply that a generated highlight is a verbatim source segment when it includes synthesized content.

Enforce tenant security and media rights

Derive tenant identity from authenticated credentials and authorize every asset, job, segment, transcript, frame, vector, clip and signed URL. Keep object prefixes and encryption keys tenant-scoped where the threat model requires it; short-lived URLs must be constrained to a specific object and operation.

Treat uploaded media, subtitles, OCR text and embedded QR codes as untrusted content. They may contain prompt injection, secrets or personal data. Model instructions cannot grant access to another asset or authorize export. Apply consent, copyright/usage rights, regional processing, retention and moderation policy before indexing or generation.

Handle deletion, retries and recovery

On deletion or ACL revocation, write a durable tombstone and deny search/playback immediately. Then invalidate caches, remove vector entries, transcripts, frame artifacts, clips and replicas asynchronously. A lineage graph lets the reconciler enumerate every descendant and verify physical deletion against retention/legal-hold rules.

Workers use leases and fencing tokens; stale workers cannot publish after a newer attempt or deletion. Persist stage checkpoints and artifact checksums. Cancellation prevents new stages and publication, but cleanup of already-created temporary files remains a separate task. Reconcile stuck jobs, missing objects, orphan vectors and partial outputs continuously.

VISUAL WALKTHROUGH

See the system in motion

Deny access immediately, fence stale workers and reconcile every derived artifact.

Process overviewAvailable even when interactive playback cannot load.
  1. Persist tombstone

  2. Deny reads and fence publication

  3. Enumerate descendants

  4. Delete and reconcile artifacts

Loading interactive walkthrough…

Estimate storage, bandwidth and compute

Estimate source storage as uploads/day × average bytes × retention days, then add replication and derived artifacts. Estimate decode cost from total media-hours, codec complexity, frame sampling rate and audio duration. Estimate embedding cost from segment count and model batch throughput; estimate GPU vision cost from selected frames/crops rather than total video duration alone.

Use separate CPU decode, GPU vision, ASR and embedding capacity pools. Queue by tenant and deadline, reserve capacity for interactive queries, and shed optional enrichment before core indexing. Track cost per processed media-minute, indexed minute, successful grounded answer and exported clip.

Measure quality and operational health

Operational signals include ingest lag, stage queue age, processing speed ratio, retries, dead letters, GPU memory/utilization, decode errors, missing modality rate, index freshness, deletion lag and signed URL failures. Trace one asset/job across every stage with tenant-safe identifiers and model/extractor versions.

Evaluate temporal retrieval recall, timestamp intersection-over-union, ASR word error rate by language/noise slice, speaker attribution, visual event recall, grounded claim precision, citation validity, abstention and user task success. Report modality-specific slices; aggregate scores can hide failures on short events, accents, low light or variable frame rate.

Walk through partial failure and reprocessing

Suppose video decode succeeds but ASR times out. Publish the visual branch as a partial artifact only if product policy permits partial search; mark audio unavailable and keep the job resumable. Retry ASR with the same immutable source and deterministic stage key. Do not rebuild already validated visual embeddings unnecessarily.

If the source is replaced while a worker is processing, the old worker may finish but cannot publish into the new version. A new sourceVersion creates a new job and index generation. If deletion arrives during processing, the tombstone fences publication and descendant cleanup proceeds from the lineage manifest.

Complete multimodal video architecture

The end-to-end path is authenticated upload API → scoped multipart object storage → durable asset/job record → event-driven workflow → media probe → parallel video/audio extraction → timestamped artifacts → modality-specific embeddings and metadata → staged temporal index → validated publication → authorized search and grounded Q&A → timestamped playback or derived clip.

The control plane owns tenant policy, job state, version pins, publication and deletion authority. The media data plane owns large objects and immutable derived artifacts. Workers own stage execution evidence; the index is a rebuildable projection. Failure paths include duplicate events, partial modality failure, stale workers, source replacement, deletion during processing and index reconciliation.

Complete animated architecture walkthrough

Follow the complete upload, temporal extraction, multimodal indexing, authorized retrieval and grounded-answer lifecycle, including version and provenance boundaries.

VISUAL WALKTHROUGH

See the system in motion

Follow the full multimodal video platform from authenticated upload through synchronized extraction, temporal indexing and timestamped grounded answers.

Process overviewAvailable even when interactive playback cannot load.
  1. Authorize and create scoped upload

  2. Store immutable source version

  3. Probe media and establish time bases

  4. Extract frames, scenes and audio in parallel

  5. Persist timestamped artifacts and transcripts

  6. Build and validate modality-specific indexes

  7. Authorize retrieval and assemble evidence

  8. Return grounded answer with playable citations

Loading interactive walkthrough…

Diagrams and relationships

Control plane and media data plane

The API controls identity and workflow; media bytes bypass application servers.

Scroll horizontally on smaller screens, or enlarge the diagram.

Control plane and media data planeThe API controls identity and workflow; media bytes bypass application servers. Uploader to Media API: authorize. Media API to Job ledger: create session. Media API to Object store: signed upload. Job ledger to Workflow: durable job. Object store to CPU / GPU workers: read media. Workflow to CPU / GPU workers: dispatch.authorizecreate sessionsigned uploaddurable jobread mediadispatchUploaderMedia APIJob ledgerObject storeWorkflowCPU / GPU workers
Control plane and media data planeThe API controls identity and workflow; media bytes bypass application servers. Uploader to Media API: authorize. Media API to Job ledger: create session. Media API to Object store: signed upload. Job ledger to Workflow: durable job. Object store to CPU / GPU workers: read media. Workflow to CPU / GPU workers: dispatch.authorizecreate sessionsigned uploaddurable jobread mediadispatchUploaderMedia APIJob ledgerObject storeWorkflowCPU / GPU workers
Temporal multimodal processing

Parallel modality branches preserve timestamps and provenance before index publication.

Scroll horizontally on smaller screens, or enlarge the diagram.

Temporal multimodal processingParallel modality branches preserve timestamps and provenance before index publication. Versioned media to Probe + timeline: source version. Probe + timeline to Frames / scenes: video timebase. Probe + timeline to ASR / speakers: audio timebase. Frames / scenes to Embeddings: visual vectors. ASR / speakers to Embeddings: text vectors. Embeddings to Temporal index: publish generation.source versionvideo timebaseaudio timebasevisual vectorstext vectorspublish generationVersioned mediaProbe + timelineFrames / scenesASR / speakersEmbeddingsTemporal index
Temporal multimodal processingParallel modality branches preserve timestamps and provenance before index publication. Versioned media to Probe + timeline: source version. Probe + timeline to Frames / scenes: video timebase. Probe + timeline to ASR / speakers: audio timebase. Frames / scenes to Embeddings: visual vectors. ASR / speakers to Embeddings: text vectors. Embeddings to Temporal index: publish generation.source versionvideo timebaseaudio timebasevisual vectorstext vectorspublish generationVersioned mediaProbe + timelineFrames / scenesASR / speakersEmbeddingsTemporal index
Grounded temporal retrieval

Search returns authorized intervals, not untraceable global summaries.

Scroll horizontally on smaller screens, or enlarge the diagram.

Grounded temporal retrievalSearch returns authorized intervals, not untraceable global summaries. User query to Tenant + ACL: authenticate. Tenant + ACL to Multi-modal retrieval: authorized scope. Multi-modal retrieval to Temporal reranker: timestamped candidates. Temporal reranker to Grounded answer: evidence pack.authenticateauthorized scopetimestamped candidatesevidence packUser queryTenant + ACLMulti-modal retrievalTemporal rerankerGrounded answer
Grounded temporal retrievalSearch returns authorized intervals, not untraceable global summaries. User query to Tenant + ACL: authenticate. Tenant + ACL to Multi-modal retrieval: authorized scope. Multi-modal retrieval to Temporal reranker: timestamped candidates. Temporal reranker to Grounded answer: evidence pack.authenticateauthorized scopetimestamped candidatesevidence packUser queryTenant + ACLMulti-modal retrievalTemporal rerankerGrounded answer

Engineer, Senior and Staff follow-ups

These are teaching lenses. Each answer changes the scope of reasoning; they are not official employer leveling criteria.

Engineer: Why use direct-to-object-store upload?

It removes large media bytes from the API tier, supports multipart/resume and lets API capacity scale independently. The API still authorizes the upload session and validates completion, checksum and object metadata.

Engineer: Why preserve separate audio and video time bases?

Streams may have offsets, drift or variable frame rate. Separate time bases preserve alignment and let citations resolve to the original media timeline.

Senior: How would you avoid duplicate processing from at-least-once object events?

Use an idempotency key based on tenant, asset, source version and checksum; persist stage state and deterministic artifact keys; fence stale attempts before publication.

Senior: How do you balance frame sampling cost and event recall?

Use a coarse scan and scene boundaries, then adaptive refinement around uncertain or high-value intervals. Evaluate short-event recall independently and cap per-asset refinement budgets.

Senior: How do you combine text and visual retrieval scores?

Do not assume raw cosine scores are calibrated across modalities. Normalize on held-out relevance data, fuse ranks or use a learned reranker, and retain modality-specific provenance.

Staff: How would you support a two-hour video without monopolizing GPUs?

Chunk by temporal windows, separate CPU decode from GPU inference, use stage-specific queues, per-tenant fair scheduling, bounded in-flight work, checkpointed artifacts and deadline-aware admission.

Staff: What happens when a user deletes a source while derived clips are being generated?

A tombstone immediately denies access and fences publication. The lineage graph identifies in-flight and completed descendants; workers check the source generation before commit and the reconciler removes all derivatives and cache entries.

Staff: How do you make a multimodal answer auditable?

Persist the exact source version, segment intervals, extraction/model versions, retrieval candidates, authorization decision and citation mapping. The answer references playable time ranges and can be recomputed from immutable evidence.

Practice and self-review

Your practice draft

Design a multi-tenant video understanding platform supporting large resumable uploads, asynchronous extraction, audio/video alignment, multimodal search, grounded timestamped Q&A and deletion. Explain capacity isolation, temporal provenance, partial failure and reprocessing.

Saved locally on this device. The worked answer does not score your draft. Existing lesson notes and completion are separate.

Progressive hints

Hint 1

Start with an immutable asset version and canonical timeline contract.

Hint 2

Keep large media bytes out of the API tier; use scoped resumable object uploads.

Hint 3

Split CPU decode, GPU vision, ASR and embedding into resource-specific stages.

Hint 4

Preserve modality-specific timestamps and provenance through the index.

Hint 5

Return authorized timestamped evidence, not unsupported global summaries.

Hint 6

Use tombstones, lineage and fencing to make deletion/recovery safe.

Reveal worked answer guidance

Authenticate the tenant, create an upload session and issue a scoped multipart capability. Validate checksum and media metadata on completion, then persist an idempotent job. Probe streams and establish time bases; branch into video frame/scene extraction and audio VAD/ASR/diarization. Write immutable timestamped artifacts with source and extractor versions. Generate modality-specific embeddings and metadata, stage a temporal index generation, validate coverage and publish atomically. Search authenticates and authorizes first, retrieves modality-specific candidates, fuses and reranks intervals, assembles bounded evidence and returns claims with playable timestamps. Clip export is a separate derived job. Use stage-specific CPU/GPU queues, leases, checkpoints and deterministic artifact keys. Deletion writes a tombstone immediately, fences workers and asynchronously removes all descendants using lineage reconciliation.

Evidence to check in your answer

  • Tenant identity and per-asset authorization are server-derived.
  • Uploads are resumable and scoped; checksum and media signature are validated.
  • Asset and derived artifacts are immutable and versioned.
  • Audio/video time bases and offsets are preserved.
  • CPU, GPU, ASR and embedding workloads are independently scheduled.
  • Every stage is idempotent, checkpointed and fenced against stale publication.
  • Temporal retrieval returns authorized evidence with playable timestamps.
  • Partial modality failures are explicit and resumable.
  • Deletion covers source, transcript, frames, vectors, caches and clips.
  • Evaluation includes temporal recall, timestamp accuracy, modality slices and groundedness.

For each item, record Missing, Prompted, Independent or Adaptive evidence. Adaptive means you can revise the reasoning under a changed constraint. Reading completion and readiness are separate.

Sources and verification scope

Original synthetic examples and reference calculations. Sources support narrowly scoped background; they do not establish production performance or independent chapter review.

Primary background links checked during authoring on 2026-10-19. Arithmetic, temporal eligibility, maturity and split reference functions are checked by the bounded local fixture suite. SQL is explanatory, not a migrated production query. No GPU, external dataset, model API or training job is required.