Prerequisites and scoped previews
Events, identity and data contracts
Define keys, event/ingestion/availability times, schema evolution, dedupe and permitted use.
Calibration, uncertainty and decision thresholds
Distinguish score quality from calibrated probabilities, uncertainty and action capacity.
Multi-tenant LLM inference gateway
Tenant-safe model routing, atomic budget admission, KV-aware serving, resumable streams and evidence-based usage settlement.
What, when, why and how
Treat media as a timeline, not a file
- what
- A video is a synchronized set of timed streams: video frames, audio segments, captions, metadata and derived artifacts.
- when
- For search, summarization, clip generation and live understanding.
- why
- Flattening everything into one transcript or one embedding loses temporal alignment and modality-specific evidence.
- how
- Use a canonical media asset ID, rational timestamps, stream metadata and versioned derived segments.
Separate control plane from media data plane
- what
- The API owns identity, job state, policy and quotas; object storage and workers move large media bytes.
- when
- For large uploads, long-running jobs and bursty GPU workloads.
- why
- Proxying multi-GB media through application servers couples API availability to media throughput.
- how
- Use resumable direct-to-object-store uploads, signed scoped URLs, durable job records and event-driven workers.
Make preprocessing reproducible
- what
- Every frame, audio segment, transcript and embedding records source version, time range, extractor/model version and configuration.
- when
- When reprocessing, debugging, comparing model versions or answering with citations.
- why
- Unversioned derived data cannot be safely invalidated or reproduced.
- how
- Store immutable artifact manifests and publish an index generation only after validation.
Choose temporal sampling by task
- what
- Frame sampling, scene-boundary detection, audio windows and adaptive crops trade compute against missed events.
- when
- For long videos, dense action recognition, OCR or sparse semantic search.
- why
- Uniform low-rate sampling is cheap but can miss short events; dense sampling can overwhelm cost and context limits.
- how
- Use coarse scan then targeted refinement, preserve timestamps and measure recall on short-event slices.
Fuse modalities with evidence provenance
- what
- Text, image/frame, audio and metadata retrieval can be fused at segment level while retaining original modality and time span.
- when
- For cross-modal search and question answering.
- why
- A single fused vector hides why a result matched and can blur contradictory evidence.
- how
- Retrieve modality-specific candidates, normalize scores, fuse/rerank and return timestamped evidence references.
Use async jobs for expensive work
- what
- Upload acceptance, processing, indexing and result publication are distinct durable states.
- when
- For minutes-long video and GPU-intensive extraction or generation.
- why
- A synchronous HTTP request cannot safely represent long work, retries or partial completion.
- how
- Persist state transitions, checkpoint stages, use idempotent artifact keys and expose progress and cancellation.
Enforce rights, privacy and deletion across derivatives
- what
- Authorization and retention apply to source media, transcripts, frames, embeddings, caches and generated clips.
- when
- For user uploads, enterprise recordings and regulated media.
- why
- Deleting the original while retaining searchable derivatives is incomplete deletion.
- how
- Maintain lineage manifests, deny access immediately on revocation, tombstone descendants and reconcile physical cleanup.
Frame the multimodal video platform
Build a platform where a tenant uploads or streams video and later searches, summarizes or asks questions about specific moments. The system must preserve temporal grounding across visual and audio evidence, process long media asynchronously, and return citations that resolve to playable time ranges.
Synthetic exercise assumptions: 50,000 tenants, 500,000 uploaded videos/month, median duration 12 minutes, p95 duration 2 hours, 20% with audio, and 10% requiring high-resolution frame analysis. These are interview inputs, not measured production facts. Model storage and compute from duration distributions, not file count alone.
Define product requirements and SLOs
Functional scope: resumable upload, metadata validation, asynchronous processing, transcript and scene extraction, multimodal indexing, natural-language search, timestamped Q&A, clip export, progress, cancellation, retention and deletion.
Separate upload availability, job acceptance latency, processing throughput (media-minutes per wall-clock minute), indexing freshness, search recall, answer groundedness, timestamp accuracy and clip export latency. A 30-minute video processed in 30 minutes is 1x real-time; do not confuse worker utilization with user-visible completion.
Define a canonical media and timeline contract
Represent the asset with tenant-scoped assetId, immutable sourceVersion, content hash, container/codec metadata, duration, frame rate, dimensions, audio stream descriptors, language hints, rights/consent metadata and retention deadline. Normalize timestamps to a canonical timebase; preserve original stream time bases and offsets for round-trip accuracy.
A derived segment records [start,end), modality, sourceVersion, extractor version, model version, configuration hash, object URI, checksum and provenance. Use half-open intervals to avoid boundary duplication. Variable frame rate and audio/video offset must be explicit rather than assuming frameNumber / nominalFps is exact.
Build resumable upload and durable ingestion
Authenticate and authorize before issuing a short-lived multipart upload capability scoped to one tenant, asset and object prefix. The client uploads directly to object storage; the API stores upload session state and validates completion, size, checksum, content signature and declared metadata before creating a processing job.
Object-created notifications are at-least-once and may arrive out of order. Deduplicate on tenant, asset, sourceVersion and object checksum. Persist an outbox/job record before acknowledging acceptance; reconcile incomplete multipart uploads and orphaned objects. Never trust a client-supplied MIME type as proof of media format.
VISUAL WALKTHROUGH
See the system in motion
Follow a scoped resumable upload into validated immutable media and a durable processing job.
Authorize upload
Upload media parts
Validate checksum and probe
Persist idempotent job
Loading interactive walkthrough…
Orchestrate the processing DAG
A durable workflow branches into media probing, video scene/frame extraction, audio extraction, ASR/diarization, OCR and metadata enrichment. Each stage writes immutable artifacts and a manifest checkpoint. A join stage waits for required branches or marks optional branches degraded with an explicit reason.
Use stage-specific queues and worker pools because CPU decoding, OCR, ASR and GPU vision have different resource profiles. Apply bounded concurrency, per-tenant fairness, queue age limits, retries with jitter and dead-letter handling. Retry only idempotent stages; deterministic artifact keys prevent duplicate billing and output corruption.
Sample frames and detect scene boundaries
Start with a coarse temporal scan, scene-change detection and task-specific sampling. High-level summaries may use sparse keyframes; action recognition, small-object detection and brief events need denser targeted windows. OCR-heavy content may need crops and resolution-aware processing.
Record each frame timestamp, source frame identity, crop transform and extractor configuration. Adaptive refinement should be budgeted: use a cheap pass to find candidate intervals, then spend additional frame/GPU budget only where uncertainty or task value warrants it. Evaluate short-event recall separately from average video retrieval quality.
VISUAL WALKTHROUGH
See the system in motion
See source-version pinning, stream probing and timestamp-preserving frame extraction.
Pin source version
Read stream time bases
Extract keyframes and scenes
Commit immutable artifacts
Loading interactive walkthrough…
Process audio without losing alignment
Extract audio with its original time base, then run voice activity detection, ASR, optional diarization and audio-event classification. Store word or phrase timestamps, speaker IDs with confidence, language, channel and alignment quality. Keep raw transcript and normalized transcript as separate artifacts.
Audio and video clocks can drift or start at different offsets. Align using container timestamps and synchronization metadata; never infer alignment solely from segment order. For live streams, use overlapping windows and revise provisional transcripts while exposing a stable committed watermark.
VISUAL WALKTHROUGH
See the system in motion
Trace audio extraction, speech recognition and alignment to the canonical media timeline.
Extract original audio stream
Detect speech windows
Transcribe and diarize
Align words to canonical time
Loading interactive walkthrough…
Build a temporal multimodal index
Create modality-specific representations: text embeddings for transcript/captions, visual embeddings for keyframes or clips, and structured metadata for time, speaker, scene, entities and access policy. Keep a segment catalog that maps each vector back to the exact source interval and artifact version.
At query time, classify intent and retrieve from relevant modalities in parallel. Normalize and fuse candidates, deduplicate overlapping intervals, rerank with a cross-modal model where justified, and enforce authorization before content hydration. Do not treat cosine similarity across separately trained embedding spaces as directly comparable without calibration.
VISUAL WALKTHROUGH
See the system in motion
Build a versioned temporal index from modality-specific artifacts and embeddings.
Load visual artifacts
Load timestamped text spans
Create modality-specific vectors
Stage and validate index
Loading interactive walkthrough…
Answer questions with timestamped evidence
Assemble a bounded evidence pack containing transcript spans, frame descriptions, keyframe references, speaker/time metadata and source versions. Ask the model to answer only from supplied evidence and return claims linked to one or more intervals. Validate that every citation resolves to an authorized asset version and playable timestamp.
When modalities disagree, expose uncertainty or retrieve more evidence rather than silently blending. If no evidence supports a claim, abstain. A generated summary is not a source of truth; preserve provenance to the underlying media and extraction versions.
VISUAL WALKTHROUGH
See the system in motion
Authorize retrieval, fuse timestamped candidates and return playable citations.
Authorize asset scope
Retrieve modality candidates
Fuse and rerank intervals
Return grounded citations
Loading interactive walkthrough…
Generate clips as a separate derived workflow
Clip creation is an asynchronous derivative job: validate requested ranges, enforce rights and policy, resolve source version, render with bounded CPU/GPU resources, store the output as a new asset and publish only after checksum and playback validation.
A clip references its parent asset and exact source interval, plus transformations such as crop, captions, redaction and audio mix. Recheck authorization at download time. Do not mutate the original media or imply that a generated highlight is a verbatim source segment when it includes synthesized content.
Enforce tenant security and media rights
Derive tenant identity from authenticated credentials and authorize every asset, job, segment, transcript, frame, vector, clip and signed URL. Keep object prefixes and encryption keys tenant-scoped where the threat model requires it; short-lived URLs must be constrained to a specific object and operation.
Treat uploaded media, subtitles, OCR text and embedded QR codes as untrusted content. They may contain prompt injection, secrets or personal data. Model instructions cannot grant access to another asset or authorize export. Apply consent, copyright/usage rights, regional processing, retention and moderation policy before indexing or generation.
Handle deletion, retries and recovery
On deletion or ACL revocation, write a durable tombstone and deny search/playback immediately. Then invalidate caches, remove vector entries, transcripts, frame artifacts, clips and replicas asynchronously. A lineage graph lets the reconciler enumerate every descendant and verify physical deletion against retention/legal-hold rules.
Workers use leases and fencing tokens; stale workers cannot publish after a newer attempt or deletion. Persist stage checkpoints and artifact checksums. Cancellation prevents new stages and publication, but cleanup of already-created temporary files remains a separate task. Reconcile stuck jobs, missing objects, orphan vectors and partial outputs continuously.
VISUAL WALKTHROUGH
See the system in motion
Deny access immediately, fence stale workers and reconcile every derived artifact.
Persist tombstone
Deny reads and fence publication
Enumerate descendants
Delete and reconcile artifacts
Loading interactive walkthrough…
Estimate storage, bandwidth and compute
Estimate source storage as uploads/day × average bytes × retention days, then add replication and derived artifacts. Estimate decode cost from total media-hours, codec complexity, frame sampling rate and audio duration. Estimate embedding cost from segment count and model batch throughput; estimate GPU vision cost from selected frames/crops rather than total video duration alone.
Use separate CPU decode, GPU vision, ASR and embedding capacity pools. Queue by tenant and deadline, reserve capacity for interactive queries, and shed optional enrichment before core indexing. Track cost per processed media-minute, indexed minute, successful grounded answer and exported clip.
Measure quality and operational health
Operational signals include ingest lag, stage queue age, processing speed ratio, retries, dead letters, GPU memory/utilization, decode errors, missing modality rate, index freshness, deletion lag and signed URL failures. Trace one asset/job across every stage with tenant-safe identifiers and model/extractor versions.
Evaluate temporal retrieval recall, timestamp intersection-over-union, ASR word error rate by language/noise slice, speaker attribution, visual event recall, grounded claim precision, citation validity, abstention and user task success. Report modality-specific slices; aggregate scores can hide failures on short events, accents, low light or variable frame rate.
Walk through partial failure and reprocessing
Suppose video decode succeeds but ASR times out. Publish the visual branch as a partial artifact only if product policy permits partial search; mark audio unavailable and keep the job resumable. Retry ASR with the same immutable source and deterministic stage key. Do not rebuild already validated visual embeddings unnecessarily.
If the source is replaced while a worker is processing, the old worker may finish but cannot publish into the new version. A new sourceVersion creates a new job and index generation. If deletion arrives during processing, the tombstone fences publication and descendant cleanup proceeds from the lineage manifest.
Complete multimodal video architecture
The end-to-end path is authenticated upload API → scoped multipart object storage → durable asset/job record → event-driven workflow → media probe → parallel video/audio extraction → timestamped artifacts → modality-specific embeddings and metadata → staged temporal index → validated publication → authorized search and grounded Q&A → timestamped playback or derived clip.
The control plane owns tenant policy, job state, version pins, publication and deletion authority. The media data plane owns large objects and immutable derived artifacts. Workers own stage execution evidence; the index is a rebuildable projection. Failure paths include duplicate events, partial modality failure, stale workers, source replacement, deletion during processing and index reconciliation.
Complete animated architecture walkthrough
Follow the complete upload, temporal extraction, multimodal indexing, authorized retrieval and grounded-answer lifecycle, including version and provenance boundaries.
VISUAL WALKTHROUGH
See the system in motion
Follow the full multimodal video platform from authenticated upload through synchronized extraction, temporal indexing and timestamped grounded answers.
Authorize and create scoped upload
Store immutable source version
Probe media and establish time bases
Extract frames, scenes and audio in parallel
Persist timestamped artifacts and transcripts
Build and validate modality-specific indexes
Authorize retrieval and assemble evidence
Return grounded answer with playable citations
Loading interactive walkthrough…
Diagrams and relationships
The API controls identity and workflow; media bytes bypass application servers.
Scroll horizontally on smaller screens, or enlarge the diagram.
Parallel modality branches preserve timestamps and provenance before index publication.
Scroll horizontally on smaller screens, or enlarge the diagram.
Search returns authorized intervals, not untraceable global summaries.
Scroll horizontally on smaller screens, or enlarge the diagram.
Engineer, Senior and Staff follow-ups
These are teaching lenses. Each answer changes the scope of reasoning; they are not official employer leveling criteria.
Engineer: Why use direct-to-object-store upload?
It removes large media bytes from the API tier, supports multipart/resume and lets API capacity scale independently. The API still authorizes the upload session and validates completion, checksum and object metadata.
Engineer: Why preserve separate audio and video time bases?
Streams may have offsets, drift or variable frame rate. Separate time bases preserve alignment and let citations resolve to the original media timeline.
Senior: How would you avoid duplicate processing from at-least-once object events?
Use an idempotency key based on tenant, asset, source version and checksum; persist stage state and deterministic artifact keys; fence stale attempts before publication.
Senior: How do you balance frame sampling cost and event recall?
Use a coarse scan and scene boundaries, then adaptive refinement around uncertain or high-value intervals. Evaluate short-event recall independently and cap per-asset refinement budgets.
Senior: How do you combine text and visual retrieval scores?
Do not assume raw cosine scores are calibrated across modalities. Normalize on held-out relevance data, fuse ranks or use a learned reranker, and retain modality-specific provenance.
Staff: How would you support a two-hour video without monopolizing GPUs?
Chunk by temporal windows, separate CPU decode from GPU inference, use stage-specific queues, per-tenant fair scheduling, bounded in-flight work, checkpointed artifacts and deadline-aware admission.
Staff: What happens when a user deletes a source while derived clips are being generated?
A tombstone immediately denies access and fences publication. The lineage graph identifies in-flight and completed descendants; workers check the source generation before commit and the reconciler removes all derivatives and cache entries.
Staff: How do you make a multimodal answer auditable?
Persist the exact source version, segment intervals, extraction/model versions, retrieval candidates, authorization decision and citation mapping. The answer references playable time ranges and can be recomputed from immutable evidence.
Practice and self-review
Your practice draft
Design a multi-tenant video understanding platform supporting large resumable uploads, asynchronous extraction, audio/video alignment, multimodal search, grounded timestamped Q&A and deletion. Explain capacity isolation, temporal provenance, partial failure and reprocessing.
Saved locally on this device. The worked answer does not score your draft. Existing lesson notes and completion are separate.
Progressive hints
Hint 1
Start with an immutable asset version and canonical timeline contract.
Hint 2
Keep large media bytes out of the API tier; use scoped resumable object uploads.
Hint 3
Split CPU decode, GPU vision, ASR and embedding into resource-specific stages.
Hint 4
Preserve modality-specific timestamps and provenance through the index.
Hint 5
Return authorized timestamped evidence, not unsupported global summaries.
Hint 6
Use tombstones, lineage and fencing to make deletion/recovery safe.
Reveal worked answer guidance
Authenticate the tenant, create an upload session and issue a scoped multipart capability. Validate checksum and media metadata on completion, then persist an idempotent job. Probe streams and establish time bases; branch into video frame/scene extraction and audio VAD/ASR/diarization. Write immutable timestamped artifacts with source and extractor versions. Generate modality-specific embeddings and metadata, stage a temporal index generation, validate coverage and publish atomically. Search authenticates and authorizes first, retrieves modality-specific candidates, fuses and reranks intervals, assembles bounded evidence and returns claims with playable timestamps. Clip export is a separate derived job. Use stage-specific CPU/GPU queues, leases, checkpoints and deterministic artifact keys. Deletion writes a tombstone immediately, fences workers and asynchronously removes all descendants using lineage reconciliation.
Evidence to check in your answer
- Tenant identity and per-asset authorization are server-derived.
- Uploads are resumable and scoped; checksum and media signature are validated.
- Asset and derived artifacts are immutable and versioned.
- Audio/video time bases and offsets are preserved.
- CPU, GPU, ASR and embedding workloads are independently scheduled.
- Every stage is idempotent, checkpointed and fenced against stale publication.
- Temporal retrieval returns authorized evidence with playable timestamps.
- Partial modality failures are explicit and resumable.
- Deletion covers source, transcript, frames, vectors, caches and clips.
- Evaluation includes temporal recall, timestamp accuracy, modality slices and groundedness.
For each item, record Missing, Prompted, Independent or Adaptive evidence. Adaptive means you can revise the reasoning under a changed constraint. Reading completion and readiness are separate.
Sources and verification scope
Original synthetic examples and reference calculations. Sources support narrowly scoped background; they do not establish production performance or independent chapter review.
Primary background links checked during authoring on 2026-10-19. Arithmetic, temporal eligibility, maturity and split reference functions are checked by the bounded local fixture suite. SQL is explanatory, not a migrated production query. No GPU, external dataset, model API or training job is required.