Key Technologies Overview
Follow one video from a durable upload request to verified playback, and see where each technology role fits.
What, why, when and how
What is it?
This map connects storage, work, stream processing, coordination and edge technologies to the responsibility each can fulfill.
Why is it needed?
Several products can satisfy one declared contract, but adding every product increases integration and recovery work.
When should I use it?
Use it when a complete baseline exposes a responsibility that needs a better mechanism.
When should I choose something else?
Do not select a product before defining access patterns, durability, ownership and operational constraints.
How do I use it?
Separate control requests, immutable bytes, durable job state, worker effects, derived search and media delivery; choose the smallest set that fulfills those responsibilities.
Can you show a concrete example?
A video upload can use a durable job store, constrained object transfer and bounded encoding. Independent retained consumers may justify a log; one encode task alone does not.
Read the diagram
Accepted upload j7: durable intent; Verified input v8: bytes complete, still not playable; Published manifest: required renditions validated
Follow the detailed diagrams below to trace the additional state changes and failure cases behind these answers.
Outcome: choose technology by the responsibility it must fulfill, and defend a small video-upload design without adding every product in the curriculum.
A creator requests an upload, transfers bytes, waits for encoding and watches the published video. The title becomes searchable. The example is original; no traffic assumptions here describe a company's production architecture. Separate control requests, large bytes, durable job facts, worker ownership, derived search and media delivery before naming products.
A minimal baseline has a complete publication boundary
Use an authenticated API, a relational job store, object storage and a bounded background worker path. A CDN can serve media when repeated delivery and geographic latency justify it. Search can begin with an indexed title lookup in the primary store if that meets the requirement. A retained event log, workflow engine and distributed coordination service are extensions whose requirements must be demonstrated.
- The API authorizes tenant t1 and atomically accepts upload job j7 with a client operation identity. Where publication to a queue is required, persist that obligation with the job.
- Issue constrained instructions for an upload session. The client transfers large parts directly to object storage under the declared size, expiry and checksum policy.
- Verify completion and freeze input generation v8. A client callback alone does not prove the bytes exist or belong to the permitted session.
- Dispatch bounded encode work for j7/v8. A current attempt owns publication; an expired attempt must not overwrite a newer result.
- Write and validate immutable rendition objects. Publish a manifest only after all required outputs are usable.
- Authorize playback and grant a scoped delivery capability for the permitted objects. Delivering a cached segment and deciding who may watch are separate responsibilities.
Read the diagram
Create durable job j7 and upload session; Transfer parts; verify immutable input v8; Current encode attempt writes rendition objects; Validate every required rendition; Publish the manifest under current ownership
Match storage to the same operations
| Candidate | Operation it can serve | What to define before choosing it | A reason to choose another path |
|---|---|---|---|
| PostgreSQL | Accept related job, operation and publication records atomically | Constraints, indexes, isolation and bounded connections | A different store may better fit a measured, key-shaped workload |
| DynamoDB | Bounded tenant/job lookups and conditional state changes | Partition/sort keys, hot-key limits and consistency scope | Routine scans or arbitrary joins indicate a poor access model |
| Cassandra | Query-shaped, partition-local histories | Bounded partition size, clustering order, repair and consistency settings | Small relational workflows may not justify distributed wide-column operations |
| Document storage | An aggregate whose fields are commonly read or changed together | Embedding versus references, document growth and update boundary | Frequently changing shared relationships can amplify duplicated updates |
| Redis | A hot, bounded lookup or useful in-memory structure | Freshness, cardinality, expiry, eviction and failover | Evictable cache state must not silently become the only job truth |
| Object storage | Immutable media bytes and versions | Upload completion, checksums, access and orphan lifecycle | It is not an arbitrary relational transaction engine |
| Elasticsearch | Text retrieval and relevance-oriented queries | Mapping, analyzer, refresh lag, deletes and query limits | Exact-key lookups or a small title filter may not require a search cluster |
Several storage choices can satisfy a declared job contract. Evaluate them on actual operations and failure policy, rather than saying one named database is always the scalable option. The curriculum has no standalone MongoDB, object-storage or CDN chapter; Data Modeling, Handling Large Blobs and Networking Essentials cover their relevant boundaries.
Distinguish jobs, retained events and workflows
A work queue gives workers tasks and exposes an acknowledgement/redelivery protocol. Standard SQS-style delivery can repeat, so j7/v8 and its effect identities must survive redelivery. Queue visibility is not an ownership guarantee for the destination that publishes a result. Define admission, queue age, retry budget and the business state machine.
A Kafka log retains ordered records per partition so independent consumer groups can progress and replay. If analytics and a search rebuild need the same retained history, that may justify a log or archive. A single encode task does not by itself require Kafka. Retention bounds replay, and an offset commit does not make an external effect atomic.
Flink is useful for stateful stream computations with event-time, window and recovery semantics. It is not a substitute for a durable record of which video is published. Temporal can own a multi-step workflow with timers and durable decisions, but activities still need safe external effect identities. For a simpler sequence, an explicit durable state machine may be enough; compare development and operational costs on the same recovery requirements.
Keep edge and coordination responsibilities precise
| Boundary | Responsibility | Guarantee it does not automatically provide |
|---|---|---|
| API Gateway | Validate identity and request policies; route control APIs | Resource-level tenant authorization or media publication |
| Load balancer | Select eligible instances and handle connection routing/drain | Durable acceptance or unlimited downstream capacity |
| CDN | Reuse and deliver eligible bytes near viewers | Immediate revocation of already-issued capabilities |
| ZooKeeper or another coordination authority | Small shared membership or leadership state | Physically stopping a paused former owner |
A leader token helps only where the protected destination enforces it. Prefer a database constraint or conditional transition when it protects the rule directly. A new coordination service is not warranted merely because two workers exist.
For private playback, define what happens when rights are revoked while a signed capability remains valid. Short expiry may bound a window, but is not an immediate-revocation guarantee. A stricter policy requires an enforceable delivery check and an honest description of bytes already delivered to clients.
Walk through a crash after partial encoding
Attempt 9 writes two required renditions, then loses its completion response. Attempt 10 obtains current ownership. It verifies any output it intends to reuse and writes under a declared generation policy. The publication owner accepts a manifest only from the current attempt and only after required objects pass validation. Late attempt 9 cannot publish over attempt 10.
If object storage and job state disagree, reconcile using job, input generation, attempt and manifest identities. Clean unpublished generations only after the retention and active-reader policies allow it. A storage PUT and a database commit are separate effects; neither the queue nor the workflow product erases that boundary.
Defend an alternative and diagnose overload
Question: Why not proxy every large upload through the metadata API?
Show answer and explanation
Answer: A proxy may be justified by a concrete scanning, transformation or trust requirement. Otherwise direct constrained transfer can reduce API memory, bandwidth and request occupancy. Compare maximum file size, connection duration, verification location and operational cost before choosing. Direct transfer still needs authorization and completion verification.
Question: Workers doubled but encoding completion did not improve. Which chapter should you revisit?
Show answer and explanation
Answer: Inspect the shared bottleneck: storage throughput, CPU/GPU budget, database connection occupancy, input size skew or output publication contention. More workers can increase pressure on the constrained resource. Bound admission and measure queue age and completion time by size class before adding another product.