Distributed Blob Store
Store large immutable objects with durable metadata, resumable transfer, integrity verification and safe garbage collection.
Store large immutable objects with durable metadata, resumable transfer, integrity verification and safe garbage collection. This worked design targets SDE-3; the scale figures below are exercise assumptions, not claims about a named company's implementation.
Clarify the prompt
| Ask the interviewer | Decision to record |
|---|---|
| Object or filesystem semantics? | Assume key/version objects, not arbitrary shared in-place writes. |
| What durability failure domain, maximum object size and consistency are required? | |
| Are overwrites, range reads, multipart uploads and deletion retention in scope? |
Requirements and scope
The core deliverable is: Store large immutable objects with durable metadata, resumable transfer, integrity verification and safe garbage collection. The first design implements the APIs below and the state constraints in the data model. Agree the latency, freshness, retention and failure policy before adding infrastructure. Every successful acknowledgment must have an explicit meaning: accepted intent, committed state or completed external work. These are separate milestones.
Capacity and bottleneck reasoning
Assume one PB of logical objects and 10 GB/s peak upload. Three full replicas require three PB before metadata and headroom. Erasure coding lowers capacity overhead at repair/read complexity; choose from measured repair bandwidth and failure assumptions.
Keep average and peak demand separate. Add replication, indexes, retries and safety headroom to raw arithmetic before selecting capacity. Load-test the busiest ownership key and recovery traffic as well as uniform happy-path throughput.
APIs and data model
- POST /uploads {key,size,checksum} -> uploadId and part URLs
- PUT /uploads/{id}/parts/{n}; POST /uploads/{id}/complete {parts} -> version
- GET /objects/{key}?version=...; DELETE /objects/{key} -> tombstone
object(bucket,key,version,manifest_id,etag,deleted_at)
manifest(id,chunk_ids,sizes,checksums,placement_generation)
upload(upload_id,owner,expires_at,state); part(upload_id,number,checksum,chunk_id)Authorize resource ownership at the API boundary; do not trust IDs or state supplied by a client. Bind retry identity to the caller and payload for mutating operations. Reject an incompatible replay instead of returning an unrelated stored result.
Component relationships
Read the diagram
Follow the labeled relationship panels; they are separate directed paths, not one global timeline.
Step-by-step design
1. Establish the authority
Separate metadata from data transfer. Clients upload bounded chunks directly to storage nodes using scoped credentials. Verify checksums and required replica/code placement before publishing a committed manifest.
2. Make retries and concurrent work safe
Commit the visible key/version pointer atomically after validating all parts. Reads resolve that immutable manifest, choose healthy chunk replicas and verify integrity. A multipart upload existing on disk is not yet a visible object.
3. Scale and recover the critical path
Repair placement after failures with throttled background work. Garbage collection traces committed manifests plus active uploads and retention/tombstone rules; delay deletion beyond in-flight reads and failed-commit reconciliation windows.
Worked request and concurrency example
A 5 GB upload uses 100 parts. Part 42 succeeds but its response is lost; retrying the same part number and checksum reuses or replaces that part under the upload contract, not a new visible object. Completion publishes one manifest only after all verified parts satisfy durability. If metadata commit fails, those parts remain unreachable and are later collected after upload expiry.
Failure and recovery walkthrough
Publishing a manifest before sufficient chunk durability can expose an object that disappears after one node loss. Conversely, deleting chunks immediately after pointer removal can break concurrent versioned reads; use retention and reference-aware collection.
Read the diagram
This numbered trace follows one scenario; explain the durable evidence at each transition.
Decisions and alternatives
| Decision | Trade-off |
|---|---|
| Alternative 1 | Replication gives straightforward reads/repair at high storage overhead. Erasure coding saves space with more CPU, network and degraded-read complexity. |
| Alternative 2 | Small chunks improve retry granularity but enlarge metadata and request count. Large chunks reduce overhead while making failed transfers and repairs more expensive. |
Operational contract
Monitor under-durable objects, checksum failures, orphan bytes, repair backlog and metadata quorum health. Test part retries, failed completion, node loss during repair and deletion with active readers.
Choose alert thresholds from the agreed user-facing objective. Assign an owner for stuck work and a replay/repair procedure that preserves the original business identity. Bound queue depth, concurrency and recovery throughput so an outage does not produce an uncontrolled catch-up burst.
Foundation, senior and staff depth
- Foundation: explain the user action, model, first complete request and the correctness constraint without relying on a list of technologies.
- Senior: defend this lesson's partition/coordination boundary, capacity estimate and failure recovery; explain the cost of the rejected alternative.
- Staff: identify failure domains and operational ownership, stage a rollout with measured guardrails, and state when rollback stops being safe.
Practice before revealing the answer
Spend 45 minutes: 5 clarifying scope, 7 on capacity/APIs/state, 12 on the first architecture, 15 on the concurrency and failure examples, and 6 on trade-offs and operations. Draw your own design before comparing it with this chapter.
Question: Why should the upload-complete response include a committed object version?
Show answer and explanation
Answer: It identifies the immutable manifest that became visible and lets a retry resolve the same commit. Upload ID alone refers to temporary work and does not prove the object reached its durability boundary.
Verification and rollout
Reproduce the worked example using two concurrent actors or a repeated request where applicable. Inject the stated failure between the durable transition and its acknowledgment; verify the resulting state and user-visible outcome. For read/ML paths, test missing or stale dependencies and verify the documented fallback still enforces permissions. Use a small canary, compare correctness and user-facing latency with the baseline, and roll back when those guardrails regress. Do not use an average throughput number as evidence that a tail-latency target is met.
Primary technical reference
Official documentation for the mechanism used in this exercise. This lesson is an original interview design, not a description of undisclosed company internals.
Related study
Before this design, review these mechanisms and identify which owns the critical state transition:
Use the Interview Playbook to adapt the depth and timing, then return to the question breakdown overview.
Shared foundation: Security and Tenant Isolation
Security and Tenant Isolation: Enforce trusted tenant, object and field permissions across APIs, caches, projections and asynchronous work. Work its failure history before extending this design.
Shared foundation: Storage Engines and Index Mechanics
Storage Engines and Index Mechanics: Connect access paths to durable acknowledgement, version retention, amplification and verified recovery. Work its failure history before extending this design.
Why should the upload-complete response include a committed object version?
Your design draft
Clarify assumptions, explain your approach, and test the difficult cases. Save your draft, then compare it with the study notes.
Self-review checklist
Self-guided practice. Automated AI feedback and code execution are not connected.