Distributed Message Queue
Build a Kafka-like partitioned replicated log with ordered partitions, producer retry identity, consumer ownership and replay.
Build a Kafka-like partitioned replicated log with ordered partitions, producer retry identity, consumer ownership and replay. This worked design targets SDE-3; the scale figures below are exercise assumptions, not claims about a named company's implementation.
Clarify the prompt
| Ask the interviewer | Decision to record |
|---|---|
| Work queue or retained log? | This design retains records and tracks consumer offsets. |
| Ordering per key or globally? | Global order limits throughput and availability. |
| How much acknowledged-data loss is acceptable on broker or region failure? | Define replication and election rules. |
Requirements and scope
The core deliverable is: Build a Kafka-like partitioned replicated log with ordered partitions, producer retry identity, consumer ownership and replay. The first design implements the APIs below and the state constraints in the data model. Agree the latency, freshness, retention and failure policy before adding infrastructure. Every successful acknowledgment must have an explicit meaning: accepted intent, committed state or completed external work. These are separate milestones.
Capacity and bottleneck reasoning
Assume one million 1 KB records/s: 1 GB/s or 86.4 TB/day raw. Three replicas require roughly 259 TB/day before compression. If one partition safely handles 10 MB/s, at least 100 partitions are needed for byte throughput, plus headroom and consumer parallelism.
Keep average and peak demand separate. Add replication, indexes, retries and safety headroom to raw arithmetic before selecting capacity. Load-test the busiest ownership key and recovery traffic as well as uniform happy-path throughput.
APIs and data model
- APPEND topic,partition,producerEpoch,sequence,records -> offsets
- FETCH topic,partition,offset,maxBytes -> ordered records
- COMMIT group,generation,partition,nextOffset -> accepted or fenced
partition(topic,id,leader_epoch,replicas,committed_offset)
record(partition,offset,key,value,producer_id,producer_epoch,sequence)
consumer_group(group_id,generation,assignments); offset(group,partition,next_offset)Authorize resource ownership at the API boundary; do not trust IDs or state supplied by a client. Bind retry identity to the caller and payload for mutating operations. Reject an incompatible replay instead of returning an unrelated stored result.
Component relationships
Read the diagram
Follow the labeled relationship panels; they are separate directed paths, not one global timeline.
Step-by-step design
1. Establish the authority
Route a key to one partition leader. Append batches to its log and replicate to followers. Acknowledge under a declared replica policy, retaining enough in-sync replicas and refusing unsafe elections rather than claiming durability while discarding acknowledged records.
2. Make retries and concurrent work safe
Use leader epochs to reject stale leaders and producer identity/sequence to recognize retries within the supported protocol. Rebalance consumer groups with generation fencing; partition order is meaningful only within that partition.
3. Scale and recover the critical path
Consumers fetch from offsets, process and then commit. A crash after an external side effect but before offset commit produces a replay; coordinate deduplication with the destination or restrict atomic consume/produce claims to supported transactional boundaries. Retention limits how far replay can recover.
Worked request and concurrency example
Leader L replicates record R to the required replicas and acknowledges offset 91. The reply is lost; the producer retries its sequence and receives the existing record identity. L fails, and an eligible up-to-date replica becomes leader under a new epoch. A stale consumer from the prior group generation cannot commit offsets. A committed nextOffset of 91 means resume at record 91 (the last processed record was 90). If the consumer applies record 91 and crashes before committing nextOffset 92, its replacement replays 91; downstream effects must tolerate that replay. A checkpoint named lastProcessed=90 instead requires conversion to the next readable position. Do not mix these conventions. Offsets can have gaps; commit the next position supplied by the consumer protocol rather than assuming every integer identifies a visible record.
Failure and recovery walkthrough
Electing an out-of-date replica may restore service while losing acknowledged data. Make that an explicit policy violation or deliberate trade-off. A slow consumer that falls behind retention needs a separate snapshot/bootstrap path; its missing data cannot be recreated by an offset number.
Read the diagram
This numbered trace follows one scenario; explain the durable evidence at each transition.
Decisions and alternatives
| Decision | Trade-off |
|---|---|
| Alternative 1 | More partitions increase concurrency but cost metadata, file handles and rebalances. A very hot key remains limited by one partition until the ordering contract changes. |
| Alternative 2 | Long retention enables replay at high disk cost. Compaction preserves a useful key history policy but is not equivalent to retaining every event forever. |
Operational contract
Monitor under-replicated partitions, replica lag, producer errors, consumer lag bytes/time, rebalance duration and disk headroom. Test leader loss after acknowledgment, stale epochs, replay and storage exhaustion.
Choose alert thresholds from the agreed user-facing objective. Assign an owner for stuck work and a replay/repair procedure that preserves the original business identity. Bound queue depth, concurrency and recovery throughput so an outage does not produce an uncontrolled catch-up burst.
Foundation, senior and staff depth
- Foundation: explain the user action, model, first complete request and the correctness constraint without relying on a list of technologies.
- Senior: defend this lesson's partition/coordination boundary, capacity estimate and failure recovery; explain the cost of the rejected alternative.
- Staff: identify failure domains and operational ownership, stage a rollout with measured guardrails, and state when rollback stops being safe.
Practice before revealing the answer
Spend 45 minutes: 5 clarifying scope, 7 on capacity/APIs/state, 12 on the first architecture, 15 on the concurrency and failure examples, and 6 on trade-offs and operations. Draw your own design before comparing it with this chapter.
Question: Is exactly-once processing guaranteed for a consumer that sends email?
Show answer and explanation
Answer: No. Broker transactions can coordinate supported log operations, but an external email provider lies outside that transaction. Use provider deduplication or accept and document the residual uncertain-delivery window.
Verification and rollout
Reproduce the worked example using two concurrent actors or a repeated request where applicable. Inject the stated failure between the durable transition and its acknowledgment; verify the resulting state and user-visible outcome. For read/ML paths, test missing or stale dependencies and verify the documented fallback still enforces permissions. Use a small canary, compare correctness and user-facing latency with the baseline, and roll back when those guardrails regress. Do not use an average throughput number as evidence that a tail-latency target is met.
Primary technical reference
Official documentation for the mechanism used in this exercise. This lesson is an original interview design, not a description of undisclosed company internals.
Related study
Before this design, review these mechanisms and identify which owns the critical state transition:
Use the Interview Playbook to adapt the depth and timing, then return to the question breakdown overview.
Shared foundation: Consensus, Leader Election and Fencing
Consensus, Leader Election and Fencing: Trace majority authority and reject a paused obsolete worker at the protected destination. Work its failure history before extending this design.
Is exactly-once processing guaranteed for a consumer that sends email?
Your design draft
Clarify assumptions, explain your approach, and test the difficult cases. Save your draft, then compare it with the study notes.
Self-review checklist
Self-guided practice. Automated AI feedback and code execution are not connected.