Question Breakdowns

Distributed Pub-Sub

Deliver each published event independently to every matching durable subscription while isolating slow or failing consumers.

A AceYourInterviews editorial · 8 min study

Deliver each published event independently to every matching durable subscription while isolating slow or failing consumers. This worked design targets SDE-3; the scale figures below are exercise assumptions, not claims about a named company's implementation.

Clarify the prompt

Ask the interviewer	Decision to record
Push or pull subscriptions, and what does acknowledgment mean?
Is ordering required per key or per topic?	Avoid imposing global order without a requirement.
How long can a subscriber disconnect, and can it replay older events?	Retention bounds recovery.

Requirements and scope

The core deliverable is: Deliver each published event independently to every matching durable subscription while isolating slow or failing consumers. The first design implements the APIs below and the state constraints in the data model. Agree the latency, freshness, retention and failure policy before adding infrastructure. Every successful acknowledgment must have an explicit meaning: accepted intent, committed state or completed external work. These are separate milestones.

Capacity and bottleneck reasoning

At 100,000 1 KB events/s and 20 matching subscribers on average, delivery reaches two million messages/s and about 2 GB/s before retries. Capacity must budget fanout amplification rather than just publisher ingress.

Keep average and peak demand separate. Add replication, indexes, retries and safety headroom to raw arithmetic before selecting capacity. Load-test the busiest ownership key and recovery traffic as well as uniform happy-path throughput.

APIs and data model

POST /topics/{id}/events {eventId,key,payload} -> durable receipt
POST /subscriptions {topic,filter,mode} -> subscriptionId
ACK subscription,eventId,generation; POST /subscriptions/{id}/seek {checkpoint}

Contract / pseudocode

event(event_id PK,topic,key,payload_ref,published_at)
subscription(id,topic,filter_version,retention,mode)
subscription_cursor(subscription,partition,next_offset); attempt(subscription,event_id,generation,state)

Authorize resource ownership at the API boundary; do not trust IDs or state supplied by a client. Bind retry identity to the caller and payload for mutating operations. Reject an incompatible replay instead of returning an unrelated stored result.

Component relationships

Distributed Pub-Sub: directed relationships and authoritative boundaries — Distributed Pub-Sub: state boundaries

Step-by-step design

1. Establish the authority

Accept a publisher event durably under its identity, then fan out matching references or advance independent cursors. Acknowledging a publish means the service accepted the event, not that every subscriber completed its side effects.

2. Make retries and concurrent work safe

Each subscription owns its delivery progress and retry budget. Multiple workers within one subscription may share partitions; separate subscriptions must each receive their own logical copy. Fence stale delivery generations where ownership changes.

3. Scale and recover the critical path

Isolate quotas, backlog and DLQs by subscription. A permanently failing webhook must not stall unrelated subscriptions. Version filters and document whether replay uses the original match decision or current filters.

Worked request and concurrency example

Event E matches subscriptions Billing and Analytics. Billing completes and acknowledges; Analytics times out. Only Analytics is redelivered. Its consumer may already have committed before its reply was lost, so it deduplicates (subscription,E) with its business update. A third subscription created tomorrow receives only future events unless an explicit retained-log replay policy includes E.

Failure and recovery walkthrough

A slow subscriber exceeds retention. Report a gap and require snapshot/bootstrap; do not move its cursor silently and claim no loss. Push endpoints need bounded concurrency, exponential backoff and a dead-letter path.

Distributed Pub-Sub: one concrete failure or state-transition sequence — Distributed Pub-Sub: concrete trace

Decisions and alternatives

Decision	Trade-off
Alternative 1	Per-subscription materialized queues simplify filtering and isolation but amplify storage. Shared retained logs with cursors reduce copies at higher cursor/filter complexity.
Alternative 2	Push reduces client polling but needs endpoint security and retries; pull gives subscribers direct flow control and requires lease/ack APIs.

Operational contract

Monitor publish latency, fanout amplification, oldest unacked age per subscription and webhook failure rate. Test one permanently failing subscriber alongside a healthy subscriber and verify isolation and replay-gap reporting.

Choose alert thresholds from the agreed user-facing objective. Assign an owner for stuck work and a replay/repair procedure that preserves the original business identity. Bound queue depth, concurrency and recovery throughput so an outage does not produce an uncontrolled catch-up burst.

Foundation, senior and staff depth

Foundation: explain the user action, model, first complete request and the correctness constraint without relying on a list of technologies.
Senior: defend this lesson's partition/coordination boundary, capacity estimate and failure recovery; explain the cost of the rejected alternative.
Staff: identify failure domains and operational ownership, stage a rollout with measured guardrails, and state when rollback stops being safe.

Practice before revealing the answer

Spend 45 minutes: 5 clarifying scope, 7 on capacity/APIs/state, 12 on the first architecture, 15 on the concurrency and failure examples, and 6 on trade-offs and operations. Draw your own design before comparing it with this chapter.

Question: How is pub-sub different from one work queue with twenty workers?

Show answer and explanation

Answer: The workers in one queue compete to handle each item once logically. Twenty independent subscriptions each need the event; each subscription may itself use a worker group.

Verification and rollout

Reproduce the worked example using two concurrent actors or a repeated request where applicable. Inject the stated failure between the durable transition and its acknowledgment; verify the resulting state and user-visible outcome. For read/ML paths, test missing or stale dependencies and verify the documented fallback still enforces permissions. Use a small canary, compare correctness and user-facing latency with the baseline, and roll back when those guardrails regress. Do not use an average throughput number as evidence that a tail-latency target is met.

Primary technical reference

Official documentation for the mechanism used in this exercise. This lesson is an original interview design, not a description of undisclosed company internals.

Before this design, review these mechanisms and identify which owns the critical state transition:

Use the Interview Playbook to adapt the depth and timing, then return to the question breakdown overview.