FB Live Comments
Design a comment stream for live videos, including durable posting, fast fanout, reconnect, and moderation.
Save your attempt before continuing. ## Interview scope and guarantees
Post comments to a live event, display recent comments with low delay, reconnect without duplicates, and remove moderated content. Ordering is per stream or shard, not necessarily global across the service. Slow consumers must not stall the broadcaster.
Capacity worksheet
Assume one event has 1 million viewers and 1,000 comments/s. Sending every 200-byte comment to every viewer would be 200 GB/s before protocol overhead. That product expectation is often unnecessary; offer a moderated or sampled view, batch updates, and define what each viewer receives.
Concrete API contract
POST /streams/{id}/comments {clientId,text} -> {commentId,sequence}
GET /streams/{id}/comments?after=...
SSE /streams/{id}/events?cursor=...
DELETE /comments/{id} -> moderation eventData model and access paths
comments(stream_id,sequence,comment_id,author_id,text,state)
stream_offsets(stream_id,partition,offset)
moderation(comment_id,version,state)
client_dedup(author_id,client_id,comment_id)Evolve a solution and explain each change
Figure — Three architecture decisions for FB Live Comments, including the pressure each introduces.
Step 1: Append comments
Authenticate the author and append a stream-scoped comment ID. Per-viewer polling wastes work and does not define replay.
Step 2: Push bounded streams
Fan out persisted comments over session connections with resume cursors. Slow viewers can accumulate unbounded buffers.
Step 3: Handle hot broadcasts
Shard fanout delivery, filter moderation changes, and shed overload predictably. Delivery shards do not automatically create a global comment order.
Responsibility overview
Figure — Connected responsibilities for FB Live Comments. Trace the authoritative and derived paths separately.
Worked end-to-end scenario
A viewer posts c20 and disconnects before the response. Retrying the client comment ID returns the stored comment. Broadcast subscribers receive c20 through fanout workers, but a slow phone fills its buffer. Rather than accumulating unlimited memory, send a resync signal or disconnect according to policy. The phone reconnects with its stream cursor and fetches a bounded persisted range. A moderator removes c20. The system emits a tombstone or removal event; reconnect replay must not resurrect it. A hot stream can use several delivery shards while one ordered ingest partition assigns stream positions.
Why these access paths matter
Persist by broadcast ID and stream position. Deduplicate author/client IDs and index moderation state. Connection registries map broadcast shards to active sessions; they are ephemeral delivery state, not authoritative history. If sampled comments are allowed under overload, state that separately from full replay and make the client aware of gaps.
Build the baseline first
- Comment API → durable append → ordered stream.
- Fanout service → connection gateway → viewers.
- Reconnect cursor → retained history → live handoff.
Evolve the design under load
- Stream partitions → regional fanout tree.
- Gateway batches → bounded client buffers.
- Moderation event → removal projection → clients.
Defend the hardest decision
An append log preserves a resumable position. Gateways subscribe once per popular stream and fan out locally, rather than establishing one broker subscription per viewer. A client that cannot keep up receives a bounded backlog or a reset-to-latest instruction. If the system samples comments, the UI must not imply a complete ordered transcript. A cursor should encode the shard positions needed to resume a partitioned stream.
Failure and recovery analysis
After a disconnect, subscribe to the live stream while fetching history with a defined high-water mark, then deduplicate the overlap. Fetching history first and subscribing later creates a gap. A moderation deletion must suppress future replay and update connected clients; removing only the database row leaves already-buffered copies visible.
Security and privacy boundary
Rate-limit authors, escape rendered content, and preserve moderator audit trails without leaking removed text to unauthorized readers.
Interview follow-ups with reasoning
Question: Can SSE replace WebSockets?
Show answer and explanation
Answer: Yes for server-to-browser updates while posting uses HTTP.
Question: How do you handle a viral stream?
Show answer and explanation
Answer: Split fanout work across regions and gateways while protecting the append authority.
Question: What do you measure?
Show answer and explanation
Answer: Publish-to-display lag, slow-client drops, reconnect gaps and moderation propagation delay.
Operate and verify the design
Publish-to-display delay, slow-client drops, replay resets and moderation lag.
Reconnect a viewer during a burst and verify the history/live handoff has no unexplained gap.
A second scenario to test transfer
A room receives 100 comments/s and has 100,000 viewers. At an illustrative 500 bytes per delivered event, raw fanout is 5 GB/s before protocol overhead. The storage write rate is modest compared with delivery bandwidth. A popular-room tier can batch or sample display events under a stated UX policy while preserving accepted comments in history. Adding database replicas alone does not solve socket fanout.
A reconnect cursor is older than retained events. What should the server return?
Show answer and explanation
Answer: An explicit reset/snapshot requirement with a new valid cursor. Silently starting at the newest event would hide a gap. The UI can show a recent window and disclose that earlier live history is unavailable.
Compare alternatives
| Concern | Mechanism | Trade-off |
|---|---|---|
| Hot room | Hierarchical fanout | More routing state |
| Slow device | Bounded buffer + resume | May need snapshot reset |
| Moderation | Removal events + authoritative history | Propagation delay |
| Ordering | Room-scoped cursor | No global ordering promise |
A design-changing exercise
A fanout node crashes after accepting a comment. Is the comment lost?
Show answer and explanation
Answer: Not if acceptance meant durable log insertion. Delivery can be reconstructed from the log and subscriber cursors; memory-only fanout is not enough.
Design workshop: replay-to-live without gaps
Core scope is posting, recent history, live viewing and moderated removal for one broadcast. Ranking every historical comment is excluded. Choose live delivery p95 under one second under admitted load, durable accepted comments and a 30-minute replay window. Popular-room display sampling, if enabled, is disclosed; it does not delete accepted history.
CONNECT /streams/id/comments accepts afterCursor. Events include streamId, sequence, commentId, type, contentVersion and payload. A cursor binds stream and position; a cursor for another stream is rejected. If it precedes retention, return reset_required with a recent snapshot and new cursor. Never silently return only newest events while pretending the suffix is complete.
Subscribe to live output, capture durable high watermark H, fetch (cursor,H], then consume live output after H. Buffer/overlap may repeat events, so deduplicate by stream/sequence. A bounded buffer overflow during handoff aborts and requires another replay. Attaching after a historical query without a watermark can lose the events between those steps.
Figure — A cursor 40 joins a stream at watermark 43.
A room log assigns sequence under one ordered append authority. Fanout delivery is then sharded across gateway groups; delivery-shard IDs are not a room-wide order. A session registry maps room→gateway populations and expiring sessions. Root fanout distributes batches to regional workers, then gateway groups, then bounded per-socket queues. Persisted history stays independent of session presence.
Figure — Fanout budget for a hot broadcast.
Sample display at a gateway only under a product contract that labels the stream as selected comments. Preserve ordering among delivered events and carry a cursor over skipped ranges so resume semantics are explicit. Full-history clients query the durable log, not the sampled socket stream. Backpressure can reject new writes with 429 under overload or accept durably and signal delayed delivery; choose the boundary honestly.
Moderation appends a versioned removal event for comment C. Clients apply newer content versions and retain a bounded tombstone so a delayed create cannot resurrect C. Reconnect snapshots exclude removed content. A moderator can race with publication; authorization and policy checks happen at the write authority and renderer. Report moderation propagation lag separately from accepted-comment latency.
Exercise: Live output is subscribed at time T, the historical query returns through 43, and events 43/44 are buffered. What is sent?
Show answer and explanation
Answer: History through 43, then only buffered events strictly after 43. If the buffer overflows, restart from a supported durable cursor rather than skip to the latest event.