Design WhatsApp: durable acceptance, ordered delivery and private content
A candidate-led conversation about online/offline messaging, device inboxes, groups, encryption and failure recovery. Workloads and service targets are illustrative assumptions.
1. Turn the brand name into a bounded problem
I would first distinguish the messaging product from a generic socket platform or cryptographic protocol. I will integrate established encryption rather than invent it. Presence and typing can be approximate; accepted messages cannot be treated the same way.
| Candidate asks | Illustrative interviewer reply | Design consequence |
|---|---|---|
| 1:1 chat or groups, and how large? | Both; groups up to 256 members. | Per-conversation order and bounded fan-out. |
| Text, media, voice/video or payments? | Text and media; others excluded. | Separate encrypted object path; no call pipeline. |
| Offline users and linked devices? | Yes; retain undelivered envelopes 30 days. | Durable per-device inboxes and replay cursors. |
| What does sent mean? | Durably accepted, not just gateway receipt. | No success ACK before durable commit. |
| What privacy and order are expected? | End-to-end encryption; consistent conversation order. | Server cannot search plaintext; explicit order authority. |
| What scale and latency? | Choose assumptions; optimize online in-region delivery. | Separate acceptance, connected delivery and offline delay. |
2. Define requirements and success boundaries
I will support text/media, small groups, online/offline delivery, receipts and linked devices. Presence/typing are best-effort; calls, stories and server-side plaintext search are excluded. Local search is compatible with encrypted server storage.
Acceptance means the encrypted message and delivery intent are durable. Transport is at least once, with stable message IDs and idempotent application. That is not exactly-once networking.
| Requirement | Proposed contract |
|---|---|
| Online latency | Illustrative p95 below 200 ms from durable acceptance to a connected in-region recipient at admitted load. |
| Availability | 99.99% send/read API monthly target; uncertain conversation ownership can temporarily reject writes. |
| Order | Monotonic sequence per conversation, not a global order. |
| Retention | Undelivered encrypted envelopes retained for 30 days; older devices need explicit recovery policy. |
| Privacy | Only endpoints hold plaintext; device/key lifecycle and metadata privacy are separate concerns. |
| Receipts | Sent = durable acceptance; delivered = recipient-device ACK; read = optional client-reported state. |
| Membership | Authorize against the membership version at acceptance; declare what departing members may receive. |
3. Separate logical messages, fan-out and connections
Assume 10M DAU send 100 messages each per day. One billion logical messages/day does not equal one billion delivery writes: assume three recipient-device envelopes on average. Group-size and linked-device distributions determine this multiplier.
A million idle TLS connections and a million active sends stress different resources. I would benchmark gateway capacity rather than invent a per-machine socket guarantee.
| Calculation | Result | Implication |
|---|---|---|
| 10M × 100 ÷ 86,400 | 11,574 sends/s average; 115,741/s at assumed 10× peak | Partition owners and bound burst admission. |
| 1B × 3 recipient-device envelopes/day | 3B envelopes/day; 34,722 writes/s average | Budget group and multi-device amplification. |
| 3B envelopes × 1 KB × 30 days | 90 TB if every envelope is retained for the full window | A conservative logical bound; indexes, replicas and media extra. |
| 1B messages × 1% media × 500 KB | 5 TB/day media ingress; 150 TB/30 days | Download fan-out and replication are additional. |
| 1M simultaneous devices ÷ 30-second heartbeat | 33,333 heartbeats/s | Jitter and separate ephemeral presence from durable writes. |
4. Model accounts, devices and authoritative order
I separate accounts from devices because each recipient may need several encrypted envelopes and independent delivery state. A device-to-gateway cache is a routing hint, never the only offline-message store.
A logical conversation owner serializes acceptance. Its durable transaction allocates a sequence, inserts the dedup/message record and appends delivery work. A sharded transactional store is a simple baseline. A log-centric or wide-column alternative must preserve the same explicit ownership and acknowledgement rules.
| Entity | Access / ownership | Invariant |
|---|---|---|
| Conversation + Member(conversationId,membershipVersion) | Conversation home partition | Membership and sequencing coordinated at acceptance. |
| Message(conversationId,sequence,messageId,ciphertextRefs) | Range by conversation and sequence | Immutable content and stable accepted order. |
| Dedup(senderDeviceId,clientMessageId,digest,result) | Unique operation key in owner transaction | Replayed input returns original result; different input conflicts. |
| DeviceInbox(deviceId,cursor,messageId,acknowledged) | Sequential replay per device | Cursor never skips unpersisted delivery work. |
| Device/public key directory | Authorized devices and public prekeys | Private keys stay on endpoints; revocation changes future delivery. |
| SessionRoute(deviceId,gateway,epoch,expiry) | Ephemeral cache | Stale routing cannot erase accepted messages. |
5. Specify message, receipt and recovery protocols
I use an authenticated persistent connection for realtime frames and HTTP for history, media capabilities and device management. Long-lived secrets should not be exposed in logged URLs; use a secure handshake or short-lived connection ticket.
Cursors are opaque and caller-scoped. Knowing a message ID or device ID grants no permission to read it.
| Interface | Payload → result | Retry / error rule |
|---|---|---|
| WS SEND | conversationId, clientMessageId, membershipVersion, encrypted envelopes → ACCEPTED(messageId,sequence) | Persist before ACK; replay same operation; refresh stale membership/key set. |
| WS DELIVERY_ACK / READ | Authenticated device, messageId or contiguous cursor → receipt state | Monotonic/idempotent; respect optional read-receipt privacy. |
| GET /v1/inbox?cursor=… | Bounded encrypted page and next cursor | Expired cursor returns recovery-required, not empty success. |
| GET /v1/conversations/{id}/messages?before=… | Authorized encrypted history | Retention bounded; client decrypts. |
| POST /v1/media/uploads | Encrypted size/type → scoped URL and mediaId | Encrypt client-side; verify object before sharing reference. |
| POST /v1/conversations/{id}/members | Expected membership version and changes → new version | Authorization plus required key/session update. |
6. Draw the durable baseline
Gateways connect clients; conversation owners accept messages; a durable outbox creates recipient inbox work. Delivery workers use routing hints for online recipients or send a minimal wake-up push. A push-provider success is not proof a recipient read the message.
When routing is down I can retain accepted messages and degrade realtime delivery. When durable acceptance is unavailable I cannot honestly return a successful sent receipt.
7. Trace a send and a lost acknowledgement
Alice persists a pending operation locally and reuses clientMessageId across reconnects. I distinguish durable service acceptance, recipient delivery and reading. Bob persists/deduplicates an envelope before delivery ACK. Receipts may also repeat.
Prepare encrypted envelopes
Use the established protocol and current authorized device/key set. Include conversation and membership versions plus a stable client operation ID.
Validate and serialize
Check sender/device membership, payload bounds and digest. Route to the fenced conversation owner; atomically allocate sequence and persist message/dedup/outbox.
Acknowledge
Return ACCEPTED only after commit. If the reply is lost, replay the same operation and return the saved ID/sequence without allocating another.
Materialize inboxes
Idempotent workers insert recipient-device items using uniqueness on (deviceId,messageId). Durable work does not depend on current socket availability.
Deliver
Try the session route. Recipient stores once by messageId, then ACKs. Stale routes leave inbox work intact and may trigger a minimal wake-up push.
Aggregate receipts
Advance per-device/per-recipient receipt state according to the agreed semantics. Define whether group delivered means any recipient or all recipients; never overload one boolean.
8. Recover offline devices and preserve conversation order
When Bob reconnects, he fetches a bounded page after his safely stored cursor, applies/persists it and acknowledges. A device offline beyond retention receives an explicit gap and a defined history-recovery flow.
Ordering cannot depend on unsynchronized device clocks. A fenced owner assigns canonical sequence; devices detect gaps and request missing data. A sequence gap may also need an authorized tombstone/skip marker rather than exposing content a device cannot read.
Groups
Accept once, then materialize recipient-device work asynchronously. Bound fan-out and isolate hot groups; the average group size hides tail cost.
Membership
Pin acceptance to a membership version and define departed-member semantics. Key changes protect future messages, not plaintext already received.
Multiple devices
Use device-specific encryption sessions and inbox cursors; sync the sender’s other devices. Define how new devices obtain old history without server plaintext access.
Owner failover
Use a lease/epoch fenced at durable writes. A stale owner cannot allocate conflicting sequences. Reject temporarily while ownership is uncertain.
9. Scale without contradicting encryption
I partition conversation writes and device-inbox reads separately. Reconnect storms need jitter, backoff, bounded replay and admission limits. I degrade presence before message durability.
Servers cannot perform ordinary plaintext search, media transcoding or content inspection on end-to-end encrypted payloads. Clients process media before encryption and search local history; abuse reporting needs an explicit user-consented disclosure path. Metadata-based rate controls have their own privacy trade-offs.
| Choice | Baseline | Revisit when |
|---|---|---|
| Small-group fan-out | Materialized per-device inbox items | Large channels need shared logs/cursors and a revised workload. |
| Regions | Home region per conversation with durable replicas | Active-active writes require a new order/conflict contract. |
| Media | Private encrypted objects, scoped URLs, keys inside encrypted messages | Align deletion, backup and recovery policy. |
| Presence | TTL heartbeats and approximate last-seen | Avoid turning every heartbeat into a durable message write. |
10. Prove the delivery guarantees
I monitor acceptance latency, connected-delivery latency, oldest pending inbox age, dedup/replay rates, fan-out backlog and device-key failures. A healthy gateway alone cannot prove delivery.
| Failure | Recovery | Assertion |
|---|---|---|
| Gateway dies after acceptance | Reconnect and retry same key | One accepted message and original sequence. |
| Worker dies during group fan-out | Replay outbox and unique inbox inserts | No skipped eligible devices or duplicate logical item. |
| Routing cache fails | Queue durably; replay on reconnect | Accepted messages retained; presence may degrade. |
| Old owner survives failover | Reject stale fencing epoch | No conflicting sequence assignment. |
| Recipient ACK is lost | Redeliver and deduplicate on device | One visible message; receipts converge. |
| Keys/membership change during send | Reject stale envelope set and refresh | Future delivery uses the current declared policy. |
11. Summarize the guarantees
I separated durable acceptance, delivery and reading. Stable IDs make repeated transport safe; inbox cursors recover offline devices; conversation ownership defines order. Encryption constrains server-side features.
A senior explanation makes those boundaries testable and addresses backpressure/failover. Staff-level discussion explores global ownership, device/key lifecycle, large-group amplification, metadata privacy and migrations of ciphertext without plaintext access.
Technical references
Primary references explain underlying mechanisms. Workloads and architecture choices above remain proposed interview assumptions.