Messaging Semantics: Queues, Logs and Pub/Sub
Messaging systems differ in retention, consumer ownership, ordering, replay and acknowledgement semantics.
Messaging Semantics: Queues, Logs and Pub/Sub
Messaging systems differ in retention, consumer ownership, ordering, replay and acknowledgement semantics. A work queue distributes each task among competing workers. Pub/sub broadcasts events to independent subscriptions. A retained log stores partitioned history so consumers track offsets and replay. At-least-once delivery is common when acknowledgement follows processing; exactly-once guarantees are scoped to a broker transaction or a specific state boundary.
Mechanism and worked example
An order-created event can feed billing, analytics and email independently. A task queue is better for assigning one image-resize job to one worker. Kafka-like logs support replay and offsets; SQS-like queues emphasize managed work delivery and visibility timeouts.
| Model | Consumer behavior | Design consequence |
|---|---|---|
| Work queue | Competing workers claim tasks | Visibility, ack, retry and DLQ |
| Pub/sub | Each subscription gets its own delivery path | Fan-out isolation and per-subscriber backlog |
| Partitioned log | Consumers advance offsets | Replay and partition ordering |
| Outbox + broker | State and event intent commit together | Relay may publish duplicates |
Failure modes and operations
A worker can commit a database update and crash before acknowledgement, causing redelivery. Poison messages need retry budgets and DLQs. Partition keys create hot partitions. Rebalances pause consumers; visibility timeout expiry can cause concurrent redelivery. Schema changes can break old consumers.
Trade-offs and design boundary
Queues simplify task distribution but are not automatically durable event history. Logs support replay but need partition and retention planning. Pub/sub fan-out increases independent consumer state and delivery cost. Ordering is scoped, not global.
Interview transfer exercise
Explain how a consumer prevents duplicate financial effects after a crash between database commit and broker acknowledgement. Commit inbox identity and business state in one transaction, then acknowledge.
Read the diagram
Producer persists intent and outbox → Broker stores queue task or partitioned event → Consumer commits idempotent state → Acknowledgement or offset advances after commit
Explain how a consumer prevents duplicate financial effects after a crash between database commit and broker acknowledgement. Commit inbox identity and business state in one transaction, then acknowledge.
Your design draft
Clarify assumptions, explain your approach, and test the difficult cases. Save your draft, then compare it with the study notes.
Self-review checklist
Self-guided practice. Automated AI feedback and code execution are not connected.