Messaging Semantics: Queues, Logs and Pub/Sub
Messaging systems differ in retention, consumer ownership, ordering, replay and acknowledgement semantics.
Messaging Semantics: Queues, Logs and Pub/Sub
Messaging systems differ in retention, consumer ownership, ordering, replay and acknowledgement semantics. A work queue distributes each task among competing workers. Pub/sub broadcasts events to independent subscriptions. A retained log stores partitioned history so consumers track offsets and replay. At-least-once delivery is common when acknowledgement follows processing; exactly-once guarantees are scoped to a broker transaction or a specific state boundary.
Mechanism and worked example
An order-created event can feed billing, analytics and email independently. A task queue is better for assigning one image-resize job to one worker. Kafka-like logs support replay and offsets; SQS-like queues emphasize managed work delivery and visibility timeouts.
| Model | Consumer behavior | Design consequence |
|---|---|---|
| Work queue | Competing workers claim tasks | Visibility, ack, retry and DLQ |
| Pub/sub | Each subscription gets its own delivery path | Fan-out isolation and per-subscriber backlog |
| Partitioned log | Consumers advance offsets | Replay and partition ordering |
| Outbox + broker | State and event intent commit together | Relay may publish duplicates |
Failure modes and operations
A worker can commit a database update and crash before acknowledgement, causing redelivery. Poison messages need retry budgets and DLQs. Partition keys create hot partitions. Rebalances pause consumers; visibility timeout expiry can cause concurrent redelivery. Schema changes can break old consumers.
Trade-offs and design boundary
Queues simplify task distribution but are not automatically durable event history. Logs support replay but need partition and retention planning. Pub/sub fan-out increases independent consumer state and delivery cost. Ordering is scoped, not global.
Interview transfer exercise
Explain how a consumer prevents duplicate financial effects after a crash between database commit and broker acknowledgement. Commit inbox identity and business state in one transaction, then acknowledge.
Read the diagram
Producer persists intent and outbox → Broker stores queue task or partitioned event → Consumer commits idempotent state → Acknowledgement or offset advances after commit