Kafka, RabbitMQ and SQS/SNS

Kafka, RabbitMQ and SQS/SNS

Kafka is a partitioned retained log; RabbitMQ is a routing-oriented broker; SQS/SNS are managed queue and pub/sub primitives. Kafka stores records in partitioned logs and consumers track offsets; partition keys define ordering scope. RabbitMQ routes through exchanges and bindings to queues; acknowledgements and prefetch govern in-flight delivery. SQS uses visibility timeouts and deletion acknowledgements; FIFO adds scoped ordering and deduplication. SNS fans out to subscriptions.

Mechanism and worked example

Use Kafka when multiple independent consumers need replayable history. Use RabbitMQ when flexible routing and work distribution are central. Use SQS for managed asynchronous tasks and SNS→SQS when independent subscribers need durable fan-out.

Failure modes and operations

Kafka rebalances move partition ownership; retention can expire data before a slow consumer catches up. RabbitMQ unacked messages and queue growth consume resources. SQS visibility expiry can cause concurrent redelivery. All systems can redeliver around crashes.

Trade-offs and design boundary

Kafka offers replay but requires partition and consumer operations. RabbitMQ offers routing but queue topology and broker capacity matter. SQS/SNS reduce broker operations but impose service semantics and limits. None makes arbitrary external side effects exactly once.

Interview transfer exercise

Choose a broker for image-resize tasks versus independently replayable order events. State retention, replay, routing, ordering, throughput and operational ownership.

Read the diagram

Producer publishes event or task → Broker stores partition log or queue item → Consumers process with offsets or acknowledgements → Retry, replay and DLQ recover failures

View full sizeOpen original

Kafka, RabbitMQ and SQS/SNS

Reset viewClose

Mechanism and decision flow for Kafka, RabbitMQ and SQS/SNS.