📝 Problem Description
Design a distributed message streaming platform like Apache Kafka. Support high-throughput message ingestion, durable storage, consumer groups, and exactly-once semantics.
👤 Use Cases
1.
Producer wants to publishes message so that message stored durably
2.
Consumer wants to reads messages so that receives in order
3.
Consumer Group wants to scales consumers so that partitions distributed
4.
Stream Processor wants to transforms data so that writes to new topic
✅ Functional Requirements
- •Publish messages to topics
- •Topic partitioning for parallelism
- •Consumer groups with rebalancing
- •Message retention (time or size based)
- •Seek to offset/timestamp
- •Replication for durability
⚡ Non-Functional Requirements
- •Throughput: 1M messages/sec
- •Latency: < 10ms for publish
- •Durability: No message loss
- •Ordering: Per-partition guarantee
⚠️ Constraints & Assumptions
- •Ordering only within partition
- •Must handle broker failures
- •Consumer rebalancing can cause delays
📊 Capacity Estimation
👥 Users
1000 producers, 10000 consumers
💾 Storage
100TB (7-day retention)
⚡ QPS
Writes: 1M/sec, Reads: 5M/sec
📐 Assumptions
- • 1M messages/sec ingestion
- • Average message: 1KB
- • 7-day retention
- • 10,000 partitions across 100 topics
- • Replication factor: 3
- • 5:1 read-to-write ratio (consumers replay)
💡 Key Concepts
CRITICAL
Log-Structured Storage
Append-only log segments for sequential I/O.
CRITICAL
ISR (In-Sync Replicas)
Replicas that are caught up with leader.
HIGH
Consumer Offset
Track consumption position per partition.
HIGH
Partition Rebalancing
Redistribute partitions when consumers change.
💡 Interview Tips
- 💡Start with the core concepts: topics, partitions, consumer groups
- 💡Emphasize the log-based architecture and its benefits
- 💡Discuss exactly-once semantics and idempotent producers
- 💡Be prepared to explain consumer group rebalancing
- 💡Know the tradeoffs between throughput and durability
- 💡Understand the difference between Kafka and traditional message queues