Partitioning, Sharding and Replication
Partitioning divides data ownership; replication copies data. Sharding is a distributed partitioning strategy, not a synonym for replication.
Partitioning, Sharding and Replication
Partitioning divides data ownership; replication copies data. Sharding is a distributed partitioning strategy, not a synonym for replication. Hash partitioning distributes keys but weakens range locality. Range partitioning supports ordered scans but risks hot ranges. A shard key routes requests to an owner. Replicas maintain copies and introduce lag, quorum and promotion concerns. Resharding must coordinate copy, concurrent changes, verification, cutover and fencing.
Mechanism and worked example
Partition chat messages by conversation_id to preserve conversation-local ordering. A very active conversation can still be hot; time buckets can distribute storage but require a merge path for reads. Adding a shard should move a bounded key range where possible.
| Term | Meaning | Question |
|---|---|---|
| Partitioning | Split a dataset into pieces | Which key routes the operation? |
| Sharding | Distribute partitions across nodes | Who owns writes during movement? |
| Replication | Maintain copies | What must be durable before success? |
| Rebalancing | Move ownership/data | How are writes captured and fenced? |
Failure modes and operations
Skewed tenants overload one shard while cluster averages look healthy. Stale routing can send writes to a former owner. A copy made while writes continue can miss updates unless CDC or a dual-write protocol closes the gap. Replica promotion can lose acknowledged writes if commit policy did not protect them.
Trade-offs and design boundary
More partitions improve parallelism but increase fan-out, metadata, rebalancing and operations. Hashing balances key counts, not necessarily bytes or request rates. Replication improves survival and read capacity but does not partition write ownership.
Interview transfer exercise
Describe an online resharding sequence. Include snapshot, change capture, parity checks, ownership epoch, old-owner fencing, routing cutover and rollback conditions.
Read the diagram
Old shard owns key range → Snapshot and change log copy to new shard → Verify data and advance ownership epoch → Fence old owner before routing cutover