📝 Problem Description
Design a system to migrate petabytes of data from on-premises data centers to Google Cloud. The system must handle live migrations with minimal downtime, ensure data integrity, and provide progress tracking and rollback capabilities.
👤 Use Cases
1.
DBA wants to initiates migration so that data transfer begins with monitoring
2.
System wants to replicates changes so that live data kept in sync
3.
System wants to validates data so that checksums match, integrity confirmed
4.
Operator wants to performs cutover so that applications switch to cloud
✅ Functional Requirements
- •Support various source systems (Oracle, MySQL, S3, HDFS)
- •Continuous data replication for live migrations
- •Schema conversion and transformation
- •Data validation and checksum verification
- •Progress monitoring and ETA estimation
- •Rollback and restart capabilities
⚡ Non-Functional Requirements
- •Migrate 1PB+ with <1 hour cutover window
- •Zero data loss (exactly-once semantics)
- •Bandwidth efficient (compression, deduplication)
- •Resume from failure without re-transferring
- •Encrypt data in transit and at rest
⚠️ Constraints & Assumptions
- •Source continues to change during migration; CDC must keep target within seconds/minutes of lag
- •Cutover window < 1 hour; rollback must be possible if validation fails
- •Network bandwidth is finite (e.g., 10Gbps); must use compression and chunking with resume
- •Exactly-once semantics for change application (idempotency + ordering per primary key)
- •Data integrity must be provable (checksums, row counts, sampled deep validation)
- •Security/compliance: encrypt in transit, least-privilege credentials, audit logs, and PII handling
- •System must checkpoint frequently and resume without re-transferring terabytes
📊 Capacity Estimation
👥 Users
Internal operations - enterprise customers
💾 Storage
Source: 1PB+; Staging: 100TB; Target: 1PB+
⚡ QPS
CDC events: 100K/sec; Validation queries: 1K/sec
🌐 Bandwidth
10 Gbps = 1.25 GB/sec = 108 TB/day
📐 Assumptions
- • 1 PB data = 1,000,000 GB = 1,000 TB
- • Compression ratio: 3:1 (transfer 333 TB)
- • Network: 10 Gbps dedicated link
- • Cutover window: 1 hour maximum
- • CDC lag target: < 1 minute
- • Validation: 100% checksums, 1% deep sampling
💡 Key Concepts
CRITICAL
Change Data Capture
Capture ongoing changes from source transaction logs
HIGH
Chunked Transfer
Transfer data in chunks with resume capability
CRITICAL
Zero Downtime Migration
Keep source and target in sync during cutover
💡 Interview Tips
- 💡Start with the phases: initial load, CDC replication, validation, cutover
- 💡Emphasize zero data loss and exactly-once semantics
- 💡Discuss checkpointing and resume capability early
- 💡Be prepared to discuss CDC mechanisms (log-based vs trigger-based)
- 💡Know the tradeoffs between downtime and complexity
- 💡Discuss validation strategies in depth - this is often underestimated