Multi-region Disaster Recovery
Recover a complete workload in another region within measured RPO and RTO objectives while preventing split-brain writes.
Recover a complete workload in another region within measured RPO and RTO objectives while preventing split-brain writes. This worked design targets SDE-3; the scale figures below are exercise assumptions, not claims about a named company's implementation.
Clarify the prompt
| Ask the interviewer | Decision to record |
|---|---|
| Which failures count: availability zone, region, account compromise or data corruption? | Replication alone cannot repair every class. |
| What are RPO and RTO per operation/data class? | Separate read-only degradation from full write restoration. |
| Must recovery work if the primary region and its control plane are unreachable? | Dependencies must exist outside it. |
Requirements and scope
The core deliverable is: Recover a complete workload in another region within measured RPO and RTO objectives while preventing split-brain writes. The first design implements the APIs below and the state constraints in the data model. Agree the latency, freshness, retention and failure policy before adding infrastructure. Every successful acknowledgment must have an explicit meaning: accepted intent, committed state or completed external work. These are separate milestones.
Capacity and bottleneck reasoning
Exercise targets: RPO <= 60 seconds and RTO <= 15 minutes for orders. At 10 MB/s writes, a 60-second replication gap represents up to 600 MB of logical changes. This is an illustrative bound under the assumed lag, not a guarantee from choosing a service name.
Keep average and peak demand separate. Add replication, indexes, retries and safety headroom to raw arithmetic before selecting capacity. Load-test the busiest ownership key and recovery traffic as well as uniform happy-path throughput.
APIs and data model
- GET /recovery/readiness -> lag,backup age,dependency status
- POST /recovery/failover {expectedEpoch,targetRegion} -> runId
- GET /recovery/runs/{id} -> phase,verification evidence
recovery_plan(workload,version,dependency_inventory,approved_rpo,approved_rto)
replication_checkpoint(source_lsn,target_lsn,observed_at)
write_owner(epoch,region,lease_or_quorum); drill(started_at,restored_at,loss_window)Authorize resource ownership at the API boundary; do not trust IDs or state supplied by a client. Bind retry identity to the caller and payload for mutating operations. Reject an incompatible replay instead of returning an unrelated stored result.
Component relationships
Read the diagram
Follow the labeled relationship panels; they are separate directed paths, not one global timeline.
Step-by-step design
1. Establish the authority
Choose backup/restore, pilot light, warm standby or active-active from the workload objectives and budget. Replicate data plus keys, secrets, artifacts, queues and configuration; ensure standby quotas and infrastructure can actually support recovery traffic.
2. Make retries and concurrent work safe
Detect and declare failure through an external control path. Fence the old write owner before promotion, select a verified replication point, restore/promote storage, start required dependencies and validate business reads/writes before shifting all traffic.
3. Scale and recover the critical path
Maintain immutable backups and point-in-time recovery separately from replication because corruption can replicate. Failback is another controlled migration: synchronize changes, fence the temporary owner and verify the new checkpoint before reversing traffic.
Worked request and concurrency example
At 10:00 the primary fails. Standby is verified through 09:59:30, so the known data gap is 30 seconds. Detection takes two minutes, promotion and fencing three, service scale-up four and validation plus routing three: 12 minutes total. Record RPO from the last recoverable business commit, not the latest heartbeat. If clients depend on an unavailable primary-region identity or key service, the apparent database recovery is not a recovered workload.
Failure and recovery walkthrough
A network partition makes both regions think the other is dead. DNS routing does not fence database writers; enforce an epoch/quorum or a documented external authority. If safe ownership cannot be established, pause writes instead of accepting divergent orders.
Read the diagram
This numbered trace follows one scenario; explain the durable evidence at each transition.
Deep dive: a recovery objective needs a complete runbook
This original exercise operates a regional order service with an asynchronous warm standby. The business accepts RPO up to 60 seconds for order intent and targets RTO of 15 minutes from first user impact to validated essential service. Payment-provider effects have a separate recovery obligation: a missing local order does not imply that an external charge never happened. Critical workloads requiring zero acknowledged-data loss need a different preconfigured durability policy or must wait for safe recovery.
Define the loss model and evidence first
The primary region may be unavailable, slow or isolated while some old workers remain alive. The standby must survive independently, including its control path, keys, artifacts and operator access. An async lag dashboard is useful but cannot prove a loss bound if the source was unobserved for an unknown interval. Store recovery watermarks and acknowledged operation evidence under a design that supports the declared disaster scenario.
| Evidence | What it establishes | What remains uncertain |
|---|---|---|
| Standby durable through source checkpoint P | Recoverable prefix in the same source history | Commits acknowledged after P |
| Durable acknowledgement ledger/checkpoints | Candidate acknowledged operations and their order | Recoverability if that evidence shares the failed domain |
| Business read/write probe | Selected paths function in the recovery region | All tenants, all background jobs or external settlement |
| Backup restore and verification | Recovery from a selected independent historical point | Changes since that point and already executed provider effects |
For an illustrative 14:00:00 outage, the latest acknowledgement in the complete available evidence is at 13:59:55 and the verified recoverable standby prefix ends at 13:59:30. There is a 25-second gap to the latest known acknowledgement, and the recovery point is 30 seconds before impact. Compare the declared 60-second objective to the correct observed boundary and record both time and missing operation identities. Do not label the result RPO=0 because a DNS switch was fast. If evidence cannot bound newer acknowledgements, report the gap as unknown rather than claiming a measured pass.
Recover dependencies, not just the database
| Dependency | Recovery prerequisite | Validation gate |
|---|---|---|
| Data and transaction/outbox history | Verified replica or restore point; writer authority | Read known records and commit a scoped test operation |
| Encryption and secrets | Recovery-region key access, grants and rotated credentials | Decrypt retained data and authenticate required dependencies |
| Artifacts and configuration | Known compatible build, schema/config versions | Boot from pinned artifacts without relying on the failed region |
| Queues, schedules and consumers | Replay/rebuild policy with original occurrence/event identities | Dedupe replay and suppress cancelled/obsolete work |
| Network, DNS and identity | Independent routing/access path, certificates and auth configuration | Real client login and essential API flow |
| Capacity and provider contracts | Standby quotas, connection pools and usable external limits | Admit representative load without exhausting recovery headroom |
A warm database with no decryption key or no executable application artifact does not meet the service's RTO. A replicated schedule store is not proof that timers were recreated or old timers stopped. Recover one authoritative schedule generation and reconcile deliveries from both regions.
Use one promotion and traffic runbook
- Detect impact and record the recovery clock start. Declare the affected scope and recovery owner; the declaration delay is part of the observed outage.
- Fence the previous write owner through the designed authority mechanism. Routing changes alone cannot revoke existing connections or queued old work. If exclusive authority cannot be established, do not accept divergent writes.
- Select a verified recovery prefix and state the known/unknown loss. Obtain the required business decision before knowingly operating outside the objective.
- Restore/promote storage and essential dependencies, establishing the new owner epoch at protected resources before admitting writers.
- Start compatible application/control workers with conservative budgets. Reconcile outbox/queue/scheduler identities, provider uncertainty and stale owners.
- Run business validation: authenticate a recovery principal, read an agreed known record, create a scoped operation and resolve its retry to the same identity. Check tenant isolation, not only TCP health.
- Shift a small traffic cohort, observe errors, correctness and saturation, then ramp within reserved headroom. Stale clients still need an ownership-safe response from the old region.
- Continue replay/repair under a separate budget and retain the incident evidence. Essential service restoration and full historical reconciliation are separate milestones.
Read the diagram
Region A loses essential service → Controller establishes exclusive owner epoch 8 → Region B restores a verified prefix and dependencies → Tenant-aware business read/write probes pass → Traffic ramps within recovery capacity
Budget RTO from impact, then measure the drill
An illustrative sequential critical-path allocation is 2 minutes detect, 1 declare/decide, 2 fence/select prefix, 4 recover dependencies, 2 business validation, 2 ramp and 2 margin: 15 minutes. Some real steps can overlap; the observed dependency graph determines the critical path. Do not add unrelated step averages or exclude a slow operator decision to manufacture a pass.
Suppose a drill takes those first six steps but dependency recovery takes 7 minutes rather than 4. Observed restoration is 16 minutes before the unused planning margin, exceeding a 15-minute objective. Record the failure and identify the bottleneck. “The database promoted in 20 seconds” does not repair the service-level result. Another drill with a nine-minute key-access wait must include that delay too.
At a sustained primary peak of 10,000 requests/s and measured recovery-region safe capacity of 6,000/s, shifting every request immediately overloads the survivor. Admission must reject/degrade at least 4,000/s until usable capacity grows, with additional space for replay and repair. Regional quota allocation and provider limits must be confirmed before the incident, not assumed from theoretical autoscaling.
Replication and backups solve different failures
An erroneous delete or compromised writer can replicate corruption quickly. Maintain independently protected backups/PITR under the recovery policy and test restoration, decryption and business validation. Restoring an older database can reintroduce already completed external actions; original operation identities and provider reconciliation remain essential.
The stated region-loss objective does not automatically cover simultaneous regions failing or inaccessible recovery credentials. Specify these exclusions and the selected response rather than silently extending the guarantee. See AWS disaster recovery guidance for strategy trade-offs and AWS recovery/failback guidance for one service's operational scope. These references do not validate this exercise's timings or transfer ownership protocol.
Failback is a second migration
While B serves writes at epoch 8, A must not resume its old epoch-7 writer role. Rebuild or reconcile A from the accepted B history, prove catch-up through an agreed checkpoint, verify compatible schema/config and prepare dependencies. Then establish a new exclusive owner epoch, such as 9, fence B's prior writer and shift traffic only after validation. There must never be an interval in which both can independently accept conflicting commands.
Keep B as the recovery path while possible, but understand the new data authority. Reverting DNS to a stale A is not rollback. If a new schema or external side effect was accepted in B, failback must preserve that fact. Repeated drills should include a failed failback, a paused old worker, missing keys and lost callbacks; a successful planned switchover alone under-tests the disaster path.
DR assessment: identify the unproven promise
Question: B has a recent replica, but A is merely unreachable. Can B start accepting writes because most users cannot reach A?
Show answer and explanation
Answer: Not without establishing exclusive authority. A may still serve another client group. Enforce fencing or stop unsafe writes according to the selected availability policy.
Question: Replication lag was 20 seconds five minutes before impact. Does that prove the 60-second RPO was met?
Show answer and explanation
Answer: No. The sample is stale and does not bound later acknowledged changes. Use recoverable-prefix and acknowledgement evidence, or report uncertainty.
Question: After B accepted new writes, is restoring A's old snapshot and switching DNS a safe failback?
Show answer and explanation
Answer: No. Catch A up from the accepted history, validate it, establish a new owner generation and fence the previous writer. Otherwise failback can discard B's accepted operations or create split-brain writes.
Score one point for measured RPO evidence, full critical-path RTO, enforced ownership, dependency/capacity validation and safe failback. A plan that tests only a database promotion fails the service-recovery criterion. The fixtures model gates and timing; they do not perform live regional disaster recovery.
Decisions and alternatives
| Decision | Trade-off |
|---|---|
| Alternative 1 | Warm standby reduces recovery time with recurring idle capacity cost. Backup/restore is cheaper but restore throughput and provisioning may exceed the target. |
| Alternative 2 | Active-active improves some availability paths but introduces conflict and ownership design; it does not automatically give zero loss or zero recovery time. |
Operational contract
Monitor replication lag, restore-test age, standby drift, key accessibility and failover ownership. Drill regional isolation and corrupted-data recovery separately. Compare measured objectives by data class and keep a rehearsed failback runbook.
Choose alert thresholds from the agreed user-facing objective. Assign an owner for stuck work and a replay/repair procedure that preserves the original business identity. Bound queue depth, concurrency and recovery throughput so an outage does not produce an uncontrolled catch-up burst.
Foundation, senior and staff depth
- Foundation: Define failure scope, recoverable point and essential-service restoration.
- Senior: Execute fencing, dependency recovery, business validation and measured RPO/RTO gates.
- Staff: Defend recovery capacity, independent backups and failback preserving accepted history.
Practice before revealing the answer
Spend 45 minutes: 5 clarifying scope, 7 on capacity/APIs/state, 12 on the first architecture, 15 on the concurrency and failure examples, and 6 on trade-offs and operations. Draw your own design before comparing it with this chapter.
Question: How do you prove a 15-minute RTO rather than just document it?
Show answer and explanation
Answer: Run representative drills that include detection, decision, fencing, dependency recovery, routing and business validation. Measure total outage from user impact to restored service and retain evidence of the recoverable data point.
Verification and rollout
Run tabletop loss/partition cases with complete and incomplete acknowledgement evidence. Reject promotion without authority or required dependencies; measure the full impact-to-service critical path. Exercise stale-owner writes, failed probes, overload during ramp and failback before catch-up. Models are not a substitute for live regional restore and key-access drills.
Primary technical reference
Official documentation for the mechanism used in this exercise. This lesson is an original interview design, not a description of undisclosed company internals.
Related study
Before this design, review these mechanisms and identify which owns the critical state transition:
Use the Interview Playbook to adapt the depth and timing, then return to the question breakdown overview.
Shared foundation: Replication, Durability and Failover
Replication, Durability and Failover: Separate durable acknowledgement, read visibility and safe promotion through a concrete replica history. Work its failure history before extending this design.