Replication, Durability and Failover
Separate durable acknowledgement, read visibility and safe promotion through a concrete replica history.
Replication, Durability and Failover
Replication maintains copies of authoritative state. Its useful guarantee depends on what must happen before acknowledgement, which copy a read uses and which survivor may become the next writer. Begin with a loss model: process crash, machine loss, zone loss, whole-region loss and logical corruption require different protection.
Start from the user-visible promise
For this original exercise, a transfer API promises that a successful acknowledgement will survive loss of its primary region. A balance display may lag by two seconds, but a new debit must consult current authoritative funds. These are three separate contracts: durable posting, bounded-stale display and safe mutation. Replication does not make them identical.
A replica in the same region can protect against a server failure while failing the region-loss requirement. A replica that has received bytes but has not durably recorded them may not survive its own crash. A replica that has durably recorded a transaction but has not applied it may preserve money while still showing an older balance.
Vocabulary before the mechanism
| Term | Meaning in this lesson |
|---|---|
| Write-ahead log | Ordered recovery records persisted before the corresponding durable change is considered safe |
| Receive, flush, apply | Obtain bytes, persist them under the storage contract, then replay them into query-visible state |
| Replication position | A protocol-specific point used to compare progress in the same history |
| RPO | The acceptable loss of acknowledged data under the declared failure model |
| RTO | The recovery objective measured from user impact to usable service |
| Epoch and fencing | Ownership generation and enforcement that rejects obsolete writers |
Positions from different divergent histories are not automatically comparable. A large timestamp or counter alone cannot prove that a replica contains every acknowledged transaction.
Follow one acknowledgement
Read the diagram
Primary commits X at position 42 → Standby remains durable through 41 → Async success returns to caller → Primary region is lost → Promoting 41 cannot recover X
| Event | Primary | Standby | What the caller may conclude |
|---|---|---|---|
| Before X | Durable through 41 | Durable/applied through 41 | Copies agree at this point |
| Commit X | Durable through 42 | Still through 41 | Only primary has X |
| Async acknowledgement | Returns success | Still through 41 | X survives primary process restart only if its durable storage survives; region durability is unproven |
| Region lost | Unreachable or destroyed | Only 41 recoverable | Promoting this copy cannot preserve acknowledged X |
| Safe synchronous alternative | Waits for required remote durability before success | X durable remotely | Declared survivor set can recover X; read visibility may still lag |
The remote acknowledgement must be part of the configured commit policy before the failure. Changing a setting during recovery cannot recreate X. If a required remote participant is unreachable, waiting, rejecting or explicitly weakening the contract are honest choices. Returning the same success while silently weakening durability is not.
Compare choices under the same workload
Assume, for this exercise, 4 MB/s of log data and a ten-minute disconnection. Catch-up requires 2.4 GB before overhead. If the recovering link can apply 10 MB/s while new changes continue at 4 MB/s, ideal net drain is 6 MB/s and catch-up takes 400 seconds. If it can apply only 4 MB/s, lag never shrinks. Synchronous commit also adds remote acknowledgement latency; raw bandwidth alone cannot predict tail latency.
| Choice | Good fit | Cost or missing guarantee |
|---|---|---|
| Async replica | Display reads and workloads accepting bounded loss | Acknowledged data can be absent after promotion |
| Required remote durable acknowledgement | Acknowledged changes must survive the declared regional failure | Higher latency; writes can block when required participants fail |
| Remote apply acknowledgement | Selected replica must expose the committed change before success | Includes replay delay; says nothing about unrelated read replicas |
| Independent backup/PITR | Restore after corruption or accidental deletion | Recovery takes work; replication itself can propagate corruption |
Do not generalize product settings from these abstract names. PostgreSQL distinguishes remote write, durable flush and apply behavior; the configured synchronous participants and storage settings matter. See the official references below before translating the design into configuration.
Read your own write without hiding lag
A caller receives a commit token for X. Its next status lookup can go to the current owner, or a replica proven to include that token in the correct history. A replica at 41 must wait within the deadline, route safely or return an honest pending/unavailable result. Arbitrary sleeping does not prove inclusion. A replica at 42 can have later application lag if 42 refers only to durable receipt, so the token check must match the actual visibility requirement.
A shared cached balance may remain stale even after a replica catches up. Document the cache policy separately. Do not authorize a debit from an old display balance.
Failover is an ownership protocol
- Detect a problem and decide whether promotion is permitted; reachability alone is not proof that the old owner is dead.
- Establish exclusive write authority through the selected coordination mechanism and fence old writers at the accepting resource.
- Select a survivor that satisfies the required acknowledged history and validate its recovery state.
- Promote, restore dependencies and run business read/write probes before shifting all traffic.
- Reconcile outstanding requests by their original identities and make old clients refresh routing.
- Rebuild a former primary from the accepted history before failback; never merge divergent money changes by last timestamp.
DNS changes route future connections but cannot stop an old process accepting writes. The fencing authority must be reachable under the chosen failure policy. If it cannot establish safe ownership, writes may need to pause. A numeric epoch in an application log is not enforcement.
Recovery drill and operations
Measure receive/flush/apply lag, retained-log runway, replica replay throughput and failed synchronous commits. Alert on the consumer promise: an apparently healthy replica can violate a two-second read contract. Keep backup restore evidence separate from replica health and test encryption keys, credentials and configuration recovery too.
Original drill: disconnect the standby, submit X, record the actual acknowledgement outcome, lose the primary and attempt promotion. A test passes only if the result matches the declared durability policy. Then restore a backup with a deliberate bad update to show why a replica alone cannot undo corruption. This lesson supplies a state model and reasoning exercise, not a production failover implementation.
Check your understanding
Question: Standby B has durably flushed position 42 but applied only 41. Does this prove it can show transfer X at 42 immediately?
Show answer and explanation
Answer: No. Durable receipt protects recovery bytes under its storage contract; read visibility requires applying X. Route a strict confirmation to an eligible reader or wait for the correct apply boundary.
Question: A and B are partitioned. Each is reachable by half the clients. Can changing DNS to B establish one writer?
Show answer and explanation
Answer: No. Existing clients can still reach A. Promotion needs authority and fencing that prevent A from committing obsolete work. If that cannot be established, stop unsafe writes.
Question: With 2.4 GB backlog, 10 MB/s replay and 4 MB/s arrivals, what is the ideal catch-up time?
Show answer and explanation
Answer: 2,400 MB divided by 6 MB/s is 400 seconds. Retries and replay overhead can extend it; if replay only matches arrivals, catch-up never finishes.
Transfer exercise and scoring
Apply this model to Global Payments Ledger and Multi-region Disaster Recovery. For each, state the failure model, acknowledgement boundary, eligible promotion set, stale-reader policy and failback rule. Award one point per item only when supported by a concrete history. Five points means the exercise is complete, not that a production system is certified.
Continue with CAP Theorem for partition operation histories and Sharding for moving ownership.
Primary references and scope
- PostgreSQL standby replication: product-specific replication behavior and synchronous acknowledgement choices.
- PostgreSQL WAL configuration: commit and durability configuration. Check the deployed version before relying on a setting.
The transfer records, positions, workloads, drill and scoring are original teaching examples. They are not production measurements.