Global Payments Ledger

Post balanced immutable financial entries, prevent duplicate business postings and recover safely across regional and provider failures. This worked design targets SDE-3; the scale figures below are exercise assumptions, not claims about a named company's implementation.

Clarify the prompt

Requirements and scope

The core deliverable is: Post balanced immutable financial entries, prevent duplicate business postings and recover safely across regional and provider failures. The first design implements the APIs below and the state constraints in the data model. Agree the latency, freshness, retention and failure policy before adding infrastructure. Every successful acknowledgment must have an explicit meaning: accepted intent, committed state or completed external work. These are separate milestones.

Capacity and bottleneck reasoning

Assume 10,000 transfers/s, two 200-byte postings per transfer: 4 MB/s or 345.6 GB/day raw before journals, indexes and replicas. Partition by ledger/account ownership; a hot shared account may dominate transaction contention.

Keep average and peak demand separate. Add replication, indexes, retries and safety headroom to raw arithmetic before selecting capacity. Load-test the busiest ownership key and recovery traffic as well as uniform happy-path throughput.

APIs and data model

• POST /journals {businessKey,currency,entries} -> posted or pending
• GET /journals/by-business-key/{key} -> authoritative status
• POST /journals/{id}/reverse {reason,newBusinessKey} -> compensating journal

journal(journal_id PK,tenant,business_key,payload_hash,currency,status)
UNIQUE(tenant,business_key); canonical payload_hash checked on every replay
posting(journal_id,account_id,amount_minor,signed_side)
balance(account_id,currency,available_minor,version)
Invariant: signed postings sum to zero per journal and currency; posted journals immutable

Authorize resource ownership at the API boundary; do not trust IDs or state supplied by a client. Bind retry identity to the caller and payload for mutating operations. Reject an incompatible replay instead of returning an unrelated stored result.

Component relationships

Read the diagram

Follow the labeled relationship panels; they are separate directed paths, not one global timeline.

View full sizeOpen original

Global Payments Ledger: state boundaries

Reset viewClose

Global Payments Ledger: state boundaries

Step-by-step design

1. Establish the authority

Validate entries and currency precision, then atomically insert the journal, balanced postings and any enforced available-balance changes. A journal is the business fact; a cached balance is a projection unless updated inside the same transaction.

2. Make retries and concurrent work safe

Bind the business key to a canonical request fingerprint. Retry with the same key returns the original journal; changed payload is a conflict. Reverse errors with new balanced entries and lineage, never edit historical money movements.

3. Scale and recover the critical path

Choose one write owner per transactional ledger boundary. A regional failover requires fencing the previous owner and verifying the replication point. Cross-shard transfers need a documented pending/clearing protocol; two independent local commits are not an atomic global transfer.

Worked request and concurrency example

Transfer X moves 100 minor units from A to B. One transaction debits A, credits B and records business key X. The response is lost and a client retries from another region. Routing finds the same authoritative owner, which returns X without another debit. If the owner is unavailable and the standby lacks the acknowledged commit, promoting it would violate the claimed RPO=0 contract; either synchronous durability had to be configured beforehand or writes must wait for safe recovery.

Failure and recovery walkthrough

A provider webhook arrives twice after a settlement timeout. Deduplicate its event identity and reconcile provider state against pending ledger intent. A unique ledger key prevents duplicate posting but cannot alone prevent two external charges; provider idempotency and reconciliation are separate controls.

Read the diagram

This numbered trace follows one scenario; explain the durable evidence at each transition.

View full sizeOpen original

Global Payments Ledger: concrete trace

Reset viewClose

Global Payments Ledger: concrete trace

Deep dive: cross-shard clearing without fictional atomicity

This original exercise transfers 100 USD minor units from A on source shard S to B on destination shard D. It uses an abstract signed-balance ledger with prefunded destination bridge capacity, no FX or fees, and a durable coordinator decision. Negative and positive entries are additive balance deltas, not a claim about legal debit/credit account classification. Each local journal sums to zero, but two local commits are not one global transaction. B cannot spend incoming funds until its local posting commits.

Both local journals balance. Across all four balances, the initial total is 1,700 and remains 1,700 after settlement or a completed cancellation. Pending value is explicit: a source debit moves value into source clearing until the matching destination leg or safe cancellation is established. Enforce nonnegative destination bridge capacity in its local posting transaction. Replenishment and settlement between bridge/clearing accounts are separate funded operations with their own identities, ownership and audit rules; this example does not supply a complete financial accounting or external settlement system.

At zero bridge capacity, a destination cannot simply credit B and promise to repair funding later. Keep the transfer pending under a deadline, fund through the declared settlement policy, or obtain a definitive cancellation before reversing the source. A simpler design colocates both customer balances in one supported atomic ledger boundary. Explain why scale or ownership requires leaving that simpler boundary.

Commit the decision and every operation identity

transfer(tenant, transfer_id UNIQUE within tenant, payload_hash, amount, currency,
         source_account, destination_account, state, decision_version)
source_posting_key = tenant/transfer_id/source
destination_posting_key = tenant/transfer_id/destination
source_reversal_key = tenant/transfer_id/source-reversal

The coordinator stores source_pending → destination_pending → settled, or a durable cancellation decision before requesting reversal. Each shard stores a unique transfer-leg record with the journal atomically. A same-key changed amount/currency is a conflict. The coordinator's process can die; its state, leg evidence and decision survive. Repeated post destination must return the existing journal rather than create another credit.

If the destination response is lost, destination_pending means unknown, not failed. Query the destination's authoritative transfer-leg record under the same identity. Never reverse the source merely because a timer elapsed while the destination may have credited B.

Make cancellation compete with destination posting

Both destination posting and destination cancellation must arbitrate on one durable destination transfer row. An atomic state transition chooses posted or cancelled; cancellation cannot be acknowledged while a concurrent posting remains free to commit. Only a definitive cancelled record allows the coordinator to reverse the source. A delayed old destination request then returns the cancelled outcome. If posted wins, complete settlement or initiate a separately defined return transfer. A source-side timeout cannot impose a destination-side fence by itself.

Read the diagram

Source posts 100 under transfer X → Destination posts but its response is lost → Coordinator queries the same X identity → Destination returns its committed journal → X settles; no source reversal

View full sizeOpen original

Global Payments Ledger: recovery decision

Reset viewClose

Query durable leg evidence before compensating an uncertain destination effect.

Reconcile identities as well as totals

At the same recovery checkpoint, compare transfer state with both leg journals, currency, amount, reversals and ownership epochs. Two duplicate balanced journals still create duplicate value movement. Monitor pending value and age per currency, unresolved cancellation requests, mismatched legs and clearing capacity. Preserve evidence longer than the maximum replay and repair horizon. Operator repairs create new immutable journals with actor and reason; they do not edit old postings.

For 10,000 transfers/s and 0.2% entering unknown state, 20 transfers/s require reconciliation. A 15-minute provider/shard incident produces 18,000 unresolved transfers. At 100 lookups/s while 20/s new unknowns continue, the ideal drain is 225 seconds. The recovery scan must share a budget with production traffic; exhaustive unthrottled lookup can prolong the incident.

Transfer assessment: defend the uncertain leg

Question: A was debited; D timed out. Can the coordinator restore A after thirty seconds?

Question: Both local journals balance to zero. Does that prove one transfer happened?

Question: Post and cancel arrive together at D. Where is the winner chosen?

Score one point each for explicit account/sign convention, unique leg identities, atomic post/cancel arbitration, uncertain-result recovery and checkpointed reconciliation. A response that reverses solely on timeout fails the recovery criterion even if every diagram contains a queue.

Decisions and alternatives

Operational contract

Continuously verify balanced journals, pending-age bounds, reconciliation differences and replication lag. Test a crash after commit, duplicate webhooks, stale-owner writes and restore from backup. Restrict posting authority and retain immutable audit evidence.

Choose alert thresholds from the agreed user-facing objective. Assign an owner for stuck work and a replay/repair procedure that preserves the original business identity. Bound queue depth, concurrency and recovery throughput so an outage does not produce an uncontrolled catch-up burst.

Foundation, senior and staff depth

• Foundation: Explain single-shard balanced posting and unique business identity.
• Senior: Work the cross-shard post/cancel race and preserve pending value.
• Staff: Defend bridge funding limits, regional durability and checkpointed repair.

Practice before revealing the answer

Spend 45 minutes: 5 clarifying scope, 7 on capacity/APIs/state, 12 on the first architecture, 15 on the concurrency and failure examples, and 6 on trade-offs and operations. Draw your own design before comparing it with this chapter.

Question: Does double-entry bookkeeping alone prevent losing or double-counting money?

Verification and rollout

Inject loss after each local leg commit and race destination post against cancellation. Verify local signed-entry sums, nonnegative funded bridge capacity, unique leg journals and exactly one destination terminal outcome. Canary routing without weakening the acknowledged durability policy; retain old evidence for the full replay horizon.

Primary technical reference

Official documentation for the mechanism used in this exercise. This lesson is an original interview design, not a description of undisclosed company internals.

Related study

Before this design, review these mechanisms and identify which owns the critical state transition:

• Payment System
• CAP Theorem

Use the Interview Playbook to adapt the depth and timing, then return to the question breakdown overview.

Shared foundation: Replication, Durability and Failover

Replication, Durability and Failover: Separate durable acknowledgement, read visibility and safe promotion through a concrete replica history. Work its failure history before extending this design.