Outbox, Inbox, Sagas and Reconciliation

Outbox, Inbox, Sagas and Reconciliation

An order can commit locally while its notification is never published. A message can be published twice while its consumer runs twice. A provider can charge successfully while the caller records an unknown result. These are different failure windows. Use an outbox for durable publication intent, an inbox for local duplicate-effect control, a saga for business progress across authorities and reconciliation for disagreements or uncertain outcomes.

Choose the smallest atomic boundary

If all changes fit in one supported database transaction, use it. The following original order exercise crosses stock and payment authorities, so no single local transaction can roll back every effect. The user-visible contract distinguishes accepted, processing, confirmed, cancelling and repair-required states. A pending state is useful only with an owner, a deadline and a recovery action.

Close the database-to-broker gap

BEGIN;
INSERT INTO orders (order_id, tenant_id, state)
VALUES (:order_id, :tenant, 'accepted');
INSERT INTO outbox (event_id, aggregate_id, aggregate_version, payload, state)
VALUES (:event_id, :order_id, 1, :payload, 'pending');
COMMIT;

Both inserts share the local transaction. The publisher claims bounded batches, sends the stable event ID and records delivery progress. Never mark an event sent before the broker's required acknowledgement. A crash after broker acceptance but before recording sent leaves a duplicate-publication window. That is expected and must be tolerated.

An outbox schema also needs uniqueness for event identity, indexes for pending work, retention and a claim/version protocol. Avoid holding a database transaction open while making a slow network call. A lease prevents permanent abandonment after publisher death, but stale publishers can still send again. Preserve event identity across them.

Read the diagram

Order O7 and event E7 commit together → Relay publishes stable event E7 → Relay crashes before recording sent → Replacement republishes the same E7 → Inbox and local effect commit once

View full sizeOpen original

Outbox, Inbox, Sagas and Reconciliation

Reset viewClose

Duplicate publication is repaired at the consumer effect boundary, not by pretending the publisher cannot crash.

Couple inbox and effect

A balance projection receives E7. In one transaction it inserts a unique inbox key and applies the versioned projection update. If E7 already exists, return the previous logical outcome. If the transaction fails, neither record may claim completion. Two workers racing on E7 need authoritative uniqueness, not independent preliminary lookups.

For entity projections, track source versions as well as event IDs. E8 arriving before E7 must not let E7 roll the entity backward. Counts may require idempotent deltas or a different representation; blindly replacing a full balance with whichever event arrived last is not a general solution. State whether ordering is per aggregate, per partition or something else.

Model the saga before choosing an engine

This exercise reserves stock H7, authorizes payment P7 and confirms order O7. Each step uses stable operation identity. A coordinator stores the next decision and outcome evidence independently of its process. The table is a scoped teaching contract, not a complete commerce implementation.

A failed reservation before any payment request follows a simpler rejection path. A charge and an authorization have different compensation actions. Partial capture/refund scenarios need additional states, not a vague generic “undo payment.”

Cancellation races with confirmation at an authoritative state/version transition. Once stock is sold to another buyer, a late success must not take it back silently. A refunded order also has a financial history; compensation does not make the original charge disappear.

Orchestration versus choreography

An orchestrator owns a durable business state machine and dispatches steps. This makes status and repair ownership explicit but concentrates workflow logic and needs compatible upgrades. Choreography lets services react to events; it can reduce central coupling while making cycles, timeouts and terminal ownership harder to see. Both require identities, version rules and observability. A queue or workflow engine does not choose those business semantics automatically.

For a three-step checkout with customer-visible status and compensation deadlines, start with an explicit coordinator. For independent analytics consumers, a saga may be unnecessary. Do not call every event subscription a distributed transaction.

Reconcile uncertain and missing evidence

Callbacks can arrive before synchronous responses, arrive twice, be reordered or never arrive. A reconciler scans old unknown operations, queries the same provider reference and stores the evidence used for each permitted transition. Comparing counts alone is insufficient: two opposite errors can cancel numerically. Compare identities, amounts, currencies, versions and settlement states as applicable.

Before repair, identify the authority for each fact. Provider evidence may establish a charge while the stock store establishes that no reservation survives. That requires a business resolution, not overwriting one system to match the other. Administrative correction records need actor, reason and a stable identity so replaying repair does not create another effect.

Bound storage and repair work

At an illustrative 500 events/s and a six-hour relay outage, the outbox holds 10.8 million events. At 400 bytes each that is 4.32 GB before indexes and replicas. If recovery publishes 800/s while 500/s new work arrives, ideal drain is 10 hours. Increase safe publication capacity, reduce admitted work or accept that delay explicitly. Faster polling alone cannot exceed the broker or consumer budget.

Track oldest pending intent, unknown payment age, compensation age, duplicate effects prevented, version conflicts and poison-event quarantine. Retain replayable evidence for the promised recovery horizon. A deletion policy must not remove the only proof needed to deduplicate or reconcile work still eligible for replay.

Check your understanding

Question: Broker accepted E7 but the publisher crashed before marking sent. Should recovery discard E7 to avoid duplicates?

Question: Can a consumer commit its inbox row, then separately apply the local balance update safely?

Question: Compensation timed out. Is the order fully cancelled and the refund definitely failed?

Transfer exercise and scoring

For Cart and Checkout, Notification System and Temporal, identify source transaction, publication identity, consumer effect boundary, unknown external state and repair owner. Score one point per concrete answer. Add a compensation-failure trace before claiming the workflow is recoverable.

Prerequisites: Idempotency and Delivery Semantics and Multi-step Processes.

Primary references and scope

• AWS transactional outbox pattern.
• AWS saga orchestration pattern.

The records, state table, calculations and repair drill are original examples. Local model tests cover defined crash boundaries; they do not simulate a real broker, workflow engine or provider.