Idempotency and Delivery Semantics

Idempotency and Delivery Semantics

A caller times out after creating an order. The server may have committed, rejected or never received it. Idempotency gives repeated attempts for the same declared operation one logical result within a documented scope and lifetime. It does not mean only one network request or only one worker execution occurs.

Start with intent, not payload equality

This original exercise creates a customer order and later charges an external provider. The local contract is: one committed order per (tenant, operation, client_key), with the same meaning on every retry. Two deliberate orders with identical contents are allowed when they use different operation keys. Therefore hashing the body alone cannot identify intent.

A unique order key cannot prevent a payment provider from creating two charges under different provider keys. Keep identities stable at every destination that owns an irreversible effect.

Store identity with the local effect

CREATE TABLE operation_result (
  tenant_id text NOT NULL,
  operation text NOT NULL,
  client_key text NOT NULL,
  payload_hash text NOT NULL,
  resource_id text NOT NULL,
  response_code integer NOT NULL,
  expires_at timestamptz NOT NULL,
  PRIMARY KEY (tenant_id, operation, client_key)
);

Canonicalize fields according to the operation contract before computing the fingerprint. Normalize only equivalences that the product actually accepts: two different currencies or shipping addresses are not harmless formatting changes. The key is bounded and validated input. Store response data needed for a consistent retry, but authorize every lookup using current caller permissions.

The following is algorithmic pseudocode. All three inserts share one supported database transaction; the network publication does not occur inside it.

begin local transaction
  attempt insert operation_result(T, "create_order", K, fingerprint, orderId, 201)
  insert order(orderId, T, immutableAcceptedLines)
  insert outbox(eventId, orderId, "order_created")
commit
return orderId

if operation uniqueness conflict:
  rollback attempted transaction
  read committed operation_result for (T, "create_order", K)
  same fingerprint -> return saved logical result
  different fingerprint -> return key-reused-with-different-intent error

A real implementation must handle conflict waiting, transaction isolation, a transaction that rolled back while the loser waited and lookup authorization. Retry a bounded fresh transaction where required. Do not insert an uncommitted placeholder in one transaction and create the order in an unrelated transaction without a recovery state machine.

Trace two callers and a crash

Read the diagram

A commits scoped key K and order O7 → Response is lost after commit → B retries K with the same fingerprint → Unique operation resolves to saved O7 → No second order; external effects need their own identity

View full sizeOpen original

Idempotency and Delivery Semantics

Reset viewClose

A lost response is resolved by durable operation identity, not by guessing whether the first request arrived.

The outbox closes the local commit-to-publication gap by preserving intent. It can still publish more than once. Broker deduplication and consumer idempotency address different boundaries; one successful publish acknowledgement does not prove the business consumer completed.

Consumer inbox and effect must share an authority

If a consumer updates an internal balance, inserting its inbox event key and applying the balance change in one transaction makes replay harmless. A preliminary seen(eventId) lookup followed by an unprotected increment does not: two workers can both see missing. If the increment commits but the inbox does not, replay increments again; if the inbox commits first and the effect does not, replay may skip required work.

For an external email or charge, the inbox cannot be atomically committed with an arbitrary provider effect. Use provider-supported operation identity and status lookup where available. Otherwise document the uncertainty and escalation policy; a local flag cannot create an atomic boundary with another service.

Resolve a provider timeout

1. Persist payment intent P7 and the exact stable provider key before sending.
2. Send the charge with P7. If the reply times out, mark the result unknown rather than failed.
3. A retry or reconciliation queries/resubmits P7 using the provider's supported contract.
4. Duplicate callbacks are recorded once by provider event identity; update payment state through allowed transitions.
5. Only after definitive evidence decide success, failure or compensation. A refund has its own identity and can itself be uncertain.

Failing over immediately to provider B under a fresh key can charge twice when provider A succeeded. If the product accepts that risk, it needs an explicit policy and durable repair ownership. “We have a queue” is not such a policy.

Identity lifetime and delayed replays

Assume local retry records live for seven days but an operator can replay messages retained for thirty days. A day-20 replay can pass an expired dedupe check and repeat the effect. Make the effect's durable identity cover the entire promised replay horizon, or reject/redesign old replay. TTL cleanup is asynchronous storage housekeeping, not a semantic proof that a key is reusable.

Define whether an expired client key is rejected, treated as new intent or resolved through a longer-lived business record. For financial operations, a long-lived business uniqueness constraint may be needed beyond the API response cache. Do not permit automatic reuse just because a cache evicted an entry.

Operate retries as finite work

At 1,000 new operations/s, a seven-day response registry holds 604.8 million entries. At an assumed 200 logical bytes each, that is about 121 GB before indexes and replicas. This estimate motivates retention tiers and compact business identity records; it does not justify forgetting an identity while replay remains permitted.

Track duplicate resolution rate, payload conflicts, in-progress age, unknown-provider age and retry budget exhaustion. Use backoff and jitter with one end-to-end deadline. A thousand retries returning the same O7 can still overload the service; correctness and capacity are different concerns.

Check your understanding

Question: Two tenants both choose key K7. Should the second receive the first tenant's order?

Question: A replayed event is 20 days old, but the inbox expires after seven days. Can it safely increment a financial balance again?

Question: The charge timed out and a new worker owns the lease. May it charge under a fresh provider key?

Transfer exercise and scoring

For Payment System, name the local operation, provider operation, callback event, retention horizon and unknown-outcome policy. Award one point for each correctly scoped identity or policy. For Notification System, explain what changes when the provider cannot deduplicate. For API Design, specify same-key changed-body and expired-key responses.

Continue with Multi-step Processes for compensation and Kafka for offsets and replay.

Primary reference and scope

AWS Builders' Library: making retries safe with idempotent APIs discusses caller intent and retry identity. This lesson's order/payment histories, schema, pseudocode, arithmetic and scoring are original examples. Provider guarantees must be checked against the specific API; do not infer them from the word idempotent.