Dealing with Contention

Contention occurs when concurrent operations compete over the same mutable fact. More servers may increase the number of contenders without increasing useful progress. Start by naming the invariant and the smallest state boundary that can enforce it. Then choose a conflict mechanism and an explicit user response when a request loses.

Learning goals

Invariants; atomic conditional writes; locks; optimistic versions; transactions; leases; fencing; contention measurement; conflict responses.

The mechanism at a glance

Figure — Concurrent callers → Atomic state owner (conditional change); Atomic state owner → Winner (predicate matches); Atomic state owner → Conflict response (predicate fails); Lease epoch → Fenced destination (current token); Winner → Fenced destination (authorized effect)

The numbered components identify responsibilities. Follow the labeled arrows rather than treating the numbers as a global execution order. The scenario later in this lesson shows one concrete sequence.

Step-by-step reasoning

1. Make the decision atomic

A strongly consistent read does not make a later write part of the same operation. Two callers can both read available before either writes. Use a conditional update, compare-and-swap version, or an appropriate transaction so checking and changing the state occur at one serialization boundary.

2. Choose optimistic or pessimistic control

Optimistic control reads a version and updates only if it still matches; conflicts require retry or user resolution. Pessimistic control locks the relevant rows while a short transaction runs. Optimistic approaches work well with uncommon conflicts, while repeated collisions can waste work. Locks also have waiting, deadlock, and throughput costs.

3. Distinguish leases from fencing

A lease expires, but the old worker may still be running after a pause. A monotonically increasing fencing token lets the destination reject work from older owners. The destination must actually validate the token; placing it in a log message is not enforcement. A lease alone is not proof of exclusive execution.

4. Reduce the hot boundary

Partition independent resources, shorten critical sections, and avoid external calls while holding locks. For commutative operations, aggregation or sharded counters may reduce contention, but not every operation is safely commutative. A last-seat decision cannot be approximated like a page-view counter.

Contracts and state

The following sketch makes the decision boundary concrete. Field names and capacity assumptions are illustrative; adapt them to the stated product contract.

UPDATE document
SET body = :new_body, version = version + 1
WHERE id = :id AND version = :expected_version;
-- zero affected rows means conflict

Fenced result write:
accept only if attempt_token == current_attempt_token

Worked example

Two editors load version 12. Alice saves and creates version 13. Bob’s save with expected version 12 fails. The interface can ask Bob to merge, reload, or retry after inspecting the new state. Silently overwriting version 13 would lose Alice’s edit. An automatic retry is safe only if Bob’s operation can be meaningfully recomputed against the new version.

Failure walkthrough

Worker A obtains lease token 8 and pauses. The lease expires; worker B receives token 9 and finishes. A resumes and tries to publish an old result. The result store rejects token 8 because token 9 is current. This protects the effect even though both workers executed. If an external provider cannot enforce fencing, use its idempotency and reconciliation mechanisms instead.

Figure — Worker A owns token 8 → A pauses past lease → Worker B owns token 9 → A resumes and writes → Destination rejects token 8

Decisions and trade-offs

Check your understanding

Why does adding a distributed lock with a 30-second expiry not guarantee only one worker writes a result?

Transfer to a new scenario

Compare a database conditional update with a distributed lease for the same scarce resource.

Why is a read-then-write check insufficient even if the read is strongly consistent?

Continue the connection

Study PostgreSQL and explain which guarantee from this lesson carries into that topic.

Put the decision where the invariant lives

A scarce resource needs one authoritative conflict decision. For a seat, use a conditional update from available to held, or lock the row and commit the hold. Reading available and later writing held without a condition allows two winners. An application-level mutex protects only the process holding it unless a broader protocol exists.

Optimistic concurrency works well when conflicts are uncommon: compare a version and retry after re-reading. Pessimistic locks make conflicts explicit but hold resources and can deadlock. Acquire multiple locks in a consistent order, keep transactions short and avoid external network calls while holding them.

A lease is temporary permission, not a guarantee that the old worker stopped. After expiry, a new worker can begin while the old one is paused. A monotonically increasing fencing token must be checked at the protected resource so stale owners cannot commit. Idempotency handles repeat identity; fencing handles obsolete ownership. They solve different problems.

Figure — A decision worksheet for Dealing with Contention: read the mechanism and its guarantee together.

Operational sketch

UPDATE seats SET state="held",hold_id=:h,version=version+1
WHERE id=:seat AND state="available";
Require affected_rows = 1
Fenced write: accept only generation >= stored_generation

A tempting mistake

A distributed lock that expires does not retract an in-flight provider call. Keep the external-effect uncertainty separate from internal write ownership.

Transfer exercise

Worker A pauses with generation 7; B acquires generation 8. What stops A from overwriting B?