Consensus, Leader Election and Fencing

Consensus, Leader Election and Fencing

Consensus lets a group agree on an ordered history despite a stated set of failures. Leader election selects who proposes work; fencing stops an obsolete owner from changing a protected resource. They cooperate, but choosing a new leader does not physically stop the old process. In interviews, name the history being agreed, the failure model and the destination that enforces authority.

Start with a concrete invariant

This original exercise has five coordination voters A–E and a separate result database. One worker at a time may publish the current export result for job J. Nodes can crash or pause and messages can be delayed; Byzantine behavior is outside scope. Coordination availability requires a reachable voting majority. The result store must independently enforce the accepted job generation.

A timeout means communication is late, not that another process is dead. A lease can expire while its holder is paused inside an external call.

Partition the voters before choosing a leader

Two size-three subsets of a fixed five-node membership intersect. Intersection supplies a useful building block, but safety also depends on durable voting state, log eligibility rules and commitment rules. Picking any three acknowledgements from arbitrary storage is not automatically a consensus algorithm. Membership changes need a protocol preserving safety between configurations; independently switching voter lists can invalidate this reasoning.

Explain a Raft-style history accurately

The selected reference is Raft, not a proposed new protocol. A candidate increments its term, persists the required state and solicits votes. Eligibility considers log freshness. Leaders replicate entries and a current-term entry can become committed through the protocol's majority rule; earlier entries are not safely declared committed merely by counting copies independently. A minority former leader cannot keep committing while isolated. On learning of a higher term it steps down. These are protocol-specific rules; a production implementation needs the full specification.

Original history: term 7 owner A pauses; C,D,E elect a term 8 leader and commit ownership generation 8. A still remembers generation 7 and may resume executing code. Coordination agrees on 8, but A's stale request is already on its way to the result database. This is where fencing matters.

Read the diagram

Worker A pauses with generation 7 → A voting majority commits generation 8 → Result store establishes current generation 8 → Worker A resumes and submits generation 7 → Atomic generation check rejects stale publication

View full sizeOpen original

Consensus, Leader Election and Fencing

Reset viewClose

Establish the new generation at the protected destination before allowing its owner to publish.

Put enforcement at the resource

For this job, a controller advances a result row's expected generation through a trusted operation and then gives the worker generation 8. Workers cannot arbitrarily advance it. Publication requires equality and the allowed state transition in one atomic decision:

UPDATE export_job
SET result_key = :immutable_key, state = 'succeeded'
WHERE job_id = :job
  AND owner_generation = :generation
  AND state = 'running';

Require exactly one affected row. A stale generation returns conflict; it does not overwrite the winner. A separate trusted assignment transition changes owner_generation and validates the coordination decision. Authorization prevents a worker from inventing a larger number. The generation and state checks must share the publication's atomic boundary.

Another resource may implement highest-seen monotonic tokens rather than exact-current equality. Do not silently substitute it: until that destination observes the new token, an older token might still be accepted. Decide the ordering guarantee required and establish the fence accordingly. Counter reset, backup restore and generation reuse can break either scheme; generations must remain meaningful across recovery.

Know the external-effect boundary

An email provider usually cannot check your job generation. Fencing your local success row prevents stale result publication, but cannot recall an email already accepted by that provider. Use its supported idempotency identity, preserve unknown outcomes and reconcile. A lock service does not extend its safety guarantee into an arbitrary external API.

Likewise, leader leases used to serve fresh reads depend on specific clock, timing and protocol assumptions. Do not infer linearizable reads from a locally cached statement that a node was leader five seconds ago. Use the selected system's supported read protocol and describe how it behaves when it cannot confirm authority.

Recovery and operating cost

Monitor election churn, quorum reachability, commit latency, log catch-up and stale-generation rejections. A successful heartbeat is not proof of durable commit. Put voters in failure domains matching the requirement; three voters in one zone cannot protect that zone's loss just because the total voter count is five.

For an illustrative 80 ms round trip to the needed majority, a fresh quorum-dependent decision cannot be assumed to complete in 5 ms. Exact latency includes disk, batching and protocol behavior. Keep high-volume application data outside a small coordination service unless its documented workload and capacity support it. Choosing a mature coordination system is usually safer than implementing consensus during an application project.

Test restart of the old owner, loss of a minority, loss of a majority and recovery from stored generation state. A safe pause when no authority can be established is different from data loss. Never lower the generation just to make a stuck worker succeed.

Check your understanding

Question: Can A,B and C,D,E both safely commit new ownership decisions during their partition?

Question: Generation 8 is current in the result row. Generation 7 resumes and finishes first. Which result wins?

Question: Does a generation check prevent a provider charge already submitted by generation 7?

Transfer exercise and scoring

For Distributed Cache, Distributed Message Queue and ZooKeeper, name the agreed state, membership, partition policy, stale owner and enforcing destination. Award one point per justified item. Include a concrete paused-worker trace; “we use a lock” earns no fencing point.

Prerequisites: Replication, Durability and Failover and Transactions and Isolation.

Primary reference and scope

Raft paper, sections 5.2–5.4, supplies the protocol rules summarized here. The export job, separate fencing contract, SQL and partition exercise are original teaching examples. Repository tests enumerate majority intersections and exercise destination generation checks; they are not a Raft implementation or distributed consensus proof.