Service Discovery, Heartbeats and Fencing

Service Discovery, Heartbeats and Fencing

Service discovery finds candidate endpoints; failure detectors estimate reachability; distributed locks coordinate ownership. These are distinct responsibilities. Discovery registers instances through DNS, registry or control plane. Health probes update suspicion. A lease grants temporary ownership, but a paused process can resume after expiry. A monotonically increasing fencing token must be checked by the protected resource so stale owners cannot commit.

Mechanism and worked example

Worker A acquires lease epoch 41 and pauses. Worker B acquires epoch 42. If A resumes, the database rejects its epoch-41 update. Without destination-side fencing, A can still write even though the lock service believes B owns the work.

Failure modes and operations

Stale DNS, delayed probes, partitions, clock drift, session expiry and paused runtimes create false assumptions. A lock service can return a new owner while the old process continues. Client-side checks alone cannot prevent stale writes.

Trade-offs and design boundary

DNS discovery is simple but can be stale. A registry adds metadata but becomes a control-plane dependency. Leases reduce coordination but need fencing. Consensus-backed locks are stronger but require quorum and operational discipline.

Interview transfer exercise

Separate endpoint discovery, health suspicion and exclusive write authority in a distributed scheduler. Explain which component prevents a stale worker from committing.

Read the diagram

Registry returns candidate endpoint → Health signal updates suspicion → Lease authority assigns increasing epoch → Database validates epoch on protected writes

View full sizeOpen original

Service Discovery, Heartbeats and Fencing

Reset viewClose

Mechanism and decision flow for Service Discovery, Heartbeats and Fencing.