Service Discovery, Heartbeats and Fencing
Service discovery finds candidate endpoints; failure detectors estimate reachability; distributed locks coordinate ownership. These are distinct responsibilities.
Service Discovery, Heartbeats and Fencing
Service discovery finds candidate endpoints; failure detectors estimate reachability; distributed locks coordinate ownership. These are distinct responsibilities. Discovery registers instances through DNS, registry or control plane. Health probes update suspicion. A lease grants temporary ownership, but a paused process can resume after expiry. A monotonically increasing fencing token must be checked by the protected resource so stale owners cannot commit.
Mechanism and worked example
Worker A acquires lease epoch 41 and pauses. Worker B acquires epoch 42. If A resumes, the database rejects its epoch-41 update. Without destination-side fencing, A can still write even though the lock service believes B owns the work.
| Concern | Mechanism | Do not assume |
|---|---|---|
| Endpoint lookup | DNS or registry | Address is healthy |
| Health | Probe/heartbeat | Timeout proves death |
| Ownership | Lease or consensus lock | Expired owner stopped running |
| Stale-write prevention | Fencing enforced at destination | Client promise is enough |
Failure modes and operations
Stale DNS, delayed probes, partitions, clock drift, session expiry and paused runtimes create false assumptions. A lock service can return a new owner while the old process continues. Client-side checks alone cannot prevent stale writes.
Trade-offs and design boundary
DNS discovery is simple but can be stale. A registry adds metadata but becomes a control-plane dependency. Leases reduce coordination but need fencing. Consensus-backed locks are stronger but require quorum and operational discipline.
Interview transfer exercise
Separate endpoint discovery, health suspicion and exclusive write authority in a distributed scheduler. Explain which component prevents a stale worker from committing.
Read the diagram
Registry returns candidate endpoint → Health signal updates suspicion → Lease authority assigns increasing epoch → Database validates epoch on protected writes