LLD-020 · DESIGN PRACTICE
Workflow / State-Machine Engine
Pin workflow definitions, fence worker attempts and repair known effects without treating timeout as proof of failure.
Local Java 21 reference verified: 36 checks. Learner drafts are not executed. Author self-review; independent review pending.
Existing introductory lesson · Version 1
Understand the problem
Design a local linear workflow protocol with versioned definitions, bounded retries, worker claims, cancellation and reverse compensation. Workers execute side effects outside the engine and report outcomes. Define stale completion, unknown effects and failed compensation before adding any orchestration features.
Before you start
- State transitions and immutable definition versions
- Task claims, leases and attempt identity
- Idempotent side effects and unknown outcomes
- Saga compensation and audit evidence
Scope and explicit decisions
- One in-memory engine for linear step lists. It returns claims but runs no worker code, provider call or timer thread. No persistence, multi-process coordination, DAG/branching, parallel joins, dynamic steps, expression interpreter, signals or arbitrary user code is implemented.
- A Definition has a unique immutable version and nonempty uniquely named steps. A step configures 1..5 attempts, positive fixed retry delay, whether external retries are safe, and whether accepted success requires compensation on failure/cancellation. Same version with changed contents rejects.
- start binds a retry key to definition version and input String. Matching retries return the current instance view. Each instance pins that exact immutable definition; registering a newer version cannot silently migrate in-flight instances. Claim carries this input; no output propagation or per-step dataflow is implemented.
- Engine operations are synchronized. claim selects one eligible due step/compensation and assigns a monotonically increasing per-instance token with checked lease deadline. RUNNING allows one active attempt. Workers act outside the engine monitor. Trusted caller time is nondecreasing; view/start do not perform automatic timeout scanning.
- Stable external operation key is instance/step/work or instance/step/undo. New attempts get fresh fencing tokens but reuse that logical side-effect key. Declaring idempotent=true is a contract on the adapter/provider, not an implementation of remote deduplication.
- SUCCESS is definite accepted effect, RETRYABLE and FATAL are trusted definite unsuccessful outcomes, UNKNOWN means outcome unresolved. UNKNOWN keeps the active attempt RUNNING so later definite evidence may arrive before lease expiry. A transport timeout cannot be reported as definite failure without external evidence.
- At now>=leaseUntil a completion cannot become accepted success under the old claim. Timeout retires its token. A safe idempotent step with budget can retry after delay; non-idempotent, exhausted or cancelled unresolved forward work enters MANUAL_REVIEW. No automated manual-review resolution API is implemented.
- Definite forward failure that cannot retry starts compensation of known completed steps configured compensate=true, in reverse success order. compensate=false must mean no repair is required under the workflow contract; it does not make irreversible harmful effects disappear.
- Running cancellation records intent and waits for definite completion. A later success is recorded then compensated; a cancelled unknown timeout needs manual review because an unobserved effect may exist. Cancellation cannot retract an external operation already committed.
- Compensation has its own stable external key and attempt budget. Definite exhausted/fatal undo failure becomes COMPENSATION_FAILED. retryCompensation is an explicit manual action resetting that undo budget; it does not bypass unresolved MANUAL_REVIEW. Terminal FAILED/CANCELLED follows completed required compensations. Histories are retained and not durable.
Worked examples
Input: reserve succeeds; idempotent charge returns RETRYABLE; retry after delay succeeds; ship returns FATAL.
Outcome: Workflow enters compensation, undoing known charge then reserve effects in reverse order. A failed undo remains visible.
Forward retry budget and reverse repair are different state machines with distinct logical external keys.
Input: Cancel while a forward worker is running. The worker later reports SUCCESS before lease expiry.
Outcome: Success becomes known history, then required compensation starts. If instead the unresolved lease expires, MANUAL_REVIEW is required.
Cancellation intent cannot prove an already issued remote action did not commit.
Your design attempt
Each language has a separate draft across all three levels. Saving is device-only and does not sync to your account. Anyone using this browser profile can load it. Loading is always explicit.
Java 21 reference code is provided for local study. Use Run program in the editor to execute your code and test driver. Program runs show output; they do not grade your design.
Run program · Java 21
No unsaved changes · 208/20000
Self-review
Checks apply to this visit. Self-review is not a passed submission or verified mastery.
Try before revealing
Hint 1
Which identity belongs to an effect, and which belongs to an attempt?
Hint 2
Does a lease expiry prove the remote operation failed?
Hint 3
Which accepted effects can you actually compensate, and what if that repair fails?
Requirements and invariants
R1 — Pin each instance to an immutable definition and typed start retry identity. start protects I1.
R2 — At most one active worker claim owns the current step, and stale completion cannot advance it. claim protects I2.
R3 — Retry only under the configured safe identity/budget and keep unknown effects explicit. timeout protects I3.
R4 — Compensate known required effects in reverse order and expose failed repair. complete protects I4.
I1: One registered version has immutable contents; an instance retains its start definition and matching start keys cannot create another instance.
I2: Only RUNNING has an activeToken; only that unexpired token may record a new definite completion. Matching recorded duplicate outcomes cannot advance twice.
I3: Logical external work identity is stable across attempts, attempt tokens change, and unresolved unsafe/exhausted work remains explicit MANUAL_REVIEW.
I4: Compensation selects only known accepted forward successes marked for repair, in reverse completion order; failed undo is never silently reported as clean rollback.
Derive the design
A loop cannot represent unresolved side effects
A for-loop calling reserve, charge and ship hides the critical case: charge commits remotely, the response is lost and the caller cannot tell whether to retry or refund. Begin with explicit instance states and immutable definition identity. Separate a logical operation from an attempt token, and let the engine accept correlated evidence instead of running network calls under its lock.
Alternative: A production durable workflow platform may provide event history, replay, timers, signals and worker routing. Its determinism/versioning rules still matter. This local reference explains the protocol boundary and deliberately does not impersonate a durable engine with a list and a boolean.
Write states before classes
List due, running, unknown, successful, failed and repairing states. A pending cancellation is intent, not proof of termination. Use MANUAL_REVIEW and COMPENSATION_FAILED as visible outcomes.
Choose identities at two levels
An instance/step key identifies the external effect. A token identifies one ownership attempt. Stable keys prevent duplicate logical work only when the destination honors them; tokens reject stale local evidence.
Pin interpretation and evidence
An instance keeps its original definition and input. Record definite outcomes for duplicate detection, but do not record UNKNOWN as a final result that would prohibit later truth.
Compensate only what is known
Build reverse repair from accepted successes. Unknown timed-out work may need reconciliation before this list is complete. Never infer a safe refund or release from a missing response.
Responsibilities and ownership
Definition / Step
Owns: Immutable version, ordered names and retry/repair policy
Methods: Validated records; register
Configuration declares external safety; it does not implement the provider contract.
Engine
Owns: Instance states, authoritative time, claims, retries and transitions
Methods: start, claim, complete, timeout, cancel, retryCompensation, view
No user code or external I/O under its monitor.
Instance
Owns: Pinned definition/input, completed history, reverse undo queue, attempts and accepted outcome evidence
Methods: Private transition state
completed count is historical forward successes, not current uncompensated count.
Claim / View
Owns: Execution capability metadata versus current observation
Methods: Immutable record accessors
Token rejects local stale completion; external destination must enforce idempotency/fencing independently.
A lease fences local reporting, not an external side effect by itself.
- Due pinned workflow step
↓ One local active owner
Claim: token + stable effect key↓ Destination honors retry identity
External execution outside engine↓ Old token cannot advance state
Accept definite evidence or review expiry
Unknown effects cannot be wished into the known-success compensation list.
Accepted undo reaches FAILED/CANCELLED
Expose COMPENSATION_FAILED / MANUAL_REVIEW
Data model, schema and relationships
Definition versions own immutable ordered steps; instances pin one definition/input. Attempts own temporary reporting authority, while compensation is separately tracked repair of known accepted effects.
Implementation boundary: The reference is a local linear protocol. Tables propose durable state/claim/outcome storage; persistence, worker execution, timers, replay and review resolution are still extensions.
These are proposed logical schemas with field contracts, not executable database migrations. PK identifies a row, FK references an existing parent, UK is a uniqueness rule. A composite PK identifies the combination. Nullable references are marked with ?. A foreign key protects identity; cross-row limits, timing and state transitions also require the transaction rules below.
Connectors name the relationship; multiplicities state parents per child and children per parent. 0..1 means optional, 1 required, 0..* any number. Self-links show two roles of the same entity.
Connectors name the relationship; multiplicities state parents per child and children per parent. 0..1 means optional, 1 required, 0..* any number. Self-links show two roles of the same entity.
Connectors name the relationship; multiplicities state parents per child and children per parent. 0..1 means optional, 1 required, 0..* any number. Self-links show two roles of the same entity.
Entity definitions and field contracts
workflow_definition
Immutable versioned execution meaning.
| Field | Type / key | Meaning and reference |
|---|---|---|
id | text · PK | Definition version |
name | text · VALUE | Workflow family |
Code mapping: Definition.
- Same version cannot change contents.
workflow_step
Ordered retry/repair policy.
| Field | Type / key | Meaning and reference |
|---|---|---|
id | uuid · PK | Definition-step identity |
definition_id | text · FK | Pinned definition · workflow_definition.id |
position | integer · VALUE | Linear order |
name | text · VALUE | Stable step name |
max_attempts | integer · VALUE | Bounded automatic budget |
retry_delay | bigint · VALUE | Positive fixed delay between safe bounded attempts |
idempotent | boolean · VALUE | Trusted external retry safety contract, not remote deduplication implementation |
compensate | boolean · VALUE | Whether known accepted success requires undo on later failure/cancellation |
Code mapping: Step.
- Unique(definition_id,position) and (definition_id,name); retain idempotency/compensation flags and retry delay.
workflow_instance
One pinned execution.
| Field | Type / key | Meaning and reference |
|---|---|---|
id | uuid · PK | Instance identity |
definition_id | text · FK | Pinned version · workflow_definition.id |
input | text · VALUE | Immutable context |
state | enum · VALUE | Explicit orchestration state |
version | bigint · VALUE | Conditional transition version |
due | timestamp · VALUE | Earliest eligible claim instant persisted for recovery |
Code mapping: Instance.
- Start key binds definition+input; RUNNING permits one active token.
workflow_attempt
Temporary worker/reporting authority.
| Field | Type / key | Meaning and reference |
|---|---|---|
id | uuid · PK | Attempt identity |
instance_id | uuid · FK | Execution owner · workflow_instance.id |
step_id | uuid · FK | Step interpretation · workflow_step.id |
token | bigint · VALUE | Fresh instance fencing epoch |
external_key | text · VALUE | Stable logical effect identity |
lease_until | timestamp · VALUE | Reporting deadline |
compensation | boolean · VALUE | Distinguishes forward work from undo using separate stable external keys |
Code mapping: Claim and accepted outcome evidence.
- Unique(instance_id,token); step must belong to instance pinned definition. Work/undo keys differ and retry tokens change.
compensation
Required repair for known success.
| Field | Type / key | Meaning and reference |
|---|---|---|
id | uuid · PK | Repair identity |
instance_id | uuid · FK | Workflow owner · workflow_instance.id |
step_id | uuid · FK | Known successful effect · workflow_step.id |
status | enum · VALUE | Pending complete failed review |
Code mapping: Undo queue; durable row extension.
- Unique(instance_id,step_id) for linear nonrepeated steps; repair order reverses accepted successes.
Relationship cardinalities and their reasons
| Parent → child | Multiplicity | FK / why |
|---|---|---|
| workflow_definition → workflow_step orders steps | 1 parent per child; 0..* children per parent | workflow_step.definition_id → workflow_definition.idDefinition owns immutable transition interpretation. |
| workflow_definition → workflow_instance pins runs | 1 parent per child; 0..* children per parent | workflow_instance.definition_id → workflow_definition.idNew versions cannot silently change existing instances. |
| workflow_instance → workflow_attempt claims attempts | 1 parent per child; 0..* children per parent | workflow_attempt.instance_id → workflow_instance.idRetained attempts explain lease/ownership history. |
| workflow_step → workflow_attempt interprets attempt | 1 parent per child; 0..* children per parent | workflow_attempt.step_id → workflow_step.idVerify same definition as instance; FK alone does not prove compatibility. |
| workflow_instance → compensation tracks repair | 1 parent per child; 0..* children per parent | compensation.instance_id → workflow_instance.idOnly known successful required effects enter compensation. |
| workflow_step → compensation has undo meaning | 1 parent per child; 0..* children per parent | compensation.step_id → workflow_step.idUndo is separate logical work and may fail independently. |
Transaction boundaries
- Claim: condition instance version/due/state/no active token, assign new token/lease and persist dispatch outbox in one transaction.
- Complete: deduplicate definite evidence, check current unexpired ownership, append outcome and advance state/repair obligations atomically. External execution stays outside transaction.
- Timeout: retire ownership; safely retry with same effect key only under idempotency/budget, otherwise retain MANUAL_REVIEW.
Indexes and uniqueness
- Unique start key plus definition/input fingerprint; unique instance/token.
- Index claimable instance(state,due), active attempt(lease_until), compensation(instance_id,status,reverse_order).
Schema traps and failure cases
- A database fencing token cannot stop a stale worker committing at an external provider unless that provider honors key/fence or is reconciled.
- A failed refund cannot be hidden as clean rollback; repair failure and unknown outcome remain explicit.
Trace the records through one operation
- Instance W pins definition v1, input and stable reserve/pay keys.
- Pay attempt expires; safe retry has fresh token and same external key.
- A later definite forward failure compensates known successes in reverse; failed undo remains visible.
Method contracts
register(definition); start(version,input,key) → View
Re-register equal version contents safely; reject changed contents. Start key binds version+input and returns one instance current view.
claim(instanceId,now) → Claim
READY/WAITING/COMPENSATING must be due with no active token. Claim increments attempt/token, supplies pinned version/input, logical external key and lease deadline.
complete(instanceId,token,outcome,now) → View
Matching accepted definite duplicate returns current view; contradiction rejects. A new outcome requires current unexpired active token. UNKNOWN retains ownership. Success advances or undoes one step; definite failure chooses safe retry or repair.
timeout(instanceId,token,now) → View
Only the expired active RUNNING attempt can timeout. Retire token and choose safe retry or MANUAL_REVIEW; timeout does not establish that the remote effect failed.
cancel(instanceId,now); retryCompensation(instanceId,now); view(instanceId)
Cancellation is intent while running and repair of known successes when safe. Failed compensation can be explicitly retried with reset budget. MANUAL_REVIEW has no automatic resolution in this reference.
What, when, why and how
Each concept is attached to the requirement it protects, its implementation, an alternative and a test.
Version-pinned state machine
- what
- Keep transition meaning stable for every in-flight instance.
- when
- Workflow definitions evolve while older instances are still running.
- why
- Loading the latest step list on every claim can reorder work or compensate a step under rules it never used.
- how
- register rejects changed contents under one version. start stores the immutable Definition and input with typed retry fingerprint. claim reads that pinned object.
Code: Definition, register, start, Instance · Invariants: I1
Alternatives, cost and failure example
Alternative: An explicit migration protocol can change running instances only with state compatibility checks and auditable mapping.
Cost: Definitions and instances remain retained; no schema migration or persistent event replay is implemented.
Counterexample: Rename charge to debit in v1 while an old instance is waiting and lose its stable external key.
Test: Register changed v1 contents rejects; register v2 then prove old instance claims v1.
Logical operation identity and fenced attempts
- what
- Separate what effect should happen from which worker currently owns reporting it.
- when
- A worker times out and another retries while the old worker may still finish.
- why
- A new random provider key on each retry duplicates effects; accepting an old token advances stale work.
- how
- Claim uses stable instance/step/work or undo key, fresh token and lease. complete accepts only current unexpired active ownership for new outcomes. Recorded definite duplicates are safe and contradictions reject.
Code: Claim, claim, complete · Invariants: I2, I3
Alternatives, cost and failure example
Alternative: A durable conditional claim row with fencing epoch implements the same local ownership idea across processes.
Cost: Local fencing cannot stop an old worker at the external destination. That destination must honor dedupe/fencing or require reconciliation.
Counterexample: Timed-out charge attempt and replacement both send independent charge IDs.
Test: Retry preserves externalKey and changes token; stale first completion rejects without advancing the replacement.
Bounded retry with honest uncertainty
- what
- Retry policy is constrained by evidence and side-effect safety.
- when
- A callback says definite retryable failure or a lease expires without a result.
- why
- Treating every exception as failure can compensate a successful charge or repeat a non-idempotent action.
- how
- RETRYABLE uses safe idempotency/budget and fixed due delay. UNKNOWN remains RUNNING. timeout retires token; unsafe, exhausted or cancelled unknown forward work becomes MANUAL_REVIEW.
Code: failed, expire, timeout, complete · Invariants: I3
Alternatives, cost and failure example
Alternative: A reconciliation worker can query external state and submit verified resolution through a carefully designed review API; none is silently invented here.
Cost: Fixed delay has no jitter/backoff classification; declared idempotency is trusted. Manual review can remain unresolved indefinitely.
Counterexample: Lease expires on non-idempotent shipment and engine assumes nothing shipped.
Test: Assert non-idempotent timeout and cancelled unknown timeout stop in MANUAL_REVIEW instead of reporting clean cancellation.
Saga compensation of known effects
- what
- Repair accepted prior successes with explicit reverse operations.
- when
- Later forward work definitely fails or cancellation arrives.
- why
- Rollback of memory cannot undo external charges or shipments, and repair itself may fail.
- how
- beginUndo builds a reverse list from completed successes with compensate=true. Compensation claims use separate stable undo keys/budgets. Definite repair failure exposes COMPENSATION_FAILED; explicit retry resets that undo budget.
Code: beginUndo, failed, retryCompensation, complete · Invariants: I4
Alternatives, cost and failure example
Alternative: A single ACID transaction is preferable when every mutation belongs to one database and no remote effects escape its transaction.
Cost: Compensation is business repair, not time travel. Refund fees, irreversible delivery and human review need explicit domain policies.
Counterexample: Mark FAILED immediately after charge then ignore a failed refund.
Test: Force forward fatal failure, then fail compensation, retry it explicitly and reach FAILED only after accepted undo.
Read and run the Java reference
Reveal complete Solution.java
Java 21. Save as Solution.java; compile with javac --release 21. The repository also contains Test.java and the lld:java verification command. This reference implements the local contract above.
claim grants one current attempt
Eligibility and active-token assignment occur together. The returned Claim carries version/input and external key; execution happens elsewhere.
complete resolves duplicates before new transitions
Previously accepted definite evidence returns its current view if identical; contradiction rejects. Fresh outcomes must own a current unexpired claim.
UNKNOWN keeps the claim unresolved
A later definite response may still resolve the active attempt before lease expiry. After expiry the old worker cannot regain ownership by reporting late success.
beginUndo and retryCompensation expose repair
Undo order is reverse known success, with fresh tokens but stable repair keys. A manual retry is explicit evidence of operator intent, not automatic cleanup of every uncertain case.
Complexity and limits
- With S steps and H retained accepted outcomes, definition validation costs O(S); start stores a definition reference rather than copying all step state. Map lookups are expected O(1).
- claim/view and ordinary completion are expected O(1), excluding initial record creation. beginUndo scans completed history O(S); the undo deque consumes O(S) and executes one claim at a time.
- Space grows with registered definitions, instances, completed/undo lists, start keys and outcome history. Attempts are bounded per automatic retry cycle, but manual compensation resets and total instance count are not memory quotas.
- One monitor serializes transitions, not external worker execution. There is no background timeout scan, durable scheduler, recovery log or distributed ownership throughput guarantee.
Walk through the actual reference
These walkthrough fixtures are emitted only after the Java harness verifies their operations and expected state. Concurrency tests additionally exercise competing callers.
Retry token versus external identity
Step 1 of 3: Reserve succeeds; pay retryable
WAITING until t=4
Invariant: Known completed steps retained
Read the complete trace table
| Action | State after action | Invariant |
|---|---|---|
| Reserve succeeds; pay retryable | WAITING until t=4 | Known completed steps retained |
| Claim retry | New fencing token, same pay external key | Retry identity and ownership epoch differ |
| Pay fatal; reserve compensation fails then retries | COMPENSATION_FAILED then FAILED after confirmed repair | Compensation is tracked business work |
Lease expiry cannot prove external failure
Step 1 of 2: Unknown idempotent attempt expires
Retry allowed within budget with fresh token
Invariant: Old completion cannot commit
Read the complete trace table
| Action | State after action | Invariant |
|---|---|---|
| Unknown idempotent attempt expires | Retry allowed within budget with fresh token | Old completion cannot commit |
| Non-idempotent or cancelled unknown attempt expires | MANUAL_REVIEW | Do not invent a safe retry or completed compensation |
Mistakes and counterexamples
Generate a new provider key for every attempt
Failing case: A late first charge and retry both commit.
Repair: Stable logical external key; fresh local attempt token.
Treat timeout as definite failure
Failing case: Charge committed but its response was lost.
Repair: Keep unknown evidence and reconcile or use proven idempotency.
Run callbacks while holding engine ownership
Failing case: A slow provider blocks cancellation and every other workflow.
Repair: Return claims and accept correlated completion later.
Hide compensation failure as rolled back
Failing case: Refund failed but workflow reports clean cancellation.
Repair: Expose repair state and explicit operator retry/reconciliation.
Change a definition in place
Failing case: A waiting old instance resumes under new step meaning.
Repair: Immutable versions and explicit migration policy.
Engineer → Senior → Staff
Implement a versioned linear state machine.
Executed: immutable definitions, start retry keys, successful progression and duplicate outcome guards.
Change exercise
Add a step output consumed by the next step.
Worked answer guidance
Define a typed immutable result schema pinned to step/definition version, store it atomically with accepted SUCCESS and supply only accepted prior outputs in the next claim. Do not let arbitrary workers mutate shared instance input.
Interview follow-ups
Why is Claim input immutable String?
Workers need start context without receiving mutable engine state. Real payload schemas need size, privacy and version contracts.
Does completed decrease after compensation?
No. It counts historical accepted forward successes. Separate undo state represents repair; history remains evidence.
Provenance and scope of evidence
Original teaching material. The local contract is implemented by Solution.java and tested by Test.java on Java 21. Distributed persistence, real provider/device adapters and deployment are not claimed.
Reference verification is not independent editorial review, production deployment verification or a claim that distributed adapters are implemented.