Prerequisites and scoped previews
Events, identity and data contracts
Define keys, event/ingestion/availability times, schema evolution, dedupe and permitted use.
Calibration, uncertainty and decision thresholds
Distinguish score quality from calibrated probabilities, uncertainty and action capacity.
AI coding assistant with repository context and safe tool execution
Commit-pinned code retrieval, safe patch generation, isolated validation and approval-gated repository writes.
What, when, why and how
Separate planning from execution
- what
- The model proposes a plan and typed tool intents; a deterministic executor validates and performs permitted operations.
- when
- Whenever an agent can call tools, mutate records or trigger external workflows.
- why
- Model output is probabilistic and must not become an authorization or execution boundary.
- how
- Use versioned tool contracts, schema validation, policy checks and a server-owned execution state machine.
Make every tool a capability
- what
- A tool is a narrow, typed operation with explicit resource scope, tenant, limits and side-effect classification.
- when
- For database reads, ticket updates, email, payments, deployment and other integrations.
- why
- Broad ambient credentials let prompt injection or confused-deputy bugs expand the blast radius.
- how
- Resolve identity server-side, mint short-lived scoped capabilities and reauthorize at execution time.
Persist workflow state
- what
- Tasks, steps, tool intents, results, approvals and checkpoints are durable records.
- when
- For work that can outlive a request, worker, browser session or provider call.
- why
- In-memory agent loops lose progress and can repeat side effects after crashes.
- how
- Use a state machine, transactional outbox, worker leases, idempotency keys and resumable checkpoints.
Treat memory as governed data
- what
- Conversation context, task state and long-lived user memory have different retention and access rules.
- when
- When agents need continuity across turns or sessions.
- why
- Unbounded memory causes privacy leaks, stale facts and cross-tenant contamination.
- how
- Keep ephemeral context separate from curated memory, attach provenance/TTL and enforce user controls.
Bound retries and compensation
- what
- Retries are safe only for idempotent operations; completed external side effects need explicit compensation or reconciliation.
- when
- Across network calls, queues, model providers and external APIs.
- why
- At-least-once delivery can otherwise create duplicate messages, tickets or financial actions.
- how
- Use operation IDs, dedupe records, retry budgets, deadlines, cancellation and saga-style compensation.
Human approval is a real boundary
- what
- Approval is persisted against an exact action payload and expires when that payload or authorization changes.
- when
- For irreversible, sensitive, costly or externally visible actions.
- why
- A conversational 'yes' or model assertion is not durable authorization.
- how
- Show exact changes, bind approval to a canonical payload hash and reauthorize immediately before execution.
Evaluate complete trajectories
- what
- Agent quality includes task completion, policy compliance, tool correctness, latency, cost and recovery behavior.
- when
- Before model/prompt/tool changes and during production rollout.
- why
- A fluent final answer can hide incorrect or unauthorized intermediate actions.
- how
- Trace every transition, replay representative tasks and score outcome plus safety invariants.
Frame the agent workflow problem
Build a platform that accepts a user goal, plans bounded work, invokes approved tools and reports a verified outcome. Example: 'Prepare a customer refund case' may require reading an order, checking eligibility, drafting a response and asking for approval before issuing a refund.
Synthetic interview assumptions: 100,000 active users, 5,000 peak concurrent sessions, 20,000 workflow starts/hour at peak, median workflow duration 45 seconds and a 10-minute p99 for human-gated tasks. These are design inputs, not production measurements. The system must support both short interactive tasks and durable workflows that wait for external events or people.
Define requirements and invariants
Functional requirements: accept a goal, clarify missing inputs, produce a plan, select tools, validate arguments, execute steps, preserve progress, request approval, resume after approval, cancel work and return evidence-backed status.
Non-functional requirements: tenant isolation, least privilege, bounded cost and latency, durable state, replayable traces, safe retries, auditability and graceful degradation when a model or tool provider is unavailable. Invariants: the model never grants itself permissions; every tool call is scoped to an authenticated principal and tenant; side effects are idempotent or explicitly reconciled; cancellation prevents new work; and every terminal result has durable evidence.
Map trust boundaries and threat model
Treat user prompts, retrieved documents, tool outputs, model responses and external web pages as untrusted data. Prompt injection can arrive through any of them. Instructions embedded in a document must never change system policy or create new capabilities.
Keep the policy engine, identity provider, secrets broker and tool executor outside the model boundary. Enforce tenant/resource checks at the tool service, not only in the planner. Redact secrets from model context and traces, constrain outbound network access and record the principal, policy version and capability used for every operation.
Choose the control-plane architecture
Separate the control plane from the execution plane. The control plane owns authentication, task admission, policy, planning requests, workflow definitions, approvals and observability. The execution plane owns durable task dispatch, worker leases, tool adapters, sandboxed code execution and external side effects.
A workflow orchestrator persists transitions. Model calls are bounded activities, not the orchestrator itself. The model can recommend the next action, but the orchestrator validates the proposed transition against the workflow state, deadline, budget and policy before dispatch.
Design typed tool contracts
Each tool contract defines a stable name/version, JSON input schema, output schema, resource scope, required capability, side-effect class, timeout, retry semantics, idempotency requirements and audit fields. Reject unknown fields and ambiguous identifiers; normalize units and timestamps before execution.
Expose narrow operations such as get_order(order_id), check_refund_eligibility(order_id) and create_refund(order_id, amount, idempotency_key), rather than a generic SQL or unrestricted HTTP tool. Tool results include source, timestamp, resource identity and a bounded payload. The model receives only the fields needed for the next decision.
| Contract field | Example | Enforcement owner |
|---|---|---|
| Input schema | order_id: UUID | Tool gateway |
| Capability | orders:read for tenant T | Policy service |
| Side-effect class | read-only / reversible / irreversible | Tool registry |
| Idempotency | task_id + step_id | Adapter + destination |
| Deadline | 3 seconds per call | Executor |
| Output provenance | source + observed_at + resource_id | Adapter |
VISUAL WALKTHROUGH
See the system in motion
Trace a model-proposed tool call through strict schema, tenant scope, policy and a typed adapter.
Receive a structured tool proposal.
Resolve the versioned tool schema.
Validate arguments and resource scope.
Evaluate current policy and side-effect class.
Dispatch only the approved typed operation.
Loading interactive walkthrough…
Use a planner with deterministic validation
The planner converts the goal into a short structured plan: intent, required facts, candidate tools, dependencies, expected outputs and risk class. It should ask a focused clarification when a required resource or decision is ambiguous.
Validate the plan against a server-owned tool registry and workflow policy. Reject unknown tools, out-of-scope resources, excessive step counts, circular dependencies, missing approval gates and attempts to bypass required steps. For repeatable business processes, prefer a deterministic workflow template with model assistance at specific decision points.
Model durable workflow state
Represent a task as accepted, planning, awaiting_input, ready, running, awaiting_approval, compensating, succeeded, failed or cancelled. Each step has queued, leased, running, succeeded, retryable_failed, terminal_failed or skipped state. Persist transition version, attempt, deadline, input hash, output reference and trace ID.
A compare-and-swap transition or optimistic version check prevents two workers from advancing the same step. Use a transactional outbox to publish dispatch events after state commits. Workers acquire expiring leases with fencing tokens; a stale worker cannot overwrite a newer attempt.
VISUAL WALKTHROUGH
See the system in motion
Follow a task through durable state, outbox dispatch, queue delivery, fenced worker lease and checkpoint.
Persist task and initial state.
Commit dispatch intent through the outbox.
Deliver the queue message.
Acquire a fenced worker lease.
Persist the completed step checkpoint.
Loading interactive walkthrough…
Separate conversation context from durable memory
Keep four stores distinct: request context for the current turn; task state for the active workflow; conversation summary for continuity within a thread; and curated long-term memory for explicitly useful, permitted preferences or facts. Do not treat the transcript as an authoritative database.
Every memory item needs owner/tenant scope, provenance, creation time, confidence, retention policy and deletion path. Retrieve only the minimum relevant memory after authorization. Expire ephemeral context, avoid storing secrets or sensitive data, support correction/deletion and prevent one user's memory from entering another tenant's prompt.
VISUAL WALKTHROUGH
See the system in motion
Keep request context, workflow state, conversation summary and curated memory separate and governed.
Load the current conversation.
Authorize tenant and user memory scope.
Retrieve only relevant permitted facts.
Attach provenance and retention metadata.
Build a bounded model context.
Loading interactive walkthrough…
Enforce identity, policy and capabilities
Authenticate the user at the API edge and derive tenant, roles and resource ownership from trusted server-side identity. The planner receives a capability summary, not raw credentials. A capability is short-lived, narrow, audience-bound and scoped to specific resources and actions.
Re-evaluate policy at execution time because roles, object ownership and account state can change while a task is waiting. Apply deny-by-default rules, per-tool quotas, tenant budgets, data classification and separation of duties. Never let the model select a tenant ID or user identity that overrides the authenticated context.
VISUAL WALKTHROUGH
See the system in motion
Enforce server-derived identity, tenant scope and short-lived capabilities at tool execution time.
Authenticate the principal.
Bind tenant and resource scope.
Evaluate current policy.
Mint a narrow short-lived capability.
Reauthorize immediately before the tool call.
Loading interactive walkthrough…
Gate side effects with explicit approval
Classify tools as read-only, reversible or irreversible. Read-only calls may run automatically within scope. Reversible changes may require policy-based approval. Irreversible actions such as refunds, external messages, account closure or production deployment require an explicit human confirmation unless a narrowly defined business policy authorizes automation.
Persist an approval request containing the exact canonical action payload, payload hash, target resource, requester, approver, policy version and expiry. Before execution, verify the hash, approval status, user authority and current resource state again. If the action changes, invalidate approval and request a new decision.
VISUAL WALKTHROUGH
See the system in motion
Bind human approval to the exact action payload and recheck authority before the side effect.
Canonicalize the proposed action.
Persist its immutable payload hash.
Present exact action and evidence to a reviewer.
Verify approval, expiry and current authority.
Execute the approved operation.
Loading interactive walkthrough…
Make retries, idempotency and compensation safe
Assume queues and network calls deliver at least once. Generate a stable operation key from task, step and logical action; persist the request fingerprint and destination result. A repeated key with the same payload returns the prior result; the same key with a different payload is rejected.
Use bounded exponential backoff with jitter only for retryable failures. Respect provider Retry-After, deadlines and per-tenant budgets. Do not retry validation errors or authorization denials. For multi-step side effects, use a saga with explicit compensating actions where possible; when compensation is impossible, create a reconciliation task and expose the uncertain outcome rather than claiming rollback.
VISUAL WALKTHROUGH
See the system in motion
Recover at-least-once work by checking the idempotency ledger and reconciling ambiguous destination outcomes.
Deliver the task at least once.
Acquire lease and attempt number.
Check the operation key and input fingerprint.
Query or execute the external destination.
Persist result or create reconciliation work.
Loading interactive walkthrough…
Handle deadlines, cancellation and backpressure
Every task has an end-to-end deadline and budget for model tokens, tool calls, retries and wall-clock time. Derive child deadlines for each step. Enforce admission control and per-tenant concurrency limits so a slow tool cannot create an unbounded queue.
Cancellation is a durable state transition. Stop scheduling new steps, signal running workers, revoke temporary capabilities and clean up sandboxes. A cancellation cannot pretend an already committed external side effect did not happen; query the destination and reconcile the result.
Support human review and escalation
Persist a review item with reason, exact proposed action, evidence, risk, deadline and owning queue. Notify through an outbox-backed channel and deduplicate notifications. The reviewer UI reads the current task version and cannot approve a stale proposal.
When a reviewer edits the proposal, create a new action version and rerun validation. On approval, resume from the checkpoint after the approval gate; on rejection, cancel or replan according to policy. If the human queue is unavailable, keep the task durably awaiting_approval and communicate that no action has been executed.
Use multi-agent coordination only when justified
Start with one orchestrator and specialized deterministic tools. Split into multiple agents only when work is genuinely parallel, requires distinct bounded contexts or benefits from independent critique. Each sub-agent should receive a narrow task contract and no broader permissions than the parent workflow.
A coordinator owns the shared deadline, budget, cancellation and final consistency check. Persist child-task IDs and outputs. Avoid free-form agent-to-agent loops, shared mutable memory and implicit delegation. Bound fan-out, depth and total model calls; merge results deterministically and reject conflicting claims without evidence.
Trace every decision and side effect
Emit one trace per task with spans for planning, policy decisions, model calls, retrieval, tool execution, approvals, retries and compensation. Record model/provider/version, prompt template hash, tool schema version, policy version, token usage, latency, outcome and resource identity.
Keep audit records append-only and privacy-minimized. Store large tool outputs and model payloads in encrypted, access-controlled object storage with retention limits; traces contain references and hashes. Redact secrets and personal data before telemetry. Dashboards should separate model quality, tool reliability, workflow throughput, safety violations, queue age and cost.
VISUAL WALKTHROUGH
See the system in motion
Trace each decision and tool action, then evaluate trajectory correctness and safety invariants.
Start a task trace.
Record model and policy spans.
Capture tool evidence and timing.
Evaluate the full trajectory.
Alert on invariant or budget violations.
Loading interactive walkthrough…
Evaluate trajectories and roll out safely
Build a task suite with normal, ambiguous, adversarial, partial-failure, duplicate-event, cancellation and approval-expiry cases. Score task success, tool selection, argument correctness, policy adherence, groundedness, side-effect correctness, recovery and user effort. Inspect intermediate actions; a correct final sentence does not compensate for an unauthorized tool call.
Run offline replay against versioned traces, then shadow mode, limited canary and gradual rollout. Use hard safety gates for unauthorized actions, cross-tenant access, duplicate side effects and missing audit records. Roll back model, prompt or tool schema independently where compatibility permits.
Estimate capacity and unit economics
Estimate workflow starts, steps per workflow, model calls per step, tool concurrency and wait time separately. At 20,000 starts/hour and an average of 6 tool steps, the platform sees about 33 starts/s and 200 tool-step/s on average before peak factors. Apply explicit burst assumptions and size queues/workers from measured service times and downstream quotas.
Track cost per completed task, not just cost per model token. Include model input/output tokens, embeddings or retrieval if used, sandbox compute, connector calls, storage, human review and failed/retried work. Enforce per-tenant budgets and stop or ask for confirmation when a workflow approaches its spend or action limit.
Walk through failures and recovery
If the model provider times out, retry within the task deadline using a bounded policy or route to a compatible fallback; never replay completed tools just because the final model response was lost. If a worker crashes after a destination commits, recover by querying the idempotency record or destination before retrying.
If permissions are revoked while awaiting approval, deny execution at the final authorization check. If an external API returns an ambiguous timeout, mark the step outcome_unknown and reconcile before another write. If the event queue is unavailable, the outbox preserves dispatch intent. If the workflow is cancelled during execution, stop future work and reconcile any in-flight side effect.
Complete agent workflow architecture
The complete path is: authenticated user → task API → durable task store/outbox → planner → policy and tool registry → workflow orchestrator → queue/worker lease → typed tool adapter → external system. Approval, memory, tracing, audit and recovery are connected control paths, not optional model plugins.
The model proposes intent and interprets results. Deterministic services own permissions, schema validation, state transitions, deadlines, retries, idempotency, approval and side effects. The final response is generated from persisted workflow state and verified tool evidence.
VISUAL WALKTHROUGH
See the system in motion
Trace the full agent workflow from authenticated goal to bounded planning, durable execution, approval, recovery and verified outcome.
Authenticate user and derive tenant scope.
Generate a bounded structured plan.
Validate tool contracts and capabilities.
Persist task state and dispatch through an outbox.
Execute typed tools under leases and deadlines.
Pause for approval on consequential actions.
Reauthorize and execute idempotently.
Persist evidence, audit and recovery checkpoints.
Evaluate the trajectory and return verified status.
Loading interactive walkthrough…
Diagrams and relationships
The model proposes; deterministic services authorize and execute.
Scroll horizontally on smaller screens, or enlarge the diagram.
Persist every transition and resume from checkpoints.
Scroll horizontally on smaller screens, or enlarge the diagram.
Engineer, Senior and Staff follow-ups
These are teaching lenses. Each answer changes the scope of reasoning; they are not official employer leveling criteria.
Engineer: How do you prevent a duplicate tool call when a worker retries?
Use a stable operation key per logical step, persist its canonical input fingerprint and destination result, and return the stored result for an identical retry. Reject key reuse with changed input.
Engineer: What belongs in a tool schema?
Typed input/output, resource scope, capability, side-effect class, timeout, retry policy, idempotency contract, version and audit metadata. Validate strictly before execution.
Senior: Why not let the LLM orchestrate the whole workflow?
The LLM is nondeterministic and cannot own durable state, authorization or exactly-once side effects. Use it for bounded planning/interpretation and a durable state machine for execution.
Senior: How do you handle an ambiguous timeout after an external write?
Mark outcome_unknown, query by idempotency key or destination reference, reconcile before retrying, and expose uncertainty until the system of record confirms the outcome.
Senior: When would you use multiple agents?
Only for independent bounded tasks or genuinely distinct contexts. Keep a coordinator for budget, deadlines, cancellation and merge; otherwise one orchestrator plus typed tools is simpler and safer.
Staff: How would you design tenant isolation across tools and memory?
Derive tenant from authenticated identity, bind it to task and capability records, enforce resource authorization inside each adapter, partition memory/index access, and test cross-tenant negative cases at every boundary.
Staff: How do you evolve tool contracts without breaking running workflows?
Version schemas and adapters, persist the selected version with each step, maintain compatibility for in-flight tasks, validate migrations and sunset old versions only after task and replay windows expire.
Practice and self-review
Your practice draft
Design a multi-tenant agent workflow platform that can plan and execute customer operations using internal tools. Include a high-risk action requiring human approval, durable recovery, retries, cancellation and a clear policy boundary. Explain why you would or would not use multiple agents.
Saved locally on this device. The worked answer does not score your draft. Existing lesson notes and completion are separate.
Progressive hints
Hint 1
Start by separating the model's proposal from the deterministic execution authority.
Hint 2
Define typed tool contracts, tenant-scoped capabilities and side-effect classes.
Hint 3
Draw the durable task/step state machine and transactional outbox.
Hint 4
Explain idempotency and reconciliation after an ambiguous external timeout.
Hint 5
Bind approval to the exact action payload and reauthorize before execution.
Hint 6
Add tracing, trajectory evaluation, budgets and multi-agent limits.
Reveal worked answer guidance
Use an authenticated Task API to persist a task and outbox event. A planner emits a bounded structured plan. The policy service validates allowed tools, schemas, tenant/resource scope, deadlines and risk. The orchestrator persists step transitions and dispatches typed calls to leased workers. Adapters reauthorize, use scoped capabilities and stable idempotency keys. High-risk actions pause at an approval gate bound to the canonical payload hash; after approval, reauthorize and verify the resource before writing. Persist tool outcomes and audit evidence. On timeout, reconcile by operation key before retry. Cancellation stops future steps and reconciles in-flight writes. Start with one orchestrator; introduce sub-agents only for bounded parallel work, under a coordinator budget and deterministic merge.
Evidence to check in your answer
- Model and executor have separate authority boundaries.
- Tool contracts are typed, versioned and least-privilege.
- Task and step state are durable and recoverable.
- Retries are bounded; side effects are idempotent or reconciled.
- Approval is bound to exact payload and rechecked at execution.
- Cancellation, deadlines and backpressure are explicit.
- Tenant isolation and memory governance are enforced server-side.
- Evaluation covers intermediate actions, safety and task outcomes.
For each item, record Missing, Prompted, Independent or Adaptive evidence. Adaptive means you can revise the reasoning under a changed constraint. Reading completion and readiness are separate.
Sources and verification scope
Original synthetic examples and reference calculations. Sources support narrowly scoped background; they do not establish production performance or independent chapter review.
Primary background links checked during authoring on 2026-10-19. Arithmetic, temporal eligibility, maturity and split reference functions are checked by the bounded local fixture suite. SQL is explanatory, not a migrated production query. No GPU, external dataset, model API or training job is required.