Prerequisites and scoped previews
Labels, delayed outcomes and human annotation
Define label windows, maturity, censoring, annotation policy and disagreement.
Calibration, uncertainty and decision thresholds
Distinguish score quality from calibrated probabilities, uncertainty and action capacity.
Slices, fairness and harm-aware decisions
Define relevant slices and competing error costs without claiming one metric certifies fairness.
Observability and operational response
Separate system, data, model, product and decision-policy signals with owners and runbooks.
Scoped prerequisite preview. The complete chapter is planned; this preview does not count as authored coverage.
What, when, why and how
Review case lifecycle
- what
- A review case is a durable unit with content revision, decision/action/policy identity, state, priority, ownership and a version for conditional changes.
- when
- Use it when human evidence or adjudication affects content actions or labels.
- why
- A retry can create duplicate cases; a delayed reviewer can act on an obsolete edit or policy. A queue acknowledgement does not mean a review decision was applied.
- how
- Define proposed → queued → claimed → resolved or escalated states, conditional claim/lease expiry, idempotent completion and result validation. Require compatible content/policy references and preserve each state transition in audit evidence.
Capacity, prioritization and overload
- what
- Review arrivals, staffed completions, backlog and oldest-case age are separate quantities. Admission and overflow policy must account for every case.
- when
- Use this for classifier uncertainty queues, urgent reports, appeals, audits and outages.
- why
- An unbounded queue cannot create human throughput. Silent dropping or treating overflow as allow changes safety policy without evidence.
- how
- Separate capacity pools and priorities, reserve urgent/appeal/audit work explicitly, cap admission and concurrency, and record pending overflow and expiry/escalation. Validate per-window units and complexity/language assumptions; high score alone need not mean highest severity.
Appeals and adversarial feedback
- what
- An appeal is a request to review an action, with provenance, timeliness, policy/revision context and a possible corrective action. Reports/appeals may be selective or coordinated.
- when
- Use this when users can challenge restrictions and when attackers adapt to action rules.
- why
- An appeal is not automatically proof the model was wrong, and reports are not unambiguous positives. A stale appeal could restore content changed after its original restriction.
- how
- Review against an explicit compatible policy, distinguish policy change from prior error, append corrected labels/actions, and invalidate downstream caches/decisions as supported. Protect reviewer access and audit integrity, rate-limit abusive inputs without assuming all repeated reports are invalid.
Work admission, pending overflow and backlog
A six-item score policy proposes three reviews C/D/E but the window has two available admission slots. C/D enter queued and E remains pending-overflow, each with a durable reason and owner. All six items have a known decision state; the pending item is neither silently lost nor labeled benign.
Suppose an operational hour begins with backlog 30, receives 120 review cases and completes 100. Ending backlog is 50 absent other arrivals/cancellations. Repeating this for three hours adds 60, leaving 90. Actual delay depends on scheduling, service-time variation and staffing, so average-rate arithmetic alone does not prove a p99 review SLA.
If twenty reviewers each have 60 productive minutes and 12-minute mean handling, nominal capacity is 100 cases/hour. Breaks, complex cases, language specialization, appeals and audit work reduce capacity available for the main queue; productive time must be an explicit assumption.
Reject a late result with a conditional state change
Case R1 is claimed for content revision 7, policy P1 and case version 3. The content is edited to revision 8 before completion. A completion expecting revision 7 is stale; do not apply its allow/restrict outcome to revision 8. Keep the result as superseded evidence and requeue/recompute under the chosen policy.
Similarly, two reviewers completing the same case must not both change the action. Use an expected case/action version and idempotency key atomically with the authoritative transition. A replay of the same accepted result may return the existing outcome; a conflicting payload under the same identity is rejected. The fixture uses optimistic version rejection, not a distributed lease implementation.
Make an overturn append-only and explainable
An appeal references the original restriction action and the reviewed revision. If completed under compatible policy with sufficient authority, it can create a new restore/corrective action linked to the original. Do not erase the restriction row or reinterpret its old score as if it never happened.
If content or policy changed, route to explicit adjudication instead of applying the old appeal outcome automatically. Corrected labels create new revisions/snapshots under availability rules. Completed-appeal overturn rate describes selected completed appeals; it is not the population false-restriction rate.
Contain outages and protect the review process
Separate ordinary model uncertainty, missing modalities and a policy-forbidden content type. Each needs an explicitly owned action under uncertainty; a universal fail-open or fail-closed rule is not justified by queue mathematics. Pending hold/restricted visibility may be appropriate in this teaching exercise but has user-impact costs and expiry/escalation.
An attacker may adapt text/images or coordinate reports. Describe adaptation and authorized evaluation without providing harmful payloads or exposing bypass thresholds in public reason messages. Use benign synthetic metadata in fixtures; do not require reviewers or learners to encounter actual harmful content.
A managed review platform reduces workflow implementation but still needs policy, authority, retention, quality and portability contracts. Building a custom queue can offer control at substantial operational cost; neither removes accountability for wrong or delayed actions.
Diagrams and relationships
A newer content/policy/action revision makes the old result superseded; corrective actions retain history.
Scroll horizontally on smaller screens, or enlarge the diagram.
Engineer, Senior and Staff follow-ups
These are teaching lenses. Each answer changes the scope of reasoning; they are not official employer leveling criteria.
Engineer: Why does E remain pending instead of becoming allowed?
Capacity is not a label or permission to change policy. The declared overload state preserves accountability and triggers an owned follow-up. Another fallback is possible only with an explicit harm/cost decision and evidence.
Engineer: Can a case for revision 7 change revision 8?
Not automatically. The expected revision/policy/version check rejects it as stale. Record the superseded evidence and make a fresh or explicitly adjudicated decision for the current content.
Senior: How do you prevent duplicate reviewer actions?
Use stable case/result identities and an expected version with the effect in one authoritative transition. Return an accepted prior result for an identical retry where supported; reject conflicting results. Queue delivery acknowledgement is separate from action completion.
Senior: How do appeals affect training labels?
A completed adjudication may create a new label revision under a declared policy and available-at time. It does not rewrite old snapshots, and its selected appeal population cannot be treated as a random evaluation sample.
Staff: How do you allocate urgent, audit and appeal capacity?
Name capacity/priority owners and explicit reservations, measure arrivals/handling/age by class, and validate trade-offs under peaks. Shared staff cannot be counted at full capacity in each queue. Record which harms and delays are accepted and when to escalate.
Staff: How do you migrate a policy without corrupting open cases?
Version guidelines, classifiers/calibrators/actions and review context, choose compatible carry-forward rules, and supersede or re-adjudicate incompatible cases. Retain evidence and corrective-action lineage, stage the transition, and monitor load and mature outcomes separately.
Practice and self-review
Your practice draft
Trace R1 through a content edit and a late review result. Show the two-slot admission, backlog arithmetic and how an appeal overturn changes state without erasing history.
Saved locally on this device. The worked answer does not score your draft. Existing lesson notes and completion are separate.
Progressive hints
Hint 1
Check content, policy, action and case versions at completion.
Hint 2
Account for pending overflow and per-window capacity.
Hint 3
Append a corrective action and a label revision; keep selected-population limitations.
Reveal worked answer guidance
Reject the revision-7 result for current revision 8, retaining superseded evidence. Admit two proposed cases and track the third as pending. Backlog 30+120-100 is 50 after one hour. A compatible adjudicated appeal appends a restore action and later label revision, while an incompatible context requires explicit re-adjudication.
Evidence to check in your answer
- Conditional/versioned state transitions.
- No silent overflow or duplicate effects.
- Append-only action/label lineage.
- Human quality, selection and ownership explicit.
For each item, record Missing, Prompted, Independent or Adaptive evidence. Adaptive means you can revise the reasoning under a changed constraint. Reading completion and readiness are separate.
Sources and verification scope
Original synthetic examples and reference calculations. Sources support narrowly scoped background; they do not establish production performance or independent chapter review.
NIST AI Risk Management Framework ↗ · Risk vocabulary and governance; not a certification or blanket legal-compliance guarantee.
Google ML evaluating for bias ↗ · General population/slice-evaluation context, not a fairness certificate or a specific moderation policy.
Primary background links checked during authoring on 2026-10-19. Arithmetic, temporal eligibility, maturity and split reference functions are checked by the bounded local fixture suite. SQL is explanatory, not a migrated production query. No GPU, external dataset, model API or training job is required.