Prerequisites and scoped previews
Linear models, trees and boosting trade-offs
Compare tabular model families by quality, calibration, serving cost and maintainability.
Scoped prerequisite preview. The complete chapter is planned; this preview does not count as authored coverage.
Classification and rare-event evaluation
Define confusion counts, precision/recall, PR/ROC metrics, slices and threshold trade-offs.
What, when, why and how
Calibration and reliability bins
- what
- A probabilistic score is calibrated when comparable predicted probabilities match observed positive frequencies under the evaluated population and label definition.
- when
- Use this when a probability drives cost-sensitive decisions, human prioritization or stable policy thresholds across releases.
- why
- A good ranking can still give misleading probabilities. Calibration on a selected, outdated or leaked dataset may not transfer to deployment.
- how
- Use held-out predictions with mature labels; inspect average prediction, observed rate and sample count by bin/slice. Fit any calibrator without reusing the model’s training outcomes or the final test as tuning data.
Threshold and action policy
- what
- A threshold converts a score into a decision. An allow/review/restrict policy may use two thresholds plus explicit missing-signal and overload rules.
- when
- Use this when actions have asymmetric consequences or a queue can process only bounded work.
- why
- Changing score calibration or prevalence can change action volume at the same numeric threshold. A model metric alone does not choose the operational policy.
- how
- Choose candidate thresholds on validation evidence, compare counts, action harms and review arrivals to capacity, then assess the selected policy on held-out mature data. Declare equality boundaries and a shortfall/overflow action.
Uncertainty, cost and capacity
- what
- Sampling uncertainty, model probability and operational capacity are distinct quantities. Expected-cost reasoning requires suitable probabilities and stated cost assumptions.
- when
- Use these when choosing among feasible policies and deciding where human evidence is needed.
- why
- A probability of .8 is not an 80% confidence interval, and review of the most uncertain cases is not automatically the highest-value allocation.
- how
- For a binary calibrated decision with false-restrict cost C_FP and missed-harm cost C_FN, compare C_FP*(1-p) with C_FN*p under the simplified assumptions. Add review delay/error/cost and constraints for a real review policy; do not reuse the binary formula as a universal review threshold.
Compute a reliability diagnostic
Four independent synthetic held-out outcomes use probabilities [0.1,0.2,0.8,0.9] and labels [0,1,1,1]. With bins [0,.5) and [.5,1], the low bin has mean prediction .15 and observed rate .5; the high bin has mean prediction .85 and observed rate 1. Counts are two each.
The illustrative count-weighted absolute bin gap is .5*|.15-.5|+.5*|.85-1|=.25. Binary Brier mean squared error is (.01+.64+.04+.01)/4=.175. Bin choices and four observations are inadequate for a population calibration claim; Brier also depends on discrimination/uncertainty, not calibration alone.
An empty bin is reported with zero count and undefined means rather than a perfect score. A calibrator can be monotonic and preserve score ordering while changing how many items cross a fixed threshold. Fit and version it as part of the release bundle.
Work threshold boundaries and review capacity
Synthetic rule: score < .40 proposes allow; .40 <= score < .90 proposes review; score >= .90 proposes restrict. These are policy assumptions, not validated safe thresholds. Non-finite/missing scores use a separately declared missing-signal policy and cannot be treated as zero.
In a six-item fixture, scores A=.95, B=.90, C=.89, D=.60, E=.40, F=.10 produce restrict A/B, review C/D/E and allow F. With two review slots, admit C/D by score and stable ID, and send E to the explicit pending-overflow state. Pending is not a negative label or a completed review.
For a one-hour window with 120 new review cases and capacity 100 completions/hour, backlog increases by 20 if no other flows occur. Twelve minutes of handle time per case implies five cases per productive reviewer-hour; 20 reviewers give 100 only under the declared productive-time assumptions. Reserve staffing for peaks, appeals, audits and complex cases.
Compare a simpler policy and calibrators
A transparent rule can be preferable when calibration evidence is weak, data is small or response timing is more important than model complexity. Sigmoid and isotonic calibration are candidate mappings with different data/shape assumptions; neither is a universal strongest method. This fixture computes diagnostics and trains neither.
The simplified binary expected-cost cutoff is C_FP/(C_FP+C_FN). With C_FP=9 and C_FN=1 it is .9; this ignores review, exposure, delays, score errors and constraints. The example derives a hypothetical policy under assumptions rather than declaring a safe moderation threshold.
Diagnose action-volume surprises
A new calibrator improves a bin diagnostic but doubles review arrivals at the unchanged numeric threshold. Compare full policies at feasible workload, recalibrate thresholds on permitted validation data and version the score/calibrator/policy bundle. Do not blame queue workers for an unbounded decision policy.
A policy-label revision or source outage can change measured rates without model degradation. Preserve label/score versions and mature cohorts, inspect counts and uncertainty, and distinguish immediate action/load containment from slower quality assessment.
Diagrams and relationships
The calibrator and threshold are part of a release; review admission is bounded and overflow stays explicit.
Scroll horizontally on smaller screens, or enlarge the diagram.
Engineer, Senior and Staff follow-ups
These are teaching lenses. Each answer changes the scope of reasoning; they are not official employer leveling criteria.
Engineer: Why is the .25 bin gap not a certification?
It depends on the selected bins, population, labels and four noisy observations. Report counts and uncertainty, inspect slices and use appropriate held-out evidence; no single scalar proves calibration or fairness.
Engineer: Where do scores exactly .40 and .90 go?
Under this explicit fixture .40 proposes review and .90 proposes restrict. Boundary conventions must match serving and evaluation; changing > to >= can change actions for tied scores.
Senior: Can you infer the workload at a new threshold from one confusion matrix?
No. You need eligible scored validation examples or a correctly defined sweep. A single operating-point matrix cannot reconstruct the underlying score distribution or new precision/recall.
Senior: How do you stop review load from silently exceeding staff?
Set bounded admission, classify overflow explicitly, record arrivals/completions/backlog/age and assign a capacity owner. Validate the final policy at the workload window and define severity-aware containment, not an implicit allow/drop.
Staff: Who can change a threshold after a score migration?
Model, policy, product and review-operation owners must agree on quality/harm/capacity evidence and change control. Validate and release the compatible bundle with rollback rather than independently editing a magic number.
Staff: What evidence is needed for a shared probability service?
Publish target/policy/population definitions, calibration scope and limitations, versioned scores and monitoring contracts. Consumers validate domain-specific costs/actions; one shared probability does not imply identical thresholds or review policy.
Practice and self-review
Your practice draft
Compute the calibration diagnostic and six-item actions. Under two review slots, account for every item and explain why this does not validate safety.
Saved locally on this device. The worked answer does not score your draft. Existing lesson notes and completion are separate.
Progressive hints
Hint 1
Bin means and observed rates use the same eligible rows.
Hint 2
Separate proposed review from admitted, completed and overflow.
Hint 3
Costs, thresholds and capacity are synthetic inputs.
Reveal worked answer guidance
The bin gap is .25 and Brier .175 under the stated convention. A/B are restricted, C/D admitted to review, E pending-overflow and F allowed. All six remain accounted for; no safe production threshold or population claim follows from the fixture.
Evidence to check in your answer
- Counts, equality boundaries and empty-bin policy.
- Calibration, ranking and uncertainty distinguished.
- Feasible review and overflow accounting.
- Held-out and versioned release evidence.
For each item, record Missing, Prompted, Independent or Adaptive evidence. Adaptive means you can revise the reasoning under a changed constraint. Reading completion and readiness are separate.
Sources and verification scope
Original synthetic examples and reference calculations. Sources support narrowly scoped background; they do not establish production performance or independent chapter review.
scikit-learn probability calibration ↗ · Reliability/held-out calibration context; original bin and Brier arithmetic. Brier is not a calibration-only measure; no calibrator is trained.
Google ML thresholds and confusion matrices ↗ · Score/threshold/count context. All action thresholds, sample populations and review budgets are synthetic teaching assumptions.
Primary background links checked during authoring on 2026-10-19. Arithmetic, temporal eligibility, maturity and split reference functions are checked by the bounded local fixture suite. SQL is explanatory, not a migrated production query. No GPU, external dataset, model API or training job is required.