Prerequisites and scoped previews
Probability, statistics and optimization essentials
Use conditional probability, uncertainty, gradients and loss interpretation only where they affect a design.
Scoped prerequisite preview. The complete chapter is planned; this preview does not count as authored coverage.
Labels, delayed outcomes and human annotation
Define label windows, maturity, censoring, annotation policy and disagreement.
Leakage, splits and evaluation contamination
Prevent future information, entity overlap, preprocessing leakage and test-set reuse.
Sampling, imbalance and observation bias
Explain selection/exposure bias, negatives, weighting and what sampling changes.
What, when, why and how
Confusion matrix and prevalence
- what
- TP/FN count actual positives captured/missed; FP/TN count actual negatives flagged/cleared under one decision threshold. Prevalence is actual positives divided by all eligible examples.
- when
- Use these for binary classification and each explicitly defined class in multi-label systems.
- why
- Accuracy can hide rare-event failure, and precision depends on the population’s prevalence. A balanced test set does not directly predict review volume in a rare-event deployment.
- how
- Define which label/action is positive, outcome maturity, inclusion/weighting and threshold convention. Compute counts first, then denominators. Return undefined when a denominator is zero unless a declared reporting convention says otherwise.
Threshold, calibration and capacity
- what
- A threshold converts a score into an action. Calibration concerns how scores correspond to observed event rates. Capacity limits how many actions a system can sustain.
- when
- Use these when choosing review, challenge or block policies and comparing model releases.
- why
- Ordering quality, probability accuracy and feasible workload are different. A numerically higher score is not automatically a reliable probability.
- how
- Choose an operating policy using validation data and costs/guardrails. Evaluate calibration and mature slice quality; validate capacity at the actual population mix. Retest a new calibration/policy bundle on held-out data.
PR/ROC curves and aggregation
- what
- PR and ROC summaries describe threshold behavior with different denominators; average precision and trapezoidal PR area are distinct summaries.
- when
- Use these to compare ranking behavior across thresholds, especially when the action is rare or costs asymmetric.
- why
- A low false-positive rate can still create overwhelming false-positive volume. A curve summary does not guarantee a useful operating point.
- how
- Report curve definition plus a chosen operating-point confusion matrix, workload and slice evidence. State micro/macro/weighted aggregation for multiple classes and handle zero-positive slices explicitly.
Compute the rare-event example
Synthetic population: 1,000,000 eligible, mature examples. Actual positives: 1,000. At the proposed threshold TP=900, FN=100, FP=9,990, TN=989,010. Counts sum to 1,000,000; prevalence is 0.1%. These are assumed counts, not a trained-model benchmark.
Precision = 900/(900+9,990) = 900/10,890 ≈ 8.264%. Recall = 900/(900+100) = 90%. Accuracy = (900+989,010)/1,000,000 = 98.991%. False-positive rate = 9,990/999,000 = 1%. Review workload is 10,890 per window. If the window is one day and review capacity is 2,000/day, overload is 8,890/day.
An all-negative predictor has accuracy 999,000/1,000,000 = 99.9%, recall 0 and undefined precision because it predicts no positives. Its higher accuracy does not make it useful for capturing fraud. We display undefined rather than silently claiming perfect precision.
| Actual / predicted | Positive | Negative |
|---|---|---|
| Actual positive | TP 900 | FN 100 |
| Actual negative | FP 9,990 | TN 989,010 |
Do not invent the next threshold result
Raising the threshold may reduce flagged volume on fixed scores, but these four counts alone do not reveal the new precision or recall. You need scored validation examples or a threshold sweep with a declared tie rule. With capacity 2,000, choose among feasible candidate policies and evaluate the selected policy on held-out mature data.
Top-K admission can bound review volume, but K cases are not guaranteed to meet a precision requirement. Define tie-breaking, queue aging and whether time-sensitive cases expire. Report missed positive risk and no-capacity fallback. A threshold calibrated last month can exceed capacity when traffic or prevalence changes.
If a score is described as 0.8 probability, test whether comparable cases are positive about 80% of the time within uncertainty and relevant slices. A monotonic calibration transform may preserve ranking but change action volume at a fixed numeric threshold; version and validate it as part of deployment.
Show why population matters
At the same assumed 90% true-positive rate and 1% false-positive rate, a balanced 2,000-example sample would contain TP=900 and FP=10, implying precision 900/910 ≈ 98.901%. The rare-event population above gives 8.264%. The conditional error-rate equality is an exercise assumption; real stratification can change other distributions.
Use representative evaluation or explicitly weighted strata and label-inclusion probabilities when justified. Always report the target population, eligibility window and uncertainty. Do not transfer balanced-sample precision into production workload. For a slice with zero actual positives, recall is undefined; retain the slice and explain its sample limitation.
Test formulas independently
The fixture exports confusion counts, precision, recall, accuracy, false-positive rate and workload. Tests compare them with hand-worked fractions, exercise ties and undefined denominators, and reject invalid scores/labels. The arithmetic fixture does not estimate confidence intervals, prove causal lift or train a classifier.
For an online rollout, keep assignment, exposure, label maturity and outcome separate. Shadow predictions assess compatibility/latency and can be reviewed against available labels; unshown alternatives do not reveal counterfactual user outcomes. A controlled exposure experiment needs its own design and analysis discipline.
Reference card: decision-policy evaluation is useful when the same score can trigger different actions. Its input is eligible scores plus labels and an explicit threshold rule; its output is counts and workload. A no-action baseline is the alternative. Its limit is observational/selection bias: correct arithmetic cannot repair invalid labels or exposure.
Diagrams and relationships
Counts refer to one mature population/window. Positive predictions become review work only under this exercise policy.
Scroll horizontally on smaller screens, or enlarge the diagram.
Engineer, Senior and Staff follow-ups
These are teaching lenses. Each answer changes the scope of reasoning; they are not official employer leveling criteria.
Engineer: Why is precision about 8.3% rather than 90%?
Ninety percent is recall: positives captured out of actual positives. Precision divides 900 captured positives by 10,890 flagged examples. The 9,990 false positives dominate the flagged population.
Engineer: What happens with no predicted positives?
TP+FP is zero, so precision is undefined. Display that explicitly; a library may offer a configured zero-division convention, but the chapter must declare it rather than hide the missing denominator.
Senior: Does a 1% false-positive rate satisfy review capacity?
No. One percent of 999,000 negatives is 9,990 cases, before adding 900 true positives. Compare absolute work per time window with staffed capacity and choose a validated feasible policy.
Senior: Can better AUC justify a release?
It is useful evidence, but does not establish the selected operating point, slice behavior, calibration, costs or mature product outcomes. Compare at operational constraints, validate the bundle and use an appropriate rollout/experiment.
Staff: How should a platform standardize metric reports?
Require population/window/label policy, inclusion weighting, threshold and zero-denominator conventions with versioned count evidence. Keep domain-specific harm and capacity ownership. A unified dashboard without denominator contracts encourages invalid comparisons.
Staff: How do you diagnose a fleet-wide precision drop?
Segment changes in prevalence, label policy/maturity, sampling, model/calibration bundle and traffic mix. Assign owners for data, model and action policy, contain harmful actions, and distinguish an observation shift from a model regression before deciding rollback or retraining.
Practice and self-review
Your practice draft
Explain why 98.991% accuracy and 90% recall still produce an infeasible review system. Give the missing evidence needed to select a better threshold.
Saved locally on this device. The worked answer does not score your draft. Existing lesson notes and completion are separate.
Progressive hints
Hint 1
Derive TP+FP rather than using accuracy.
Hint 2
Compare workload and capacity in the same window.
Hint 3
Confusion counts at one threshold do not reconstruct scores at other thresholds.
Reveal worked answer guidance
The policy flags 10,890/day against 2,000 slots, with only 8.264% precision. Obtain eligible scored validation examples, threshold/tie definitions, costs and guardrails; select a feasible operating point and evaluate held-out mature outcomes. Do not claim that an arbitrary higher threshold achieves a desired recall. Retain undefined metrics for empty denominators and test traffic/prevalence sensitivity.
Evidence to check in your answer
- Counts and denominators correct.
- Workload exceeds capacity by 8,890.
- No invented threshold performance.
- Sampling, maturity and calibration caveats are explicit.
For each item, record Missing, Prompted, Independent or Adaptive evidence. Adaptive means you can revise the reasoning under a changed constraint. Reading completion and readiness are separate.
Sources and verification scope
Original synthetic examples and reference calculations. Sources support narrowly scoped background; they do not establish production performance or independent chapter review.
Metrics and scoring ↗ · Metric definitions and implementation conventions; pin versions and specify formula/edge policies.
Primary background links checked during authoring on 2026-10-19. Arithmetic, temporal eligibility, maturity and split reference functions are checked by the bounded local fixture suite. SQL is explanatory, not a migrated production query. No GPU, external dataset, model API or training job is required.