Prerequisites and scoped previews
Probability, statistics and optimization essentials
Use conditional probability, uncertainty, gradients and loss interpretation only where they affect a design.
Scoped prerequisite preview. The complete chapter is planned; this preview does not count as authored coverage.
Labels, delayed outcomes and human annotation
Define label windows, maturity, censoring, annotation policy and disagreement.
Leakage, splits and evaluation contamination
Prevent future information, entity overlap, preprocessing leakage and test-set reuse.
What, when, why and how
Population versus observed sample
- what
- The target population is the set of eligible decisions you want to describe. A sample includes a subset under an observation/selection mechanism.
- when
- Use this when training/evaluation data comes from reports, model-selected review, clicked items or deliberately sampled classes.
- why
- A model evaluated only on reviewed cases can look strong while missing errors outside the queue. A balanced sample generally does not preserve deployment prevalence.
- how
- Record eligibility, time window, inclusion rule/probability where known, label provenance and completeness. Separate training convenience samples from a defensible evaluation sample and name groups with no support.
Inverse inclusion weighting
- what
- A known inclusion probability q can attach weight 1/q to a sampled record under a valid sampling design. Weighted counts estimate population totals; they are not extra independent observations.
- when
- Use this for a declared probability sample or retrospective stratified audit where inclusion probabilities and target population are known.
- why
- Ignoring sampling changes denominators; pretending weights solve missing coverage conceals unobserved populations. Large weights can make estimates noisy.
- how
- Compute weighted TP/FP/FN/TN with the same eligible label/threshold contract. Report raw sample size, inclusion assumptions, uncertainty and any excluded group. Do not infer propensities from arbitrary model confidence.
Unobserved is different from negative
- what
- An unreviewed, unexposed or immature case lacks the evidence required for the target label. Negative sampling is a modeling choice, not automatically ground truth.
- when
- Use this for recommendations, moderation, abuse detection and delayed outcomes.
- why
- Treating every unselected record as negative trains the model to reproduce the selector and can hide harmful or legitimate content outside its support.
- how
- Maintain pending/unknown status, audit permitted random or stratified cases outside the operational selector, and keep review/appeal labels separate from population evaluation. Explain where identification remains impossible.
Work a balanced sample and its population weights
Synthetic retrospective audit: a known frame has 100 positive and 9,900 negative items. The sample includes all 100 positives and 100 randomly selected negatives, so q_positive=1 and q_negative=100/9,900=1/99. These class labels are known in this pedagogical frame; an operational sampler cannot identify true classes before annotation without additional assumptions.
On the 200 sampled rows, counts are TP=90, FN=10, FP=10, TN=90. Unweighted precision=90/100=90%, recall=90/100=90%. Weight positives by 1 and negatives by 99: estimated population counts TP=90, FN=10, FP=990, TN=8,910. Weighted precision=90/1,080≈8.333%, recall=90%.
This calculation assumes random sampling within the declared classes and valid labels. It illustrates denominator reconstruction, not a guarantee that a small audit yields accurate population error rates. The 990 estimated false positives are not 990 independent observed examples.
Inspect support and label mechanisms
If only reported items are reviewed, the probability of observation may depend on exposure, language, controversy and coordinated reports. An unreported language group may have no adjudicated examples. No finite inverse weight creates information for a group with zero inclusion probability.
Reserve a separate permitted audit process with known selection design where feasible. Stratify on observable cohorts and record how examples are chosen. Active-learning uncertainty samples may help training, while evaluation still needs its own population contract.
Appeals overrepresent people who were restricted and chose to appeal. Their overturn rate describes completed appeals, not all restrictions or all content. Report cases awaiting appeal/review and the time horizon.
Choose a sampling strategy for a question
A simple random sample is conceptually clear but can observe too few rare positives. Stratified sampling can target underrepresented cohorts, provided inclusion/weighting and uncertainty are documented. A case-control sample can support some modeling/evaluation tasks but does not directly preserve action precision or workload.
Downsampling for training, reweighting the loss, calibration and changing the decision threshold are different operations. Evaluate the final score/action policy on appropriately representative held-out evidence; do not assume one class-weight parameter corrects every shift.
Diagnose an apparent win
A candidate improves review-queue precision from 80% to 90%, but a new selector stopped sending difficult cases. Keep the selector version and audit unseen cases before crediting the model. A clean metric calculation can still answer the wrong population question.
A policy change can redefine which examples count as positive. Partition labels by compatible policy, snapshot their availability and compare like with like. Do not describe every changed denominator as model drift.
Diagrams and relationships
Training samples and independent audits have different purposes; neither invents labels outside observed support.
Scroll horizontally on smaller screens, or enlarge the diagram.
Engineer, Senior and Staff follow-ups
These are teaching lenses. Each answer changes the scope of reasoning; they are not official employer leveling criteria.
Engineer: Why does weighted precision fall from 90% to 8.333%?
Negative examples are heavily undersampled. Each sampled negative represents 99 target-frame negatives, so ten false positives estimate 990. TP remains 90; the flagged denominator becomes 1,080.
Engineer: Can I set q=0 and use an infinite weight?
No. Zero support means the sample cannot identify that group’s outcome under this design. Obtain appropriate evidence or state the limitation; an infinite numeric value is not a correction.
Senior: Can the completed-appeal overturn rate measure all false restrictions?
No. It conditions on restriction, appeal submission and completion. Nonappealers and unresolved cases may differ. Track appeal coverage/delay and use separate audit evidence for broader error claims.
Senior: How do you compare two review selectors?
Freeze the population/label policy and inspect selection/inclusion changes, not just queue metrics. Use common permitted audit evidence, report overlap and unsupported groups, and account for delayed labels and correlated samples.
Staff: Who owns a shared sampling contract?
Data/evaluation owners define population, inclusion and label revision semantics with domain/policy owners. Consumers preserve those fields through snapshots and reports. Changes require versioning and migration evidence rather than a silent dashboard update.
Staff: How do you prioritize audit spending?
Separate training utility from population error estimation. Allocate audit coverage to consequential, weakly observed cohorts under an explicit design, report uncertainty and trade-offs, and avoid promising that a fixed budget guarantees rare-harm detection.
Practice and self-review
Your practice draft
Reconstruct weighted counts from the balanced audit. Explain why these weights cannot repair a group with no labels and why appeal outcomes need a separate denominator.
Saved locally on this device. The worked answer does not score your draft. Existing lesson notes and completion are separate.
Progressive hints
Hint 1
Start with the target-frame size and each inclusion probability.
Hint 2
Apply weights to counts before taking ratios.
Hint 3
Selection into review or appeal is not random by default.
Reveal worked answer guidance
Use weights 1 and 99 to obtain 90/10/990/8,910 for TP/FN/FP/TN. Precision becomes 8.333%, recall 90%. This depends on a valid probability design and labels; it does not recover a zero-support group or make completed appeals representative of all restricted items.
Evidence to check in your answer
- Population and sample are distinct.
- Weights and raw counts explicit.
- Unsupported/unknown cases retained.
- Evaluation and training sampling separated.
For each item, record Missing, Prompted, Independent or Adaptive evidence. Adaptive means you can revise the reasoning under a changed constraint. Reading completion and readiness are separate.
Sources and verification scope
Original synthetic examples and reference calculations. Sources support narrowly scoped background; they do not establish production performance or independent chapter review.
Metrics and scoring ↗ · Metric definitions and implementation conventions; pin versions and specify formula/edge policies.
Google ML evaluating for bias ↗ · General population/slice-evaluation context, not a fairness certificate or a specific moderation policy.
Primary background links checked during authoring on 2026-10-19. Arithmetic, temporal eligibility, maturity and split reference functions are checked by the bounded local fixture suite. SQL is explanatory, not a migrated production query. No GPU, external dataset, model API or training job is required.