Prerequisites and scoped previews
Candidate retrieval and ANN indexes
Separate eligible-corpus recall from latency/memory and filtering effects.
Ranking, reranking and multi-stage systems
Bound candidate sets, score/rerank and apply policy without confusing retrieval with ranking.
Classification and rare-event evaluation
Define confusion counts, precision/recall, PR/ROC metrics, slices and threshold trade-offs.
What, when, why and how
Relevance recall and precision at K
- what
- Recall@K measures the fraction of judged relevant eligible IDs retrieved; precision@K measures the fraction of the selected K slots that are relevant under a declared shortfall convention.
- when
- Use these when assessing candidate coverage and final list relevance with trustworthy judgments.
- why
- Duplicates, filtered items, unjudged items and no-positive queries can change a denominator without changing the visible list.
- how
- Deduplicate output, apply the same eligibility set to judgments and predictions, and declare binary relevance. Here a short list keeps denominator K for precision and a zero-positive query has undefined recall, reported with a coverage count.
Graded DCG and normalized DCG
- what
- DCG sums a gain for each item discounted by position. NDCG divides by the ideal DCG for the same query/corpus and cutoff.
- when
- Use graded judgments when putting highly relevant items earlier matters more than treating all positives equally.
- why
- Different gain and tie conventions yield different numbers. An ideal list computed only from retrieved items hides missing relevant candidates.
- how
- This course fixture uses gain=2^grade-1, discount=log2(rank+1), nonnegative integer grades and deterministic item-ID score ties. Build ideal ordering from all eligible judgments. Return undefined if ideal DCG is zero.
Query aggregation and incomplete evidence
- what
- Metrics can be averaged equally over defined queries or weighted by traffic/users; neither silently represents the other. Missing judgments are a data limitation.
- when
- Use explicit aggregation for head/tail queries, user cohorts and repeated sessions.
- why
- Ignoring undefined queries without reporting their count can make coverage failure look like improvement; traffic weighting can conceal poor tail behavior.
- how
- Report defined/undefined counts, query population, weights and slices. The toy fixture gives unjudged eligible IDs zero gain as an explicit convention; production assessment should distinguish genuinely judged negatives from unknown judgments.
Calculate NDCG@3 and missing-candidate loss
For one eligible query, grades are A=3, B=2, C=1 and D=0. Returned ranking is B,A,D. Gains at ranks 1,2,3 are 3,7,0. DCG=3+7/log2(3)+0/log2(4)≈7.416508. The ideal A,B,C gives IDCG=7+3/log2(3)+1/2≈9.392789. NDCG≈0.789596.
Relevant means grade>0 in this fixture. Recall@3=2/3, precision@3=2/3, and reciprocal rank of the first relevant result is 1. If retrieval produced only B/A/D, the ranker cannot recover C. Compute candidate relevance recall before interpreting final NDCG.
If a slate policy removes B due to a creator cap, evaluate the displayed constrained order against a clearly declared reference. This chapter’s unconstrained IDCG remains an explicit reference and can penalize the policy; a separately named feasible-policy ideal is possible but must not be silently substituted.
Declare duplicates, ties and undefined values
Returned A,A,B deduplicates to A,B before metrics. For K=3, recall is 2/3 and precision is 2/3; the missing slot is not filled by counting A twice. Equal model scores are broken by item ID in our fixture. A system that averages over tied permutations is another valid convention with a different result.
A query with all grade-zero judgments has IDCG=0 and NDCG is undefined, not perfect. When averaging this query with the worked positive query, report mean≈0.789596 over one defined query and one undefined query. Do not write “two-query NDCG” without that coverage statement.
For K=0 or negative/invalid grades, reject the input contract. K larger than the corpus is allowed: missing positions have zero gain, and recall still uses the actual relevant set. Do not invent extra negatives or relevant items to fill K.
Keep evaluation independent of selection
Use a temporal, entity/duplicate-aware split matching the deployment question. Freeze relevance labels and eligible corpus at the declared cutoff. Fit/tune on training/validation only and hold final test evidence apart from repeated design selection.
For ANN evaluation, separately compare the approximate search to exact vector top K. For ranking evaluation, use relevance evidence independent of vector scores. Defining relevance as “the exact vector top K” only measures search fidelity and says nothing by itself about user satisfaction.
Online product evidence needs eligible assignment/exposure/outcomes and a defensible experiment. Offline NDCG on selected historical impressions cannot prove a causal retention improvement. Shadow ranking can measure compatibility and cost; it does not expose the user to the alternate order.
Use metric libraries with an explicit contract
The standalone Python reference recomputes DCG from a hand-written grade list and compares it with the JavaScript rank trace. It needs no data download or ML library. Every result is synthetic and small-data arithmetic, not a production benchmark.
A library metric may use linear gain or its own tie treatment. Validate convention compatibility before comparing values. Cite metric definitions narrowly and include your gain, discount, ideal-corpus and undefined-query policy beside the result.
Diagrams and relationships
Freeze the eligible corpus and relevance policy before computing rank metrics.
Scroll horizontally on smaller screens, or enlarge the diagram.
Engineer, Senior and Staff follow-ups
These are teaching lenses. Each answer changes the scope of reasoning; they are not official employer leveling criteria.
Engineer: Why is ideal DCG based on A/B/C rather than B/A/D?
The reference covers all eligible judged items, including missed C. Using only the returned list would erase part of retrieval loss and inflate the metric.
Engineer: Does A/A/B contain three relevant hits?
No. One item identity counts once. Deduplicate to A/B, retain the declared K=3 precision denominator and report the shortfall. Repeated IDs cannot improve recall.
Senior: How do you aggregate a zero-positive query?
Under this contract its NDCG/recall are undefined. Report defined-query mean and both counts. Use separate coverage/error analysis; assigning zero or excluding it without disclosure changes the interpretation.
Senior: Why might two libraries disagree on NDCG?
Check gain transformation, log discount, ties, judged corpus, aggregation and zero-IDCG behavior. Our exponential gain is an explicit authored convention; do not assume a library’s default matches it.
Staff: How do you stop benchmark drift across teams?
Version judgment policy, eligibility, split, gain/discount, cutoff and weighting as a shared evaluation contract. Retain exact fixtures and publish changes with migration evidence. A number without its population and convention is not comparable.
Staff: How would you handle a head-query win and tail-query loss?
Report query-weighted and traffic-weighted views with population counts, inspect relevance/judgment coverage and uncertainty, then negotiate product guardrails. An aggregate gain does not establish acceptable tail impact or causal user benefit.
Practice and self-review
Your practice draft
Compute NDCG@3 for B/A/D, then assess A/A/B and an all-zero relevance query. State your gain, tie and aggregation conventions.
Saved locally on this device. The worked answer does not score your draft. Existing lesson notes and completion are separate.
Progressive hints
Hint 1
Use gains 3 and 7 in the returned order, not raw grades.
Hint 2
Use the ideal A/B/C from all eligible judgments.
Hint 3
Keep undefined-query and duplicate counts visible.
Reveal worked answer guidance
NDCG@3 is about .789596 for B/A/D under exponential gains and log2 discount. A/A/B counts two unique positives, with precision/recall 2/3 under the declared convention. An all-zero query has undefined NDCG and recall; report defined mean and coverage rather than replacing it with an unexplained perfect score.
Evidence to check in your answer
- Correct DCG/IDCG formula and cutoff.
- Unique IDs and stable ties.
- Eligible corpus and incomplete judgments explicit.
- Aggregation and causal limitations stated.
For each item, record Missing, Prompted, Independent or Adaptive evidence. Adaptive means you can revise the reasoning under a changed constraint. Reading completion and readiness are separate.
Sources and verification scope
Original synthetic examples and reference calculations. Sources support narrowly scoped background; they do not establish production performance or independent chapter review.
Metrics and scoring ↗ · Metric definitions and implementation conventions; pin versions and specify formula/edge policies.
scikit-learn NDCG API and references ↗ · Metric comparison context; this fixture explicitly chooses exponential gain and deterministic ID ties rather than claiming library-default equivalence.
Primary background links checked during authoring on 2026-10-19. Arithmetic, temporal eligibility, maturity and split reference functions are checked by the bounded local fixture suite. SQL is explanatory, not a migrated production query. No GPU, external dataset, model API or training job is required.