Metric Sheet
The artefact written before the first prompt: one metric per family (accuracy, latency, cost, safety, security), each carrying a definition, a measurement method, a baseline, and a target. Anthropic's guidance puts the order plainly — define success criteria, then design evaluations to measure against them.
Exam context: Options that defer metrics until after launch "when real data exists" invert the discipline: the bar becomes whatever the system happens to achieve. A single aggregate accuracy number is the other planted answer.
See also: 4.1 Defining Evaluation Metrics
Guard Metric
What a proposed change must not damage, as distinct from the success metric it is meant to improve. Guards are usually cost per task, tail latency, and the safety rates. A variant that lifts the headline while breaching a guard has failed.
Exam context: Every experiment in 4.3 travels with both kinds, and production alerts in 4.6 fire on the guards. A change proposal that names only its win condition is incomplete.
See also: 4.1 Defining Evaluation Metrics
Tail Percentile
The p95 or p99 of a latency distribution, where the pain the mean conceals actually lives. Standard SLO practice puts targets and alert thresholds here, because the users experiencing the tail are the ones who churn.
Exam context: Mean latency offered as the SLA metric or the alert threshold is a recurring distractor. The professional answer states the percentile and which latency the consumer feels.
See also: 4.1 Defining Evaluation Metrics
Time to First Token (TTFT)
The interval between sending the prompt and the first token of the response arriving, as distinct from baseline latency, which covers processing the prompt and generating the whole response. Anthropic's latency documentation treats them as separate measurements.
Exam context: Interactive surfaces feel TTFT; pipelines feel total completion time. They are different promises, and streaming moves only the first of them.
See also: 4.1 Defining Evaluation Metrics
Proxy Trap
The standing failure of any stand-in metric: response length as a completeness proxy, judge score as a satisfaction proxy. Optimised hard enough, a proxy stops tracking the outcome it was chosen to represent, and the dashboard keeps improving while the outcome does not.
Exam context: The discipline is a periodic audit of each proxy against the real outcome, with retirement when they decouple. Every metric sheet contains at least one proxy waiting to be gamed.
See also: 4.1 Defining Evaluation Metrics
Case Populations
The four groups that make an eval dataset risk coverage rather than a sample: representative cases mirroring the production distribution, edge cases at the boundaries where behaviour changes, adversarial cases probing safety and security, and regression cases minted from production failures.
Exam context: When a stem pairs recurring production incidents with a green suite, the missing populations are the answer. Anthropic names the forgotten edges: irrelevant data, overly long inputs, poor user input, ambiguous cases.
See also: 4.2 Designing Evaluation Datasets
Golden Set
The reference evaluation dataset, trusted because of three properties: versioned, so scores compare across time; held out of the prompt, so passing proves capability rather than recall; and refreshed from production, so it ages with the traffic instead of away from it.
Exam context: A static launch set keeps scoring green while real-world quality falls, because the distribution it certifies no longer arrives. Refresh cadence is a design decision, not an accident.
See also: 4.2 Designing Evaluation Datasets
Test-Set Leakage
The quiet invalidation of an eval suite: a case that also appears in the prompt as a few-shot example stops being a test and becomes a memory. Held out means held out of the prompt, not only out of the training data.
Exam context: Leakage inflates every score the suite reports, so the pass rate rises while capability does not. Near-duplicates leak too, which is why the check belongs in the harness rather than in a visual skim.
See also: 4.2 Designing Evaluation Datasets
Judge Calibration
Validating an LLM judge against human labels on a sample before its scores are allowed to gate anything, and defining in advance what level of agreement is good enough. The judge is a model with its own failure modes, scoring a system it resembles.
Exam context: An uncalibrated judge produces confident numbers with no evidential weight, however precise the dashboard looks. Anthropic also calls it best practice to use a different model as judge than the one that generated the output.
See also: 4.2 Designing Evaluation Datasets
Statistical Sizing
Sizing and reporting an eval with the machinery non-determinism demands: standard errors of the mean and confidence intervals rather than bare scores, clustered standard errors when questions are related, resampled outputs averaged per question, and a power analysis run before the sample size is fixed.
Exam context: Anthropic's statistical evals research notes that clustered standard errors on popular evals can be over three times the naive ones. Growing a dataset to a round number is aesthetics; sizing follows risk and variance.
See also: 4.2 Designing Evaluation Datasets
Gate-Then-Validate
The two-stage evidence rule for shipping changes: the offline eval suite gates every candidate, and an online A/B against the live baseline validates the survivors. The set approximates production; users are production.
Exam context: A passing suite offered as licence for a full user-facing rollout is the planted answer. Passing the gate is necessary, never sufficient, for a behaviour change users will feel.
See also: 4.3 A/B Testing and Iterative Improvement
One-Variable Rule
Changing prompt or model or retrieval in a given experiment, never two at once, because attribution requires isolation. Land the first change, establish the new baseline, then test the second against it.
Exam context: "Since we're touching it anyway" is the exam's favourite bundling cue. A combined win cannot be credited and a combined loss cannot be unpicked, so the next decision inherits a superstition.
See also: 4.3 A/B Testing and Iterative Improvement
Paired-Difference Test
Comparing two variants on the same questions and analysing the per-question differences, which removes the variance that question difficulty contributes. Anthropic's evals research recommends it for model comparisons as a free variance reduction, since frontier models correlate roughly 0.3 to 0.7 on question scores.
Exam context: The named machinery behind rejecting the eyeballed winner. When a stem reports a clear improvement on a handful of cases, the keyed answer reaches for sample size and significance.
See also: 4.3 A/B Testing and Iterative Improvement
Staged Exposure
Releasing a risky change in stages rather than all at once: shadow runs the new variant on real traffic with outputs compared offline and no user exposure, then a canary hands the variant to a small live slice, with guards watched and rollback armed. Standard deployment practice, taught here at professional register.
Exam context: A rollback plan is not a substitute — shipping to everyone with a revert path still exposes everyone to an unvalidated change. Model-version migrations default to shadow first, because behaviour can shift broadly and silently.
See also: 4.3 A/B Testing and Iterative Improvement
Re-Baselining
Re-measuring every downstream metric after a legitimate floor change, such as a model-version migration or a corpus refresh that genuinely improved retrieval. Without it, later comparisons run against a baseline that no longer exists.
Exam context: Skipping it makes trends and regressions indistinguishable from the migration's own shift. It is a decision with an owner and evidence, not an automatic step.
See also: 4.3 A/B Testing and Iterative Improvement
Diagnostic Order
The sequence this objective examines: trace and reproduce, isolate the stage the evidence implicates, classify against the failure signatures, run the cheapest decisive test, fix at the failing layer, and mint regression cases so the diagnosis outlives the incident.
Exam context: Options that jump to a fix without a diagnosis are distractors even when the fix might incidentally help. One bad output proves nothing in a non-deterministic system; confirm a pattern across cases.
See also: 4.4 Diagnosing System Issues
Prompt Failure
The class where the instructions are the problem. Signature: format drift, violated output contracts, rules applied inconsistently, and behaviour that changes with phrasing on the same inputs, with failures clustering wherever the instructions underdetermine something.
Exam context: The decisive test is a clarified, structured prompt with everything else held constant. Fix at the cheapest layer of the technique ladder — a larger model obeys ambiguity more expensively.
See also: 4.4 Diagnosing System Issues
Grounding Failure
Hallucination caused upstream: the context fed to the model was wrong, stale, or missing, and the model answered faithfully from a bad basis. The cue is wrongness correlating with a corpus event (a refresh, a re-import, a migration) or with degraded retrieval-quality signals.
Exam context: The guide's own Domain 4 sample is this case, keyed to investigating retrieval and indexing first. No instruction restores passages the retrieval step no longer supplies, which is why prompt-patching is the planted answer.
See also: 4.4 Diagnosing System Issues
Generation Failure
Hallucination with correct context: the trace shows the right passages present and the answer contradicts or exceeds them. The fix is grounding discipline — allow "I don't know", quote word-for-word first on long documents, require citations and retract unsupported claims, restrict answers to the provided text, verify with chain-of-thought or best-of-N.
Exam context: The grounding/generation split decides where the fix goes: pipeline versus prompt-level discipline. Collapsing both into one word guarantees fixing the wrong layer half the time.
See also: 4.4 Diagnosing System Issues
Model Mismatch
The class where the task exceeds the tier. Signature: uniform mediocrity on the hard cases across prompt variants, persisting with a clean prompt and verified-correct context, with quality tracking task difficulty rather than phrasing.
Exam context: The decisive test is running the failing set on a stronger tier; clearing the bar makes the upgrade a diagnosed, eval-gated decision, and failing to clear it sends the search back upstream. Mismatch runs both directions — an over-provisioned tier is the same diagnosis with the opposite prescription.
See also: 4.4 Diagnosing System Issues
Cheapest Decisive Test
The lowest-cost experiment that can confirm or eliminate a hypothesis, run with everything else held constant. Reading existing traces is cheaper than any experiment, so the evidence already in hand gets exhausted first.
Exam context: Classification is only half the skill; items follow up by asking which test confirms the diagnosis. A test that changes two things at once decides nothing.
See also: 4.4 Diagnosing System Issues
Faithfulness
The measured rate at which output stays within what its supplied context supports. Turning fabrication from an anecdote into a number is what makes a generation-failure fix provable rather than hopeful.
Exam context: The metric that closes the loop on grounding discipline. Without it, "we told the model not to invent things" is an intention with no evidence attached.
See also: 4.4 Diagnosing System Issues
Bias Disaggregation
Splitting eval results across cohorts and content types instead of reporting one aggregate. A system accurate on average and systematically wrong for one population is failing, and only disaggregated reporting shows it.
Exam context: Detection is Domain 4's job; the governance response (policy, accountability, disclosure) belongs to Domain 5. Items that blur the boundary are testing whether you know which half you are being asked about.
See also: 4.4 Diagnosing System Issues
Dominant Term
The largest contributor to a workload's cost or latency, identified from traces and usage data before any lever is chosen. If most of the spend is input tokens on one stage, tuning output tokens elsewhere is activity rather than improvement.
Exam context: Optimising the most visible line while the dominant term sits elsewhere is this objective's quietest anti-pattern. Anthropic's Usage and Cost Admin API exists to name the term programmatically.
See also: 4.5 Optimising Tokens, Latency, and Cost
Eval Re-Gate
Re-running the quality suite on the optimised path before an optimisation ships. The bar from the metric sheet is a constraint, not a preference, so "cheaper and still passes" is the only win condition.
Exam context: An optimisation that skips its re-gate is a regression with good intentions — the saving is provable and the quality loss is unmeasured. Blind downsizing is the canonical example.
See also: 4.5 Optimising Tokens, Latency, and Cost
Cost-Performance Frontier
The framing that every optimisation moves along a trade-off curve rather than repealing it: batch spends turnaround time, downsizing spends capability headroom, streaming spends nothing and changes only perception. The professional answer names what is spent to save what.
Exam context: An option promising savings with no named cost and no eval evidence is the planted distractor. Selection cues matter too: a waiting user rules out batch, and unique inputs rule out caching.
See also: 4.5 Optimising Tokens, Latency, and Cost
Continuous Evaluation
Scoring sampled production traffic automatically between releases: programmatic checks where the property is deterministic, the calibrated judge where it is not. Quality then reads as a trend line against the launch baseline rather than a point-in-time certificate.
Exam context: "We eval before every release" is the planted answer, because drift arrives between releases. Anthropic's guidance to automate and prioritise volume is what makes continuous scoring affordable at production scale.
See also: 4.6 Monitoring System Performance
Drift Indicator
A leading production signal that moves before user complaints do. Each one maps to a pipeline stage: token spend per request, refusal rate, tool-failure rate, input-length distribution, retrieval-quality signals.
Exam context: The stage mapping is what makes diagnosis fast when one fires. Waiting for concrete complaints is framed as prudence in distractors and is in fact the most expensive, most lagging signal available.
See also: 4.6 Monitoring System Performance
Incident Loop
The operational cycle that makes monitoring compound: alert, traces, diagnose with the diagnostic order, fix at the failing layer, mint the failing cases into the regression population, and decide whether the floor legitimately moved.
Exam context: The last two steps are the ones teams skip. Closing an incident without regression cases leaves the fix unguarded, so the team pays for each bug twice — once to fix it, once to rediscover it.
See also: 4.6 Monitoring System Performance