What You Need to Know
An offline eval tests a candidate change on a fixed dataset before users see it. An online A/B test splits live traffic between the current baseline and the candidate. A guard metric must not get worse. With the 4.1 metrics and 4.2 datasets, these form a loop that turns a hypothesis into an attributable, statistically honest, reversible production change.
A cleaning firm uses Claude to draft customer quotes. The team wants a rewritten quoting prompt and a price-list lookup.
The Gate and the Validation
Offline evals gate; online experiments validate. The offline suite is quick and repeatable enough to run on every candidate. Failing it stops the candidate. Passing is necessary but not sufficient for a user-facing behaviour change. The eval set approximates production, and users are production. The surviving candidate proves its real-world impact in an A/B against the live baseline, on real metrics and traffic. The firm's new prompt faces past quote requests, then live customers.
Anthropic's model-selection guidance sets the same order for model changes:
- Build benchmark tests specific to your use case.
- Test with your actual prompts and data.
- Weigh the performance and cost trade-offs.
The gate is not the validation, and a rollback plan substitutes for neither.
One Variable at a Time
Only an isolated change can be credited with a result. If the prompt is rewritten and the model upgraded together, a win cannot be credited and a loss cannot be unpicked. The next decision inherits a superstition, not a finding. So land the first change, establish the new baseline, then test the second against it. The firm tests its new prompt alone, then the lookup against that baseline.
This feels slower and costs less. Unattributed wins get reverted by accident, and unattributed losses get debugged blind. "Since we're touching it anyway" is how experiments become deployments.
The Evidence Bar
Non-deterministic outputs make measurements noisy, which raises the bar for declaring anything. Anthropic's statistical approach to evals supplies the methods:
- Report the standard error of the mean and confidence intervals, not bare scores.
- When two variants answer the same questions, use a paired-difference test. Question difficulty is correlated across variants, and pairing uses that correlation to cut variance at no cost.
- Resample non-deterministic outputs and average per question.
- Run a power analysis first, so the sample size can resolve the effect you care about.
Judging by eye is not evidence, even if the firm's new prompt "looks clearly better on these twenty cases".
Staged Exposure for Risky Swaps
Standard deployment practice stages exposure for risky changes, above all model-version migrations, since behaviour can shift broadly and silently. In a shadow run, the new variant handles real traffic and its outputs are compared offline, unseen by users. That tests behaviour on reality at zero user risk. In a canary, a small live slice gets the variant first, with guard metrics watched and rollback armed. The firm shadows a newer model on live requests before any canary.
Both need versioned prompts and configurations with clean rollback. An experiment that cannot be reversed cleanly is a deployment. After a migration lands, re-baseline every downstream measurement. Comparisons against the old model's baseline are now noise.
In the exam
Stems show a team like the firm's about to do something efficient-looking and wrong.
- bundling two changes into one test
- shipping on a green offline suite
- declaring a winner from a handful of cases
When a stem offers a clear improvement on a small sample, the keyed answer names sample size and significance.
The Improvement Loop
The full loop the exam expects: measure, hypothesise, change one thing, eval offline, A/B online with success and guard metrics. Then ship or revert, and mint the losing variant's failures as 4.2 regression cases. Teams skip the last step. It is what makes the loop compound instead of repeat.
Exam Traps
Bundling a model migration with a prompt rewrite because the system is being touched anyway
Two variables moved at once means no attribution: a win cannot be credited, a loss cannot be unpicked. The exam keys single-variable, staged migration — first the model against the old prompt, then the rewrite against the new baseline.
Treating a passing offline suite as licence to roll a user-facing change to everyone
The eval set approximates production; the A/B against the live baseline is the proof of real-world impact. Skipping it converts an approximation into an assumption at full user exposure.
Declaring a winner because the difference looks clear on a handful of cases
LLM variance makes small samples noisy, and a difference that looks clear is often within noise. The professional answer names the machinery: paired-difference tests, standard errors, and a power analysis run before the experiment.
Offering a rollback plan as a substitute for staged exposure
Shipping everything at once with a rollback still exposes all users to an unvalidated change, and detection after the fact is the most expensive way to learn it failed. Shadow and canary exist to buy the evidence before the exposure.
Practice Scenario
A team wants to migrate a support agent to a newer model version and simultaneously rewrite the system prompt, since they are touching it anyway. They must be able to say which change caused any shift in behaviour, and no change may reach every user untested. What is the right approach?
Build Exercise
Write the Experiment Plan for Your Next Change
What you'll learn
- How to isolate a single experimental variable and hold the rest constant
- How to pair success metrics with guard metrics and their baselines
- How to justify a sample size with a power analysis
- How to choose an exposure strategy proportionate to the risk of the change
- How to define the decision rule, rollback, and regression-case minting in advance
- Name the single variable the experiment changes, and what is deliberately held constant
Why: Attribution requires isolation — the plan should make it impossible to smuggle a second change in later.
You should see: One named variable and an explicit held-constant list covering prompt, model, retrieval, and parameters.
- State the success metric and every guard metric, with the current baseline for each
Why: A variant that lifts the headline while breaching a guard has failed — the experiment must be able to detect that outcome.
You should see: One success metric plus guards (cost per task, tail latency, safety rates), each with a measured baseline.
- Justify the sample size: what effect size matters, and what does a power analysis say you need to resolve it
Why: Sample size decided after seeing results is how noise gets promoted to signal.
You should see: A minimum effect size worth acting on, and the case count required to resolve it with confidence.
- Choose the exposure strategy (offline only, shadow, canary, full A/B) and defend the choice against the risk of the change
Why: Exposure should be proportionate: a copy tweak and a model migration do not deserve the same blast radius.
You should see: A named strategy with one sentence tying it to the risk class of the change.
- Define the decision rule in advance (ship, revert, or iterate), the rollback mechanism, and where failures from the losing variant will be minted as regression cases
Why: Decision rules written after the results arrive are rationalisations; and the regression-case step is what makes each experiment compound.
You should see: An if-then decision rule, a tested rollback path, and the destination population for the failures.
Sources
- A statistical approach to model evaluations — Anthropic
- Define success criteria and build evaluations — Anthropic
- Choosing the right model — Anthropic
- Claude Certified Architect Professional Exam Guide (v1.0) — Domain 4 — Anthropic