Quick Reference: Domain 4 — Evaluation, Testing & Optimisation

Order of operations: define success criteria → design evaluations → build. Metrics chosen after launch are calibrated to the accident of the build, not the business need.

Every metric has three parts: definition + measurement method + target with baseline. "Accurate" is a wish.

• Success versus guard metrics: what a change should improve versus what it must not damage (cost per task, tail latency, safety rates). No guards means licensed regressions.
• Leading signals (drift indicators) warn early; lagging proof (formal eval results) licenses decisions. You need both.
• Proxy trap: any proxy optimised hard enough stops tracking the outcome it proxied. Audit proxies against the real outcome; retire them when they decouple.
• Anthropic's named dimensions include task fidelity, consistency, latency, and price, with criteria that are specific, measurable, achievable, and relevant.
• Latency vocabulary: baseline latency is the time to process the prompt and generate the response; time to first token is the time to the first token of that response, and it is the only one streaming moves.

Four populations, each buying different risk coverage: representative (the bulk, mirroring the production distribution) · edge (irrelevant data, overly long inputs, poor user input, ambiguous cases) · adversarial (injection attempts, leading questions, malformed input) · regression (one minted per production failure).

Golden-set discipline: versioned · held out of the prompt · refreshed from production traffic. Held out means out of the few-shot examples, not only out of the training data.

The pairing rule: deterministic where possible, judge where not, humans to calibrate the judge and audit the tail.

• Anthropic's design principles: be task-specific, include edge cases, automate when possible, and prioritise volume — more questions on slightly lower-signal automated grading beat fewer hand-graded ones. Best practice is a different model as judge than the one that generated the output.
• Statistics, not round numbers (Anthropic's statistical evals research): report the standard error of the mean with confidence intervals; cluster standard errors on related questions, because clustered errors on popular evals can be over three times the naive ones; resample non-deterministic answers and average per question; run a power analysis before choosing the sample size.
• Both extremes get planted as options. An uncalibrated judge is decoration; grading everything by hand mis-spends the gold standard.

Offline evals gate; online experiments validate. Failing the suite ends the conversation. Passing it is necessary but not sufficient for a user-facing behaviour change, because the set approximates production and users are production.

• One variable at a time: prompt OR model OR retrieval. "Since we're touching it anyway" is how an experiment becomes a deployment.
• Every experiment carries success and guard metrics. A headline win with a guard breach is a failed variant.
• Evidence bar: a paired-difference test when both variants answer the same questions (question-score correlation on frontier models typically runs 0.3 to 0.7, so pairing is free variance reduction), standard errors, resampled outputs, and a power analysis run first. "Clearly better on twenty cases" is folklore.
• Staged exposure, proportionate to risk: shadow (new variant runs on real traffic, outputs compared offline, zero user exposure) → canary (small live slice, guards watched, rollback armed) → full A/B. A rollback plan substitutes for neither.
• Model-version migrations shadow by default, and once a migration lands, re-baseline every downstream measurement: the floor moved, so old comparisons are noise.
• Anthropic's upgrade order: build benchmark tests for your use case → test with your actual prompts and data → compare accuracy, response quality, and edge-case handling → weigh performance against cost.
• The loop: measure → hypothesise → change one thing → eval offline → A/B online → ship or revert → the losing variant's failures become regression cases.

The objective that punishes fluent guessing hardest, and the guide's own Domain 4 sample sits here: confident wrong answers after a document refresh, model and latency unchanged, keyed to the most likely first place to investigate (retrieval/indexing) rather than the best eventual fix.

Diagnostic order: trace and reproduce → isolate the stage → classify against the signatures → cheapest decisive test → fix at the failing layer → mint regression cases.

• Generation-failure repertoire (Anthropic's reduce-hallucinations guidance): allow "I don't know" explicitly, extract word-for-word quotes first on long documents, require citations and retract any claim without a supporting quote, restrict answers to the provided text, verify with chain-of-thought or best-of-N.
• Prompt-failure repertoire (increase-consistency guidance): specify the output format precisely, constrain with examples, use structured outputs where schema conformance must be guaranteed.
• Mismatch runs both directions. A premium tier sailing far above a simple task is the same diagnosis with the opposite prescription; 4.5 owns the downsizing lever.
• Inconsistency is a symptom, not a class: underdetermined instructions, retrieval varying per run, or genuine variance at the edge of capability. Trace, cluster the divergent cases, find what differs between runs.
• Bias detection is disaggregation: split eval results across cohorts and content types. A system accurate on average and wrong for one population is failing. Detection belongs to Domain 4; the governance response belongs to Domain 5.
• Planted distractors: the reflex model upgrade, prompt-patching a broken index, temperature zero as a correctness fix, and one word ("hallucination") for every wrong answer.

Two iron rules bracket every optimisation. Measure first: traces plus usage data name the dominant term. Re-gate with evals: "cheaper and still passes the bar" is the only win condition.

Figures as of 30 September 2026. Re-verify before quoting; the reasoning outlives the numbers:

• Batch processing: 50% off all usage (input, output, and special tokens); most batches finish in under 1 hour; results are readable when every request completes or after 24 hours, whichever comes first, and a batch expires at 24 hours; results stay downloadable for 29 days after creation; one batch caps at 100,000 requests or 256 MB, whichever is reached first; stream is unsupported inside a batch. Caching discounts stack with the batch discount, with best-effort hit rates typically between 30% and 98%.
• Prompt caching: reads 0.1× base input price on most models (0.05× on Opus 5.5 and Sonnet 5.5, 0.025× on Fable 5.1 and Mythos 5.1); five-minute writes 1.25×, one-hour writes 2×; up to 4 breakpoints; five-minute TTL by default, refreshed free on use. Minimum cacheable prefix: 512 tokens (Fable 5.1, Opus 5.5, Sonnet 5.5, Haiku 5.5, Opus 5, Fable 5), 1,024 (Sonnet 5, Sonnet 4.6, Opus 4.8), 2,048 (Opus 4.7), 4,096 (Haiku 4.5, Opus 4.6, Opus 4.5). Below the minimum the request runs uncached, with no error.
• Effort levels, highest token spend first: max · xhigh · high (the default on every model that supports effort except Opus 5.5 and Haiku 5.5, which default to medium; sending the default is identical to omitting the parameter) · medium · low. Effort shapes all tokens: text, tool calls, and thinking. Anthropic states that tuning effort is often a better lever than switching models. Hold it constant inside a cached conversation, because changing it invalidates the cached prefix.
• Prices per million tokens, input/output: Fable 5.1 $10/$50 · Opus 5.5 $4/$20 · Sonnet 5.5 $2/$10 · Haiku 5.5 $0.10/$0.50 (prompts up to 100,000 tokens; $0.50/$2.50 above). Halve each for batch. Fast mode (research preview, Opus 5.5, Opus 5 and Opus 4.8) buys up to 2.5× output speed at premium pricing.
• Errors worth recognising: 400 invalid_request_error (an over-long prompt lands here) · 413 request_too_large (Messages 32 MB, Batch 256 MB, Files 500 MB) · 429 rate_limit_error · 500 api_error · 504 timeout_error · 529 overloaded_error. The official SDKs retry transient failures twice by default with exponential backoff, honouring retry-after.

Distractors: batch offered for an interactive path, blind downsizing with no re-gate, and caching offered for window overflow, wrong answers, or a workload of unique one-off documents.

Domain 3 built the plumbing — traces, sampling, aggregate metrics. Domain 4 runs the quality practice on top of it. Uptime does not measure wrongness: a system can be fully up, fast on average, and confidently incorrect.

• Continuous evaluation: sample production traffic and score it automatically — programmatic checks for deterministic properties, the calibrated judge for the rest. Quality becomes a trend line against the launch baseline instead of a release-day certificate.
• Leading indicators, each mapped to a stage: token spend per request (inputs grew, or a prompt edit landed) · refusal rate (traffic shifted, or a policy prompt interacts badly with new inputs) · tool-failure rate (an upstream API changed) · input-length distribution (a new client integrated) · retrieval-quality signals (a corpus refreshed — the guide's own sample scenario, caught early).
• Alerts fire on guard metrics at tail percentiles, with thresholds derived from measured baselines. Averages fire late or never; alerting on everything trains the team to ignore alerts, so choose deliberately what not to page on.
• Incident loop: alert → traces → diagnose with 4.4's order → fix at the failing layer → mint regression cases → re-baseline if the floor legitimately moved. Skip the minting and the team pays for each bug twice.
• Instruments as of mid-2026: the Usage and Cost Admin API reports usage grouped and filtered by model, workspace, API key, service tier, context window, inference_geo (data residency — global, us, or not_available), and speed, in one-minute, one-hour, or one-day buckets (cost is daily only, in USD), with data typically visible within five minutes; its named use cases include monitoring product performance, cache efficiency, and budget alerting. Both the speed group-by and the speeds[] filter require the fast-mode-2026-02-01 beta header (Anthropic, Usage and Cost API). Guardrails guidance adds continuous monitoring of outputs for signs of successful injection.
• Planted distractors: re-running the launch suite as proof of present health, waiting for concrete user complaints, and the undiagnosed model upgrade.