What You Need to Know
An evaluation metric measures whether a Claude system meets its need. Each entry on a metric sheet has three parts: the definition, the measurement method, and the target with its baseline. "Accurate" is a wish. "At least 95% field-level extraction accuracy on the held-out set, measured by exact match" is a metric. Decision-grade metrics are specific enough to measure and complete enough to catch what a single number hides.
A yoga studio plans a Claude assistant that answers members in a website chat and drafts the weekly newsletter overnight. The owner's whole brief is "accurate".
Metrics Before the Build
Define metrics before the system is built; the rest of Domain 4 depends on it. Anthropic's evaluation guidance gives the order: clearly define success criteria, then design evaluations to measure against them. Domain 4's blueprint encodes the same sequence: define metrics, design datasets, then test.
A system built first and measured later inherits whatever bar it happens to hit. Its metrics are calibrated to the status quo, and no design decision was tested against the business need. Write the sheet before the first prompt, as the studio does.
Key Concept
Anthropic's guidance is that success criteria should be specific, measurable, achievable, and relevant. Most use cases need multidimensional evaluation. Its named dimensions include task fidelity, consistency, latency, and price.
The Five Metric Families
The guide names five metric families:
| Family | The professional distinction | The single-number mistake |
|---|---|---|
| Accuracy | Match the metric to the task shape: exact match for deterministic outputs, per-field precision and recall for extraction, rubric or judge scores for open-ended generation | One aggregate "percent correct" over heterogeneous tasks hides which capability is failing |
| Latency | Report percentiles, and distinguish time-to-first-token from total completion time, which are different promises to different consumers | A mean that conceals the tail the churning users live in |
| Cost | Per task at projected volume, the unit-economics framing Domain 1's cost pillar demands | Per-token figures with no volume attached |
| Safety | Refusal correctness in both directions: harmful-output rate and over-refusal of legitimate requests | Counting refusals as if more were always safer |
| Security | Resistance to injection and data leakage, measured under adversarial testing (objective 4.2 designs the cases) | Resistance claimed on paper, never attacked |
The studio checks chat class times by exact match and newsletter drafts by rubric.
Latency targets and alerts belong on tail percentiles such as p95 and p99. This is standard SLO practice: means conceal the pain of the users most likely to churn. Interactive surfaces feel time-to-first-token, while pipelines feel total completion time. Anthropic's latency documentation treats them as separate measurements. Studio chat members feel the first token; the newsletter job feels the total.
A safety scorecard with no over-refusal metric lets the system "get safer" by refusing everything. A studio assistant declining every class question would never produce harmful output. Measuring refusal correctness in both directions makes over-refusal visible as a failure.
Success Metrics and Guard Metrics
The success metric is what a proposed change should improve. Guard metrics are what it must not damage: typically cost per task, tail latency, and the safety rates. Declaring a success metric without guards licenses regressions. A variant that lifts the headline while breaching a guard has failed. A longer studio prompt that lifts accuracy but breaches the p95 guard fails. Objective 4.3's experiments carry both, and objective 4.6's production alerts fire on the guards.
Leading Signals, Lagging Proof, and the Proxy Trap
Leading signals are drift indicators from monitoring, such as token spend per request, refusal rate, and retrieval-quality signals. They move before quality visibly drops. Lagging proof is a formal evaluation result. You need both: leading signals for early warning, lagging proof for decisions. A rising refusal rate warns the studio; a formal evaluation decides.
Any metric sheet risks the proxy trap: a proxy metric optimised hard enough stops tracking the outcome it proxied. Response length as a completeness proxy and judge score as a satisfaction proxy are useful until the system learns to move the proxy without moving the outcome. Reward longer studio answers and the assistant can pad them. Audit each proxy periodically: check it against the real outcome, and retire it when they decouple.
In the exam
Questions on this objective test whether your metrics are decision-grade. A stem offering one aggregate number, or proposing to define metrics "once real data exists", is offering you the trap. Two distinctions carry recurring weight: latency on tail percentiles with the felt latency named, and over-refusal measured alongside harmful output.
Exam Traps
Accepting mean latency as the latency metric or SLA target
Means conceal tail pain; the users experiencing the p99 are the ones who churn. Targets and alerts belong on tail percentiles, and the professional answer also states which latency — time-to-first-token or total completion — the consumer actually feels.
Deferring metric definition until after launch, when real production data exists
Post-hoc metrics are calibrated to whatever the system happens to achieve, not to the business requirement. Anthropic's guidance and the blueprint order both put success criteria before the build — the exam rewards that sequence.
Scoring safety as harmful-output rate alone
A system can get safer by refusing everything. Refusal correctness runs in both directions: over-refusal of legitimate requests is a failure with a metric attached, not a virtue.
Practice Scenario
A team launching a contract-extraction service proposes one metric: overall accuracy of at least 90%, measured on requests they spot-check. The sponsor will sign off only on targets that are fixed before launch, can be re-measured the same way after every release, and show where the service fails. What should the architect do?
Build Exercise
Write the Metric Sheet for One Claude System
What you'll learn
- How to define a metric with a definition, measurement method, baseline, and target
- How to match an accuracy metric to the task shape
- How to state latency as a percentile and name which latency the consumer feels
- How to separate success metrics from guard metrics
- How to spot the proxies in your own metric sheet before they decouple
- Pick a production Claude system you know (or design one) and write one metric per family: accuracy, latency, cost, safety, security
Why: The five families come straight from the exam guide, and the sheet forces multidimensional evaluation instead of one aggregate number.
You should see: Five named metrics, one line each — no metric shared between families.
- For each metric record the definition, the measurement method, the baseline, the target, and whether it is a success or a guard metric
Why: A metric without a measurement method cannot be run; a target without a baseline cannot show progress; and the success/guard split is what every experiment in 4.3 will inherit.
You should see: A five-row table with all five columns filled — no row reading simply accurate or fast.
- For accuracy, match the metric to the task shape and justify it: exact match, per-field precision and recall, or rubric score
Why: Task-shape matching is the recurring accuracy distinction: the wrong grader produces a number that means nothing.
You should see: One sentence naming the task shape and the grader it implies.
- For latency, state the percentile and whether the consumer feels time-to-first-token or total completion time
Why: These are different promises to different consumers: interactive surfaces feel the first token, pipelines feel the total.
You should see: A percentile target with a named consumer, for example: p95 time-to-first-token for the chat surface.
- Mark each metric as a leading signal or lagging proof, and name one proxy in your sheet that could decouple from its outcome if optimised hard
Why: Leading signals buy early warning, lagging proof licenses decisions, and every sheet contains at least one proxy waiting to be gamed.
You should see: Each row tagged leading or lagging, plus one sentence naming the proxy and the audit that would catch it decoupling.
Sources
- Define success criteria and build evaluations — Anthropic
- Reducing latency — Anthropic
- Mitigate jailbreaks and prompt injections — Anthropic
- Claude Certified Architect Professional Exam Guide (v1.0) — Domain 4 — Anthropic