What You Need to Know
Launch-time evals prove readiness once; monitoring proves it stays true in production. Drift is post-launch change that moves quality. A baseline is a metric's measured level; a guard metric is one a change must not damage. This objective closes Domain 4's loop, putting 4.1 metrics, 4.2 graders and 4.4 diagnosis to work in production. Every incident feeds cases back into the dataset.
An accountancy firm's Claude assistant answers client questions from its tax guidance library. Since launch, the firm has refreshed the library, edited the prompt and added a client portal.
The Quality Practice on Top of Domain 3
Domain 3 built the plumbing: what a trace captures, how sampling works, and aggregate metrics infrastructure. Domain 4 runs the quality practice on top: what dashboards show and what happens when a line moves. The dashboards track the 4.1 metric sheet against its targets, not just infrastructure health: accuracy signals, latency percentiles, cost per task, refusal and error rates.
Uptime does not measure wrongness. A green uptime dashboard proves the service is up, not that it is right. A system can be fully up, fast on average and confidently incorrect. So quality monitoring is its own discipline, with its own scored signals. The firm's assistant can answer quickly and still quote last year's guidance.
The Domain 3/4 Boundary
Trace capture and sampling design belong to Domain 3's plumbing. Which quality signals to score, alert on and act on belongs here.
Continuous Evaluation
Formal eval runs are lagging proof at release boundaries; drift arrives between them. Continuous evaluation scores sampled production traffic automatically. Programmatic checks score deterministic properties; the calibrated judge from 4.2 scores the rest. Quality becomes a trend line against the launch baseline, not a point-in-time certificate. At the firm, a check confirms each answer cites a guidance note; the judge scores the advice.
Anthropic's evaluation guidance (automate when possible, prioritise volume with automated grading) makes this affordable at production scale. The Usage and Cost API documentation lists monitoring product performance, and measuring whether system changes improved it, as use cases. The guardrails guidance adds security: continuously monitor outputs for signs of successful injection.
Drift and Leading Indicators
Input distributions shift, corpora refresh, models migrate and prompts accumulate edits, on no release schedule. Leading indicators move before user complaints do. Each maps to a pipeline stage, so 4.4's stage isolation is fast when an alert fires.
- Token spend per request: inputs grew, or a prompt edit changed behaviour
- Refusal rate: traffic shifted, or a policy prompt interacts badly with new inputs
- Tool-failure rate: an upstream API changed
- Input-length distribution: a new client integrated
- Retrieval-quality signals: a corpus refreshed
At the firm, the library refresh maps to retrieval-quality signals and the new portal to input-length distribution.
Waiting for concrete user complaints is the expensive alternative. Complaints are the most lagging signal, and each is a user the leading indicators could have saved.
Alerts, Baselines, and the Incident Loop
Alerts fire on guard metrics at tail percentiles, with thresholds from measured baselines. Alerting on averages hides the pain, so alerts fire late or never. Alerting on everything trains the team into alarm fatigue until the real alert is ignored. Protecting signal means deliberately choosing what not to alert on. The firm sets tail thresholds from its baselines and names what pages nobody.
When an alert fires, the incident loop runs:
- Alert
- Traces
- Diagnose, in 4.4's order
- Fix at the failing layer
- Mint regression cases (4.2)
- Re-baseline, if the floor legitimately moved
The last two steps make monitoring compound. Skip the regression cases and the same failure returns under a new ticket number. Skip the re-baseline after a legitimate change (a model migration, a corpus refresh that genuinely improved things) and every future comparison is noise against a floor that no longer exists. When the firm traces wrong answers to the refreshed library, it fixes retrieval, adds regression cases, and re-baselines if the refresh improved things.
In the exam
The exam tests running this practice, and where Domain 3 ends and Domain 4 begins. Uptime versus wrongness carries direct exam weight. When a stem asks "what should the team monitor?", check whether it wants instrumentation (Domain 3) or practice (Domain 4). The guide's own sample scenario is a corpus refresh surfacing as confident wrong answers.
Exam Traps
Presenting green uptime and mean-latency dashboards as evidence the system is healthy
Uptime does not measure wrongness. A system can be fully up, fast on average, and confidently incorrect — quality needs its own scored signals, tracked against the 4.1 targets.
Skipping continuous evaluation because the team evals before every release
Drift arrives between releases: corpora refresh, traffic shifts, and prompts accumulate edits on no release schedule. Launch evals certify the past; only continuous scoring of sampled traffic sees the present.
Closing incidents without minting regression cases from the failures
The fix is unguarded: nothing detects the same failure returning, so the team pays for each bug twice — once to fix it, once to rediscover it. The regression step is what makes the incident loop compound.
Practice Scenario
Three months after launch, a procurement assistant's satisfaction scores sag. Release evals all passed; the uptime and mean-latency dashboards are green. The team must learn what has changed before altering the system, and catch any further decline before users report it. What should the team do?
Build Exercise
Write the Monitoring Page for One System
What you'll learn
- How to choose what gets continuously scored on sampled production traffic and with which grader
- How to select drift indicators and map each to the pipeline stage it implicates
- How to set alert thresholds on guard metrics at tail percentiles from measured baselines
- How to write the incident loop as an operational runbook
- How to test a monitoring design against the on-call engineer question
- For a Claude system you know, list what is continuously scored on sampled production traffic, and with which grader (programmatic check or the calibrated judge from your 4.2 exercise)
Why: Continuous evaluation is the practice that turns quality from a release-time certificate into a trend line.
You should see: A short list of scored signals, each paired with its grader and sampling approach.
- Choose the drift indicators you will watch (token spend per request, refusal rate, tool-failure rate, input-length distribution, retrieval-quality signals) and map each to the pipeline stage it implicates
Why: Leading indicators move before complaints, and the stage mapping is what makes diagnosis fast when one fires.
You should see: Each chosen indicator with the stage it implicates, for example: retrieval-quality signals implicate the index and embedding pipeline.
- Set alert thresholds: which guard metrics, at which tail percentiles, derived from which measured baselines — and name one thing you deliberately will NOT alert on, to protect signal
Why: Alerts on averages fire late; alerts on everything train the team to ignore them. Both failures are design choices.
You should see: A threshold table (guard metric, percentile, baseline, threshold) plus one explicit non-alert with its justification.
- Write the incident loop as a runbook: alert, traces, diagnose, fix, regression cases, re-baseline decision
Why: The loop is what makes monitoring compound — and it only runs under pressure if it was written down before the pressure.
You should see: Six runbook steps with owners, ending in two explicit questions: which cases get minted into the eval set, and did the floor legitimately move.
- Defend the page against the 2am question: when this system fails three months from now, what will the on-call engineer see, and how fast does it lead them to the failing stage?
Why: A monitoring design is only as good as the path from signal to stage for someone who did not build the system.
You should see: A walkthrough from a plausible failure to the alert that fires, the dashboard line that moved, and the stage the indicator implicates.
Sources
- Usage and Cost API — Anthropic
- Define success criteria and build evaluations — Anthropic
- Mitigate jailbreaks and prompt injections — Anthropic
- Claude Certified Architect Professional Exam Guide (v1.0) — Domain 4 — Anthropic