What You Need to Know
A trace records one request in full, from input to response. LLM systems break the debugging instincts that server software trained into us. The same input can produce different outputs. So a single anecdote proves little, and a re-run proves less. A green uptime dashboard says nothing about whether the answers are right.
Take a flooring firm whose Claude assistant retrieves product sheets, calls a stock tool and answers customers. Customers report wrong prices while the uptime dashboard stays green.
Non-Determinism Breaks Anecdote Debugging
Agents make dynamic decisions and behave differently between runs, even on identical prompts. Anthropic's multi-agent research team reported this from production: users saw agents "not finding obvious information". Engineering could not see why until full production tracing let them diagnose failures systematically.
Re-running a failing input to "see if it happens again" is the anti-method. Without the trace, a reproduction proves nothing and a non-reproduction proves less. A wrong flooring quote that comes out right on a re-run teaches nothing. Reason over distributions, and keep the full context of each request so any single failure can be explained.
The Trace Is the Unit
A failure you cannot trace is a failure you cannot diagnose. Capture these per request:
- prompt version
- model id
- retrieved context
- tool calls and their results
- token usage
- cost
- latency
- the response
Correlation ids stitch the stages of a multi-call pipeline (or the iterations of an agent loop) into one navigable record. A trace that covers single calls in a five-stage pipeline cannot tell you which stage failed. At the flooring firm, one id joins retrieval, stock call and reply.
Suppose the wrong prices began right after a product-sheet refresh, model and latency unchanged. The trace shows what retrieval returned, so start there, not at the model or temperature.
Two Tiers at Scale
Nobody reads a million traces. Logging everything forever fails twice: on cost, and on privacy, because full prompts and outputs capture user data.
Volume forces a two-tier design. Tier one is cheap aggregate metrics on every request: latency percentiles, token spend, error and refusal rates, and tool-failure rates. Tier two is expensive full payloads on a sample, plus every detected failure. Sampling rates, retention windows and redaction of sensitive fields, such as flooring customers' addresses, are architecture decisions made up front. The compliance requirements they serve are Domain 5's territory. Designing for them is yours.
Quality Is Not an Infrastructure Metric
Uptime, error rate and latency can all be green while the system confidently produces rubbish, as at the flooring firm. A hallucinating system is "up". Quality needs its own instrumented signal: automated evaluation on sampled production traffic, scored against a rubric or an LLM judge that Domain 4's formal evals have calibrated. That turns "it sometimes makes things up" into a measurable, alertable trend between formal eval runs. Behavioural monitoring can work at the pattern level without reading individual conversations, which is how it coexists with privacy constraints.
Green Dashboards, Bad Answers
Complaints alongside healthy service metrics mean quality instrumentation is missing. More uptime alerting adds more of the metrics already green.
Drift Shows in the Cheap Signals
Distribution shifts appear in cheap metrics before quality complaints arrive, because those metrics count every request. Token spend per request creeps up, refusal and tool-failure rates rise, and input lengths drift. Alerting on these catches a broken document refresh (like the flooring firm's), a prompt regression or changed user behaviour early.
Alert on tail percentiles, not averages. A stable mean can hide a doubled p99, and the users on the tail file the tickets.
In the exam
The stem: support reports bad answers while engineering sees green dashboards. The keyed answer adds instrumentation that would see the failure; the distractors add more of the metrics already green. "Add more uptime alerting" is the planted distractor in nearly every observability question. The guide's Domain 4 sample has the same shape: confident wrong answers right after a document refresh, model and latency unchanged, so start at retrieval.
Exam Traps
Adding more uptime and latency alerting when support reports wrong answers
Infrastructure metrics cannot see fabricated answers — the dashboards are already green while the failure continues. The gap is quality instrumentation: sampled, automated evaluation of what the system actually said.
Debugging a reported failure by re-running the input a few times
Non-determinism makes single re-runs uninformative in both directions. The trace already holds the retrieved context, tool results, and prompt version that explain the failure — read it instead.
Persisting full prompts and outputs for every request indefinitely
Cost scales with volume and the payloads capture user data, so privacy and retention obligations both fail. The balanced design aggregates everything, samples payloads, and captures every detected failure.
Tracing single calls in a multi-stage pipeline without correlation ids
Without correlation ids the failing stage cannot be found — five healthy-looking call logs do not reconstruct one broken request. The exam rewards the end-to-end trace.
Practice Scenario
A document-assistant serves 200k requests per day across a five-stage pipeline. Support reports that it sometimes makes things up; engineering sees green dashboards (uptime, mean latency). Engineering needs to find where in the pipeline each invented answer starts and how often it happens, without waiting for users to notice. What should the team build?
Build Exercise
Design the Observability Plan for One Pipeline
What you'll learn
- How to define a trace schema that makes any single failure diagnosable
- How to split monitoring into cheap aggregates and sampled payloads
- How to build a calibrated quality signal on production traffic
- How to choose drift alerts on tail percentiles and write a payload privacy policy
- Take a multi-stage Claude pipeline and define its trace schema: prompt version, model id, retrieved context, tool calls and results, tokens, cost, latency, response, correlation id
Why: The trace is the unit of observability — a failure it cannot explain is a failure the team cannot diagnose.
You should see: A schema listing every field with its source stage, and a correlation id that stitches all stages into one record.
- Decide the two tiers: which aggregate metrics run on every request, and what sampling rate plus failure-capture rule gets full traces
Why: Nobody reads a million traces, and logging everything forever fails on cost and privacy at once.
You should see: An aggregates list (latency percentiles, token spend, error, refusal, and tool-failure rates) and a written sampling-plus-failures rule.
- Design the quality signal: what a sampled request is scored against — rubric or judge — and how the scorer is calibrated against formal evals
Why: Quality is not an infrastructure metric; without a scored sample, wrong answers surface as support tickets instead of alerts.
You should see: A scoring definition per sampled request and a calibration note tying the scorer to the formal eval set.
- Choose the drift alerts: thresholds on token spend per request, refusal rate, tool-failure rate, and input-length distribution — on tail percentiles rather than means
Why: Distribution shifts announce themselves in cheap signals before quality complaints arrive, and tails carry the pain that means conceal.
You should see: Three or four alert definitions, each naming a signal, a percentile, and a threshold.
- Write the payload policy: what is redacted, how long traces are retained, and which fields never leave the trace store
Why: Full payloads capture user data — sampling, retention, and redaction are design decisions, not afterthoughts discovered by an auditor.
You should see: A short policy covering redaction rules, a retention window, and the fields with restricted access.
Sources
- How we built our multi-agent research system — Anthropic
- Usage and Cost API — Anthropic
- Claude Certified Architect Professional Exam Guide (v1.0) — Domain 3 — Anthropic