What You Need to Know
An operational incident is a production agent behaving in a way nobody expected. Resolving one is a team skill: who can see the evidence, what the first move is, and what the incident leaves behind. Containment asks whether harm is ongoing. Diagnosis asks what is wrong. Blast radius is how far the failure reaches outside the team.
Take an ice cream parlour whose agent writes online cake orders onto the kitchen sheet. Since last week, some orders arrive with the wrong flavour. Staff check the sheet before making anything, so the errors stay internal and reversible. That week the parlour also loaded a summer menu and updated its till connector.
Read the Evidence First
When a production agent starts behaving unexpectedly, the first move is to read what happened. That means the inputs, the context retrieved, the tool calls and what they returned, the outputs, and the versions in play.
The tempting first move is to edit the agent's instructions. That is wrong for three reasons, and they stack. Nothing yet implicates the instructions. The edit changes the system under investigation. And the failing runs that would have identified the cause become unreproducible, so the parlour could no longer replay last week's orders. A fix applied before the failure is understood usually changes something and occasionally fixes something. It always destroys the evidence.
Containment Is a Different Decision From Diagnosis
Different facts decide containment and diagnosis. Blast radius decides containment. An external effect still accumulating gets contained immediately: money moving, messages sent, records altered outside the team's control. Revert to the last known-good version, disable the capability, or gate the action behind an approval, then diagnose with the pressure off.
An internal, reversible degradation is worth more as evidence than as a restored system. The parlour's wrong flavours are of this kind. Had the agent been charging cards twice, the owner would contain first.
Containment is not a diagnosis either. A reverted system is quiet, not understood. Blanket containment also costs attribution, because several changes reverted together teach nothing about which one was responsible.
Blast Radius Decides, Not Discomfort
Ongoing external effect: contain first, diagnose second. Internal and reversible: the failing cases are the asset, and a rollback spends them.
Change History Narrows the Search Before Any Trace Opens
Most operational incidents are caused by a change, so the fastest question is what changed and when the symptom started. Instruction versions, model versions, corpus refreshes, configuration changes, and dependency and connector updates all deserve a version and a timestamp. Correlating symptom onset against that timeline eliminates most of the search space without touching the system.
When two changes land in the same window, test them against the evidence one at a time rather than reverting both. Two variables moved together cannot be attributed. So the parlour tests the summer menu and the till connector separately. This is Domain 4's one-variable rule. Anthropic's configuration debugging uses the same method: remove variables until the behaviour changes, then reintroduce them one at a time.
The Agent Investigates; It Does Not Adjudicate
AI-assisted tooling shortens root-cause work in five ways:
- reading logs and traces at volume
- reconstructing a timeline
- correlating a symptom against a deployment
- reproducing a failing case
- proposing hypotheses faster than a human enumerates them
Correlated telemetry makes this possible by pulling everything triggered by a single prompt into one story.
The tooling does not settle the question. Asked why it behaved unexpectedly, an agent gives a coherent account of its own reasoning. That account is a hypothesis. It comes from the same context that produced the behaviour, with no privileged access to the cause, and it persuades because it is fluent. If the parlour's agent blames the new menu, that stays unproven until a human confirms it against the trace. Non-determinism raises the bar. One clean re-run does not clear a hypothesis, and one failing run does not prove it. Confirmation means a pattern across collected cases.
Enablement Is Built Before the Incident
Everything above assumes a capability somebody designed in advance:
- traces that exist and are readable
- a pager-holder who can open them at 2am without waiting on an approval
- rollback available to more than one named individual
- a runbook that names the first three moves
- shared investigation tooling, rather than whatever gets assembled privately
An on-call engineer without trace access has no first move, and no diagnostic discipline makes up for it. If only the parlour's owner can open the traces, a 2am failure waits until morning.
What the Incident Must Leave Behind
A fix at the failing layer, verified against the real case that failed. A regression case in the suite Domain 4 owns. A runbook update while the gap is fresh. Any access gap closed or recorded.
In the exam
Domain 4 diagnoses a product's quality failures against evals and traces. Where it meets this objective, the item asks about the practice and the people, not the classification. Reading the evidence first is the most reliably keyed answer in the domain. Editing the agent's instructions is the most reliably planted distractor. Containment is tested through blast radius, so read the stem for what the failure is doing outside the team. The option that ships the fix and closes the ticket is the near-passer trap: one step short.
Exam Traps
Rewriting the prompt or the shared instructions as the first move
Nothing yet implicates the wording, the edit changes the system under investigation, and the failing runs that would have identified the cause become unreproducible. It usually changes something, occasionally fixes something, and always destroys the evidence.
Reverting several recent changes at once as a precaution, with no ongoing harm to stop
Containment is warranted by blast radius, not by discomfort. Blanket reversion may quieten the symptom while destroying attribution, so nobody can say which change caused the failure and the incident returns the next time either one lands.
Accepting the agent's account of its own behaviour as the root cause, or clearing a hypothesis on one clean re-run
The account comes from the same context that produced the behaviour and is persuasive because it is fluent, so it is a hypothesis to confirm against the trace. Under non-determinism a single run proves nothing either way.
Closing the incident with a fix and nothing else, or planning an investigation that assumes access nobody has granted
A fix with no regression case is unguarded, and a team with no runbook update is no faster next time. An on-call engineer who cannot read traces or roll back has no first move, whatever the plan says.
Practice Scenario
An internal reporting agent has started skipping its source-verification step on a minority of nightly runs. The effect is internal and every affected report has been regenerated without loss. Two things landed this week: a model version bump and a rewrite of the shared project instructions. Nobody has opened a trace, and asked about a failed run the agent gives a fluent account of why it judged the step unnecessary. The team wants to know today which of this week's two updates caused the skip, before altering anything. What should the team do first?
Build Exercise
Write the Incident Page for One System
What you'll learn
- How to name the first three pieces of evidence an on-call engineer reads
- How to test blast radius to decide between containment and diagnosis
- How to make change history usable as a diagnostic instrument
- How to separate an agent hypothesis from a confirmed finding
- How to close an incident so the team is faster the next time
- For a production Claude system you know, name the first three things an on-call engineer reads when it misbehaves, and the access each one requires
Why: Every diagnostic move assumes evidence that somebody made readable in advance.
You should see: Three named sources in reading order, each with the permission or tool needed to open it.
- Record who holds that access today and how they obtain it at 2am, then flag every step that depends on one particular person being awake
Why: An investigation plan that assumes access nobody granted is not a plan.
You should see: An access list with at least one uncomfortable single point of dependency flagged.
- Write the containment decision rule: which failures are contained before diagnosis, which are diagnosed on live evidence, and the blast-radius test that separates them
Why: Containment and diagnosis answer different questions, and the exam tests whether you can tell which one the facts call for.
You should see: One rule a tired engineer can apply, with a worked example on each side of the line.
- List every change stream that could move behaviour - instruction versions, model version, corpus refresh, configuration, dependency and connector updates - and say where each version and deployment time is recorded today
Why: What changed and when the symptom started narrows the search before a single trace is opened.
You should see: A stream-by-stream table with honest gaps wherever nothing is timestamped.
- Audit the last real incident the team handled: what regression case it left, what runbook update it left, and what access gap it exposed
Why: An incident closed with no artefact is paid for a second time at full price.
You should see: Three answers, some of them blank, and a short list of what to write down before the next one.
Sources
- Debug your configuration — Anthropic
- Monitoring — Anthropic
- Best practices for Claude Code — Anthropic
- Claude Certified Architect Professional Exam Guide (v1.0) — Domain 7 — Anthropic