What You Need to Know
Diagnosis is a discipline, not a lookup: you find why a Claude system fails before changing anything. In a prompt failure, the instructions are the problem. A hallucination is confident fabrication: invented entities, citations, figures or clauses, fluent enough that a non-expert would accept them. In a model mismatch, the task exceeds the model tier.
A pub's Claude assistant answers customers from retrieved passages of its menu, allergen sheet and booking rules. After a winter menu load, it starts calling a dish with nuts nut-free. Model and latency are unchanged.
The Diagnostic Order
- Trace and reproduce. A trace records what went into the request (prompt version, retrieved context, tool results) and what came out. Domain 3 built this plumbing; Domain 4 consumes it. One bad output proves nothing in a non-deterministic system, so confirm a pattern across cases, not a single run. The pub pulls the trace of every wrong allergen answer since the menu load.
- Isolate the stage. Let the evidence implicate a pipeline stage before you name a failure class. The pub's errors began with the menu load, with model and latency unchanged. That implicates retrieval and indexing before anyone says the word "hallucination".
- Classify against the signatures below: prompt failure, hallucination (grounding or generation), model mismatch. Each has a recognisable signature.
- Run the cheapest decisive test for the hypothesis: the check that confirms or rules it out.
- Fix at the failing layer, never at the layer that is merely easiest to reach. The reflex fixes, "upgrade the model" and "rewrite the prompt harder", are conclusions without a diagnosis. They skip steps 2 to 4, so they can land on a layer that never broke. This objective's distractors are built from them.
- Mint regression cases (objective 4.2) from the failing inputs, so the diagnosis outlives the incident.
Prompt Failure
The signature: format drift, violated output contracts, rules ignored or applied inconsistently, and behaviour varying with phrasing on identical inputs. Failures cluster where the instructions underdetermine. The pub's booking replies switch between list and paragraph as wording changes.
The decisive test: a clarified, structured prompt (explicit rules, delimited data, an exact-format example) fixes the failing cases, all else held constant. Anthropic's consistency guidance is this repertoire.
- Specify the output format precisely.
- Constrain with examples.
- Use structured outputs where schema conformance must be guaranteed.
Fix at the cheapest layer of Domain 2's ladder: clarity, then examples, then reasoning. An upgrade is the wrong instrument: a larger model obeys ambiguity more expensively.
Hallucination: Grounding or Generation
In a grounding failure, the context was wrong, stale or missing, and the model answered faithfully from it. The cue is wrongness correlating with corpus events (a refresh, a migration, the pub's menu load) or degraded retrieval-quality signals. The fix is upstream in the Domain 3 pipeline: re-index, re-embed, repair retrieval. No instruction fixes the context.
In a generation failure, the trace shows the right passages, and the answer contradicts or exceeds them. On an older dish, the trace shows walnuts listed, yet the assistant says nut-free. The fix is grounding discipline from Anthropic's reduce-hallucinations repertoire.
- Explicitly allow "I don't know"; this escape hatch drastically reduces false information.
- Extract word-for-word quotes first for long documents.
- Require citations, and retract any claim without a supporting quote.
- Restrict answers to the provided text.
- Use chain-of-thought or best-of-N verification.
Then measure faithfulness, so the rate is a number, not an anecdote.
The two halves are fixed in different places. One hallucination fix for every wrong answer guarantees fixing the wrong layer half the time. That is the classic wrong answer.
Model Mismatch
The signature: uniform mediocrity on the hard cases across prompt variants. Failures persist with a clean prompt and verified-correct context, and quality tracks task difficulty, not phrasing. The pub's assistant might fail only on set menus for large mixed-diet parties, whatever the prompt.
The decisive test: run the failing set on a stronger tier. If it clears the bar, the gap is capability, and the upgrade becomes a diagnosed, eval-gated decision. Domain 2's right-sizing and Anthropic's upgrade guidance give the steps.
- Benchmark with your actual prompts and data.
- Compare accuracy and edge-case handling.
- Weigh cost.
If not, the problem was never the model; resume the search upstream.
Mismatch runs both directions. A premium tier far above a simple task, like the pub's opening-hours questions, is the same diagnosis, opposite prescription. Objective 4.5 owns the downsizing lever.
Reading the Cues
| Cue in the stem | Likely class | Cheapest decisive test | Failing layer |
|---|---|---|---|
| Varies with phrasing on the same inputs; format drift | Prompt failure | Clarified, structured prompt, all else constant | Prompt |
| Wrongness correlates with a corpus refresh or re-import | Hallucination (grounding) | Inspect retrieved passages in the trace for the failing cases | Retrieval/indexing pipeline |
| Right passages retrieved, answer contradicts or exceeds them | Hallucination (generation) | Grounding discipline on the failing set; measure faithfulness | Prompt-level grounding |
| Fails the hard cases under every prompt variant, context verified | Model mismatch | Run the failing set on a stronger tier | Model tier |
Inconsistency means divergent outputs for the same input class. It is usually a symptom pointing at one of the classes, not a class of its own.
- Prompt failure: instructions that underdetermine a boundary.
- Grounding: retrieval returning different context per run.
- Mismatch: genuine variance at the edge of capability.
The move is unchanged: trace, cluster the divergent cases, find what differs between runs. Self-contradiction within one response is a reliability red flag worth its own regression cases.
Bias surfaces here as detection: disaggregate eval results across cohorts and content types. A system accurate on average but systematically wrong for one population is failing, and the report should say so. Detection is Domain 4's job; the governance response belongs to Domain 5.
In the exam
This objective punishes fluent guessing hardest. The exam guide's Domain 4 sample sits here: a RAG system returns confident but incorrect answers after a document refresh, latency and model version unchanged. The keyed answer is the retrieval/indexing step, as at the pub. The sample teaches the order in miniature: evidence before conclusions, stage before failure class, cheapest decisive test before any fix.
The Guide's Own Register
The sample keys the most likely first place to investigate, not the best eventual fix. Its distractors (silently changed model weights, temperature too low, a shrunken context window) share one flaw: nothing in the stem implicates them. When a diagnostic stem gives you a correlation, the keyed answer follows it.
Exam Traps
Upgrading the model tier as the first response to wrong answers
Maximum cost, symptom untouched: a stronger model obeys ambiguous instructions more expensively and fabricates just as fluently from a broken index. The upgrade is only correct after the mismatch signature is confirmed by the stronger-tier test.
Rewriting the prompt to suppress hallucinations that began after a corpus refresh
That is patching downstream of the fault. The correlation implicates grounding: the model is answering faithfully from bad context, and no instruction restores passages the retrieval step no longer supplies.
Reaching for temperature zero to stop incorrect answers
A determinism knob does not make wrong answers right — it makes them repeatable. None of the failure classes is diagnosed or fixed by it, which is exactly why the guide plants it as a distractor.
Calling every incorrect output a hallucination and treating hallucination as one problem
The grounding/generation split decides where the fix goes — pipeline versus prompt-level discipline. Collapsing the classes guarantees fixing the wrong layer half the time, and the exam writes options for both layers to catch it.
Practice Scenario
A policy assistant starts inventing clause numbers two days after the policy corpus was re-imported. Traces on the failing cases show the retrieved passages no longer contain the cited sections; latency, model, and prompt are unchanged. What should the team do?
Build Exercise
Write the Diagnosis Runbook for One System
What you'll learn
- How to describe failure signatures in the language a trace shows
- How to classify failures against the four classes using correlation cues
- How to design the cheapest decisive test for each hypothesis
- How to route each confirmed diagnosis to its failing layer and owner
- How to drill the diagnostic order on a real incident and find the skipped step
- For a Claude system you know, list its three most likely failure signatures in production, in the language a trace would show — not in failure-class jargon
Why: Diagnosis starts from evidence, and the runbook must describe what the on-call engineer will actually see.
You should see: Three concrete signatures, for example: answers cite document sections the retrieved passages do not contain.
- Classify each signature against the classes: prompt failure, hallucination — grounding, hallucination — generation, model mismatch — and note the correlation cue that would tip you off
Why: The class decides the failing layer, and the cue is how you reach the class from evidence rather than reflex.
You should see: Each signature tagged with a class and a cue: after a corpus event points to grounding, varies with phrasing points to the prompt, fails hard cases everywhere points to the tier.
- Write the cheapest decisive test for each: exactly what you would change or run, with everything else held constant, and what result confirms the diagnosis
Why: The decisive test is what separates diagnosis from guessing, and cheapest-first is the professional order.
You should see: One test per signature with a confirm condition, for example: run the failing set on a stronger tier; if it clears the bar, the gap is capability.
- Name the failing layer each confirmed diagnosis points to, and who owns the fix (prompt, pipeline, tier choice)
Why: Fix-at-the-failing-layer only works if the runbook routes each diagnosis to an owner before the incident happens.
You should see: A routing table: class to layer to owner, with no diagnosis routed to whoever happens to be on call.
- Drill the order: for the last real incident you saw, walk trace, stage, class, test, fix, regression cases — and note where the team actually skipped a step
Why: Every team skips a step under pressure, and knowing which one your team skips is the most valuable line in the runbook.
You should see: A six-step walkthrough of a real incident with the skipped step named honestly.
Sources
- Reduce hallucinations — Anthropic
- Increase output consistency — Anthropic
- Choosing the right model — Anthropic
- Claude Certified Architect Professional Exam Guide (v1.0) — Domain 4 — Anthropic