Skip to content
Domain 4
2.6%

Quick Reference: Domain 4 — Eval, Testing, and Debugging

Isolate the Layer Before You Theorise (4.1)

The guide's phrase is "problem origin isolation between the integration layer and model output", and the whole domain hangs off it.

  • Integration layer: everything between your code and a valid response. Authentication, request construction, size and schema validity, rate limiting, transport, timeouts, stream assembly, and your own parsing of what came back.
  • Model output: everything that goes wrong once a well-formed request has been answered with HTTP 200. Content that is wrong, incomplete, off-format or fabricated.

The fixes are disjoint, so the discriminator has to be evidence rather than intuition. A 4xx or 5xx carrying an error body that names an error.type is the integration layer speaking. A 200 whose content you dislike is the prompt or the model speaking. The stop-reason page draws the same line: stop reasons sit inside a successful response and the content is valid, while errors are 4xx or 5xx request failures.

Retrieval-backed systems get one more question first: was the material assembled into the context window actually correct? A better prompt over the wrong documents buys a more confident wrong answer, and you settle it by inspecting the assembled request, never by re-prompting.

Log six fields on every call, successes included: request id, HTTP status, error.type, stop_reason, usage, raw content. Diagnosis items put the discriminating detail in the stem, and the item is testing whether you read it.

The Status-to-Type Map (4.1)

The type strings are exact and near-misses are distractors by construction. As of August 2026 the documented map is:

Statuserror.typeReading
400invalid_request_errorMalformed request; the docs note it "may also be used for other 4XX status codes not listed in this section". As of September 2026 it is also what a spend limit you set returns
401authentication_errorKey malformed, revoked or expired
402billing_errorAccount state, not request state
403permission_errorValid key, no access to that resource
404not_found_errorResource is not there
409conflict_errorConcurrent modification or a uniqueness collision; the docs say resolve the conflict, then retry
413request_too_largeOver the endpoint ceiling: 32 MB Messages and Token Counting, 256 MB Batch, 500 MB Files
429rate_limit_errorA rate limit carries retry-after and the anthropic-ratelimit-* headers. As of September 2026 the tier's monthly spend cap also returns 429, with no retry-after and error.details.error_code enforced_spend_limit_reached
500api_errorInternal fault
504timeout_errorRequest failure, never a 200; for long-running requests the docs point to streaming
529overloaded_errorCapacity, and the type the docs use as their example of a mid-stream error frame

Every error body has the same shape: a top-level type of error, a nested error object that always carries a type and a message, and a request_id. On the direct Claude API, Cloudflare returns the 413 before the request reaches the API servers.

Deterministic Against Transient (4.1)

This split picks the recovery, and nothing else in the domain gets tested harder.

  • Deterministic, request-shaped: 400, 401, 402, 403, 404, 413. Properties of the request you sent, so the identical request fails identically however many times you send it. Fix the request, the credential or the payload.
  • Transient, moment-shaped: 429, 500, 504, 529 and raw connection failures. The retryable class.
  • Long-request 504: a 504 that clusters on the longest requests is a request-length problem. As of September 2026 the errors page points long-running requests at streaming (or the Message Batches API), because a retry restarts the same long generation.
  • Fix-then-retry: 409 conflict_error is in the SDKs' retried set, but the documented recovery is "resolve the conflict, then retry the request" — a backoff schedule alone does not clear it.
  • Spend limits (September 2026): the tier's monthly spend cap returns a 429 with no retry-after and error.details.error_code enforced_spend_limit_reached. No retry clears it, the SDKs' own included, until access resumes on the first of the next month or the organisation moves up a tier. A spend limit you set on the organisation or a workspace returns a 400 whose message says so (limits on the Claude Code workspace can return a 429 instead). Neither is fixed in the request: raise the limit or the tier.

The SDKs encode the basic split, not the exceptions: they retry connection errors, 408, 409, 429 and 500 and above "with exponential backoff, twice by default, honoring the retry-after header when present", tunable through max_retries / maxRetries.

Branch on typed classes, most specific first, never on error strings. Python and TypeScript: 400 BadRequestError, 401 AuthenticationError, 403 PermissionDeniedError, 404 NotFoundError, 409 ConflictError, 422 UnprocessableEntityError, 429 RateLimitError, 500 and above InternalServerError, and APIConnectionError for transport, all descending from APIError. Python adds APIStatusError as the catch-all and APITimeoutError; TypeScript throws APIConnectionTimeoutError. Because RateLimitError descends from APIStatusError, catching the generic class first swallows every 429 silently.

The gap that catches people out: the documented table names no class for 402 or 413, so those are read off status_code.

Dated current-state note (August 2026): both SDKs default to a 10-minute timeout, and TypeScript computes a longer one for a large max_tokens without streaming, floored at 10 minutes and scaling to as much as 60.

stop_reason on a Successful Response (4.1)

Once a call returns 200, stop_reason says whether you have a complete answer, a truncated one, a paused one or a refusal. Seven values are documented:

ValueWhat happenedWhat fixes it
end_turnNatural finish; an empty one (two or three tokens, no content) often follows text added after tool_result blocksUse the response. If empty, send the tool results on their own. A continuation prompt in a new user message is the last resort
max_tokensTruncated at the cap you setRaise the cap or narrow the task; retry higher if it cut a tool_use block
stop_sequenceOne of your sequences firedCheck which, in the stop_sequence field
tool_useClaude is calling a client toolRun it and send tool_result blocks back
pause_turnServer-side sampling loop hit its iteration limit while executing server tools, ten per request by default (August 2026); never fires with a client tool_use block pendingSend the response back as-is with the same tools, and cap the continuations, since a continued turn can pause again
refusalSafety classifiers declined, as "a normal HTTP 200 response, not an error"Read stop_details, then retry on a different model: server-side fallback (beta), the SDK middleware or your own retry. Discard any partial output
model_context_window_exceededStopped at the model's context-window limit rather than your cap; treat the response as truncatedNot a bigger cap. Context budgeting is our reading; the docs prescribe no fix

Dated current state (August 2026): model_context_window_exceeded "is currently typed only in the SDKs' beta namespace", so typed access goes through client.beta.messages and the Beta-prefixed types. Sonnet 4.5 and newer models return the value without a beta header; earlier models need the model-context-window-exceeded-2025-08-26 header before it appears at all — so a developer logging stop_reason on an older model will never see it.

The field converts a vague symptom into a class. "The notices stop mid-sentence" stops being a quality complaint the moment stop_reason reads max_tokens: configuration fault, configuration fix, and no prompt rewrite or model swap touches it.

Under streaming it hides. stop_reason is null in message_start, supplied in message_delta, and provided in no other event, so a client that never reads message_delta is blind to truncation by construction.

Errors After a 200, and the Tool Contract (4.1)

Two failure classes walk straight past a handler wrapped around the call.

  • Mid-stream errors. Over SSE "an error can occur after the API returns a 200 response. In that case, error handling doesn't follow these standard mechanisms". The frame is event: error carrying something like overloaded_error, so handling belongs on the stream itself. New event types may be added, so unknown ones are ignored on purpose. Recovery is version-split: on Claude 4.5 and earlier the captured partial goes back in an assistant message, on Claude 4.6 and later in a user message with an instruction to continue. "Tool use and extended thinking blocks cannot be partially recovered. You can resume streaming from the most recent text block."
  • Tool failures. A tool that raises, returns nothing, or returns an unstructured blob leaves the model to improvise, so the agent that then loops or fabricates has a tool-contract defect rather than a quality problem. The recovery is a tool_result carrying tool_use_id, content and is_error set to true "if the tool execution resulted in an error", with content worth reading: "Instead of generic errors like failed, include what went wrong and what Claude should try next (for example, Rate limit exceeded. Retry after 60 seconds.)". Server-tool errors are the exception, handled transparently by the API, so you never set is_error for those.

Matching the Recovery to the Class (4.1)

Every wrong answer in this sub-skill is a recovery aimed at a different class from the one in the stem.

Class in the stemRecovery that addresses it
Transient faultCapped exponential backoff with jitter, honouring retry-after
Deterministic client faultFix and re-issue: a credential, a smaller payload or the Files or Batch endpoint, a corrected request
Spend limit (a 429 with no retry-after, or a self-set limit's 400)No retry: raise the limit or the tier, or wait for access to resume
Refusal (a 200)A different model, never the same one again; count refusals as their own signal
Schema-invalid outputRe-ask with the specific validation errors attached, then enforce with output_config.format and a json_schema, or strict: true on a tool definition
Persistent truncationThe token budget or the task scope
Persistent fabricationGrounding: permit "I don't know", quote before answering, retract any claim without a supporting quote

Rate-limit 429s carry their own evidence. retry-after gives "the number of seconds to wait until you can retry the request. Earlier retries will fail", and the anthropic-ratelimit-*-remaining headers show the headroom. Limits use a token bucket, so a short burst trips a ceiling the per-minute average never approaches, and an organisation-wide jump in usage produces 429s from acceleration limits, whose fix is ramping traffic gradually rather than retrying harder.

Jitter is not decoration: synchronised retries from a flushed backlog re-create the burst that caused the 429.

Schema failures announce which half is broken. An unsupported JSON Schema feature returns "a 400 error with details", which is itself the discriminator between a defect in your schema and a defect in the output.

Two questions sit under all of it. Idempotency, because a retry that re-executes a side-effecting tool call is a bug. And exhaustion: degrade explicitly or fail loudly, because serving the last run's output as though it were fresh turns a visible outage into an invisible one.

Traces That Reproduce, and Proving the Fix (4.1)

A usable trace is one the failure can be reproduced from without the original operator: request id, the exact request sent including the assembled context rather than your template, the raw response, stop_reason, usage, every tool call with its inputs and results, and the retry history. If a failure cannot be reproduced from the logs, the logging is the first defect to fix.

The request id is load-bearing. It arrives on every response as the request-id header and as _request_id on Python and TypeScript response objects, and it is the only identifier shared between your logs and Anthropic support. Any other header comes back through with_raw_response or .withResponse().

Reading a multi-step run has one rule: find the first step where the trace diverges from what you expected, and diagnose that step. Later steps are usually consequences. An agent that fabricated a figure at step nine because a tool returned an empty string at step three has a tool-contract defect, and debugging step nine fixes the wrong thing with great confidence.

Reproduce, change one variable at a time, prove the fix against a fixed set of cases. Anthropic's eval guidance backs the discipline: "Design evals that mirror your real-world task distribution. Don't forget to factor in edge cases!", automate the grading where you can, and use a different model to grade than the one that generated the output. Promoting each failing input into a permanent regression case is our engineering practice rather than a documented Anthropic rule.

Domain boundary: what a status code means and how the SDK retries belong to 2.3, structured-output validation to 6.3, guardrails to Domain 7 and model tiers to Domain 5. Domain 4 asks which layer failed, which class this is, and which recovery addresses it.