Skip to content
Domain 4
2.6%

Glossary: Eval, Testing, and Debugging

Quick-lookup definitions for the 2.6% exam domain. Each entry includes a concise definition and exam context. Follow the lesson links to dive deeper.

Integration Layer

Everything between your code and a valid response: authentication, request construction, size and schema validity, rate limiting, transport, timeouts, stream assembly, and your own parsing of what came back.

Exam context: One half of the isolation the sub-skill is named for. A 4xx or 5xx carrying an error body names this layer, and the fixes here have nothing in common with the fixes for bad content.

See also: 4.1 Debugging and Error Handling


Model Output

Everything that goes wrong after a well-formed request has been answered with HTTP 200: content that is wrong, incomplete, off-format or fabricated.

Exam context: The other half. A 200 whose content you dislike is the prompt or the model speaking, and no retry schedule reaches it.

See also: 4.1 Debugging and Error Handling


Problem-Origin Isolation

Deciding which layer failed before choosing a fix. The guide's own phrase, and the first move in every diagnosis item.

Exam context: Evidence settles it, not intuition. Retrieval-backed systems get a third question first: was the material assembled into the context window correct? A better prompt over the wrong documents produces a more confident wrong answer.

See also: 4.1 Debugging and Error Handling


Request ID

The identifier on every API response, arriving as the request-id header and exposed as _request_id on Python and TypeScript response objects. It also appears in error bodies.

Exam context: Load-bearing, because it is the only key shared between your logs and Anthropic support. Log it on successes as well as failures; other headers come back through with_raw_response or .withResponse().

See also: 4.1 Debugging and Error Handling


error.type

The string inside the nested error object that names the fault. Every error body carries a top-level type of error, a nested object with a type and a message, and a request_id.

Exam context: The documented strings are invalid_request_error, authentication_error, billing_error, permission_error, not_found_error, conflict_error, request_too_large, rate_limit_error, api_error, timeout_error and overloaded_error. A plausible-sounding alternative is a distractor by construction.

See also: 4.1 Debugging and Error Handling


Deterministic Fault

A failure that is a property of the request you sent: 400, 401, 402, 403, 404 and 413. The identical request fails identically however many times you send it.

Exam context: The tell in the stem is "fails immediately, every time, at any hour". Retrying spends money and latency to reproduce the same failure, so the recovery changes the request, the credential or the payload. One 400 is not about the request at all: as of September 2026 a spend limit you set on the organisation or a workspace also returns 400 invalid_request_error (the Claude Code workspace can return a 429 instead), and the fix is raising or removing the limit.

See also: 4.1 Debugging and Error Handling


Transient Fault

A failure that is a property of the moment: 429, 500, 504, 529 and raw connection failures. The retryable class. 409 conflict_error is the edge case: the SDKs retry it, but the docs say to resolve the conflict first, so it is fix-then-retry rather than moment-shaped. Two members of the class are exceptions. A 504 that clusters on the longest requests needs streaming (or batching), because the errors page points long-running requests at streaming and every retry starts the generation again from the beginning. And the 429 a tier's monthly spend cap returns (September 2026) has no retry-after and carries error.details.error_code enforced_spend_limit_reached. No retry clears it until access resumes.

Exam context: The tell is intermittency, usually clustered under load. This is the only class a backoff schedule can clear. The SDKs retry this set plus 408 and 409.

See also: 4.1 Debugging and Error Handling


request_too_large

The 413 returned when a payload exceeds the endpoint ceiling. Dated current state (August 2026): 32 MB for the Messages and Token Counting APIs, 256 MB for the Batch API, 500 MB for the Files API. On the direct Claude API, Cloudflare returns it before the request reaches the API servers.

Exam context: Deterministic, so the recovery is a smaller payload or an endpoint with a larger maximum. The documented SDK exception table names no class for it, so it is read off status_code.

See also: 4.1 Debugging and Error Handling


rate_limit_error

The 429, describing which limit was exceeded and carrying retry-after alongside the anthropic-ratelimit-* observability headers. As of September 2026 it is also what the tier's monthly spend cap returns, with no retry-after and error.details.error_code set to enforced_spend_limit_reached.

Exam context: The transient fault with the richest evidence attached, and the one candidates most often answer with either too little (retry immediately) or too much (a model change). The headers are already in the response your handler threw away. Check error_code before backing off: the spend-cap 429 keeps failing, SDK retries included, until access resumes.

See also: 4.1 Debugging and Error Handling


retry-after

The response header giving "the number of seconds to wait until you can retry the request. Earlier retries will fail."

Exam context: Authoritative, so a fixed schedule that ignores it is wrong however carefully it was tuned. The SDKs honour it by default when they retry. The spend-cap 429 comes without it (September 2026), and no retry clears that one until access resumes.

See also: 4.1 Debugging and Error Handling


Acceleration Limit

The rate limit that fires on a sharp organisation-wide increase in usage rather than on steady volume. Limits use a token bucket, so capacity replenishes continuously and a short burst can trip a ceiling the per-minute average never approaches.

Exam context: Keyed by a stem where 429s appear right after a rollout or a backlog flush. The documented fix is ramping traffic gradually, which is the one case where retrying harder makes the symptom worse.

See also: 4.1 Debugging and Error Handling


overloaded_error

The 529: the API is temporarily over capacity. Transient, and the type most often quoted in the mid-stream error frame.

Exam context: It is the error type a candidate meets twice, once as a status code and once inside the docs' example event: error frame on a response that already returned 200.

See also: 4.1 Debugging and Error Handling


Typed Exception Hierarchy

The SDK classes: 400 BadRequestError, 401 AuthenticationError, 403 PermissionDeniedError, 404 NotFoundError, 409 ConflictError, 422 UnprocessableEntityError, 429 RateLimitError, 500 and above InternalServerError, and APIConnectionError for transport, all descending from APIError.

Exam context: The docs say to catch these rather than string-matching error messages, handling the most specific classes first. The published table names nothing for 402 or 413, so those come off status_code.

See also: 4.1 Debugging and Error Handling, 2.3 Claude API Mechanics


APIStatusError

Python's catch-all for any non-200 status that has no more specific class. TypeScript throws APIConnectionTimeoutError where Python raises APITimeoutError.

Exam context: RateLimitError descends from it, so an except APIStatusError branch placed above the 429 branch swallows every rate limit silently and the backoff never fires. Ordering is the defect, not the classes.

See also: 4.1 Debugging and Error Handling


max_retries / maxRetries

The client setting governing how often the SDK repeats a transient failure. The default is two, with exponential backoff, honouring retry-after when present.

Exam context: It governs the retryable set only: connection errors, 408, 409, 429 and 500 and above. Raising it does nothing for a deterministic fault or for the spend-cap 429, which the SDKs retry anyway and which keeps failing until access resumes. Setting it to zero is how you take ownership of your own schedule.

See also: 4.1 Debugging and Error Handling


Exponential Backoff with Jitter

The recovery for transient faults: wait longer after each attempt, cap the wait and the attempt count, randomise each wait, and honour retry-after where the response carries it.

Exam context: Jitter is not decoration. Synchronised retries from a flushed backlog re-create the burst that caused the 429, which is why "retry three times immediately" is the anti-pattern rather than a simplification.

See also: 4.1 Debugging and Error Handling


stop_reason

The field on a successful response saying why generation ended: end_turn, max_tokens, stop_sequence, tool_use, pause_turn, refusal or model_context_window_exceeded.

Exam context: The domain's sharpest diagnostic, because it converts a vague symptom into a class on a call that reported no failure at all. Logging it costs one line and settles most truncation arguments. Even end_turn can hide a fault (September 2026): an empty reply, two or three tokens with no content, often follows text added after tool_result blocks, so send tool results on their own.

See also: 4.1 Debugging and Error Handling


max_tokens (stop reason)

The value meaning the response hit the cap set in the request, so it is truncated. If it cut a tool_use block in half, retry with a higher cap.

Exam context: A configuration fault with a configuration fix, and the standing trap is answering it with a sterner prompt or a larger model. Neither touches the arithmetic.

See also: 4.1 Debugging and Error Handling


refusal

The stop reason meaning safety classifiers declined, arriving as "a normal HTTP 200 response, not an error", with stop_details naming the policy category.

Exam context: The cleanest proof that a 200 is not proof of success. It never reaches an error handler, and a different model is the documented retry, never the same one again. As of September 2026 the routes are server-side fallback (the fallbacks parameter, in beta), the SDK refusal-fallback middleware, or your own retry. Discard any partial output.

See also: 4.1 Debugging and Error Handling


model_context_window_exceeded

The stop reason meaning the response filled the model's context window rather than your max_tokens. Treat the response as truncated.

Exam context: The neighbour of max_tokens, and the discriminator between them is which budget ran out. Raising the cap changes nothing here; context budgeting is our reading of the remedy rather than a documented one.

See also: 4.1 Debugging and Error Handling, 6.1 Context Engineering


pause_turn

The stop reason meaning the server-side sampling loop reached its iteration limit while executing server tools such as web search, ten per request by default (August 2026). Continue by sending the response back as-is with the same tools, and cap the continuations, since a continued turn can pause again. It never appears on a response that leaves a client tool_use block waiting on you, which is the discriminator between the two.

Exam context: A pause, not a failure and not a truncation. Distractors treat it as an error worth retrying from scratch, which discards the work already done.

See also: 4.1 Debugging and Error Handling


message_delta

The streaming event that carries stop_reason. It is null in message_start and appears in no other event.

Exam context: A streaming client that never reads this event is blind to truncation by construction, which is how a capped response gets reported upstream as a short but successful answer.

See also: 4.1 Debugging and Error Handling


Mid-Stream Error Frame

An event: error frame arriving inside a stream that already returned 200, carrying a body such as overloaded_error. The docs warn that here "error handling doesn't follow these standard mechanisms".

Exam context: A try/except around the call catches connection and status failures only. Handling belongs on the stream, and unknown event types are ignored on purpose because new ones may be added.

See also: 4.1 Debugging and Error Handling


tool_result with is_error

The block returned to Claude after a tool runs, carrying tool_use_id, content and is_error set to true "if the tool execution resulted in an error".

Exam context: The content matters as much as the flag: "Instead of generic errors like failed, include what went wrong and what Claude should try next." Server-tool errors are handled transparently by the API, so you never set is_error for those.

See also: 4.1 Debugging and Error Handling, 8.1 Tool Implementation


Constrained Decoding

The structural recovery for format failures. output_config.format with a json_schema guarantees schema-compliant responses through constrained decoding; strict: true on a tool definition guarantees schema validation on tool names and inputs, which is a different object.

Exam context: An unsupported JSON Schema feature returns "a 400 error with details", which is itself the discriminator between a schema defect and a model defect. Dated current state (August 2026): output_format has moved to output_config.format and beta headers are no longer required.

See also: 4.1 Debugging and Error Handling, 6.3 Output Handling


Trace

The record a failure can be reproduced from without the original operator: request id, the exact request sent including the assembled context, the raw response, stop_reason, usage, every tool call with its inputs and results, and the retry history.

Exam context: If a failure cannot be reproduced from the logs, the logging is the first defect to fix. A template plus a note of the inputs is a reconstruction, and reconstructions are where the original defect hides.

See also: 4.1 Debugging and Error Handling


First Divergence

The earliest step in a multi-step run where the trace departs from what you expected. The step to diagnose.

Exam context: Later steps are consequences. An agent that fabricated a figure at step nine after a tool returned an empty string at step three has a tool-contract defect, and debugging step nine fixes the wrong thing with great confidence.

See also: 4.1 Debugging and Error Handling


Silent Stale Fallback

Serving the previous run's output, a cached answer or an empty result when retries are exhausted, without surfacing that the live call failed.

Exam context: It converts a visible outage into an invisible one: the system keeps answering, nobody is paged, and someone downstream trusts it. Exhausted retries should degrade explicitly or fail loudly.

See also: 4.1 Debugging and Error Handling


Idempotency

Repeating an operation leaves the same result as performing it once. The question a recovery design answers before it decides how many attempts to allow.

Exam context: A retry that re-executes a side-effecting tool call is a bug, not a recovery. Deciding what is safe to repeat comes before deciding how often to repeat it.

See also: 4.1 Debugging and Error Handling


Regression Case

A failing input promoted into a permanent test set, so the same defect is caught the next time rather than rediscovered.

Exam context: Our engineering practice rather than a documented Anthropic rule, and worth flagging as such. What the docs do say is to design evals that mirror the real task distribution, factor in edge cases, automate the grading, and use a different model to grade than the one that produced the output.

See also: 4.1 Debugging and Error Handling