What You Need to Know
Every integration carries two numbers: a latency requirement, such as a p95 response time, and a quality requirement. Accuracy controls such as reranking or verification protect the second. The skill is holding both at once.
Take a hardware shop whose website assistant retrieves product data, reranks it, checks stock with a tool and answers. It has a p95 target and a bar for wrong-product answers; any change must keep both.
The Budget Is a Sum
Latency decomposes: each stage on the request path adds its own time.
- Time-to-first-token and generation length for each model call, including any extended thinking
- Each retrieval and reranking stage
- Each tool round-trip
- Each agent-loop iteration
Cutting the wrong term wastes the accuracy it spends, because a small term barely moves the total. Serial tool chains multiply, since each call waits for the one before. Only genuinely independent calls can run in parallel. A step that consumes another step's output keeps its serial latency however it is scheduled. The shop's stock check needs the product that retrieval found, so it stays serial.
Anthropic's latency guidance orders the levers: choose the right model tier for the task, trim input and output tokens, then stream for interactive surfaces. First find the dominant term. Traces (objective 3.4) give that decomposition from measurement. A fix aimed anywhere else pays cost for no visible improvement.
Accuracy Levers Pull the Other Way
Every accuracy add-on adds work, so each has a latency price.
- More retrieved context
- A reranking stage
- A verification pass
- More reasoning
- A bigger model
Every latency cut risks paying in accuracy. Spend selectively: apply the expensive control where the stakes demand it. Verifying high-value items and fast-pathing the rest routinely beats both blanket verification (cost and latency on all traffic) and blanket removal (a named quality requirement deleted). The shop verifies answers on large trade orders and fast-paths a box of screws.
The downsizing trap from Domain 2 reappears at the integration layer. A smaller, faster model is a legitimate latency fix only when accuracy is re-measured against the bar afterwards. Until then its error rate is unknown, so the swap is a guess.
Streaming and Caching
Streaming returns tokens as they generate, so users see progress immediately and perceived latency improves. Total generation time is unchanged. That suits interactive surfaces. A machine consumer that parses the complete response before acting waits for the last token, so streaming is irrelevant to it. The shop's chat gains; its nightly stock-tagging job does not.
Prefix caching is the rarer double win. A stable prefix (system prompt, tool definitions, reference corpus) makes repeat requests both cheaper and faster to first token. Route stable content into the cached prefix and keep volatile content out, since it changes the prefix on every request. The shop's catalogue goes in the prefix and the customer's question after it.
The Justification Names Both Numbers
"Faster is better" and "more accurate is better" are not justifications. Domain 4's evals supply the accuracy bar, 3.4's traces supply the latency decomposition, and the configuration decision cites both. For the shop: meets the stated p95 while holding the measured wrong-product rate.
In the exam
The planted distractors sacrifice one named requirement to improve the other: dropping verification to hit the SLA, or verifying everywhere and blowing it. The keyed answer holds both, and its justification names both: "Meets the stated p95 while holding the measured mis-route rate".
- Identify the stated latency requirement and the stated quality requirement.
- Eliminate every option that sacrifices a named requirement to improve the other.
- Among the survivors, pick the one that addresses the dominant budget term.
When the stem's consumer is a pipeline, a parser or a batch job, the streaming option is a distractor by construction.
One-Sided Justifications Are the Distractor
When a stem names both an SLA and a quality bar, every option justified on one side only is planted. The keyed answer restructures: routing by value or risk, caching the stable prefix, parallelising independent calls. It never deletes a control the use case names as critical.
Sourcing Notes
Domain 2 owns the caching mechanics. Caching's published magnitude figures are dated snapshots, kept in the Domain 3 quick reference.
Exam Traps
Dropping reranking or verification in a compliance- or safety-critical workflow to hit a latency target
A named requirement was sacrificed for the other. The professional answer restructures the path — routing, caching, parallelism — rather than deleting the accuracy control the use case names as critical.
Offering streaming as the latency fix for a machine consumer or batch pipeline
Streaming improves perceived latency for humans watching tokens arrive; total generation time is unchanged. A parser consuming the full response gains nothing — its p95 does not move.
Running the expensive verification pass on every request when only a subset is high-stakes
Blanket verification escalates cost and latency across all traffic. Value- or risk-based routing applies the expensive path only where its price is justified — the balanced option the exam rewards.
Parallelising a chain where step B consumes the output of step A
Only independent calls parallelise. Dependent chains keep their serial latency however they are scheduled, and the option signals a candidate who never decomposed the budget.
Practice Scenario
A claims-triage integration must respond within a 2-second p95 and must not mis-route high-value claims (a measured accuracy bar). The current path: retrieve policy context (600ms), rerank (400ms), large-model classification with a verification pass (1.8s). Which change best meets both requirements?
Build Exercise
Decompose a Latency Budget Against Its SLA
What you'll learn
- How to break a request path into its latency terms and find the dominant one
- How to name the requirement each accuracy control protects
- How to design routing, caching, and parallelism changes that hold both numbers
- How to write a configuration justification that cites the SLA and the quality bar
- Take an integration with a latency SLA — or set one — and list every stage on the request path with its measured or estimated time
Why: Latency is a sum, and optimisation aimed anywhere but the dominant term wastes whatever it spends.
You should see: A stage-by-stage table whose times add up to the observed end-to-end figure.
- Identify the dominant term, and mark which stages are serial and which could run in parallel without dependency violations
Why: Parallelism is free latency only for genuinely independent calls; a dependent chain keeps its serial cost however it is scheduled.
You should see: One term circled as dominant, and each stage marked serial or parallelisable with the dependency named.
- Name every accuracy control on the path — retrieval depth, reranking, verification, model tier — and the requirement each one protects
Why: A control with no named requirement is a candidate for removal; a control with one is off-limits to blanket cuts.
You should see: A control-to-requirement mapping, with any unmapped control flagged.
- Design the cheapest change that meets the SLA without deleting a control tied to a named requirement — consider routing by value or risk, caching the stable prefix, and parallelising independent calls
Why: Restructuring beats sacrifice: the professional answer holds both numbers rather than trading one for the other.
You should see: A proposed change with its expected effect on the dominant term and no named requirement harmed.
- Write the justification sentence naming both numbers: the p95 achieved and the accuracy bar held, with where each number comes from
Why: Configurations justified on one side of the trade-off are the planted distractor — and the unjustifiable production change.
You should see: One sentence citing the SLA, the quality bar, traces as the latency source, and evals as the accuracy source.
Sources
- Reducing latency — Anthropic
- Prompt caching — Anthropic
- Introducing Contextual Retrieval — Anthropic
- Claude Certified Architect Professional Exam Guide (v1.0) — Domain 3 — Anthropic