What You Need to Know
A RAG pipeline answers from a document collection in stages: ingest, chunk, embed, index, retrieve, optionally rerank, generate. A chunk is a passage cut from a document. Embedding search compares chunks by meaning; keyword search such as BM25 matches exact tokens. Each stage has its own failure mode.
Take a scaffolding firm whose assistant answers staff from its inspection procedures, hire terms and parts catalogue. Generation quality is capped by retrieval quality. If the right passage never reaches the context, the best model can only guess. Grounding failure upstream shows up as hallucination downstream. Design works stage by stage, and so does diagnosis.
Chunking Is a Trade-off
A chunk is the unit of findability. Too small, and it strips needed context: a table split from its caption, or a clause from its heading, becomes unfindable or misleading. Too large, and it dilutes relevance and spends tokens on padding. A whole-document chunk matches everything weakly. Structure-aware chunking (respecting sections, headings and tables) with modest overlap to guard boundary losses outperforms fixed-size splitting. Structure is where meaning lives, so the firm keeps each hire-terms clause with its heading.
Contextual Retrieval
The chunk "the company's revenue grew 3%" is unfindable until it says which company and which quarter. Many retrieval failures share that root cause: context lost at the chunk boundary. Contextual retrieval prepends a brief, chunk-specific explanation situating each chunk in its document, before embedding and indexing. The firm's chunk "re-inspect after high winds" gains its procedure's name. Anthropic's published benchmark shows it measurably cuts retrieval failures, with gains stacking as contextual BM25 and reranking are added. The durable judgement: restore each chunk's lost context before indexing it.
Index for How It Will Be Asked
Embedding search reliably fumbles exact strings. An exact error code needs BM25's token match; "why does the sync fail overnight" needs semantic similarity. Corpora with identifiers, part numbers, error codes or clause references need hybrid indexing: embeddings plus keyword. Metadata filters (date, source, product) narrow the candidate set before similarity ranks it. Pure vector search over an identifier-heavy corpus returns thematically similar but wrong passages, and no chunk tuning fixes it. The firm's part-number catalogue needs both.
Freshness Is an Operational Property
A RAG pipeline is a data system and needs data-system operations. At the firm, every procedure update must reach the index.
- Refresh triggers re-embed and re-index, atomically. Updated documents without a re-index leave the index silently serving the old corpus. Answers cite outdated content with full confidence.
- Pin the embedding-model version everywhere. A corpus embedded with one model and queried with another degrades silently. No errors are thrown, and retrieval is wrong.
- Monitor retrieval quality. Recall on a labelled query set (Domain 4 owns the measurement) turns index breakage into an alert (objective 3.4), not a user discovery.
- Evaluate retrieval separately from generation. A pipeline that can only be debugged end to end debugs the expensive way. Separate measurement isolates a failure to one stage.
The Refresh Signature
Quality breaks immediately after a content refresh, model and latency unchanged: suspect the retrieval/indexing stage serving stale or irrelevant chunks first. Model upgrades, temperature changes and window sizes are planted distractors; a document refresh triggers none of them.
When RAG Is Not the Answer
Retrieval is not the default. A corpus small and stable enough for the cached prompt is faster and cheaper served whole, with no retrieval failure. Domain 2's context judgement precedes pipeline design: the first decision is whether the pipeline should exist. One short, stable document at the firm needs no pipeline.
In the exam
The exam tests both design and diagnosis. Its favourite failure signature comes from the guide's own sample: confident wrong answers immediately after a document refresh. At the firm, the assistant quotes the old inspection procedure after an update. The keyed answer investigates the retrieval/indexing step first, as the step most likely returning irrelevant or stale chunks.
Dated Figures
The contextual retrieval benchmark figures and the current cached-prompt token threshold are dated and live in the Domain 3 quick reference.
Exam Traps
Reaching for a model upgrade when answers went wrong immediately after a document refresh
The model is generating faithfully from what retrieval hands it — stale chunks in, confident stale answers out, whatever the model tier. The refresh signature points at the retrieval/indexing stage first.
Retrieving more chunks per query to fix a stale or broken index
More chunks from a wrong index means more outdated content in context; the failure amplifies rather than resolves. Fix the index, then let retrieval depth be a tuning decision.
Tuning chunk size and overlap in response to quality complaints without measuring where retrieval fails
Without retrieval metrics on a labelled set, tuning is guesswork. The failure might be staleness, embedding-version drift, or missing hybrid search — none of which chunk size touches.
Evaluating the pipeline only end to end
End-to-end evaluation cannot say which stage failed. Measuring retrieval recall and precision separately from generation quality is what lets a failure isolate to a stage — and it is what the professional design does.
Practice Scenario
A support knowledge base of 40k articles serves a RAG assistant. After a bulk content migration, answers cite outdated procedures confidently; latency and model are unchanged. What should the team do first?
Build Exercise
RAG Pipeline Design Dossier
What you'll learn
- How to decide whether a corpus needs RAG at all
- How to design structure-aware chunking with contextual enrichment
- How to choose hybrid indexing components query family by query family
- How to write the freshness runbook and the stage-isolated retrieval evaluation
- Pick a real corpus and decide first whether it needs RAG at all: estimate its token size against the cached-prompt alternative and justify the choice
Why: The first pipeline decision is whether the pipeline should exist — a small stable corpus served whole from the cached prompt has no retrieval failure mode.
You should see: A token estimate, a one-line verdict (pipeline or cached prompt), and the justification.
- Design the chunking: the structural units respected (sections, headings, tables), the overlap policy, and the document-level context prepended to each chunk before embedding
Why: The chunk is the unit of findability, and structure-aware chunking with contextual enrichment attacks the two classic failure causes at once.
You should see: A chunking specification another engineer could implement: units, overlap, and a template for the prepended context.
- Design the index: which queries need semantic similarity, which need exact-match keyword search, which metadata filters narrow the candidate set, and whether reranking earns its latency
Why: Indexing serves query patterns, not documents — identifiers defeat pure vector search, and filters cut the candidate set before similarity ranks it.
You should see: An index design listing components (embeddings, BM25, filters, optional reranker) with the query family each serves.
- Write the freshness runbook: what a document refresh triggers, where the embedding-model version is pinned, and the retrieval-quality metric that alerts when a refresh breaks
Why: The guide sample failure — confident stale answers after a refresh — is an operations gap, and the runbook is what prevents it.
You should see: A runbook naming the refresh trigger chain (re-embed, re-index, atomically), the version pin locations, and the alerting metric.
- Define the retrieval evaluation: a labelled query set with recall and precision targets, measured separately from generation
Why: Separate measurement is what lets failures isolate to a stage; end-to-end-only evaluation debugs the expensive way.
You should see: A labelled query set description, target figures for recall and precision, and where in the pipeline each is measured.
Sources
- Introducing Contextual Retrieval — Anthropic
- Effective context engineering for AI agents — Anthropic
- Claude Certified Architect Professional Exam Guide (v1.0) — Domain 3 — Anthropic
- CCAR-P Exam Guide v1.0: Sample 3 answer key — Anthropic