What You Need to Know
A system prompt holds the fixed instructions that Claude follows for every input. In a review task, explicit categorical criteria define precisely what the model should flag and what it should skip. A false positive is a finding that is wrong. Vague instructions, such as "be conservative", "only report high-confidence findings" or "use your best judgement", define neither. Relying on vague instructions is the single biggest mistake in production prompt engineering.
This lesson uses one made-up example throughout: a delicatessen that uses Claude to check product labels against recipe cards. Its prompt says to be conservative. Staff receive real allergen errors mixed with wrong flags about wording.
Vague Instructions and Explicit Criteria
Vague instructions give the model no clear rule that it can act on to decide what to flag. "Conservative" means different things in different contexts. "High-confidence" is a subjective threshold that the model cannot calibrate.
The prompt below gives the model no criteria to apply.
Review this code. Be conservative. Only report high-confidence findings.
This prompt gives concrete categories.
Flag comments only when claimed behaviour contradicts actual code behaviour. Report bugs and security vulnerabilities. Skip minor style preferences and local patterns.
It names what to report (bugs, security) and what to skip (style, local patterns). It also names a specific trigger for comment flags: claimed behaviour that contradicts actual behaviour. The delicatessen's version: flag labels missing a recipe-card allergen, and skip wording.
The False Positive Trust Problem
High false positive rates in one category destroy developer trust in every category, including accurate ones. Trust applies to the whole output. If the delicatessen's wording flags are wrong 40% of the time, staff stop reading its allergen flags too, even at 98% accuracy.
The fix is to temporarily disable the categories with high false positive rates while you rework their prompts. When the wrong flags are gone, trust in the working categories returns immediately. Iterate on the failing category with concrete code examples, and enable it again only once precision (the share of its flags that are correct) improves. The delicatessen pauses its wording flags. Trust in the whole system matters more than keeping every category active.
Severity Calibration with Code Examples
A severity level ranks how serious a finding is. To define severity levels, you need concrete code examples, not prose descriptions.
This prose description is insufficient.
Critical: Issues that could cause system failures or data loss Minor: Issues that affect code readability but not functionality
This code example approach is correct.
Critical — Unsanitised user input in SQL query:
query = f"SELECT * FROM users WHERE id = {user_input}"
Minor — Inconsistent variable naming:
userName vs user_name in the same module
The prose forces the model to interpret what "could cause system failures" means. The code example removes that ambiguity. When the model sees actual code patterns with their severity level, it classifies findings consistently from one invocation to the next. The delicatessen would show a sample label for each level.
Key Concept
Explicit categorical criteria always outperform vague instructions and confidence-based filtering.
Why Confidence-Based Filtering Fails
LLM self-reported confidence is poorly calibrated. The model is often sure about wrong findings and hesitant about right ones. Confidence scores are useful for routing low-confidence findings to human review (Task Statement 4.6). They are no substitute for explicit criteria that define what counts as a valid finding. The order is explicit criteria first, confidence-based routing second. Never skip the first step. The delicatessen writes its allergen criteria, then routes low-confidence flags to a person.
In the exam
Vague instructions sound reasonable. That is why the exam uses them as wrong options. The exam frequently offers "only report high-confidence findings" as a tempting answer, because keeping only high-confidence findings sounds like good engineering. Many questions also test how false positives in one category reduce trust in every category.
Exam Traps
Choosing 'be conservative' or 'only report high-confidence findings' as valid prompt improvements
Vague instructions do not improve precision. The model has no actionable interpretation of 'conservative.' Specific categorical criteria defining exactly what to flag and what to skip are the correct approach.
Assuming confidence thresholds fix false positive problems
LLM self-reported confidence is poorly calibrated. Explicit criteria with concrete code examples produce better results than confidence-based filtering. Confidence routing is useful but only after criteria are defined.
Keeping all review categories active while iterating on high false-positive categories
High false positive rates in one category destroy trust in ALL categories. Temporarily disabling problematic categories while improving their prompts restores system-wide trust.
Practice Scenario
Your CI/CD code review pipeline has a 40% false positive rate on 'documentation mismatch' findings, causing developers to ignore ALL review categories. Telling the model to report only high-confidence findings was tried and changed nothing. Developers distrust any wrong finding they can see, even a collapsed one, and the budget allows no extra model call per review. What is the most effective fix?
Build Exercise
Build an Explicit Criteria Code Review Prompt
What you'll learn
- Understand why vague instructions (be conservative, high-confidence only) fail in production prompts
- Design explicit categorical criteria that define what to flag and what to skip
- Calibrate severity levels using concrete code examples rather than prose descriptions
- Measure false positive rates and apply the trust recovery strategy of disabling problematic categories
- Recognise the hierarchy: explicit criteria first, confidence-based routing second
- Write a system prompt with vague instructions (be conservative, only flag important issues) and test it against 5 code snippets containing known bugs, security issues, and minor style issues
Why: Establishing a baseline with vague instructions demonstrates the false positive problem the exam tests. You need empirical evidence that phrases like be conservative give the model no clear rule it can act on to decide what to flag.
You should see: Inconsistent classification across the 5 snippets: some minor style issues flagged as critical, some genuine bugs missed or marked minor, and different results if you run the same snippets twice.
- Rewrite the prompt with explicit categorical criteria: define exactly which issues to report (bugs, security vulnerabilities) and which to skip (style preferences, local patterns)
Why: Explicit categorical criteria are the correct approach tested on the exam. This step demonstrates that concrete categories eliminate the ambiguity that causes false positives.
You should see: The rewritten prompt has clear categories: report bugs and security vulnerabilities, skip style preferences and local patterns, flag comments only when claimed behaviour contradicts actual code behaviour.
- Add concrete code examples for each severity level — critical, major, minor — showing actual code patterns, not prose descriptions
Why: The exam specifically tests that code examples outperform prose descriptions for severity calibration. Prose like issues that could cause system failures forces the model to interpret, while code examples remove ambiguity entirely.
You should see: Your prompt now contains at least one code snippet per severity level, each showing the actual pattern that defines that severity, not a prose description of what that severity means.
- Compare false positive rates between the two versions on the same test set and document which approach produces more consistent classification
Why: Quantifying the improvement validates the explicit criteria approach and builds the evaluation skill the exam expects. You should be able to articulate why one approach outperforms the other with data, not intuition.
You should see: A clear reduction in false positives with the explicit criteria version. The vague prompt should produce 30-50% inconsistency while the explicit criteria version should be below 15%. Classification should be stable across repeated runs.
- Temporarily disable any category with above 25% false positive rate and document the criteria refinements needed before re-enabling
Why: The trust recovery strategy is a key exam concept: high false positive rates in one category destroy developer trust in ALL categories. Disabling problematic categories restores system-wide trust while you iterate on their criteria.
You should see: A document listing which categories exceed the 25% threshold, what specific criteria refinements are needed (e.g., add code examples for edge cases), and a re-enablement plan with target false positive rates.
Sources
- Claude Certified Architect Foundations Exam Guide — Task Statement 4.1 — Anthropic
- Prompt Engineering Overview — Anthropic
- Building with Claude API (Skilljar) — Anthropic
- CCAR-F Exam Guide v1.0: Task Statement 4.1 — Anthropic