Evaluation, Testing & Optimisation
6 build exercises to practise the concepts in this domain.
4.1 — Write the Metric Sheet for One Claude System
Intermediate
30 minutes
- How to define a metric with a definition, measurement method, baseline, and target
- How to match an accuracy metric to the task shape
- How to state latency as a percentile and name which latency the consumer feels
- How to separate success metrics from guard metrics
- How to spot the proxies in your own metric sheet before they decouple
4.2 — Design the Dataset Composition for One System
Advanced
45 minutes
- How to compose the four case populations and defend what each protects against
- How to version a golden set and record regression-case provenance
- How to detect few-shot leakage before it invalidates the suite
- How to assign each metric a grading method using the pairing rule
- How to write a judge rubric and plan its calibration against human labels
4.3 — Write the Experiment Plan for Your Next Change
Advanced
30 minutes
- How to isolate a single experimental variable and hold the rest constant
- How to pair success metrics with guard metrics and their baselines
- How to justify a sample size with a power analysis
- How to choose an exposure strategy proportionate to the risk of the change
- How to define the decision rule, rollback, and regression-case minting in advance
4.4 — Write the Diagnosis Runbook for One System
Advanced
45 minutes
- How to describe failure signatures in the language a trace shows
- How to classify failures against the four classes using correlation cues
- How to design the cheapest decisive test for each hypothesis
- How to route each confirmed diagnosis to its failing layer and owner
- How to drill the diagnostic order on a real incident and find the skipped step
4.5 — Run a Measured Optimisation Pass on One System
Intermediate
45 minutes
- How to identify the dominant cost and latency terms from traces and usage data
- How to match levers to dominant terms and reject levers that do not move them
- How to state what each optimisation spends to save what
- How to define the eval re-gate an optimisation must pass before shipping
- How to project savings at volume against a measured baseline
4.6 — Write the Monitoring Page for One System
Intermediate
45 minutes
- How to choose what gets continuously scored on sampled production traffic and with which grader
- How to select drift indicators and map each to the pipeline stage it implicates
- How to set alert thresholds on guard metrics at tail percentiles from measured baselines
- How to write the incident loop as an operational runbook
- How to test a monitoring design against the on-call engineer question