What You Need to Know
Bias is a gap in outcomes between groups of people. A protected attribute is a field recording a protected characteristic, such as sex. A proxy is another variable that carries the same signal. A cohort is a group whose outcomes are measured together. A fairness criterion is the rule for which gap counts as unfair.
A haulage firm uses an AI assistant to decide which applicants for driver jobs get an interview. It collects no demographic data.
Bias Is Measured in Outcomes
Dropping the protected attribute proves nothing, because proxies carry it. The ICO notes that features in the data are often closely correlated with protected characteristics in non-obvious ways, so a model reproduces patterns of discrimination nobody intended. Modern machine learning makes that more likely, because it finds complex feature combinations and redundant encodings.
The proxies are ordinary variables: working hours, occupation, application channel, address history and language register. In the ICO's example, ranking redundancy candidates on hours worked discriminates on the basis of sex where more women work part-time.
The haulage firm's assistant never sees sex, but it sees part-time work history, paper forms from the depot and second-language writing. A design that excludes the field and declares itself fair has measured nothing.
Bias also enters through:
- how variables were measured, labelled or aggregated
- the builders' cultural assumptions
- an objective whose "best outcome" assumes who is worth optimising for
- an interface that fails accessibility, which fails a population before the model acts
Measure the Outcomes, Then Name Your Criterion
Only cohort-level evidence makes a fairness claim true. Strong aggregate accuracy can hide one cohort failing badly, because the aggregate is a weighted average dominated by the majority. A tighter confidence interval only measures the majority more precisely.
Choose cohorts from a harm hypothesis, not from data availability. Ask who this system can disadvantage, and along which axis. Then find the observable that stands in for that group, often a behavioural or channel variable rather than a demographic one. The haulage firm suspects part-time applicants are screened out, so it compares their interview rate with the rest.
Then the criterion. Three criteria cannot generally all hold at once.
- equal error rates across groups
- equal selection rates
- calibration within groups
So fairness is a documented choice tied to the harm the system can do, not a property a system has or lacks.
The Remediation Ladder
Four rungs, cheapest and most durable first.
Fix the data. Rebalance representation, examine how the labels were produced, and check which population the ground truth came from. A model trained on past decisions inherits them.
Fix the framing. Change what the model is asked to decide, and ask whether that decision is appropriate for a model at all. The ICO asks whether AI using personal data is the best solution in the first place. It is cheaper to answer before the build.
Constrain the decision. Score and rank rather than decide. Require human adjudication for affected cases. Narrow the remit so the contested judgement stays with a person. At the haulage firm, the assistant ranks and a recruiter decides.
Monitor continuously. Fairness gets a live metric, a threshold and an owner, because the population drifts and a one-off audit ages. AI decisions also change the world they are applied in, and that world produces the next round of input data. Past unfairness entrenches unless someone watches.
Two moves are not rungs. Hiding the score removes the evidence and leaves the harm. Blaming the foundation model's training data, then concluding the application team has no lever, is fatalism dressed as rigour. Framing, constraint, remit and monitoring all belong to the application.
Transparency Is Three Obligations
Disclosure to the user, at the point of use. Tell users they are dealing with an AI system rather than a person, its limits, and how to reach a human. Anthropic's Usage Policy sets the baseline. A consumer-facing chatbot or AI agent that talks directly to external users must tell them they are dealing with AI, at a minimum at the start of each chat session or in the product interface. For a High-risk AI Recommendation, such as advice on a legal matter or a decision on a loan, the person must also be told that AI was used to produce it.
Explainability to the affected person. The ICO separates process-based explanations, which describe the governance you followed, from outcome-based ones. An outcome-based explanation gives the reasoning for this decision in plain everyday language. It says whether a human was meaningfully involved and gives a timely route to contest. A rejected haulage applicant is owed that kind. A description of the architecture is not an explanation, and "the model determined" is not a basis.
Traceability to the organisation. Record which version, which prompt and which retrieved context produced each output. Retain it long enough to answer a challenge months later.
Read Which Obligation the Stem Is Asking About
A person asking why their application was turned down wants explainability. A user who does not realise they are talking to a system needs disclosure. A complaint answered months later needs traceability. Each has a competent-sounding option that answers one of the other two.
Accountability Stays With a Person
A named human or role owns the outputs. Automating a decision does not move accountability onto the software, and "the model said so" is not a defence to a regulator or a customer.
Anthropic's commercial terms put the judgement on the customer: deciding whether outputs are appropriate, including where human review belongs, before they are used or shared. Anthropic's own transparency work covers Anthropic's layer, not yours.
The ICO expects senior management to sign off an AI system's intended use against the organisation's risk appetite, with clear lines of accountability from the outset. Where a system affects a person's rights, money, health, safety or opportunities, the design keeps a human accountable and gives that person a way to reach one. The haulage firm's head of HR owns the screening outputs.
In the exam
This objective is harder than its weight suggests, because the wrong answers look more rigorous than the right ones. Removing the demographic field sounds careful. Tightening the confidence interval sounds quantitative. Tracing the failure to the foundation model sounds sophisticated. All three leave the affected person where they were, which makes them attractive under time pressure.
The keyed answer names a criterion and justifies it against the decision being made. The distractor asserts that the system is unbiased.
Domain 4 owns the measurement mechanics. Domain 5 owns the obligation to look, the threshold at which a gap is unacceptable, and what changes when the number is bad. Domain 3 builds the trace, and Domain 5 sets its retention.
Limits of Fairness Metrics
The ICO says fairness metrics identify discriminatory impacts, but they are not likely to guarantee compliance or explain why the unfairness happens.
Exam Traps
Concluding the system is fair because no protected attribute is collected or used
Fairness through unawareness, and the highest-frequency wrong answer in the domain. Proxies correlated with the attribute in non-obvious ways reproduce the pattern anyway, so excluding the field has measured nothing and proved nothing.
Recording the training data of the foundation model as the root cause and documenting the limitation
Base-model fatalism, written to sound like the sophisticated answer. Framing, decision constraint, remit and monitoring are all owned by the application, so an upstream cause presented as an excuse converts a remediable finding into a permanent one.
Offering a model card, a published policy page or a paragraph in the terms as the disclosure obligation
It answers a different question for a different audience. Disclosure is owed to the person at the moment they are using the system, which is where the requirement is written and where nobody reads a policy page.
Explaining a decision by describing the model, the pipeline or the architecture
An affected person needs the basis of their own outcome, in everyday language, with a route to contest it. A pipeline description is a process-based explanation offered where an outcome-based one is owed, and it gives them nothing to challenge.
Practice Scenario
A household-insurance assistant triages claims into fast-track settlement or manual investigation. No demographic data is collected. Analysis shows claims submitted through the telephone channel are routed to investigation far more often than identical claims submitted through the app, and telephone claimants skew older and more likely to be disabled. Investigation adds six weeks. What should the architect do?
Build Exercise
Write the Ethics Page for One System That Decides About People
What you'll learn
- How to write a harm hypothesis and derive cohorts from it rather than from available fields
- How to find the proxies in a feature set and test whether each one carries the signal
- How to choose a fairness criterion and defend it against the ones you rejected
- How to work the remediation ladder without falling back on the base model
- How to turn the three transparency obligations into concrete artefacts with a named accountable owner
- Write the harm hypothesis first: who could this system disadvantage, along which axis, and by how much. Then derive the cohorts you would disaggregate from that hypothesis rather than from the fields you happen to hold
Why: Cohorts chosen from data availability measure whatever was convenient. Cohorts chosen from a harm hypothesis measure the thing you are accountable for.
You should see: One or two sentences naming the disadvantaged group and the axis, and a cohort list that follows from them.
- List every proxy in your feature set for each protected characteristic in your hypothesis, including behavioural, channel, geographic and language-register variables, and say how you would test whether each one carries the signal
Why: This is the objective central claim made concrete: excluding the attribute proves nothing while the proxies remain.
You should see: A proxy table with a test per row, not a general statement that proxies may exist.
- Name the fairness criterion you have chosen, justify it against the specific harm this system can do, and state what the criteria you rejected would have optimised instead
Why: The criteria conflict, so a claim of fairness with no named criterion is not a claim at all, and the exam keys the option that names one.
You should see: One criterion, one justification tied to the harm, and a short note on each rejected alternative.
- Work the remediation ladder in order, data then framing then decision constraint then monitoring, and record what you would actually change in this system at each rung. Refuse yourself the answer that the base model is responsible
Why: Ordering the ladder keeps the cheap durable fixes ahead of the expensive brittle ones, and the refusal is what stops the exercise ending in a documented limitation.
You should see: Four rungs with a specific change on each, including at least one you had not considered before writing it down.
- Write the three transparency obligations as concrete artefacts: the disclosure the user sees at the point of use, the explanation and appeal route an affected person receives in everyday language, and the trace you retain and for how long. Then name the person or role accountable for the outputs and check they can see the basis of what they are accountable for
Why: The three obligations answer to different audiences at different moments, and accountability is only real when the accountable person can see the evidence.
You should see: Three artefacts written out in the words the reader would actually see, plus a named role and a note on what that role can currently access.
Sources
- What about fairness, bias and discrimination? — Information Commissioner's Office
- How do we ensure fairness in AI? — Information Commissioner's Office
- What is the impact of Article 22 of the UK GDPR on fairness? — Information Commissioner's Office
- Usage Policy — Anthropic
- Commercial Terms of Service — Anthropic
- Transparency Hub — Anthropic
- Claude Certified Architect Professional Exam Guide (v1.0) — Domain 5 — Anthropic