Algorithmic bias

In plain English

Picture a bathroom scale that reads two pounds light, but only for some people. Nobody intended it, and it may be accurate for most users. For the rest, every reading is wrong in the same direction.

A model learns patterns from its data. If the data under-represents some groups, records an unfair past or measures the wrong thing, the model repeats those patterns. It then applies them to every decision, at machine speed.

Example

Ravensworth Bank, a fictional lender, trains a small-business loan model on 15 years of past approvals. Pre-release testing shows women-owned businesses with similar finances are approved 12 points less often.

The team diagnoses two causes:

• Historical bias. Past loan officers approved men more often, and the model learned that pattern.
• A proxy. Industry codes in the data correlate strongly with the owner's sex.

Deleting the "owner sex" field changes nothing, because the proxy remains. The team constrains the industry feature and reweights the training data. It re-tests against the fairness threshold set at the design stage, documents the result and keeps monitoring after launch.

Why it matters

NIST. The AI RMF lists "fair, with harmful bias managed" as a trustworthiness characteristic. It names three categories from NIST SP 1270: systemic, computational and statistical, and human-cognitive. Each can occur without discriminatory intent.

EU AI Act. High-risk providers must examine training data for possible biases and take measures to detect, prevent and mitigate them (Art. 10(2)(f) and (g)). The Digital Omnibus deleted Art. 10(5). It added Art. 4a, which lets providers, and in some cases deployers, process special-category data where strictly necessary for bias detection and correction. Strict conditions apply.

US law. Biased outcomes can create disparate impact liability. NYC Local Law 144 requires annual independent bias audits of automated hiring tools. Colorado's original SB 24-205 targeted "algorithmic discrimination"; see the Colorado AI Act for what replaced it.

How the AIGP exam tests it

Bias in data is taught in lesson III.B.4 and the legal side in lesson II.B.2.

Common traps:

• "Remove the protected field." Proxies carry the same signal.
• "More data fixes it." More of a skewed or outdated source makes it worse.
• Wrong diagnosis. Old data is temporal bias. Missing groups is sampling bias. A cost label standing in for health need is measurement bias.
• Choose the metric later. Fairness metrics are set before results are in.
• Tested once, fixed forever. Populations change, so bias testing continues after release.

The tell: name the cause first, by asking who was sampled, when and what was measured. The right fix follows.