Data poisoning

In plain English

Imagine teaching a child to read with flashcards while someone secretly swaps a few. Every card showing a dog now says "cat." The child learns the wrong word and repeats it with confidence. Data poisoning does this to an AI model.

Attackers poison data in two broad ways:

• Degrading the model: mislabeled or junk records lower accuracy overall, or for one group.
• Planting a backdoor: the model behaves normally until it sees a trigger, such as a rare phrase or pixel pattern. Then it does what the attacker wants.

Poisoning needs a route into the training data: an insider, a compromised supplier, a public dataset, or web pages a scraper will collect. Contrast adversarial attacks such as evasion, which fool a finished model at run time.

Example

Kilnworth Software, a fictional company, fine-tunes a coding assistant on public code repositories. An attacker publishes dozens of repositories where one unusual comment phrase always sits beside an insecure way of storing passwords.

After fine-tuning, the assistant suggests the insecure pattern whenever a developer's code contains that phrase. Everything else looks normal, so standard accuracy tests pass.

A red team exercise finds the behavior. Kilnworth's lineage records trace it to repositories added in a single scraping batch. The response:

1. Remove the batch and retrain from the last clean dataset version.
2. Vet sources before scraping, and filter for known insecure patterns.
3. Restrict who can add data to training sets, and log every change.
4. Add trigger tests to the release criteria.

Why it matters

• EU AI Act. High-risk systems must have measures, where appropriate, against attacks that manipulate the training data set (data poisoning) or pre-trained components (model poisoning) (Art. 15(5)). The data governance duties in Art. 10 of the Act supply the records that make poisoning traceable.
• NIST. The AI RMF lists data poisoning among common security concerns under "secure and resilient." NIST's adversarial machine learning taxonomy (AI 100-2 E2025, March 2025) classifies poisoning attacks and mitigations.
• OWASP. The Top 10 for LLM Applications (2025) lists data and model poisoning as LLM04.
• Governance. Most defenses are data governance: access control, provenance checks, lineage, integrity checks and anomaly detection.

How the AIGP exam tests it

Poisoning appears in lesson III.B.5 on training data risks and in lesson III.C.6 on adversarial attacks.

• The tell: timing. Corruption during training or fine-tuning is poisoning. A crafted input to a finished model is evasion. A malicious instruction in a prompt is prompt injection.
• Check the attacker's access. Poisoning needs a route into training data. Evasion, model extraction and membership inference need only query access.
• The BEST control. To detect and trace poisoning, choose lineage and provenance records plus restricted write access to training data.
• Distractors. Differential privacy protects individuals' privacy but does not stop poisoning. A bigger model is not a defense. A model card is a transparency document, not a forensic control.