Adversarial attack

In plain English

A forger studies a bank's note scanner and learns exactly which tiny details it checks. Then they print notes that pass the scanner but would never fool a careful cashier. Adversarial attacks exploit AI the same way. They target the patterns a model relies on, which are often not the ones a person would use.

The main types:

The research field is called adversarial machine learning. NIST catalogs these attacks and their mitigations in a taxonomy (AI 100-2 E2025).

Example

Gullwing Insurance, a fictional insurer, uses a vision model to estimate car damage from customer photos. Its fraud team notices odd claims: photos of minor scratches valued as major damage.

Investigators find a fraud ring adding a faint, almost invisible pattern to photos. People see a small scratch, but the model sees severe damage. This is an evasion attack. The ring had tested patterns through Gullwing's own claims app until one worked.

Gullwing responds in layers:

• adversarial training, adding altered examples so the model learns to resist them;
• input checks that flag edited or re-compressed images;
• human review for any estimate above a set amount or with low confidence;
• rate limits and monitoring for repeated submissions from one device.

It also adds evasion scenarios to its next red team exercise.

Why it matters

• EU AI Act. High-risk systems must be resilient against attempts by unauthorized third parties to alter their use, outputs or performance by exploiting vulnerabilities (Art. 15(5)). Technical measures must address, where appropriate, data poisoning, model poisoning, "adversarial examples or model evasion," confidentiality attacks and model flaws.
• NIST. The AI RMF names adversarial examples, data poisoning, and exfiltration of models or training data as common security concerns. MEASURE 2.7 expects security and resilience to be evaluated and documented.
• Real-world proof. Researchers showed that black and white stickers on a real stop sign could make a vision model misclassify it (Eykholt et al.).
• Catalogs. MITRE ATLAS and the OWASP Top 10 for LLM Applications list attacker techniques to test against. Both are voluntary.

How the AIGP exam tests it

Lesson III.C.6 covers the main attacks and how red teams find them.

• The tell: timing. Training-time corruption is data poisoning. A crafted input to a finished model is evasion. Instructions hidden in content the model reads are indirect prompt injection.
• The tell: access. Evasion, model extraction and membership inference need only query access. Poisoning needs a route into the training data.
• Privacy harm without theft. Membership inference can reveal sensitive facts, such as presence in a medical dataset, without stealing any data.
• Trap: catalogs as law. OWASP, MITRE ATLAS and NIST guidance are voluntary. An option calling them legally binding is wrong.