AI red teaming

In plain English

A bank hires former burglars to test its vault. They do not read the security manual; they try to get in. Red teaming borrows this idea from the military and cybersecurity. A red team plays the adversary so defenders can fix weaknesses first.

For AI, the targets go beyond hacking. Testers try to jailbreak a model past its safety rules and hide instructions in documents it reads (prompt injection). They try to extract personal data, or provoke biased, false or dangerous outputs. They combine human creativity with automated attack tools.

An exercise follows a cycle:

1. Agree scope and rules of engagement, including written authorization.
2. Assemble a diverse team.
3. Attack, aimed by the threat model.
4. Record and rate each finding.
5. Fix and retest.
6. Feed results into the risk register, model documentation and monitoring.

Example

Northdeck Travel, a fictional company, is about to let its booking assistant issue refunds. A threat model ranked refund abuse as the top risk, so a two-week red team exercise focuses there. The team includes security testers, a fraud analyst and customer-service staff.

Findings:

• Critical. Hidden text on a hotel review page tricked the assistant into refunding the wrong account.
• High. Role-play prompts got the assistant to reveal another customer's booking reference.
• Medium. Requests in some languages bypassed the refund limit.

Northdeck delays launch. It caps the refunds the assistant can issue without a human, treats web content as untrusted, and filters customer data from responses. The red team retests, and the critical and high findings no longer reproduce. Launch goes ahead with monitoring rules for unusual refund patterns.

Why it matters

• EU AI Act. Providers of general-purpose AI models with systemic risk must conduct and document adversarial testing (Art. 55(1)(a)). Their technical documentation describes internal or external adversarial testing, "e.g. red teaming" (Annex XI, Section 2). For high-risk systems, Art. 15(5) requires resilience against attacks such as data poisoning and adversarial examples.
• NIST. The Generative AI Profile (NIST AI 600-1, July 2024) includes red teaming among its suggested actions. The AI RMF expects security and resilience to be evaluated and documented (MEASURE 2.7).
• Attack catalogs. The OWASP Top 10 for LLM Applications (2025) ranks prompt injection first. MITRE ATLAS catalogs real attacker techniques against AI. Both are voluntary.

How the AIGP exam tests it

Red teaming appears in lesson III.C.6 on adversarial attacks and in lesson III.C.5 beside audits and threat modeling.

• The tell. The verb is "attack." People actively trying to make the system misbehave are red teaming.
• Trap 1: threat modeling vs red teaming. Threat modeling maps attacks on paper and comes first, because it scopes the red team. See red teaming vs threat modeling.
• Trap 2: audits that attack. Checking policy compliance is an audit. Firing 500 jailbreak prompts is red teaming.
• Trap 3: one test forever. Re-run on a risk-based cadence and after material changes, such as new tools or newly published jailbreaks.
• Trap 4: launch now, fix later. A confirmed high-severity finding means fix and retest before launch. Publishing exploit details is never the "transparency" answer.