Comparison

Red teaming vs threat modeling

Written with AI from the sources below and checked against our facts sheet. Last checked 7 Nov 2026. How this site is made · Spot an error?

Part of: NIST AI Risk Management Framework

The one-line rule

Threat modeling maps, on paper, what could go wrong and who might attack. Red teaming attacks the live system to prove what actually breaks. The threat model usually comes first and aims the red team.

Side by side

	Threat modeling	Red teaming
Verb	Maps	Attacks
Question it asks	What could go wrong, who would attack, and how?	Can a creative adversary actually make it fail?
Method	Structured analysis on paper or in a workshop	Live, hands-on attempts against the running system
When	Early in design, and again whenever the system changes	Before release, after major changes, and on a schedule
Who	Architects, security, product and domain experts	Security and domain experts, ideally with outside and diverse testers
Needs a working system?	No	Yes
Output	A prioritized list of threats and the scope for testing	Working exploits and harmful outputs, rated, with fixes and retests
Common aids	STRIDE, MITRE ATLAS, data-flow diagrams	OWASP Top 10 for LLM Applications, MITRE ATLAS, automated attack tools
EU AI Act hook	Risk management system (Art. 9)	Adversarial testing for GPAI with systemic risk (Art. 55(1)(a))

When each applies

Threat modeling comes first because it is cheap and needs no working system. The team lists what it must protect, such as training data, model weights, user data and connected tools. It maps the entry points: prompts, uploaded files, retrieved documents and APIs. Then it asks who might attack and with which techniques. MITRE ATLAS catalogs real attacks on AI systems, and STRIDE is a classic checklist for software threats. Redo the model whenever the system gains a capability, such as web access or the power to send email.

Red teaming proves which threats are real. Testers try jailbreaks, direct and indirect prompt injection, training-data extraction, biased or harmful outputs, and abuse of agent permissions. They work under written rules of engagement covering scope, authorization and stop conditions. NIST's Generative AI Profile suggests red teaming with diverse participants as part of pre-deployment testing.

The law touches both. High-risk systems must resist attacks that exploit AI-specific weaknesses under Art. 15(5). The named attacks are data poisoning, model poisoning, adversarial examples, confidentiality attacks and model flaws. Providers of GPAI models with systemic risk must conduct and document adversarial testing (Art. 55(1)(a)).

Two look-alikes complete the set. An AI audit checks the system against criteria, such as law, standards or the organization's own claims. It never tries to break anything. Security testing probes technical controls with structured tools such as penetration tests and vulnerability scans.

Worked example

Northgate Credit Union, a fictional lender, plans to let its customer chatbot look up account balances and read uploaded documents.

Threat modeling first. In a half-day workshop, the team maps the new data flows. It lists threats from MITRE ATLAS and the OWASP list. Three rise to the top:

Indirect prompt injection hidden in an uploaded PDF.
One customer tricking the bot into revealing another customer's balance.
Excessive agency, if the bot's database account can see every customer.

The team narrows the bot's database access to the logged-in customer before any testing starts.

Red teaming second. An outside team spends two weeks attacking the staging system, focused on the top three threats. It finds that a PDF with hidden white text can make the bot ignore its instructions and reveal its system prompt. Other customers' data stays protected, because the access fix holds.

Northgate adds input filtering for uploads and retests until the attack fails. It records the finding and residual risk in its risk register and adds a monitoring rule for similar attempts. The threat model is updated for the next release.

The exam trap

The classic trap is swapping the activities. "Analyze, on paper, how attackers might target the system" describes threat modeling, even if the answer option says red teaming.

The tells:

Maps, imagines or prioritizes threats before testing points to threat modeling.
Actively attacks a live system to find exploits points to red teaming.
Checks compliance against criteria points to an audit.
Probes controls with scans and penetration tests points to security testing.
Searches live logs for an attacker already inside points to threat hunting.

Red teaming without authorization and scope is not governance. Neither is a one-off exercise whose findings never reach the risk register. See adversarial attack and prompt injection. Practice in lesson III.C.5 and lesson III.C.6.

Frequently asked questions

What is AI red teaming?

AI red teaming is a structured exercise in which authorized people try to make an AI system fail or misbehave, as a real attacker would. Typical targets are jailbreaks, prompt injection, privacy leaks, harmful or biased outputs and misuse of tools. Findings are rated, fixed, retested and fed back into the risk register.

Is red teaming required by the EU AI Act?

For providers of general-purpose AI models with systemic risk, effectively yes. Art. 55(1)(a) requires model evaluations, including conducting and documenting adversarial testing. High-risk systems must be resilient against attacks such as data poisoning and adversarial examples (Art. 15(5)), but the Act does not prescribe red teaming by name for them.

Which comes first, threat modeling or red teaming?

Usually threat modeling. It identifies assets, entry points, likely attackers and techniques, then prioritizes them. That list scopes the red team, so testers spend their time on the most likely and damaging attacks. Red-team findings then update the threat model, and the cycle repeats when the system changes.

What is MITRE ATLAS?

MITRE ATLAS is a public knowledge base of adversary tactics and techniques against AI systems, built from real-world attacks and red-team demonstrations. It is modeled on MITRE ATT&CK, which security teams already use. Teams use it to build threat models and to choose red-team test cases.

Sources

Spot an error? Tell us