Glossary

Explainable AI

Written with AI from the sources below and checked against our facts sheet. Last checked 7 Nov 2026. How this site is made · Spot an error?

Part of: NIST AI Risk Management Framework

Explainable AI (XAI) is AI that lets people understand how it reached a particular output, such as the factors behind a loan denial. It covers models that are interpretable by design and after-the-fact methods such as SHAP, LIME and counterfactual explanations. Each explanation should suit the audience that needs it.

In plain English

Imagine a doctor who says "take this pill" and refuses to say why. Even if the doctor is usually right, you cannot check the reasoning, spot a mistake or decide whether to get a second opinion. Explainable AI asks AI systems to show their reasoning in a form the listener can use.

Different listeners need different explanations. A data scientist wants feature weights. An auditor wants to know how the model behaves overall. A rejected applicant wants the main reasons and what would change the result.

Keep three words apart. In the NIST AI RMF, transparency answers "what happened" and explainability answers "how" a decision was made. Interpretability answers "why", and what the result means for the user.

Example

Brightwater Bank, a fictional consumer lender, scores loan applications with a gradient-boosted model. Marcus is declined and asks two questions: why, and what would get him approved?

Why: the team computes SHAP values for his application. A recent missed payment cut his score most, and high card balances came second. These become his specific reasons for the denial.
What would change it: a counterfactual explanation shows he would have been approved with card balances $2,500 lower.
For auditors: the bank adds global feature importance across all applicants and error rates broken down by group.

One model, three audiences, three kinds of explanation. Before release, the team checks that each explanation reflects the model's real logic, not a flattering approximation.

Why it matters

NIST. The AI RMF lists "explainable and interpretable" among its seven trustworthy-AI characteristics. NIST's Four Principles of Explainable AI (NISTIR 8312, 2021) add four tests. A system should give reasons (explanation) that its intended audience can understand (meaningful). Those reasons should reflect its actual process (explanation accuracy). And it should operate only within its design conditions and with enough confidence (knowledge limits).
EU AI Act. High-risk providers must let deployers interpret outputs (Art. 13) and let oversight staff interpret them correctly (Art. 14(4)(c)). Art. 86 adds a right to explanation for significant decisions based on Annex III systems (EUR-Lex). After the 2026 Digital Omnibus, Arts. 13 and 14 apply to Annex III systems from 2 Dec 2027.
GDPR. For solely automated decisions with legal or similarly significant effects, people are owed meaningful information about the logic involved (Arts. 13 to 15). Art. 22 adds safeguards such as human intervention (GDPR).
US. Colorado's AI law (SB 26-189), applying from 1 Jan 2027, requires an explanation within 30 days of an adverse consequential decision.

How the AIGP exam tests it

Explainability appears in Domain I as a responsible-AI principle (lesson I.A.4) and in Domain III as a testing method (lesson III.B.7).

Two traps come up most:

Transparency vs explainability. A website that discloses AI use and lists data categories offers transparency. A rejected person asking why her own application failed is asking for explainability.
The wrong method. Global feature importance describes the model on average, not one person's outcome. SHAP explains one prediction. Someone asking "what would I need to change?" needs a counterfactual explanation.

Watch for trade-off questions too: more interpretability can cost some accuracy, and NIST expects organizations to balance this in context. The tell: match the explanation to the audience and to the exact question being asked.

Frequently asked questions

What is the difference between explainability and interpretability?

In the NIST AI RMF, explainability describes the mechanisms behind how a system works and how a decision was made. Interpretability describes what an output means in the context of its purpose, and why it matters to the user. A loan model's factor weights are an explanation. Telling an applicant what her score means for her is interpretation.

What is the difference between transparency and explainability in AI?

Transparency tells people what is happening: that AI is in use, its purpose, its data and its limits. Explainability tells a person how a specific output was produced. A notice that AI screens all job applications is transparency. Telling one rejected applicant which factors drove her result is explainability.

Does the EU AI Act require explainable AI?

It does not name XAI or mandate a technique. For high-risk systems, providers must make outputs interpretable for deployers (Art. 13) and let oversight staff interpret them correctly (Art. 14(4)(c)). Art. 86 gives people affected by significant decisions based on Annex III high-risk systems a right to a clear, meaningful explanation from the deployer.

What are SHAP and LIME?

Both are post hoc methods that explain individual predictions of complex models. SHAP (SHapley Additive exPlanations) uses game theory to show how much each input pushed one prediction up or down. LIME (Local Interpretable Model-agnostic Explanations) fits a simple stand-in model around one prediction. LIME results can vary between runs.

Sources

Spot an error? Tell us