AI audit

In plain English

An AI audit is like a building inspection. The inspector does not try to knock the house down. They check it against the building code, the approved plans and safety rules, then write up what fails.

An AI audit compares what is actually happening with a set of criteria. The criteria can be a law, such as New York City's bias audit rule, or a standard, such as ISO/IEC 42001. They can also be the organization's own AI policy, a contract, or claims made in a model card.

Audits come in three broad types:

• Internal (first-party): run by the organization itself, ideally by a team that did not build the system.
• Second-party: run by a customer on its supplier.
• Third-party: run by an independent firm or an accredited certification body.

Scope varies too. A technical audit tests one model's performance and bias. A governance audit tests whether policies, roles and records work in practice.

Example

Thistlewood Staffing, a fictional recruitment firm, uses an AI tool to rank job applicants for employers in New York City. It runs two audits every year:

1. Bias audit. An independent auditor checks the tool for bias every year, because Local Law 144 requires an audit within one year before use. Thistlewood posts a summary of the results and notifies candidates.
2. Internal audit. The firm's risk team, which did not choose or configure the tool, checks practice against its AI policy and ISO/IEC 42001 controls. Are vendor updates re-tested? Are recruiter overrides logged? Is the AI inventory current?

The internal audit finds that a vendor update went live without re-testing. The finding goes into the risk register with an owner and a deadline. Next year's audit checks that it was closed.

Why it matters

• US local law. New York City's Local Law 144 bars employers and employment agencies from using an automated employment decision tool unless it had a bias audit within one year of use. A summary of results must be public. Enforcement began on 5 July 2023.
• ISO/IEC 42001. Clause 9.2 requires internal audits of the AI management system. Accredited certification bodies run certification audits, and ISO/IEC 42006 (July 2025) sets requirements for those bodies.
• EU AI Act. The Act has no general "AI audit" duty. Where a notified body assesses a high-risk provider's quality management system, it must carry out periodic audits (Annex VII, point 5.3).
• NIST. MEASURE 1.3 calls for regular assessments by internal experts who were not front-line developers, or by independent assessors.

How the AIGP exam tests it

Audits sit beside red teaming, threat modeling and security testing in lesson III.C.5 and lesson IV.C.3. The exam swaps them.

• The tell. The verb is "check against criteria." If the question mentions conformance with policy, law, contract or a standard, the answer is an audit.
• Trap 1: auditors who attack. Trying hundreds of jailbreak prompts is red teaming, not auditing. See red teaming vs threat modeling.
• Trap 2: an audit replaces monitoring. A clean annual audit is a snapshot. Continuous monitoring still runs.
• Trap 3: marking your own homework. Markers for an independent or third-party audit include consequential decisions, sensitive data, high-risk domains, large scale, potential for bias, critical infrastructure and influence on public discourse.