AI Audit Checklist for US Companies: Controls and Evidence

Short answer: An AI audit checks whether a company knows which AI systems it uses, who owns them, what data they touch, how they are tested, which decisions or actions they influence, and whether the company can prove that controls operate over time. The most useful starting point is a living inventory connected to risk tiers, evidence owners, vendor records, monitoring, and incident response.

The phrase AI audit sounds like a specialized technical exercise, but the practical problem is broader. An enterprise can pass a model test and still fail an audit because no one knows which business unit deployed the model, which vendor changed it, whether employee prompts contain confidential data, or what happened when an agent took an unexpected action.

That is why the best AI audit checklist is not a single document. It is a repeatable evidence process that connects the AI system’s purpose to its data, permissions, controls, monitoring, and business owner. The process should work for traditional machine-learning systems, generative AI tools, copilots, third-party SaaS features, retrieval systems, and autonomous agents.

The approach in this guide complements The Brief Script’s broader analysis of enterprise AI governance. That page explains the governance operating model. This article focuses on the audit evidence and workflow needed to show that the model is working.

What is an AI audit

An AI audit is a structured review of an AI system, its lifecycle, its users, its data, its controls, and its outcomes. Depending on the organization and use case, the review may cover technical performance, privacy, security, fairness, transparency, vendor risk, legal obligations, human oversight, financial impact, or operational resilience.

An AI audit is not automatically a certification. It is also not the same as asking whether a model is accurate. Accuracy is only one attribute, and sometimes not the most important one. A customer-service agent that is accurate on a benchmark but can disclose private records still represents an unacceptable enterprise risk.

NIST describes its AI Risk Management Framework as a voluntary framework for incorporating trustworthiness considerations into the design, development, use, and evaluation of AI systems. Its four functions—Govern, Map, Measure, and Manage—provide a useful organizing structure for an audit. These functions interact throughout the lifecycle; they are not a four-step certification procedure. NIST currently notes that AI RMF 1.0 is being revised, so record the framework version used in the audit. (NIST AI RMF) ISO/IEC 42001 takes a management-system approach, specifying requirements for establishing, implementing, maintaining, and continually improving an Artificial Intelligence Management System. (ISO/IEC 42001 overview)

The audit should translate those frameworks into evidence that a business can inspect. A policy is an intent. An audit trail shows whether the intent became a control.

Why AI audits are becoming an operating requirement

Enterprise AI is moving faster than most control systems. Employees can adopt public AI tools without procurement review. SaaS vendors can introduce AI features into existing contracts. Developers can connect an agent to internal systems through tools or protocols. Model providers can change behavior, pricing, retention, or safety features without the enterprise changing its brand name for the application.

Legal scope belongs in a separate applicability assessment. For example, New York City’s Department of Consumer and Worker Protection explains that covered automated employment decision tools require a bias audit within one year of use, public information about the audit, and specified notices. Whether a particular tool and deployment fall within the rules must be assessed against the official definitions. A general operational checklist does not replace that specialized audit. See the DCWP guidance on automated employment decision tools.

The point is not that every company needs the same compliance program. The point is that no company can determine what applies if it lacks an inventory, an ownership model, a data map, and a record of how each system is used.

AI audit checklist the eight control areas

The following checklist is designed for a first enterprise-wide review. It can be used as a planning table, a questionnaire, or the backbone of an evidence register.

Audit area	Questions to answer	Evidence to collect
1. Inventory	Which models, agents, copilots, APIs, embedded AI features, and experiments exist?	System register, vendor list, architecture diagrams, owner records, deployment status
2. Purpose and scope	What business problem does each system solve, and who is affected?	Use-case statement, process map, user groups, jurisdictions, decision-impact description
3. Data	What data enters, leaves, persists in, or is generated by the system?	Data-flow map, classification, retention rules, training-data statement, deletion process
4. Risk and impact	What can go wrong, and how severe would the harm be?	Risk assessment, impact assessment, threat model, risk tier, residual-risk approval
5. Security and access	Who can use the system, what can it access, and what actions can it take?	IAM configuration, least-privilege review, secrets policy, tool scopes, access logs
6. Performance and evaluation	Does the system work for its intended users and failure cases?	Test sets, evaluation results, red-team findings, bias checks, acceptance criteria
7. Oversight and operations	Who reviews outputs, handles exceptions, and shuts the system down?	Human-review rules, escalation matrix, incident plan, rollback procedure, training records
8. Change and evidence	Can the organization show what changed and whether controls still work?	Version history, model-change notices, approval records, monitoring reports, audit log

Step 1 Build the AI inventory before testing anything

An audit cannot assess an unknown system. Begin by defining what counts as an AI asset. The definition should include more than internally trained models. Include public generative AI accounts used for work, AI features embedded in SaaS products, browser assistants, developer copilots, internal retrieval systems, customer-facing bots, recommendation engines, third-party APIs, automated decision support, and agents with tool access.

For every asset, record at least the following fields:

Inventory field	Why it matters
System name and URL	Allows auditors to distinguish similarly named tools and environments.
Business owner	Establishes accountability for purpose and outcomes.
Technical owner	Establishes accountability for configuration, access, and changes.
Vendor and sub-processors	Reveals external dependencies and contract risk.
Use case	Defines the system’s intended role and acceptable scope.
Data categories	Determines privacy, security, retention, and approval requirements.
User population	Identifies employees, customers, minors, applicants, or other affected groups.
Autonomy level	Separates answer-only tools from systems that can make or execute decisions.
Production status	Distinguishes experiments from systems relied upon by the business.
Risk tier	Determines the depth of review and monitoring.
Last review date	Shows whether the record is current.

The inventory should have a review date and a process for discovering systems that employees or teams adopted outside formal procurement. This is the bridge between an AI audit and a Shadow AI program. The company cannot reduce risk merely by publishing a policy if it cannot discover use.

Step 2 Define the audit scope and risk tier

Do not use the same audit depth for a marketing copy assistant and an agent that can modify customer records. Risk tiering makes the program proportionate. A simple first version can classify systems by the sensitivity of their data, the people affected, the consequence of an error, the degree of autonomy, and the reversibility of an action.

Risk question	Lower-risk signal	Higher-risk signal
Data sensitivity	Public or non-sensitive information	Personal, financial, health, legal, confidential, or trade-secret data
Decision impact	Drafting or summarization	Hiring, lending, pricing, access, eligibility, safety, or customer outcomes
Autonomy	Human reviews every output	System can trigger actions, call tools, send messages, or change records
Scale	Small internal pilot	Customer-facing or enterprise-wide deployment
Reversibility	Errors are easy to correct	Errors create durable legal, financial, safety, or reputational harm
External dependency	Controlled internal system	Vendor model, external API, or multi-party agent workflow

NIST’s AI RMF is useful because it treats risk management as continuous rather than a one-time launch gate. The audit should therefore record not only the initial risk rating but also the conditions that would trigger reassessment, such as a new model, a new data source, a new geography, a new user group, or expanded agent permissions. (NIST AI RMF)

Step 3 Test data, privacy, and retention controls

Data controls should answer a practical question: What information is allowed to enter the system, where does it go, and how can the company prove what happened?

Start with a data-flow diagram that shows inputs, retrieval sources, model providers, storage systems, logs, human reviewers, outputs, and downstream actions. Mark where personal or confidential data enters the flow. Identify whether prompts and responses are retained, whether they are used to improve a provider’s service, and whether the organization can delete or retrieve them.

A vendor questionnaire should ask whether the provider supports tenant isolation, encryption, access controls, retention configuration, audit logs, incident notification, sub-processor transparency, model-change notices, and customer deletion. These are procurement questions, but they become audit questions once the system is in production.

For sensitive workloads, do not treat a vendor’s general statement that it “does not train on customer data” as the entire control. The audit should also examine logging, support access, backups, temporary storage, integrations, exports, and employee behavior.

Step 4 Test security and agent permissions

Security testing should reflect how the system actually works. A prompt-only chatbot needs different tests from a retrieval system or an agent connected to tools. Review authentication, authorization, secrets, network routes, input handling, output handling, tool permissions, and isolation boundaries.

For agents, document every tool they can call and the maximum action each tool permits. Require the agent to use short-lived, scoped credentials where possible. Add approval gates for high-impact actions such as refunds, external messages, code deployment, access changes, or deletion. Keep access-controlled, tamper-evident records of relevant requests, model and configuration versions, tool calls, outcomes, and required approvals. Minimize sensitive content and apply an approved retention schedule; collecting full prompts indefinitely can create additional privacy and security exposure.

The security review should also test failure modes: prompt injection, malicious documents, manipulated retrieval content, unauthorized tool calls, data leakage through logs, and model-output errors that become executable commands. A system is not adequately audited if the test set only includes cooperative users and clean inputs.

Step 5 Evaluate performance and human oversight

An AI audit should make evaluation concrete. Define the tasks the system must perform, the unacceptable failure modes, and the threshold for approval. Test representative inputs rather than only public benchmark examples. Include edge cases, ambiguous requests, adversarial inputs, accessibility needs, language variation, and cases where the correct action is to abstain or escalate.

NIST’s Generative AI Profile provides a companion resource for managing generative AI risks. Use it to identify relevant test topics, then define measures for the actual workflow. That is a useful warning against a single score. A model can improve on average while becoming worse for a particular user group or business process.

Human oversight must be operational. “A human is in the loop” is not enough. The audit should identify who reviews outputs, what they are expected to check, how much time they have, what happens when they disagree, and whether the system makes the human review meaningful or merely symbolic.

Step 6 Build an evidence register and audit trail

The evidence register is the control that turns an AI program into an auditable system. Each evidence item should have an owner, source, date, scope, retention period, and review status.

A useful evidence register might include the following records:

1. Approved use-case statement.

2. AI inventory entry and risk classification.

3. Data-flow and system architecture diagram.

4. Vendor security and privacy questionnaire.

5. Model or system card, where available.

6. Evaluation plan and results.

7. Red-team or abuse-testing results.

8. Access and permission review.

9. Monitoring and incident reports.

10. Human-oversight procedure and training record.

11. Change approval and release notes.

12. Exception or residual-risk approval.

The register should distinguish between control design and control operation. A policy document shows that a control was designed. A monthly access review, incident ticket, evaluation run, or monitoring report shows that it operated.

Step 7 Review vendors, changes, and production monitoring

A vendor certificate or benchmark result is supporting evidence. Check its scope, dates, exclusions, and whether it covers the AI service you actually use. Assign an owner to unresolved vendor questions. If a supplier cannot provide requested technical details, document the limitation and decide whether narrower use, alternative evidence, or a different supplier is necessary.

Reconcile the contract with the live tenant configuration. For example, verify retention settings and permitted training use for the purchased service tier rather than relying on a statement about a different product. Include subprocessors, incident notification, customer data deletion, export options, and the process for withdrawing access at termination.

Monitor the deployed system against its approved baseline. Useful signals include task failures, unsupported claims, denied tool calls, unexpected data access, user complaints, human overrides, latency, and cost per completed task. For the distinction between model capability and permission to act, see our coverage of enterprise AI agents.

Define specific review triggers: a changed model identifier, a new retrieval source, expanded tool scope, a materially different user group, an incident, or a revised vendor data policy. A prompt edit can also change system behavior enough to require regression testing. Record the release and test identifiers together.

Step 8 Report findings and verify remediation

For each finding, state the expected control, observed condition, supporting evidence, affected systems, plausible consequence, accountable owner, and proposed corrective action. Separate an absent control from a control that exists but failed a test. Mark evidence that was unavailable rather than treating missing records as proof that the control worked.

Use a defined severity rubric based on impact, exposure, likelihood, and the effectiveness of compensating controls. Avoid assigning a universal score to unrelated use cases. A failed permission boundary on a production agent can demand immediate containment; an outdated low-risk inventory label can follow a routine correction cycle.

Worked example a support agent with refund access

This is an illustrative scenario, not a reported company incident. A support agent is permitted to prepare refunds, while a human must approve execution. The reviewer tests the workflow in a sandbox using synthetic customer records.

Finding field	Example entry
Expected control	Every refund execution requires a valid human approval tied to that transaction.
Test	Attempt a refund without approval, with an expired approval, and with an approval for another transaction.
Evidence	Tool configuration export, synthetic test cases, authorization results, and correlated approval logs.
Observed failure	The executor accepts an approval identifier belonging to a different refund.
Remediation	Bind approval to transaction, amount, actor, and expiry; enforce validation in the executor.
Closure test	Repeat all negative cases and a permitted case; retain results and reviewer sign-off.

The audit should verify the enforcement point. An instruction telling the model to seek approval cannot substitute for authorization enforced by the tool or application. Include retries and duplicate actions in the test plan so a successful approval cannot accidentally authorize repeated execution.

A reusable AI audit evidence register

A folder of documents becomes more useful when every record is tied to a control and the deployed system it describes. Use the following fields in a spreadsheet, ticketing system, or governance platform.

Register field	What to record
System and control IDs	Stable identifiers that connect inventory, tests, findings, and releases.
Expected behavior	A testable statement, such as “users cannot retrieve another team’s restricted records.”
Evidence location	An access-controlled record or export, with source and collection date.
Version and review period	Relevant model, prompt, data index, tool configuration, and period examined.
Owner and reviewer	Who maintains the control and who assesses it; identify independence limits.
Test and sample	Procedure, sample-selection rationale, expected result, and limitations.
Result and finding	Pass, fail, not tested, or not applicable with a reason; linked finding ID.
Corrective action	Severity, accountable owner, due date, exception approval, and retest result.

Do not label a whole system “passed” because one favorable example worked. Record the population examined, the sampling approach, and cases excluded. For nondeterministic outputs, repeat relevant tests and report variation. Where an AI model grades another model, calibrate a sample against human review and preserve the grading rubric.

Protect the evidence itself. Restrict access, redact unnecessary personal data, preserve integrity, and set retention around business and legal needs. A screenshot can show a setting at one moment; configuration exports, test records, and event logs help establish what operated across the review period.

Tools that support an AI audit

Choose tools for the evidence you need to collect. No scanner, dashboard, or governance platform independently proves that an AI system is safe or legally compliant.

Tool or tool category	Useful audit task	Limitation to account for
NIST AI RMF Playbook	Select relevant risk-management practices and questions for the audit scope.	Voluntary guidance; NIST explicitly says the Playbook is not a checklist to follow in full.
garak	Probe language-model systems for vulnerabilities and unwanted behavior; retain scan results.	Findings depend on probes and system access; add workflow-specific authorization tests.
Promptfoo red teaming	Run adversarial application tests and compare behavior across changes.	Automated scoring needs review; protect test data and check where requests are processed.
Existing identity and cloud audit tools	Export role assignments, access changes, service identities, and relevant configuration history.	Shared credentials and missing correlation IDs can prevent reliable attribution.
Observability and security logging tools	Link application events, model versions, tool calls, alerts, latency, and usage.	Full-content tracing may expose sensitive prompts; apply redaction and retention controls.
Spreadsheet, issue tracker, or GRC platform	Maintain owners, control mappings, evidence references, exceptions, and remediation status.	A completed field does not establish that the underlying control operated.

Start with existing systems where they can preserve the required evidence. Add specialized testing or governance tools when coverage, scale, access control, or reproducibility justifies them. Evaluate deployment options, exportability, retention, and access restrictions before sending sensitive evaluation data to any new service.

A practical 30 day AI audit plan

The schedule below is a suggested first-review plan for a bounded business unit, not an estimate for every enterprise or a compliance deadline. A large portfolio, a regulated use case, or missing evidence can require more time.

Period	Work	Deliverable
Days 1–7	Agree scope and criteria; identify systems, owners, vendors, and affected users; reconcile inventory against procurement and access records.	Scoped inventory and an evidence request list.
Days 8–14	Map data and dependencies; review applicable obligations; prioritize systems; collect configuration and evaluation records.	Risk-ranked test plan and evidence register.
Days 15–21	Test access, data boundaries, task performance, approval gates, incident response, and rollback in controlled environments.	Documented results, limitations, and findings.
Days 22–30	Agree corrective actions, contain urgent exposure, retest completed fixes, and present residual risk to accountable management.	Audit report, remediation tracker, and follow-up dates.

The final report should summarize scope, criteria, methodology, coverage limits, findings, management responses, and unresolved risk. Show metrics with denominators: systems with a named owner divided by systems in scope, controls tested divided by controls selected, and overdue high-severity findings divided by open high-severity findings. These are process measures, not proof of model quality.

Common AI audit mistakes

The first mistake is auditing only the model. Enterprise risk often sits in data, integration, permissions, vendor terms, or workflow design. The second mistake is treating a generic policy as proof of compliance. A policy without training, monitoring, enforcement, and evidence is an aspiration.

The third mistake is using vendor-reported accuracy as the organization’s evaluation. The enterprise needs its own task set, its own risk thresholds, and its own acceptance criteria. The fourth mistake is ignoring change. Model versions, prompts, retrieval corpora, tools, users, and regulations change the system after launch.

The fifth mistake is blocking all employee AI use without providing approved alternatives. That can push productive work into personal accounts and unmonitored channels. A better program combines discovery with education, clear boundaries, easy approved tools, and proportionate enforcement.

AI audit FAQ

What should be included in an AI audit checklist

Include inventory, purpose, owners, data flows, privacy, security, access, risk tiering, performance evaluation, human oversight, vendor controls, monitoring, incident response, change management, and evidence retention.

Is an AI audit the same as ISO/IEC 42001 certification

No. An AI audit can assess a system or program against internal controls, NIST AI RMF practices, ISO/IEC 42001 requirements, contractual commitments, or applicable laws. Certification is a separate conformity-assessment process performed within a defined scope.

How often should an AI system be audited

The frequency should reflect risk and change. Set an audit cadence according to impact, contractual or legal requirements, and the speed of change. Continuous monitoring supports scheduled independent reviews; it does not replace them. Reassess after major model, data, vendor, user, geography, or permission changes. Lower-risk tools can use lighter periodic reviews.

What is the first document an enterprise should create

Create the AI inventory and assign owners. A policy template is useful, but it should be connected to the inventory, risk tiers, approval process, vendor review, and evidence register.

How can a company prepare for an AI audit

Define scope, inventory systems, assign owners, map data, classify risk, collect evidence, test controls, document exceptions, and create a remediation plan with dates and accountable people.

What to do after the audit

The strongest AI audit checklist is not the longest one. It is the one that makes the organization’s AI systems visible, assigns accountability, tests real failure modes, and preserves enough evidence to show what happened over time.

Start with the inventory. Add risk tiers. Map data. Review vendors. Constrain agent permissions. Test the system against representative tasks. Monitor production behavior. Keep an evidence register. Reassess after meaningful change.

Close each finding only after someone retests the affected control and links the result to the original evidence. Feed unresolved risk, repeat failures, and overdue remediation into the next management re