Enterprise AI Security: Protect Data, Models and Agents

Enterprise AI security protects the data, identities, software and actions behind an organization’s AI systems. Start by controlling what each user or agent can access, treating model output as untrusted input to other software, and enforcing action permissions outside the model. Then test whether those controls hold when content is malicious, access changes or a connected service fails.

The practical goal is to prevent unauthorized disclosure and changes, limit disruption and make incidents recoverable. A safer model can help, but the surrounding application still needs authentication, authorization, monitoring and secure software practices.

Our enterprise AI overview explains the wider category. This guide focuses on protecting those systems. Our separate coverage of AI applications in cybersecurity examines how defenders use AI.

Which controls does each type of enterprise AI need?

Begin with the capability being deployed. An assistant drafting text, a search system retrieving documents and an agent updating business records create different exposure.

AI capability	Main security boundary	What to verify first
Purchased assistant	Information sent to and retained by the service	Approved accounts, permitted data, provider terms and administrator controls
Internal knowledge search	Source information delivered to a particular user	Permissions before content reaches the model, including caches and derived summaries
Agent with connected tools	Business actions taken through APIs or applications	Workload identity, narrow tool permissions, independent authorization and revocation
Custom predictive or generative model	Training data, model artifacts and serving infrastructure	Dataset provenance, artifact integrity, restricted access and secure deployment

Inventory the actual integrations, including AI features enabled inside existing SaaS products. Record the owner, data sources, provider, credentials, permitted actions and deployment version. Include unattended and scheduled jobs, whose authority may differ from that of the person reading their output.

Security requirements should enter the enterprise AI strategy before a pilot receives broad access. The additional cost belongs in the enterprise AI deployment budget, including testing, monitoring, support and incident recovery.

What are the main enterprise AI security risks?

The OWASP Top 10 for LLM applications provides a useful vocabulary, including prompt injection, sensitive information disclosure, supply-chain weaknesses and excessive agency. Its published 2025 list addresses LLM applications; it is not a complete inventory for every AI deployment.

The following operational map connects common failure paths to controls. It is an editorial guide for planning tests, rather than a ranked threat forecast.

Failure path	Example consequence	Control to verify
Prompt injection	Instructions in a retrieved file redirect the assistant’s behavior	Restrict available data and actions; test malicious content in an authorized environment
Broken access controls	A summary exposes another team’s restricted information	Enforce the requesting identity’s permissions before retrieval content enters the model
Sensitive data handling	Inputs, logs or support exports retain prohibited information	Minimize collection and verify storage, access, retention and deletion behavior
Excessive agent authority	A task assistant edits records outside its assigned scope	Use narrowly scoped credentials and authorize each action at the service boundary
Unsafe output handling	Generated text becomes executable code or an unsafe database command	Validate structured fields and use safe execution interfaces for the destination
Compromised data or dependencies	An altered dataset, package, connector or model changes behavior	Verify provenance and integrity, restrict changes and track deployed versions
Resource abuse	Repeated calls or retries exhaust capacity or spending limits	Enforce bounded execution, aggregate quotas and safe retry behavior

The distinction between a bad answer and an unauthorized action matters. A model can make a factual mistake without an attacker. Conversely, a technically accurate answer can still disclose information to someone who should not receive it. Test quality and security separately.

Can prompt injection be completely prevented?

Treat prompt injection as a risk to reduce and contain. Direct attacks arrive in a user’s request. Indirect attacks can arrive through email, documents, websites, tool responses or other material the model processes.

The UK National Cyber Security Centre explains why prompt injection differs from SQL injection. An LLM does not provide the same firm separation between instructions and data that a parameterized database query can enforce. The NCSC recommends designs that limit the consequences of a manipulated model.

Instruction hierarchies, content screening and attack detectors can reduce exposure. A declaration in a system prompt that certain actions are forbidden does not establish authorization at the connected service. A second model checking the first can also make mistakes.

For example, a hypothetical support assistant may read an external troubleshooting attachment. Even if the attachment persuades it to request a prohibited action, the application should reject that request through independent permission and policy checks.

Run these tests with synthetic material in an authorized test environment. Record whether the model followed the unwanted instruction, whether it attempted a tool call, whether the service rejected it and whether the event appeared in the logs. Those are different outcomes.

A model refusal is evidence about behavior in that test. A denied service action is evidence about an authorization control. Keep both results so a change in model behavior does not hide a broken permission boundary.

How should companies protect data in AI applications?

Map where information travels through the complete workflow. Include prompts, uploaded files, retrieved passages, generated answers, embeddings, memory, logs, backups and downstream exports. An embedding or summary should not automatically be treated as non-sensitive because it differs from the original document.

For purchased services, verify the specific product, plan and deployment configuration. Public statements about a vendor’s enterprise offering may not apply to a personal account, optional connector or separate model provider.

Data question	Evidence to request
Can inputs or outputs be used for model training?	Applicable contract terms, enabled settings and the providers covered by them
What is retained and where?	A data-flow description covering operational storage, logs, backups and subprocessors
Who can access customer content?	User, administrator, support and service-account access rules, with review records
What happens when access is revoked?	A timed test covering retrieval, caches, conversation history and derived content
What happens on deletion or disconnect?	Scope, retention exceptions, backup handling and a verifiable completion process

Use corporate identity controls and appropriate multifactor authentication. Keep credentials in managed secret storage and inject them through the application, rather than placing them in prompts. Restrict outbound destinations and apply data-loss controls where the workflow requires them.

Does permission-aware retrieval solve oversharing?

It addresses one part of the problem. Source permissions themselves may already be too broad. Review sensitive repositories and sharing groups before connecting them, then verify that the AI application respects the corrected permissions.

For retrieval-augmented generation, or RAG, authorized content is added to the model’s context to help it answer a question. Enforce access before that context is constructed. Removing a restricted citation from the final answer is too late if its contents have already been disclosed to an unauthorized processing path.

Include user identity, tenant and relevant access state in cache isolation and invalidation design. A cache keyed only by question text can create a disclosure path when different users ask the same question.

Revocation also needs a defined scope. Removing source access may prevent new retrieval without erasing a copy previously included in conversation history. Agree how the product handles that history and test it explicitly.

The detailed enterprise AI search evaluation should include permission freshness and derived information alongside answer quality.

What changes when AI agents can take actions?

Execution adds consequences beyond the generated answer. Identify which workload is making each request and whose authority it carries. Give it only the tools, records, environments and operations required for its task.

The May 2026 joint guidance on careful adoption of agentic AI services, co-authored by agencies including ASD’s ACSC, CISA, NSA and the UK NCSC, urges caution. It recommends low-risk, non-sensitive tasks, restricted privileges, ongoing oversight and human control. Approval checkpoints alone do not establish that a sensitive autonomous workflow is appropriate.

For API tools or Model Context Protocol servers, review the server operator, available methods, credentials and network access. A connector’s presence does not make every exposed action suitable for the agent. Review changed tool descriptions and permissions as configuration changes.

What should an approval actually authorize?

An approval should bind to the specific action being reviewed, with its target, parameters, destination, identity and validity period. Recheck authorization immediately before execution, including any relevant business state that may have changed.

In this hypothetical ticketing example, a reviewer approves closing ticket T-104. If the target changes to T-105, the earlier approval must not authorize the changed request. The backend should reject it or require a new approval.

The application also needs a replay policy. A one-time approval should not become reusable permission for an unlimited sequence of actions. Where retries are possible, use an operation identifier and idempotency controls so a timeout does not cause the same business action twice.

Our recommended review question is whether the system can perform a different action from the one the person approved. Test altered targets, changed fields, expired approval and repeated execution. A generic confirmation button does not answer that question.

Per-action limits also need an aggregate boundary. In a hypothetical notification system, allowing 200 recipients per call still permits 800 recipients across four calls. A daily or job-level cap must be enforced across requests if it is part of the policy. These figures illustrate the distinction, rather than recommended limits.

For the wider implementation process, agent deployment from pilot to production provides the operational context. The security review should record the exact authority tested and approved for the release.

How should models and their supply chains be secured?

Apply normal software-security practices to model-serving applications, packages, containers, APIs and administrative interfaces. Protect them against ordinary vulnerabilities as well as AI-specific attacks. An exposed management endpoint does not become safer because the service hosts a model.

For custom training or fine-tuning, document dataset origin, authorized modifications and evaluation versions. Protect model weights and checkpoints as controlled artifacts. Verify integrity and provenance through a trusted release process; a checksum supplied beside an untrusted download does not establish who produced it.

Treat changes to retrieval content, memory, prompts, tools and dependencies as possible behavior changes. Maintain a tested release configuration and a rollback path. Where a provider controls model updates, establish notice and retesting arrangements rather than assuming the model will remain unchanged.

Monitor for compromised sources and corrupted persistent memory. Rolling back a model will not remove malicious content left in a connected knowledge base. Recovery needs to address the component that carried the problem.

What should a practical enterprise AI security test include?

Use an authorized staging environment with synthetic records and isolated destinations. Run allowed operations as well as prohibited ones, so a system that blocks everything cannot pass as a working deployment.

This five-phase test is an editorial starting framework. Tailor it to the architecture and consequences of failure; it does not replace a system-specific threat assessment.

Test phase	Observable test	Passing evidence
1. Identity and retrieval	Use two test identities with different permissions; then revoke one identity’s access	Allowed content works; restricted material does not enter unauthorized context or outputs; revocation behavior matches the agreed scope
2. Untrusted content	Add harmless adversarial instructions to a test document or tool response	Attempts and outcomes are recorded; prohibited data access and actions remain blocked independently of the model’s response
3. Action authorization	Change the target after approval, repeat an approved request and attempt an out-of-scope operation	Changed or unauthorized actions are denied; replay and duplicate-side-effect controls work
4. Failure and resource limits	Simulate a tool timeout, exhausted quota and unavailable policy service	Execution stays within agreed limits, retries are bounded and protected actions fail safely
5. Containment and recovery	Disable the affected capability while a test job is active, then restore a known-good configuration	New and queued actions stop within the defined scope; evidence is preserved and completed side effects are reconciled

Record the software version, model, source configuration, identities, expected outcome, actual outcome and underlying service result. Repeat relevant tests after material changes. An average pass rate should not hide a single unresolved path to prohibited disclosure or action.

Retain evidence in the AI audit checklist process. Assign remediation and release decisions through enterprise AI governance, with a named owner for any remaining risk.

What should happen during an AI security incident?

Predefine who may disable tools, revoke credentials, isolate a data source and move the workflow to a manual route. Waiting to discover those permissions during an incident slows containment.

A pause in the chat interface may leave scheduled jobs, queues or existing tokens active. Test containment at the components that perform the work. Separately verify what already happened, because revocation cannot reverse a completed disclosure or transaction.

Incident task	What to establish
Contain execution	Which tools, identities, sessions and queued jobs have actually been stopped
Preserve evidence	Correlated request, retrieval, policy, approval and service-side action records
Establish impact	Information exposed, records changed, recipients reached and work still in flight
Remove the cause	Affected source, memory, credential, connector, configuration or software defect
Restore service	Corrected controls, successful regression tests and reconciliation of completed actions

Log enough to reconstruct an event: the user or workload identity, relevant versions, source identifiers, authorization result, requested action, approval reference and actual tool result. Correlate application records with the downstream service where feasible.

Avoid collecting every prompt and answer by default without a retention and access plan. Security logs can themselves contain sensitive material. Use redaction and restricted evidence capture where suitable, while preserving the information necessary for investigation.

Measure the time until effective containment, rather than the time until someone clicked a pause button. Also track unresolved high-impact test failures, unauthorized requests that succeeded and the time required for permission changes to take effect. Zero observed incidents is not proof that the system is secure.

Which security work should come before wider rollout?

Prioritize the paths with the most consequential access and authority. Before expanding a deployment, establish an accountable owner, known data flows, working identity controls and a tested permission boundary. Verify action controls wherever tools can change something.

Then demonstrate that operators can detect, stop and investigate a failure. A team that cannot account for its active connectors or disable an unattended job needs that operating capability before granting more reach.

Tie these checks to the enterprise AI adoption process so training explains the actual approved workflow, exceptions and reporting route. A product’s certifications and a vendor questionnaire can inform the review, but the deployment still needs evidence from its own configuration.

Frequently asked questions about enterprise AI security

Is enterprise AI security the same as AI governance?

Security focuses on protecting data, systems and operations against misuse and compromise. Governance establishes decision rights, responsibilities and lifecycle oversight. They meet when an owner approves a deployment, reviews evidence and decides how remaining risk will be handled.

Is a private or self-hosted model automatically secure?

No. Hosting choice changes the exposure and responsibility. A private deployment still needs protected endpoints, scoped identities, secure dependencies, access controls and tested operations. It can disclose data or execute unauthorized actions through a vulnerable surrounding application.

Does RAG prevent prompt injection or data leakage?

RAG adds retrieved information to the model’s context. Its security depends on the sources, access checks, isolation and handling of that information. Retrieved content may contain malicious instructions, and retrieval cannot repair permissions that were incorrectly configured.

Should an AI agent hold administrator credentials?

Start with the minimum authority needed for its approved task. Broad persistent credentials enlarge the consequences of error or compromise. If a task genuinely requires privileged operations, assess that workflow separately and enforce appropriate scope, approval, credential lifetime and monitoring outside the model.

What is the most useful first security check?

Identify the data and actions the application can reach, then test an operation it must refuse. Confirm the denial in the enforcing service and its logs. Pair that test with an allowed operation so the result demonstrates a usable boundary.