Generative AI Governance (2027): Framework, Risks and Controls
How to govern generative AI and LLMs: the 12 NIST AI 600-1 risks, ISO/IEC 42001, EU AI Act GPAI and transparency dates, a risk-control-evidence table and what a gen AI governance platform does. Checked November 2026.
In short: Generative AI governance is the set of policies, controls and evidence that keep LLMs, copilots and AI agents safe to use at work. It differs from classic AI governance because the risk sits in open-ended outputs and prompts: hallucinations, data leaking into prompts, prompt injection, copyright, shadow AI and models that change under you. The practical baseline is NIST AI 600-1 for the risks, ISO/IEC 42001 for the management system and the EU AI Act’s GPAI and transparency rules for the deadlines.
This guide is the gen AI layer. For the organisation-wide programme (committees, policy, model risk), start with our enterprise AI governance playbook. Regulatory dates below were checked on official EU, NIST and ISO pages on November 13, 2026.
What makes generative AI governance different
Classic model governance assumes a model with a fixed task, a test set and a measurable error rate. A large language model has none of these. The same model drafts contracts, answers customers and writes code, and every prompt is a new input you never tested. That shifts governance in seven ways:
- Output risk, not just model risk. NIST calls hallucination confabulation: “confidently stated but erroneous or false content.” You govern each answer, not only the model.
- Data leaks in through the prompt. Staff paste customer records, code and contracts into chat windows. The data-loss risk sits at the moment of use.
- Prompt injection. Text hidden in a web page, email or PDF can override your instructions. OWASP ranks it first in its Top 10 for LLM applications.
- IP and copyright cut both ways: what went into the model, and whether its output copies protected work.
- Shadow AI. IBM’s 2026 breach study found 43% of breached organisations had a shadow AI incident, up from 20% a year earlier. Our shadow AI guide covers detection.
- Models and vendors change under you. Providers retrain, fine-tune and retire models. NIST tells buyers to cover “GAI system changes over time (e.g., fine-tuning, drift, decay)” in vendor contracts.
- Agents act. Once an LLM can call tools, send email or move money, a bad output becomes a bad action. OWASP lists this as Excessive Agency.
The frameworks: NIST AI 600-1, ISO/IEC 42001 and the EU AI Act
NIST AI 600-1: the risk catalogue
NIST published the Generative AI Profile (NIST AI 600-1) on July 26, 2024 as a companion to the voluntary AI RMF 1.0. It names 12 risks that are “unique to or exacerbated by” generative AI: CBRN information, confabulation, dangerous or hateful content, data privacy, environmental impacts, harmful bias and homogenization, human-AI configuration, information integrity, information security, intellectual property, obscene or abusive content, and value chain and component integration.
It then lists suggested actions under the AI RMF’s Govern, Map, Measure and Manage functions. We counted 211 unique actions (Govern 57, Map 39, Measure 72, Manage 43) and tallied the risk tags NIST attached to each.
Two things stand out. Information integrity (provenance, labelling, misinformation) gets the most attention, with 72 actions. Confabulation, the risk most business users worry about first, gets only 18. If hallucination is your main exposure, the NIST profile alone will not give you enough controls; you will need output checks and human review on top.
ISO/IEC 42001: the management system
ISO/IEC 42001:2023, published in December 2023, sets requirements for an AI management system (AIMS). It applies to any organisation that develops, provides or uses AI and follows the Plan-Do-Check-Act cycle familiar from ISO 27001. In practice, NIST AI 600-1 tells you what can go wrong with generative AI, and ISO/IEC 42001 gives you the auditable process for managing it. ISO/IEC 42005:2025 adds guidance on AI system impact assessments.
EU AI Act: the GPAI and transparency deadlines
The EU AI Act entered into force on August 1, 2024. For generative AI, three sets of rules matter:
- General-purpose AI (GPAI) model providers have had obligations since August 2, 2025. Under Article 53 they must keep technical documentation, give downstream integrators enough information to comply, adopt a copyright policy that respects text-and-data-mining opt-outs, and publish a summary of training content. Models trained with more than 10^25 FLOPs are presumed to carry systemic risk and face extra duties.
- Enforcement started on August 2, 2026, when the Commission’s AI Office gained powers to request documentation, evaluate models and fine providers. Models placed on the market before August 2, 2025 must comply by August 2, 2027.
- Article 50 transparency has applied since August 2, 2026. Providers must mark synthetic content in a machine-readable way, and deployers must label deepfakes and AI-written text on matters of public interest. Systems already on the market before August 2, 2026 have until December 2, 2026 to add marking. Fines reach EUR 15 million or 3% of worldwide turnover.
Most enterprises are deployers, not GPAI providers. Your direct duties are mainly Article 50 labelling, plus the high-risk rules if you use gen AI in areas such as hiring or credit. After the AI Omnibus (in force July 27, 2026), those high-risk rules apply from December 2, 2027. You still need your vendors’ Article 53 documentation to do your own risk assessment. Our AI regulation tracker follows other jurisdictions.
A generative AI governance framework: risk, control, evidence
A framework is only useful if every risk has a control and every control leaves evidence an auditor can check. This is the compact version we would start from:
| Gen AI risk (NIST AI 600-1) | Control | Evidence |
|---|---|---|
| Confabulation (hallucination) | Grounding checks against source documents; human review for high-impact outputs | Grounding scores, review logs, error-rate tests |
| Data privacy (leakage in prompts) | DLP on gen AI sites and copilots; PII masking at the gateway | DLP match reports, masked-entity logs |
| Information security (prompt injection) | Prompt-attack filters; treat retrieved content as untrusted; least-privilege tools | Red-team results, blocked-attack logs |
| Intellectual property | Approved-model list; vendor IP terms; output similarity checks for published content | Vendor contracts, model register |
| Value chain (model and vendor change) | Change clauses for fine-tuning, drift and deprecation; re-test on model updates | Contract clauses, regression test records |
| Human-AI configuration (over-reliance, agents) | Human approval for consequential actions; deactivation procedure | Approval logs, kill-switch runbook |
| Information integrity (labelling) | Machine-readable marking; visible labels on deepfakes and public text | Provenance metadata, label audit |
| Shadow AI | Inventory and discovery of unsanctioned tools; sanctioned alternatives | AI inventory, discovery reports |
For agents specifically, see our study of AI agent governance controls.
What a gen AI governance platform does
No single product covers the whole table. Buyers looking for a “gen AI governance platform” are usually choosing across four categories:
- AI governance and GRC platforms hold the inventory, policies, risk assessments and audit evidence. IBM’s watsonx.governance page, for example, lists a governance graph, shadow AI detection, policy enforcement, obligation mapping and compliance evidence capture. IBM also says it was named a Leader in Gartner’s first Magic Quadrant for AI Governance Platforms (2026)†, a vendor claim we have not checked against the Gartner report. We compare 16 tools in our AI governance tools guide.
- Runtime guardrails sit between users and the model. Amazon Bedrock Guardrails documents content filters including a Prompt Attack category, denied topics, PII masking, contextual grounding checks for hallucinations and Automated Reasoning checks. Google Cloud Model Armor screens prompts and responses for prompt injection and jailbreaks, sensitive data and malicious URLs, including text inside PDFs.
- Data security for AI (DSPM and DLP) watches what flows into prompts. Microsoft Purview DSPM shows prompts and responses sent to generative AI sites that contained sensitive information, flags DLP matches and runs oversharing risk assessments.
- Evaluation and observability tools test and trace outputs over time. They supply the regression evidence when a vendor swaps the model behind an API.
Before you buy, ask each vendor which NIST AI 600-1 risks and EU AI Act articles their controls map to, and ask to see the evidence export, not the dashboard.
A 30/60/90-day rollout
If you are starting from a policy and little else, sequence the work so you can see your AI use before you try to control it, and control it before you try to prove it.
FAQ
What is generative AI governance?
It is the set of policies, technical controls and records that let an organisation use LLMs, copilots and AI agents within its risk appetite and the law. It covers outputs (hallucination, harmful or infringing content), inputs (data leakage, prompt injection) and the supply chain (model and vendor changes).
Is NIST AI 600-1 mandatory?
No. It is a voluntary profile of the NIST AI RMF. It works well as a gen AI risk register because it is specific and free to download, and it can sit alongside an ISO/IEC 42001 management system.
Does the EU AI Act apply to companies that only use ChatGPT or Copilot?
Mostly through Article 50 and the high-risk rules. A company that deploys a gen AI tool must label deepfakes and AI-generated public-interest text from August 2, 2026, and must meet high-risk duties from December 2, 2027 if the use case is high-risk. The GPAI obligations fall on the model provider.
What is the difference between AI governance and gen AI governance?
AI governance is the whole programme for every model you run. Gen AI governance is the layer for open-ended models: prompt and output controls, guardrails, provenance labelling and vendor model change. Our enterprise AI governance playbook covers the wider programme.
Sources
- NIST: Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1): https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-generative-artificial-intelligence
- NIST: NIST AI 600-1 (PDF; risk definitions and suggested-action tables): https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf
- OWASP: Top 10 for LLM Applications 2025: https://genai.owasp.org/llm-top-10/
- IBM: Cost of a Data Breach Report 2026 (PDF): https://assets.ibm.com/adobe/assets/urn:aaid:aem:21111142-1251-4369-86fb-57b82f5bb108/original/as/Cost%20of%20a%20Data%20Breach%20Report%202026.pdf
- ISO: ISO/IEC 42001:2023, AI management system: https://www.iso.org/standard/42001
- European Commission: AI Act (application timeline and AI Omnibus): https://digital-strategy.ec.europa.eu/en/policies/regulatory-framework-ai
- AI Act Service Desk: Article 53, Obligations for providers of general-purpose AI models: https://ai-act-service-desk.ec.europa.eu/en/ai-act/article-53
- AI Act Service Desk: Article 51, GPAI models with systemic risk: https://ai-act-service-desk.ec.europa.eu/en/ai-act/article-51
- European Commission: Guidelines for providers of general-purpose AI models: https://digital-strategy.ec.europa.eu/en/policies/guidelines-gpai-providers
- AI Act Service Desk: Article 111, AI systems and GPAI models already on the market: https://ai-act-service-desk.ec.europa.eu/en/ai-act/article-111
- European Commission: Quick facts, transparency rules for AI systems: https://digital-strategy.ec.europa.eu/en/factpages/quick-facts-transparency-rules-ai-systems
- IBM: watsonx.governance: https://www.ibm.com/products/watsonx-governance
- AWS: Amazon Bedrock Guardrails (user guide): https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails.html
- Google Cloud: Model Armor overview: https://cloud.google.com/security-command-center/docs/model-armor-overview
- Microsoft Learn: Data Security Posture Management (Microsoft Purview): https://learn.microsoft.com/en-us/purview/data-security-posture-management-learn-about