When AI fails publicly, the brand pays the bill.
From Air Canada's chatbot inventing refund policies to deepfake video calls authorizing $25M transfers, generative AI failures are no longer hypothetical. They are documented, litigated, and expensive. In 2025 alone, 346 AI incidents were recorded, with nearly half involving deepfakes. The question is not whether your AI will make mistakes. It is whether your governance can catch them before they reach the customer.
Executive Summary . What you will learn
- AI failures are now documented across every modality. Text chatbots have invented policies and sworn at customers. Voice AI has been used for election interference. Deepfake video has authorized millions in fraudulent transfers. Image generation has caused brand crises and operational disasters.
- Seven failure mechanisms repeat across every modality: hallucination as policy, prompt injection, ambiguity at the edges, over-automation, weak provenance, data leakage, and model drift. If your governance does not address each one, the failure is a matter of when, not if.
- The EU AI Act is now enforceable. Prohibited AI practices became enforceable February 2, 2025. High-risk AI system obligations take effect August 2026, with penalties up to 35 million euros or 7% of global turnover.
- NIST AI RMF provides the implementation framework. The four core functions, Govern, Map, Measure, and Manage, form the backbone of any enterprise AI risk program. March 2025 updates expanded the threat taxonomy to include generative AI risks.
- Eight governance controls form the minimum viable posture. System-of-record rule, evidence-first UX, tiered automation, eval gates, conversation firebreaks, human fallback, audit logging, and change management.
- Two questions separate governance theater from production control: Can your AI system cite the approved source for every claim it makes to a customer? When the AI fails, how fast can a human take over, and who owns that handoff?
Contents
346 incidents in 2025. The pattern is clear.
AI governance failures are no longer edge cases. They are a documented, growing category of enterprise risk with financial, legal, and reputational consequences.
In 2025, researchers recorded 346 AI incidents, nearly half of which, 179, involved deepfakes for voice, video, or image impersonation. Q3 2025 alone saw 2,031 verified deepfake incidents, with 48.3% targeting businesses. The shift from consumer-targeted scams to industrial-scale enterprise attacks is now complete.
The incidents span every modality. Text chatbots have invented policies, sworn at customers, and advised businesses to break the law. Voice AI has been used for election interference and drive-thru ordering failures. Image generation has caused brand crises when outputs were historically inaccurate or when marketing assets could not be operationally delivered. Deepfake video has authorized millions in fraudulent transfers by impersonating executives on Zoom calls.
The common thread across these incidents is not the technology. It is the governance gap. Most organizations lacked the rigorous AI governance needed, failing to implement basic security controls like multi-factor authentication for high-value approvals, runtime AI monitoring, and deepfake detection. Regulatory frameworks like the EU AI Act and NIST AI RMF are only now catching up, but many incidents revealed gaps between policy and technical implementation.
Every incident in this report shares the same root cause: the system was deployed without the governance controls that would have caught the error before it reached the customer. The model worked in the demo. It failed in production. The organization paid the bill.
The chatbot said it. Now you own it.
Customer-facing chatbots have hallucinated refund policies, sworn at customers, advised businesses to break the law, and provided harmful health guidance. In each case, the organization was held responsible for the output.
Air Canada: Chatbot invented refund policy (2024)
A customer relied on a chatbot response about bereavement fares and refunds. The advice turned out to be made up. A tribunal held the airline responsible for the misinformation and required honoring the fabricated policy. The legal precedent is now clear: "the bot said it" becomes a product promise when it is on your site, even if your terms say otherwise.
DPD: Chatbot swore at customers (2024)
A customer prompted a support chatbot into profanity and insults. DPD disabled it after the incident went viral. Brand and trust damage happened instantly when guardrails proved brittle. The exchange demonstrated that "brand voice" can be hijacked by prompt interactions unless guardrails are designed and tested like any other interface.
NYC MyCity: Chatbot advised breaking the law (2024)
New York City deployed a chatbot to answer business questions. Investigations found it produced incorrect guidance, including advice that could violate regulations. The city defended it publicly while issues persisted. Hallucinated "official" guidance creates legal exposure and erodes trust in public-facing systems.
NEDA: Eating disorder chatbot paused (2023)
The National Eating Disorders Association's chatbot, intended as a support tool, reportedly provided problematic dieting and ED-related guidance. The organization suspended it. In high-risk domains, "edge cases" are the cases that matter. Testing must include adversarial and vulnerable-user scenarios.
Unbounded generation presented as authoritative output is the failure mode. Chatbots and copilots can sound definitive even when guessing, inventing policy, or hallucinating citations. The governance control is simple: the model never invents policy. It can only quote and cite approved sources, and it must link to the canonical page.
Looks right is not evidence.
When LLMs are used for legal research, compliance advice, or decision support, the failure mode shifts from embarrassment to liability. Fabricated citations, leaked confidential data, and insecure code have all reached production.
Mata v. Avianca: Fake citations, real sanctions (2023)
Lawyers filed a brief with citations generated by ChatGPT that did not exist. The court sanctioned counsel, demonstrating a simple rule: if an output must be true, it must be verified against primary sources. LLMs can produce confident, plausible references. "Looks right" is not evidence.
Samsung and enterprise bans: Sensitive data pasted into chatbots (2023)
Reports emerged of employees at multiple companies pasting proprietary code and data into public chat tools, triggering internal bans and policy changes. The incident pattern is not malice. It is convenience plus unclear boundaries. Governance is not just the model. It is data handling, tooling, and policy enforcement.
AI-assisted code: Security quality regression (Research)
Multiple studies and practitioner analyses report that AI assistance can increase vulnerability rates and reduce secure-by-default behavior, especially for novices. The risk is systematic: speed goes up, review burden rises, and insecure patterns spread faster. The governance control is eval gates: offline eval sets, adversarial tests, and red-teaming before production.
Every claim needs a citation. Every citation needs verification. Every high-risk output needs human review. The failures often occur at the handoff to action: legal filings, money movement, public guidance, or health advice. Weak "last-mile" controls are the common thread.
I saw them on Zoom is no longer a control.
Voice cloning and speech-to-intent systems have moved from research curiosity to operational threat. Voice AI pilots have failed at scale, and deepfake voice has been used for election interference.
McDonald's ended AI drive-thru voice ordering pilot (2024)
McDonald's ended a voice AI test with IBM after inconsistent ordering performance. Public examples circulated showing errors. Speech-to-intent systems fail at the messy edges: accents, noise, ambiguity, and interruptions. The governance gap was unclear human fallback when the system failed.
New Hampshire "Biden" robocalls: AI voice deepfake (2024)
An AI-generated voice impersonating President Biden was used in robocalls to mislead voters. Enforcement actions followed. Voice cloning makes identity a weak signal. The governance control is out-of-band verification for high-risk instructions. "I heard their voice" is no longer proof of identity.
High-risk approvals need out-of-band verification. Voice and video identity are now weak signals. People over-trust what looks or sounds real and underestimate how easy it is to spoof identity and intent. The governance control is explicit: any approval above a threshold requires verification through a separate channel.
Synthetic images create expectation debt.
AI-generated images have caused brand damage when historically inaccurate outputs went viral, and operational chaos when marketing assets promised experiences that could not be delivered.
Google paused Gemini image generation (2024)
Google paused and limited Gemini's ability to generate images of people after widely shared inaccuracies in historically sensitive contexts. Brand and legal risk spikes when synthetic images are mistaken for "depictions of reality," especially around protected classes and sensitive events.
Willy's Chocolate Experience: AI marketing disaster (2024)
Promotions used AI-generated or AI-style visuals. Customers reported the real event did not match expectations and went viral as a "bait-and-switch." Refunds were demanded, police were called, and the story made global news. Synthetic marketing assets can create an expectation debt that operations cannot repay, triggering reputational blowback.
Every synthetic image needs human review before publication. Marketing assets must match operational reality. The governance control is tiered automation: low-risk outputs can be automated; high-risk outputs, anything customer-facing, anything that could be mistaken for reality, requires human review.
$25 million authorized by a video call that never happened.
Deepfake video has moved from theoretical threat to operational reality. Finance workers have authorized millions in fraudulent transfers after video calls where participants appeared to be executives but were entirely synthetic.
Hong Kong / Arup deepfake video call fraud (2024)
Reports describe fraudsters using deepfake video to impersonate executives in calls, leading to approximately $25 million in unauthorized transfers. The vulnerability was not only the media. It was the payment authorization workflow. "I saw them on Zoom" is no longer a control.
Q3 2025 alone saw 2,031 verified deepfake incidents, with 48.3% targeting businesses. The shift to industrial-scale attacks is complete. Corporate environments are now the primary target, with real-time deepfake CEO impersonation during video calls becoming a documented attack vector.
Approvals need cryptographic or out-of-band verification. The governance control is explicit: any financial approval above a threshold requires verification through a separate channel, not just visual or audio confirmation on the same call. Multi-factor authentication for high-value transactions is no longer optional.
Seven mechanisms. Every modality. Same patterns.
These failure mechanisms are portable across text, audio, image, and video. If your governance does not address each one, the failure is a matter of when, not if.
1. Hallucination as policy
Confidently wrong answers become product promises. Legal and brand exposure follows. The Air Canada case established the precedent: the organization owns what the chatbot says, even if the chatbot invented it.
2. Prompt injection / instruction hijack
Outputs and actions bypass intent and policy. Users can steer systems into off-brand or unsafe responses unless guardrails are designed and tested like any other interface. The Chevrolet incident, where a chatbot was manipulated into selling a product for $1, demonstrated the risk.
3. Ambiguity at the edges
Accents, sarcasm, mixed intents, and partial context cause operational failure. Speech-to-intent systems fail at the messy edges. The McDonald's drive-thru pilot demonstrated that "mostly right" is not a control when errors are visible to customers at scale.
4. Over-automation (no human fallback)
Small errors become incidents at the speed of automation. Without explicit escalation paths, SLAs, and ownership, the blast radius of a single failure expands to every user the system touches.
5. Weak provenance
No citations, no source-of-truth linkage. "Looks correct" becomes "is correct" in the user's mind. The Mata v. Avianca case demonstrated that fabricated citations can reach court filings when provenance is not enforced.
6. Data leakage
Employees paste secrets; logs retain sensitive data. Compliance and security incidents follow. The Samsung ban and similar enterprise responses demonstrate that convenience plus unclear boundaries creates systematic risk.
7. Model drift and vendor updates
Silent behavior changes break contracts and cause regressions. When the vendor updates the model, the behavior your users depend on can change without notice. The governance control is version pinning, regression tests, and release notes for every model and configuration change.
These mechanisms are not unique to any modality. They repeat across text, audio, image, and video. They repeat across customer service, legal, finance, and marketing. The governance controls that address them are also portable. The question is whether your organization has implemented them.
The EU AI Act is now enforceable. NIST AI RMF provides the framework.
Regulatory frameworks are catching up to the technology. Enterprises operating AI systems in or accessible to the EU market face enforceable obligations with significant penalties.
EU AI Act: Key Deadlines
- February 2, 2025: Prohibited AI practices became enforceable. Social scoring, real-time remote biometric ID in public, manipulative AI. Non-compliance can result in fines up to 35 million euros or 7% of global turnover.
- August 2, 2025: Rules for General Purpose AI (GPAI) and related governance obligations take effect.
- August 2, 2026: Original deadline for compliance with high-risk AI system obligations: risk management, conformity assessment, registration, transparency.
- December 2, 2026: Transparency requirements for AI-generated content, including watermarking and content marking.
- December 2, 2027: Extended deadline for high-risk AI systems in Annex III, including biometric ID, employment and recruitment algorithms, credit scores, and public services.
High-Risk AI Compliance Checklist
- Risk Management System: Enterprise-wide, documented, continuous risk assessment and mitigation plan.
- Data Governance: Training data must be of high quality and managed to eliminate bias and security risks.
- Technical Documentation: Comprehensive system documentation covering design, purpose, functioning, and limitations.
- Transparency: Users must be provided clear information about AI system usage and limitations.
- Human Oversight: Demonstrate real, effective human oversight is feasible and practiced.
- Accuracy, Robustness, Security: Implement safeguards to ensure system accuracy and resilience to attacks.
- Conformity Assessment: Most high-risk AI will need a CE marking via internal assessment or external notified body.
- Registration: List high-risk AI systems in the official EU database before placing them on the EU market.
- Post-Market Monitoring: Ongoing tracking and reporting for deployed systems, including incident reporting.
- Fundamental Rights Impact Assessment: Assess and document impacts on fundamental rights, separate from GDPR DPIAs.
NIST AI Risk Management Framework
The NIST AI RMF provides the implementation framework for enterprise AI risk management. The four core functions form the backbone of any AI risk program:
- Govern: Establish robust governance structures. Define clear policies for AI use, accountability, and risk management. Assign roles and responsibilities, ensure oversight, and foster a culture of risk awareness.
- Map: Systematically identify AI systems, their context, data sources, stakeholders, dependencies, and intended uses. Document potential risks and impacts.
- Measure: Continuously monitor and evaluate AI system performance, risks including bias, security, safety, and fairness, and outcomes using metrics and tests.
- Manage: Prioritize risk mitigation actions, implement controls, and establish continuous improvement. Monitor third-party AI risks and supply chain vulnerabilities.
March 2025 updates expanded the AI threat taxonomy to include generative AI risks: data extraction, poisoning, evasion, model manipulation, and supply chain vulnerabilities.
Compliance is required if your AI system is placed on or used in the EU market, accessible by EU users even via API from outside the EU, output is used within the EU, or you have an EU establishment. The reach is broader than many enterprises initially assumed.
Eight controls. One governance posture.
Each control maps to one or more failure mechanisms. Together, they form the minimum viable governance for any production AI system.
If you cannot check every box, you are not ready for production. The cost of governance is always less than the cost of a public failure. Air Canada learned this in court. The $25M deepfake fraud learned this in the wire transfer. The question is whether your organization will learn it before or after the incident.
Gate 0 to Gate N: Audit-safe AI delivery.
If you want audit-safe AI delivery, treat it like any other high-risk change: gated, evidenced, and reversible. This path works for both predictive models and LLM features.
Gate 0: Problem and Risk Framing
Confirm this is the right intervention. Clear user and business objective. Failure modes listed. Risk tier assigned for privacy, safety, and regulatory. Baseline process documented. No-go triggers: "we will know it when we see it," unclear owner, cannot state harms.
Gate 1: Data and Access Readiness
Ensure data can legally and ethically support the use. Data inventory and lineage. Legal basis and consent. Retention policy. PII handling. Training and serving parity plan. No-go triggers: unknown provenance, policy gaps, cannot reproduce dataset.
Gate 2: Evaluation Plan
Define how you will decide "works." Offline metrics tied to objective. Human review rubric where needed. Acceptance thresholds. Bias and safety checks scoped. No-go triggers: no thresholds, only anecdotal demos, no plan for edge cases.
Gate 3: Prototype and Red-Teaming
Stress the design before it hardens. Prototype meets minimum thresholds. Red-team findings triaged. Mitigations implemented: filters, guardrails, UX. No-go triggers: unmitigated critical harms, prompt-only "controls" for high-risk use.
Gate 4: Pilot (Limited Exposure)
Prove in-context performance. Instrumentation in place. Shadow or A/B where feasible. Operational runbook. Rollback path tested. No-go triggers: no monitoring, no rollback, ambiguous ownership of incidents.
Gate 5: Production Launch
Deploy responsibly. Go-live checklist complete. On-call and incident process active. Change management for model updates. Post-launch review scheduled. No-go triggers: "ship and hope," no communications plan, unreviewed last-minute model swap.
Gate N: Continuous Assurance
Keep it safe and useful over time. Drift checks. Periodic evals. Re-approval triggers. Audit trail functioning. No-go triggers: silent regressions, untracked prompts and models, missing audit evidence.
The goal is not bureaucracy. It is making decisions falsifiable. Each gate answers "what would make us stop?" When someone asks "why did you ship this?" you want a shelf of receipts, not a story.
Two questions separate governance theater from production control.
These questions expose the gap between vendor capability claims and what actually happens when the system reaches a customer.
Question 1: Can your AI system cite the approved source for every claim it makes to a customer?
This sounds technical. It is not. It is the single most consequential governance question an executive can ask about their AI program right now.
If the answer is no, then your system can invent policy. Air Canada's chatbot invented a refund policy. The tribunal held the airline liable. The legal precedent is now clear: you own what the chatbot says, even if the chatbot made it up.
A governed system looks different. Every claim traces to an approved source document. The user can see the citation. The system cannot generate claims that are not grounded in the source of truth. If the system does not know, it says so and escalates to a human.
Question 2: When the AI fails, how fast can a human take over, and who owns that handoff?
This is the blast radius question. Without explicit escalation paths, SLAs, and ownership, small errors become incidents at the speed of automation.
If the answer is unclear, then your system has no human fallback. The McDonald's drive-thru pilot ended because the fallback was unclear. The DPD chatbot went viral because there was no circuit breaker. The $25M deepfake fraud succeeded because the approval workflow did not require out-of-band verification.
A governed system has explicit escalation. There is a human who can take over. The handoff is fast. The SLA is defined. The ownership is clear. The incident response is documented.
Ask your platform owner: "Show me the audit log for an action the AI took on behalf of a user yesterday. I want to see the source citation, the escalation path, the human fallback owner, and the time to handoff if the system failed." If they cannot produce that log without engineering work, your platform does not yet have production-grade governance.
Sources & references.
All facts in this report are drawn from public disclosures, regulatory documents, vendor announcements, and named research outputs as of May 2026.
- Cybernews, 2025. 346 AI incidents recorded in 2025, from deepfakes and fraud to dangerous advice. Includes breakdown of incident types and modalities.
- Resemble AI Q3 2025 Deepfake Incident Report. 2,031 verified deepfake incidents in Q3 2025, with 48.3% targeting businesses.
- Air Canada Chatbot Case, 2024. Ars Technica, BBC, McCarthy Tetrault legal analysis. Tribunal held airline responsible for chatbot-invented refund policy.
- DPD Chatbot Incident, 2024. Reuters, ITV, The Guardian. Support chatbot prompted into profanity and insults, disabled after viral spread.
- NYC MyCity Chatbot, 2024. Reuters, The Markup, AP News. Business chatbot provided guidance that could violate regulations.
- NEDA Eating Disorder Chatbot, 2023. NPR, New York Times, Wired. Chatbot paused after harmful responses to vulnerable users.
- Mata v. Avianca, 2023. Reuters, New York Times. Lawyers sanctioned for filing brief with ChatGPT-generated fake citations.
- Samsung and Enterprise Chatbot Bans, 2023. Reuters, CNBC. Employees pasted confidential data into public chat tools, triggering internal bans.
- AI-Assisted Code Security Research. ACM, Schneier on Security. Studies report AI assistance can increase vulnerability rates.
- McDonald's AI Drive-Thru Pilot, 2024. CNBC, AP News, New York Times. Voice AI test ended after inconsistent ordering performance.
- New Hampshire Biden Robocalls, 2024. NH DOJ, NPR, BBC. AI-generated voice impersonation used in election interference, enforcement followed.
- Google Gemini Image Generation Pause, 2024. Reuters, CNBC. Image generation of people paused after historically inaccurate outputs.
- Willy's Chocolate Experience, 2024. BBC, Business Insider. AI-style marketing images created expectation debt, event failed, went viral.
- Hong Kong / Arup Deepfake Video Call Fraud, 2024. CNN. Approximately $25M transferred after deepfake video call impersonating executives.
- EU AI Act. Travers Smith, GDPR Register, Enzai, Axis Intelligence, Captain Compliance. Enforcement timeline, high-risk AI requirements, penalties.
- NIST AI Risk Management Framework. NIST official documentation, EPC Group, Cloud Security Alliance, ISPartners. Four core functions, March 2025 updates.
- InspectAgents, 2025-2026. Complete list of AI chatbot failures including prompt injection attacks and governance gaps.
- Adversa AI Top Security Incidents Report, 2025 Edition. Comprehensive analysis of AI security incidents and attack vectors.