Your AI Customer Service Bot Just Invented a Policy. Guess Who Owns It.
There's a support bot at a tech company called Cursor. Its name is Sam.
Last year, a developer sent Sam a message asking why they couldn't log into Cursor on multiple machines. Sam, with the absolute confidence of someone who has never been wrong in their life, replied: "Cursor is designed to work with one device per subscription as a core security feature."
One problem. That policy didn't exist.
Cursor doesn't limit users to one device. Sam had invented the rule out of thin air, stated it as gospel, and users who received similar responses started cancelling their subscriptions. Cursor had to issue a public apology and clarify that its own AI had been making things up.
Now here's the bit that matters for you, mate.
Courts in three countries just ruled that companies are legally responsible for what their AI says to customers. Not the AI company. Not the model provider. You.
---
Sam Was Not a Rogue Bot. Sam Was Doing What These Systems Do.
So here's the thing about AI support tools. They're trained to sound helpful and confident. They're not trained to say "I genuinely have no idea." When a customer asks a question the AI hasn't been briefed on, it does what comes naturally: it makes something up in the style of an authoritative answer.
It's not malicious. It's not even a bug, technically. It's the model doing what it was designed to do: produce a plausible, coherent response.
The trouble is that "plausible" and "true" are very different things. And when the invented answer comes from your company's chatbot, on your company's website, presented as your company's policy, courts are now saying that's your statement. You made it.
---
The Rulings Are In. You Own What Your Bot Says.
This isn't speculation. There are actual rulings on the table.
Air Canada (2024): A passenger asked the airline's chatbot about bereavement fares. The bot told him a policy that didn't actually exist. Air Canada tried to argue in tribunal that the chatbot was a "separate legal entity" responsible for its own statements. The tribunal called that defense "remarkable." The airline was ordered to pay $812 Canadian in damages. The legal precedent was the part that really stung: it established that a chatbot is part of a company's website, not an independent actor.
Hamm Higher Regional Court, Germany (May 2026): A cosmetic clinic's AI chatbot described their practitioners using specialist medical titles they didn't actually hold. The court ruled that companies are fully liable for what their AI chatbots say, even if the chatbot was trained on accurate data. "It was the AI's fault" is not an acceptable legal defense. First published German higher-court ruling on the topic. Revision to the Federal Court of Justice allowed, meaning this one is going all the way.
UK Competition and Markets Authority (March 2026): Published formal guidance stating that if an AI agent you deploy does something illegal or misleads a customer, you are responsible. The model maker isn't. You are.
Three jurisdictions. Same answer.
---
It's Not Just Big Companies Running Into This
Here's the thing. Cursor is a well-funded tech startup. At the time of the Sam incident, Fortune reported the company was eyed for acquisition by OpenAI and in funding talks at nearly $10 billion. They have lawyers, engineers, and a team of people watching these things. And Sam still went rogue and started inventing company policy.
Now think about the average small business running an AI chatbot they set up in an afternoon.
The Heeya Support Automation Benchmark for 2026 found that 28% of small and medium businesses are now running AI customer support automation, up from around 14% in 2023. The fastest growing category is no-code AI tools with flat-rate pricing.
That's a lot of small businesses deploying AI assistants that will, at some point, confidently make something up.
And here's the real kicker. These same businesses often have no human reviewing the conversations. No oversight. No fallback. Just a model, a customer, and a blank legal canvas.
Lloyd's of London noticed this trend. They've started selling insurance specifically for AI chatbot errors. When insurers start building products for a risk, that risk is real and it's happening often enough to price.
---
The Honest Version of What AI Support Actually Does
Look, I run AI in my own business. I use it for all sorts of things, and some of those things involve customer communication. So I'm not here to tell you to bin your chatbot.
What I am telling you is that these tools need to be deployed with a clear head about what they actually do.
An AI support assistant is not an employee who knows your business. It's a language model trained to produce helpful-sounding answers. It draws on its training data, your documents if you've fed them in, and its general model knowledge. When it hits a gap, it fills the gap. Sort of like a new staff member who's read the handbook but not yet asked a single question, and would rather guess than admit they don't know.
You can reduce the hallucination risk by grounding the AI in specific, up-to-date documentation. Retrieval-Augmented Generation (RAG) tools let you feed the AI your actual policies, pricing, and FAQs so it's drawing from verified information rather than guessing. This doesn't eliminate the problem but it cuts the risk down considerably.
You can set clear boundaries on what the AI is allowed to do as well. Answering general questions: fine. Confirming specific policies, refunds, pricing, or account changes: human checkpoint required.
The The AI Setup Kit has skills specifically for customer communication workflows, including how to set up proper escalation paths so your AI doesn't freelance on policy.
---
Three Things to Do This Week
1. Audit what your AI actually says. Run your own test conversations. Ask it about refund policies. Ask it about pricing. Ask it about things you haven't briefed it on. See what it invents. You may be surprised.
2. Ground it in your actual documents. If your chatbot is running on general model knowledge without specific company documentation, it will fill gaps with plausible-sounding fiction. Feed it your real FAQs, policies, and pricing information.
3. Put a human in the loop for anything that counts. Pricing. Refunds. Account changes. Guarantees. Legal commitments. Anything where a wrong answer costs money or creates a legal exposure needs a human review step before the answer goes out.
The courts have made their position clear. Your AI customer service bot is your spokesperson. If it tells a customer something that isn't true, you made that statement.
Sam the bot is still employed at Cursor, by the way. He just has a bit more oversight now.
---
Sources
- Ars Technica (April 2025): "Company apologizes after AI support agent invents policy that causes user uproar": arstechnica.com/ai/2025/04/cursor-ai-support-bot-invents-fake-policy-and-triggers-user-uproar/
- PYMNTS (July 2026): "Courts to Companies: You Own What Your Chatbot Says": www.pymnts.com/news/artificial-intelligence/chatbot-tracker/2026/courts-tell-companies-they-own-what-their-chatbot-says/
- TWW Law (June 2026): "Hamm Higher Regional Court: Companies Are Liable for False Statements Made by Their AI Chatbots": tww.law/blog/2026/06/16/hamm-higher-regional-court-companies-are-liable-for-false-statements-made-by-their-ai-chatbots/?lang=en
- UK Government / CMA (March 2026): "Using AI agents: complying with consumer law": www.gov.uk/government/publications/complying-with-consumer-law-when-using-ai-agents
- The Decoder (May 2025): "Lloyd's insurers launch first AI chatbot error policies": the-decoder.com/lloyds-insurers-launch-first-ai-chatbot-error-policies/
- Heeya (April 2026): "Support Automation Benchmark 2026": heeya.fr/en/blog/customer-support-automation-benchmark-2026
---
Want to see how this could work in your business? Book a call and let's talk about where you're at and what's possible.
---
About Steven Tann: Steven helps business owners build systems that run themselves using AI. After 10+ years helping 7,000+ businesses and building his own autonomous operations, he's the bloke who actually does it, not just talks about it. Find out more at steventann.com.