Something weird happened on a client call last week.
We were reviewing a chatbot trial for a mid-sized logistics firm. The bot was supposed to answer simple questions about shipping rates and delivery times. Instead, it spent twenty minutes trying to convince a prospect from Birmingham that it was "deeply passionate about the ethical sourcing of gravel" and then offered him a 400% discount if he could name three types of igneous rock.
The client looked at me. I looked at the screen. The bot, presumably, sat there in its digital void, feeling quite proud of its sudden expertise in geology.
We call these "hallucinations," which is a polite, Silicon Valley way of saying the software has decided to drop acid and start making things up. In any other industry, if a staff member started shouting about gravel and inventing non-existent discounts, you'd call an ambulance or at least suggest a very long holiday. In AI, we just call it a "probabilistic outlier" and hope nobody notices.
What are AI hallucinations in business?
To understand why your chatbot occasionally tries to sell your soul or invent new physics, you have to understand how it thinks. It doesn't actually "know" anything. It is essentially a very sophisticated version of the autocomplete on your iPhone, but instead of suggesting "ducking" when you're angry, it’s predicting the next most likely word in a vast mathematical sea of data.
Sometimes, the most likely word is just wrong. It’s like a waiter who doesn't speak your language but is so desperate to please that they just start nodding and bringing you random plates from the kitchen. "I didn't order the gravel, Pierre." "Yes, but it is excellent gravel, monsieur."
The problem is that we’ve spent years being told that computers are logical. We expect them to be cold, hard calculators. When they start being creative with the truth, it feels like a personal betrayal. But here is the secret: the hallucination is where the humanity lives.
If you want to read more articles on AI that don't involve geology, you'll see a pattern: the businesses that succeed aren't the ones trying to build a perfect robot. They're the ones who know how to put a leash on a smart but slightly eccentric one.
Why do AI bots lie to customers?
I’ve watched people treat ChatGPT like an Oracle. They ask it for legal advice, medical diagnoses, or—heaven forbid—market research. The bot, being a people-pleaser, won't say "I haven't got a clue, mate." It will invent a study from the University of Nowhere, cited by Dr. Imaginary, and present it with the confidence of a man who just won a pub quiz.
In a sales context, this usually manifests as:
- Inventing features your product doesn't have.
- Quoting prices that would bankrupt your company in a week.
- Promising delivery dates that require the use of a TARDIS.
- Developing an inexplicable obsession with gravel.
The reason it lies is simple: it is trained to be helpful, not necessarily truthful. If the truth is boring or unavailable, it fills the gap. It's essentially that one friend we all have who tells "true" stories about meeting celebrities, except the friend is plugged into your customer service portal.
How to stop your AI from making things up
You can't 100% cure an AI of its imagination, but you can give it a much tighter script. This is what we call "grounding." If you just let an LLM loose on the internet, it's a disaster. If you lock it in a room with only your company handbook and tell it "if it isn't in these pages, shut up," it becomes significantly more useful.
Here is a quick checklist for avoiding a digital PR disaster:
- Restrict the Knowledge Base: Use RAG (Retrieval-Augmented Generation). It's a fancy term for "only let the bot read your stuff."
- The "I Don't Know" Directive: Explicitly tell the AI it is allowed to say "I don't know." In fact, reward it for saying it.
- Temperature Settings: Lower the "temperature" in the API settings. A high temperature is a poet; a low temperature is an accountant. In sales, you usually want the accountant.
- Human-in-the-loop: For anything high-stakes, have a human click "approve" before the message goes out.
On Mondays, I usually spend my time looking at how SalesM8 handles these types of interactions. The goal isn't to replace the human; it's to give the human a superpower that doesn't occasionally hallucinate about igneous rocks.
The unexpected upside of AI blunders
There is a strange sort of charm in a bot getting it wrong. Last year, a major airline's chatbot reportedly promised a customer a refund that didn't exist. The customer took them to court and won. While the airline's lawyers were likely crying into their lattes, the rest of us got a timely reminder: AI is a tool, not a replacement for common sense.
When an AI messes up, it forces us to look at our processes. If a bot can't explain your pricing, it’s probably because your pricing is too complicated for a human to understand, let alone a machine. The bot isn't just failing; it's acting as a stress test for your business logic.
I’ve found that some of the best client breakthroughs happen right after an AI does something stupid. It breaks the tension. It reminds everyone that we are still in charge. It’s a lot easier to book a consultation to fix a broken bot than it is to fix a broken business model, but usually, we end up doing both.
Turning the "oops" into an "aha"
The next time your AI tells a prospect that you offer free unicorns with every subscription, don't panic. Take a screenshot, have a laugh, and then fix the prompt.
We are living through a period where the tech is smarter than us in some ways and dumber than a bag of hammers in others. Navigating that gap is where the real money is made. It’s about building trust, not just through perfect code, but through how you handle the imperfections.
If you can explain to a customer why the bot went off the rails, you’ve shown more transparency and personality than a thousand "perfect" automated emails ever could. People don't buy from robots; they buy from people who know how to use robots.
If you’re reading this on a Thursday, you might want to get the free book to see how we handle these quirks in the real world of sales. It contains significantly fewer mentions of gravel than this blog post, I promise.
In the end, AI is a bit like a toddler with a PhD. It can explain quantum physics to you, but you still shouldn't leave it alone with the crayons and a white sofa. Keep a watchful eye, keep your sense of humour, and for heaven's sake, check your discount settings.
Your igneous rock collection will thank you.
Want more stories like this? I share observations about AI, business, and life on the narrowboat at steventann.com.
About the Author
Steven Tann is an AI consultant, author of "You're Selling AI Wrong", and founder of SalesM8. He writes about AI, sales, and running a business from a narrowboat on the English canals. Connect with him at steventann.com.
