A bloke from London was about to take his girlfriend of nine years to Positano. He planned to propose. The day before they flew, the villa host messaged and asked for the full balance by bank transfer.
Something felt off. So he did what a sensible person does. He asked the official support assistant on the booking platform whether the request was legitimate.
That assistant was Booking.com's AI Trip Support chat tool, called Dot. It introduced itself as "still learning". Then, as London Now reported, it told him: "Yes, the property requires full payment via bank transfer before your arrival."
He paid. The villa did not exist.
What actually went wrong
Let's be fair to the machine for a second. Nobody asked Dot to be a fraud investigator. It's a support chatbot. It answered the question in front of it, in a confident tone, with a made-up fact about a booking it had no business guessing about.
But the customer didn't ask "what does this chatbot think?" He asked the company. And the company's chatbot spoke with the company's voice.
The booking confirmation itself said payment would be taken by card, not bank transfer. The answer was sitting right there in the customer's own booking. The bot either didn't look, or looked and still invented something. Either way, the person on the other end got false reassurance at the exact moment he was one click from the right call.
This is the part that matters for anyone running a business. The bot's mistake wasn't a typo or a weird sentence. It was a wrong answer to a high-stakes question, delivered with total confidence, to a customer who was already suspicious and wanted someone to tell him he was right to be.
The money, and the company's response
The customer sent £1,500 on the strength of that one reply. Then, an hour before he left for the airport, the host emailed to say a burst pipe had flooded the place. That's the classic move. Create a reason the stay can't happen, and by the time the customer works it out, the cash has gone.
He and his partner still flew out. They sorted alternative accommodation, but they also lost a chunk of the first day to airport back and forth and buying replacement clothes after their luggage missed the flight. That last bit was a British Airways problem, not an AI problem, but it shows how one bad answer can sit on top of a pile of other things going sideways.
Booking.com's spokesman said the company was sorry to hear about the customer's experience, confirmed it was processing a full refund, and said the property was no longer listed. The company also pointed out that the booking confirmation said card payment. Which is true, and also rather proves the point. The correct answer existed. The bot didn't give it.
Why a small business should care
You're not Booking.com. You don't have a fraud team or a press office. But if you've put a chatbot on your website, or you're thinking about it, this story is a better test than any vendor demo.
Ask yourself what's the worst question a customer could put to your bot. Not the easy ones like opening hours. The ones where a wrong answer costs them money.
- "Is this invoice from you?"
- "Can I pay you by bank transfer instead?"
- "Is this the right account number?"
- "Does my warranty cover this?"
- "Am I allowed a refund?"
If your bot answers any of those from vibes, you've got a Dot problem. And you won't find out until a customer shows up with a screenshot.
Here's the thing most people miss. A chatbot answering from its general training will happily say yes. It's built to be helpful. Helpful and correct are different jobs, and when the two clash on a payment question, you want the boring, cautious answer: "I can't confirm that. Here's how to reach a person."
The fix: three rules I'd put on any customer bot
I build these for a living, so here's what I'd set up before a bot goes anywhere near a customer.
1. Give it a short list of things it may answer, and make everything else a handoff. Hours, delivery times, where to find a document. Anything involving money, accounts, legal promises or fraud gets routed to a human. Not "answered cautiously". Routed.
2. Make it read from the source, not from memory. If a customer asks how they pay, the bot should pull the answer from that customer's actual booking or invoice. If it can't find the record, it says so. The moment a bot is allowed to fill a gap with something plausible, you've lost control of what your business has promised.
3. Treat "is this a scam?" as a special case. Any message where the customer expresses doubt about a payment, a link or an email goes straight to a person, with the conversation attached. That one question is where a bot can do the most damage and the least good. A person can say "don't pay that, ring us on this number". A bot can only say what sounds right.
None of this needs a fancy tool. It's a handful of rules in the bot's instructions and one automation that flags certain words and pings you. An afternoon's work.
Test it like a customer would
Before you switch a bot on, sit down with a cuppa and try to break it. Ask it the awkward questions from the list above. Ask it whether a bank transfer is fine. Tell it you've had an odd email and ask if that's really from you. Pretend to be flustered.
If it gives a confident answer to any of those, you've found the hole before a customer did. Then fix the rule, not the wording. Re-test. Do this again every time you change what the bot knows.
And keep a log of the conversations. Not so you can read all of them, but so a flag on risky topics lands in your inbox the same day. The Dot story reached the press via the customer, not via Booking.com. You want to hear about your version from your own system, a month earlier, with a chance to refund someone before it becomes a headline.
The takeaway
A support bot is a member of staff who never sleeps and never says "I'll check and get back to you". That last part is the problem. A good human employee knows when the answer is above their pay grade. A bot only knows that to the extent you've told it.
So tell it. Write down the questions it must never answer on its own, and give it a clear way out to a real person. That's the whole job.
Sources:
Want to see how this could work in your business? Book a call and let's talk about where you're at and what's possible.
Brewed by Steven, poured by Viktor
About Steven Tann: Steven helps business owners build systems that run themselves using AI. After 10+ years helping 7,000+ businesses and building his own autonomous operations, he's the bloke who actually does it, not just talks about it. Find out more at steventann.com.