AI Made Up a Court Case. A Consulting Firm Charged $290,000 for It.
There's a 237-page report sitting on an Australian government website right now.
It was commissioned by the Department of Employment and Workplace Relations. It cost $290,000. It was produced by one of the biggest consulting firms in the world.
And it contained references to court cases that don't exist.
Not misquoted. Not misattributed. Made up entirely. Fabricated case citations. A quote attributed to a federal court judge who never said it. References to academic research papers that were never written by anyone, mate.
Deloitte Australia paid a partial refund after a researcher at Sydney University named Chris Rudge read the report and flagged it as "full of fabricated references."
Deloitte reviewed it, confirmed some footnotes and references were incorrect, published a revised version, and added a disclosure that Azure OpenAI had been used in its creation.
The substantive findings stood. The citations were just... wrong.
Now. Before you read this and think "serves the big consultancies right," I want to make sure you understand what actually happened here.
Because it's not a story about a dodgy firm cutting corners.
It's a story about how AI fails in a way that most people don't sort of expect.
---
AI Doesn't Know What It Doesn't Know
Here's the thing about AI hallucinations. The model doesn't know it's making something up. There's no moment of "hmm, I'm not sure about this one, I'll guess." The model produces the fabricated citation with exactly the same confidence as a real one. Same formatting. Same tone. Same weight.
That's what makes it dangerous.
But if AI gave you wrong answers in an obviously uncertain way, like a blank response or "I'm not sure about that," you'd verify everything. You'd treat it like a draft, not a deliverable. But AI is very good at sounding authoritative. It fills in gaps with things that look right.
Stanford HAI researchers tested this on legal AI tools specifically built to reduce errors. Not general-purpose chatbots. Specialist legal research tools from LexisNexis and Thomson Reuters, two of the most established names in legal information. Even those tools produced incorrect information more than 17% of the time on one platform and more than 34% on another.
One in six queries, at minimum. On tools designed to be accurate.
And that's legal AI, where accuracy matters enough that companies have invested heavily in reducing hallucinations. Apply that failure rate to a general-purpose AI writing a 237-page report and you have a maths problem.
---
The Company That Tried to Blame Its Own Chatbot
It gets stranger when companies realise what they've built.
In 2024, a passenger named Jake Moffatt asked Air Canada's chatbot about bereavement fares after a family member died. The chatbot told him he could book a full-price ticket now and apply for the discounted bereavement fare afterwards.
That wasn't Air Canada's policy. The discount had to be requested before the flight.
Moffatt booked the ticket, made the trip, then applied for the discount. Air Canada refused it and pointed out that the chatbot had been wrong. Their actual position, argued before the British Columbia Civil Resolution Tribunal, was that the chatbot was a "distinct legal entity" accountable for its own conduct.
Meaning: don't blame us, blame the bot.
The Tribunal dismissed that claim entirely and ordered Air Canada to pay Moffatt the difference.
The ruling was blunt. It should be evident to Air Canada that it is responsible for all information on its website, whether that information comes from a human or a chatbot.
That's the precedent now. If your AI tells a customer something, your business told that customer something.
---
What This Actually Means for Your Business
Right. So you're not Deloitte. You're not Air Canada. You're running a real business with real customers and you're probably using AI in at least a few places already.
Here's what the Deloitte story and the Air Canada ruling actually mean for you.
First: AI output needs a human somewhere in the chain.
The Deloitte problem wasn't that they used AI. It was that the AI output went from generation to a 237-page published report with no one checking the citations. The research may have been solid. But citations are the kind of thing that look right even when they're wrong. A human reading the final draft for accuracy would have caught it. Presumably, anyway.
The lesson is not "don't use AI for research." It's "don't ship AI-assisted work without a verification step."
Second: know which parts of your work carry legal or reputational weight.
Using AI to draft social posts? Lower stakes. An error gets corrected. Using AI to draft contracts, legal summaries, policy documents, client-facing advice? Higher stakes. That's where you need verification built into your process, not added as an afterthought.
The Air Canada situation arose because a chatbot was given the job of answering policy questions and no one had thought through what happened when it was wrong.
Third: if you're using AI to write anything that cites sources, check the sources.
This applies whether you're a solo consultant writing articles or a team using AI to produce reports for clients. AI will confidently cite things that don't exist. Not always. But often enough.
According to research compiled by Suprmind's 2026 hallucination statistics report, 47% of organisations using AI have seen at least one negative consequence from it, with inaccuracy being among the most commonly cited causes. That stat comes from McKinsey's 2025 Global Survey on AI.
Nearly half. And those are the organisations that noticed.
---
The Actual Fix (It's Not Complicated)
I run AI across most of my business. Writing, research, operations, client work. I have not sworn off AI because Deloitte had a bad citation. That would be like swearing off spreadsheets because someone put a wrong formula in a cell.
What I have done is built a habit of knowing where AI is reliable and where it isn't.
AI is reliable at structure, tone, synthesis, and summarising things it has solid context on.
AI is unreliable at specific facts, citations, statistics, legal references, and anything requiring it to "remember" something from outside its training data or the context you've given it.
So the rule is simple. If you're using AI to produce something that contains specific facts, citations, or legal statements, those need to be verified by a human against the actual source.
Not skimmed. Checked.
If you're using AI to talk to customers, test what it actually says in edge cases before you deploy it. Not just the happy path. What does it say when someone asks for a refund? What does it say when your return policy doesn't cover their situation? What does it say when it doesn't know the answer?
Because it will give an answer. That's the thing. It won't say "I don't know." It'll sound helpful right up to the moment you're standing in front of a tribunal explaining why your bot promised something you didn't authorise.
---
The Bigger Picture
AI is genuinely useful. I say this as someone who's built an operations model around it. I'm not writing this post to scare you off it.
But useful doesn't mean infallible. And the way AI fails, confidently, fluently, with great formatting, is different from how humans fail. When a person makes something up, there's usually a hint of uncertainty. When AI makes something up, it looks exactly like something real.
That difference changes how you need to verify AI output.
Deloitte learned that the expensive way.
Air Canada learned that in a tribunal.
You don't have to.
---
Sources:
- Fortune, "Deloitte was caught using AI in $290,000 report to help the Australian government," October 7, 2025: https://fortune.com/2025/10/07/deloitte-ai-australia-government-report-hallucinations-technology-290000-refund/
- BBC Travel, "Airline held liable for its chatbot giving passenger bad advice," February 23, 2024: https://www.bbc.com/travel/article/20240222-air-canada-chatbot-misinformation-what-travellers-should-know
- Stanford HAI, "AI on Trial: Legal Models Hallucinate in 1 out of 6 (or More) Queries," May 23, 2024: https://hai.stanford.edu/news/ai-trial-legal-models-hallucinate-1-out-6-or-more-benchmarking-queries
- Suprmind, "AI Hallucination Statistics 2026: 50+ Sourced Data Points," May 16, 2026 (citing McKinsey 2025 State of AI): https://suprmind.ai/hub/insights/ai-hallucination-statistics-research-report-2026/
---
If you want to talk this through with people who are actually doing it, come join us in The Autonomous Business on Facebook. No gurus, no fluff, just real conversations with business owners building systems that run themselves.
---
About Steven Tann: Steven helps business owners build systems that run themselves using AI. After 10+ years helping 7,000+ businesses and building his own autonomous operations, he's the bloke who actually does it, not just talks about it. Find out more at steventann.com.