There's a bloke I know, decent small business owner, runs a fit-out company in Bristol. He asked his AI assistant a fairly simple question about whether he should take drawings or salary from his limited company given his current turnover.
The AI answered immediately. No hesitation, no caveats. It walked him through the logic with bullet points, used his actual numbers, and concluded he should increase his salary. He followed the advice. He didn't call his accountant. He figured the AI had sorted it.
Six months later, his accountant looked at the year and told him he'd been paying himself in the least tax-efficient way possible for his situation. Somewhere between three and four thousand pounds worse off than he needed to be.
The AI wasn't guessing. It wasn't uncertain. It gave him a confident, well-structured, entirely wrong answer.
And that's the thing about AI that nobody talks about clearly enough.
Confidence Is Not Accuracy
AI tools don't signal when they're unsure the way a person does. When your accountant isn't certain, she'll say "I want to check that before I advise you." When your mate at the pub is guessing, you can usually tell from his face. When an AI is working from incomplete or misapplied information, it produces the same smooth, well-formatted, authoritative-sounding output as when it's absolutely right.
This is the problem economists call a "credence good". Like a mechanic's diagnosis or a doctor's recommendation, you often can't tell whether the advice was good until much later, sometimes years. A bad tax decision might not surface until an audit. A dodgy cash flow model might not bite until the economy turns.
And research published in the June 2026 edition of the Journal of Financial Planning by professors at the University of Rome Tor Vergata and the University of Georgia tested seven of the most popular AI platforms, including ChatGPT, Claude, Gemini, Copilot, DeepSeek, Meta AI, and Perplexity. They gave each platform identical financial planning scenarios covering emergency savings, retirement withdrawals, and investment allocation.
The results weren't reassuring. The platforms gave inconsistent answers to the same questions. And in some scenarios the advice shifted when the demographics of the hypothetical person asking changed. Same question about retirement savings. Different answer depending on whether the head of household was described as a man or a woman, or as white or Black.
Same question. Different answer. No warning that the answer might vary.
One in Five People Following AI Financial Advice Have Already Lost Money
It's easy to treat this as a theoretical risk. It isn't.
A 2025 survey of 2,000 US adults by Pearl.com, a professional services platform, found that 19% of people who used AI chatbots for financial advice lost more than $100 following that advice. Among Gen Z investors, the figure rose to 27%.
That's not a fringe failure rate. That's roughly one in five people who asked an AI about their money and got burned.
And the numbers using AI for financial questions are going up fast. Pew Research in 2025 found that 34% of US adults had used ChatGPT, roughly double the share from two years earlier. Among under-30s, it was 58%.
More people asking. More confident wrong answers going out. Fewer people checking.
The "Confident Wrong" Problem Goes Beyond Finance
So here's where it gets broader than just tax advice.
VentureBeat Pulse Research from June 2026 surveyed enterprises using AI agents and found that 57% had watched their AI agents be "confidently wrong," producing incorrect outputs with no sign of uncertainty.
And the Stanford HAI 2026 AI Index Report found that AI agents, even the best ones, still fail roughly one in three attempts on real-world structured tasks. They improved from 12% task success to 66% over a few years, which is a genuine achievement. But 66% success is another way of saying it fails one in three times.
In a benchmark. Under controlled conditions.
In your actual business, with your actual messy data and edge cases, the failure rate is almost certainly higher.
AlixPartners' July 2026 analysis of the AI confidence problem puts it plainly: the task for business leaders isn't to eliminate errors. It's to build operations that benefit from useful AI without being compromised by fallible AI. Those are two different things, and most businesses aren't doing the second one.
The Quiet Failure Is the Dangerous One
There's a version of AI failure that's obvious. The AI tells you a meeting is at 11am and it's actually at 3pm. You notice immediately, you fix it, you stop trusting the AI for calendar management.
That's the safe failure. You catch it quickly.
The dangerous failure is the one that sounds right. The confident tax advice. The coherent cash flow projection. The authoritative-sounding market sizing estimate. These don't set off any alarms because they look exactly like good work. The bloke in Bristol didn't think to question it because it came back with his actual numbers in a neat, logical structure.
The AI didn't know what it didn't know. And it didn't tell him that, either.
Three Practical Things You Can Do Right Now
This isn't an argument against using AI. I use it constantly, and so should you. It's an argument for using it with the right mental model.
1. Know which decisions are credence goods.
Tax, legal, medical, financial. These are areas where the feedback loop is slow and the cost of being wrong is high. Use AI to learn and to generate options. Don't use it as the final word. Run it past a qualified human before you act, especially when the AI is giving you specific numbers or recommendations.
2. Test your AI's answers against something real.
Before you rely on an AI output for anything important, cross-reference it. Ask the AI to cite its sources. Google the specific claim. If the AI can't point you to a real source for a specific number or recommendation, treat that output as a starting point for research, not a conclusion.
3. Build verification into your workflow, not just your intentions.
Good intentions don't hold up under time pressure. If you want your team to verify AI outputs before acting on them, you need to make verification part of the process itself. That means a checklist, a second step in the workflow, a human sign-off before anything goes live. The goal of The Autonomous Business isn't to replace human judgment. It's to free up human judgment for the things that actually need it.
Your AI is confident. That's sort of the whole point of it. And it's also the thing you need to watch most carefully.
---
Sources
- When managing your money, take a chatbot's 'confidence' with a grain of salt, The Conversation, July 2026. Cites Pearl.com 2025 survey (2,000 US adults, 19% lost money), Pew Research 2025 (34% US adults using ChatGPT).
- Do Different Generative Artificial Intelligence (GenAI) Tools Provide Different Financial Recommendations?, Journal of Financial Planning, June 2026. University of Rome Tor Vergata and University of Georgia.
- AI Financial Advice May Be Biased, Study Finds, Money.com, July 2026.
- 57% of Companies Experience AI Agent Errors: Understanding Ownership of the Context Layer, Hitechub (citing VentureBeat Pulse Research, June 2026).
- Stanford AI Index 2026: Agent Task Success Hits 66%, Arahi AI, April 2026. Sourced from Stanford HAI 2026 AI Index Report.
- Why AI Is Still Confidently Wrong, AlixPartners, July 2026.
---
Want to try this yourself? Grab the free AI Setup Kit. 77 plug-and-play skills for content, sales, marketing, and operations. Works with Viktor, Claude, or ChatGPT. No course, no tool to configure, just markdown files and your AI.
---
About Steven Tann: Steven helps business owners build systems that run themselves using AI. After 10+ years helping 7,000+ businesses and building his own autonomous operations, he's the bloke who actually does it, not just talks about it. Find out more at steventann.com.