Australia spent $565 million finding out what happens when you automate a decision without keeping a human in the loop. I want to walk you through it, because the lesson isn't a government-scale lesson. It's a lesson for anyone running a business who's about to hand a process over to a machine.
Between 2015 and 2019, the Australian government automated welfare debt recovery. The system compared a person's reported fortnightly income against their annual tax data, averaged it out, and if the numbers didn't match, it fired off a debt notice. No caseworker checked it first. No one asked whether averaging a year's income across a fortnight actually reflected what someone earned in any given week. It didn't, and it couldn't, not for anyone with irregular hours, casual shifts, or a job that started partway through the year.
The scheme became known as Robodebt. It issued 453,000 unlawful debt notices before a Royal Commission tore the whole thing apart, calling it, in the words of Commissioner Catherine Holmes, "a crude and cruel mechanism, neither fair nor legal." The net cost to the Commonwealth: $565 million. Not in fines. In cleanup, court costs, and repayments to the people it wrongly chased.
Here's the bit that matters for you and me. The Royal Commission's finding wasn't that the algorithm was broken. The maths did exactly what it was told to do. The failure was structural: three specific things went wrong, and every one of them is a thing small business owners do when they rush to automate.
The three failures, and why they're not just a government problem
One: there was no feedback loop. Nobody experiencing the wrongly issued debts had a fast way to tell the system "this is wrong" and have that actually change how the system behaved. Complaints went into a queue, got looked at eventually, and by the time anyone acted, thousands more notices had already gone out. Compare that to a business that automates its customer replies, its pricing, or its lead follow-up, then only checks the results once a quarter. The damage compounds every day between checks.
Two: institutional silos. The people who designed the policy weren't the people who ran it day to day, and neither group had full visibility of what was happening on the ground. Sound familiar? It's the same reason a business owner's automated invoicing system can go on double-charging a client for three months before anyone notices, because the person who built the workflow isn't the person who reads the invoices, and the person who reads the invoices assumes the workflow is fine because nobody told them otherwise.
Three: rules were automated that needed judgement, not rules. Working out whether someone genuinely owed money involved context: irregular income, gaps in employment, individual circumstances. That's a judgement call. Robodebt turned it into a formula. The formula couldn't ask "does this actually make sense for this person," so it didn't.
And that third one is the one I see business owners get wrong constantly. Automating a repetitive, rule-based task, like sending a reminder email or logging an expense, is low risk because there's no judgement required. Automating a decision that affects someone's money, their contract, or their trust in you, without a human checking the output, is a different animal entirely.
This isn't just a government-scale problem
Look, it's tempting to read the Robodebt story and think "that's a massive government IT disaster, nothing to do with my 12-person business." But the pattern shows up at every scale. RAND Corporation's 2024 research on AI project failures, based on interviews with 65 experienced data scientists and machine learning engineers, found that more than 80% of AI projects fail, roughly twice the failure rate of ordinary IT projects that don't involve AI at all. And RPA, the more basic rules-based automation that came before the AI boom, has its own well-documented track record: Ernst & Young's analysis, based on RPA projects delivered across 20 countries, found that 30 to 50% of initial RPA projects fail. Not because the technology is bad. Because of the same three structural mistakes as Robodebt, just at a smaller scale: no feedback loop, no one owning the whole picture, and rules automated that actually needed a person to think about them.
I build automated systems for a living now. The Autonomous Business framework I use has four pillars: Memory, Automation, Integration, and Direction. Most people who fail at this only ever build the second one. They automate a task and walk away. They skip Direction entirely, which is the pillar where a human still decides what matters, checks the output, and steers the thing when it starts drifting.
What I actually do differently
When I automate something in my own operation, whether it's a client's CRM follow-up sequence or the pipeline that writes and publishes this very blog post, I ask three questions before I switch it on:
Where's the feedback loop? If the system gets something wrong, how fast do I find out, and how fast can I fix it? For low-stakes stuff, a weekly review is fine. For anything touching money, contracts, or a client's public reputation, I want same-day visibility, not "I'll notice next month when someone complains."
Who owns the whole picture? One person, not a committee, needs to understand the entire workflow end to end, as well as the bit they didn't personally build. If I hand a client's lead qualification off to an automated system, I make sure someone (often me) can trace a lead from first contact through to the human who eventually talks to them, and see every automated decision along the way.
Does this step need judgement or does it need a rule? Sending a booking confirmation is a rule. Deciding whether a client's complaint warrants a refund is judgement. I automate the first kind freely. The second kind gets a human in the loop, always, no exceptions, even if it slows things down by a day.
But none of this is complicated, not really. It's sort of the same rule every time: the difference between automating a task and automating a decision. Tasks are safe to hand over. Decisions that affect someone's money, their standing, or their trust in you need a person checking the output, at least until the system has proven itself over hundreds of cases with a human watching.
So the businesses I see get burned by automation aren't the ones who moved too slowly. They're the ones who built the Automation pillar and skipped Direction, then wondered why the system had gone sideways for six weeks before anyone caught it. Robodebt ran for four years before it was stopped. Most small business mistakes get caught faster than that, thankfully, but "faster than a $565 million government scandal" is a low bar to clear.
So build the feedback loop before you build the automation. Know who owns the whole picture. And keep a human on anything that requires judging a person's specific situation rather than following a rule. Get those three right, and you'll avoid the version of this story that shows up in your own business, at whatever scale you're operating at.
If you want to talk this through with people who are actually doing it, come join us in The Autonomous Business on Facebook. No gurus, no fluff, just real conversations with business owners building systems that run themselves.
Brewed by Steven, poured by Viktor
About Steven Tann: Steven helps business owners build systems that run themselves using AI. After 10+ years helping 7,000+ businesses and building his own autonomous operations, he's the bloke who actually does it, not just talks about it. Find out more at steventann.com.
---
Sources:
- Holmes, C. (2023), quoted in "3 lessons from a $565M automation failure for governing AI", Zen Ex Machina, 19 Jul 2026: https://zenexmachina.com/three-lessons-from-a-565m-welfare-automation-failure-for-governing-ai/
- RAND Corporation, "The Root Causes of Failure for Artificial Intelligence Projects and How They Can Succeed", 2024 (via Healthcare Reimagined summary): https://healthcarereimagined.net/2024/08/15/why-ai-projects-fail-and-how-they-can-succeed-rand/
- Ernst & Young RPA project failure analysis, cited in "Why RPA Implementation Projects Fail", CMSWire: https://www.cmswire.com/information-management/why-rpa-implementation-projects-fail/