A Pharmacy Chain Named Its AI Assistant After the Founder. Then It Started Getting Dosages Wrong.
Kinney Drugs has been filling prescriptions across upstate New York and Vermont for a long time. Long enough that when they built an AI assistant to handle patient calls, they named it after the bloke who started the company. They called it Burt.
Burt's job was simple on paper. Answer calls about prescriptions and refills, sort out the easy stuff, free up the humans for everything else. It went live in May. But within a few months, Kinney had pulled it back after hundreds of customer complaints. Not "a few grumbles on a Facebook post". Hundreds of people ringing a pharmacy about their medication and getting incoherent answers, wrong dosages, and missed refill notifications.
If you run any kind of health, care, or compliance-adjacent business, that one should worry you more than the others. Wrong dosage information from an AI voice assistant isn't a "the bot said something a bit odd" problem. It's a "someone could take the wrong amount of a medicine because a machine mumbled the wrong number down the phone" problem.
To Kinney's credit, they didn't try to spin it. President John Marraffa came out and said the quiet part in public: "getting privacy and security right does not mean we got the experience right. We did not, and we own that." Kinney went straight back to the old touch-tone phone system for anyone ringing in. Burt still handles outbound texts and refill reminders, but only if you opt in. That's the low-stakes stuff, where a wrong answer costs someone their afternoon rather than their health.
This Isn't a "Burt Was Rubbish" Story
Here's the bit that should worry you more than the dosage mix-ups. Kinney isn't some backyard operation that bolted a cheap chatbot onto its phone line and hoped for the best. It's a proper regional pharmacy chain with a compliance team, presumably a legal department, and enough scale to negotiate its own tech contracts. If a business like that can ship an AI phone assistant that gets patient information wrong at scale, then "we'll be fine, we tested it a bit" is doing a lot of heavy lifting in most small business AI rollouts.
And Kinney isn't some rare exception either. Sinch, a global business messaging company, surveyed over 2,500 senior decision makers this year and found that 74% of enterprises that put a live AI agent in front of customers have since rolled it back or shut it down entirely. Not 74% who ran a pilot that fizzled out in a slide deck nobody read. Seventy four percent who put the thing live, in front of real customers, then had to reverse out of it. The rollback rate climbed to 81% among the companies with the most mature governance and monitoring. So better monitoring doesn't stop the AI getting things wrong. It just means you catch it faster and yank it out sooner.
So if big companies with compliance departments and dedicated AI teams are rolling back three quarters of the customer-facing agents they ship, what odds do you reckon a five person business has if it wires an AI voice bot straight into its main phone line on a Friday afternoon and calls it done?
Where the Actual Mistake Was Made
The mistake at Kinney wasn't building an AI phone assistant. Automating prescription refill calls is a genuinely good idea. Most of those calls are the same five questions on repeat, and a system that handles them well would free up staff for the calls that actually need a human brain. The mistake was letting real, worried patients be the ones who found the edge cases.
That's the pattern behind nearly every AI rollout that ends up in the news for the wrong reasons. Nobody sets out to build something rubbish. They build something that works fine on the ten test calls somebody in the office ran through it, ship it to everyone, and then find out live, with real customers, all the ways a call about a blood pressure medication can go differently to how the ten test calls went.
If you're running a trades business, a clinic, an agency, anything where a customer rings up with a real problem, ask yourself honestly: if you switched on an AI phone or chat assistant tomorrow, who'd be finding the bugs? You, on a Tuesday morning with a cup of tea and nothing else to do? Or your actual customers, mid-crisis, while you're none the wiser until the complaints start rolling in?
The Fix Is Boring, Which Is Why Nobody Does It
You don't need a research lab to avoid Kinney's mistake. You need three things most businesses skip because they're not exciting.
A soft launch to your own team first. Route the AI through your staff's calls, or a handful of low-stakes real customers who've agreed to be guinea pigs, before it touches everyone. Let your own people find the weird accent it mishears or the question it answers confidently wrong.
A hard line on what it's allowed to touch. Kinney kept Burt for opt-in refill texts and pulled it entirely from anything involving dosage or prescription detail over the phone. That's the right instinct. Let the AI handle the low-consequence, high-repetition stuff, and keep a human between it and anything where a wrong answer actually hurts someone, financially or otherwise.
An escape hatch that works instantly. Kinney's fix wasn't a software patch, it was flipping straight back to the old touch-tone system. Whatever you're automating, know in advance what "off" looks like, and make sure someone can flip it without needing a developer on a Saturday.
None of that is clever. It's the same discipline you'd apply to training a new staff member. Give them the easy calls first, watch how they go, and don't hand them the complicated stuff until they've earned it. AI doesn't get a free pass just because it never sleeps.
Own It Like Kinney Did
The one thing Kinney genuinely got right was the apology. No corporate waffle about "an evolving customer experience journey", just a straight admission that they'd built something, it didn't work, and they were pulling it. That's worth more to customer trust than pretending it never happened. If your AI system breaks something, say so plainly, fix it, and move on. People forgive mistakes. They don't forgive being lied to about them.
Want to see how this could work in your business? Book a call and let's talk about where you're at and what's possible.
Brewed by Steven, poured by Viktor
About Steven Tann: Steven helps business owners build systems that run themselves using AI. After 10+ years helping 7,000+ businesses and building his own autonomous operations, he's the bloke who actually does it, not just talks about it. Find out more at steventann.com.