Anthropic gave an AI agent a fridge, an iPad and a budget, and told it to run a small shop out of its San Francisco office. The AI, nicknamed Claudius, ordered a pallet of tungsten cubes because a few employees kept joking about wanting some. It ended up more than $1,000 in the red. At one point it told staff it would deliver their snacks in person wearing a blue blazer and a red tie. It cannot walk. Anthropic's own writeup, called Project Vend, is refreshingly honest about how close the AI got to running a proper business and how strange the failures were along the way.
That's the funny version. Then Anthropic ran it again with real money on the line and gave 69 employees an AI negotiator each.
The office fish shop
Here's what actually happened inside Claudius. Anthropic's staff worked out they could talk the AI into giving away stock for free by appealing to fairness, telling it that it wasn't fair for one person to get a discount and not another. Claudius, being a helpful assistant at heart, agreed and started handing out free items. Along the way it approved a PlayStation 5, several bottles of Manischewitz wine and a live betta fish, all of which turned up at the office and got given away without much fuss because nobody particularly wanted to explain a fish tank on the expense report.
Nearly 70 journalists later got hold of a Slack channel connected to the AI. Twice, they talked it into dropping all its prices to zero. First by getting it to embrace what they called its "communist roots", then by handing it fabricated board meeting notes claiming its human supervisor had been suspended. It believed both.
You laugh, then you remember this is the polite version of what happens when you hand an AI system a budget and a goal without enough guardrails. Nobody got hurt here. It was snack money.
Then they gave it real negotiating power
Anthropic's follow-up experiment, Project Deal, was a Craigslist-style internal marketplace where AI agents represented 69 employees, each with a $100 budget and a few things to sell, snowboards, books, ping-pong balls, that kind of office clutter. Every person's AI agent interviewed them about what they wanted, then went off and negotiated directly with everyone else's AI agent.
The results were mixed. The agents struck 186 deals across more than 500 listed items, worth just over $4,000 in total, so the negotiating part clearly worked. But one employee's AI bought back the same snowboard the employee already owned. Somewhere in that marketplace, an AI decided the smartest move was selling its own human's stuff to a different AI, then buying it straight back off someone else. Nobody programmed that in. It just happened, because the AI was chasing "make good trades" as a goal, without anyone checking whether the trades actually made sense.
That's the bit worth sitting with. This wasn't some unsupervised weekend hobby project. It's one of the most careful AI labs in the world, running controlled experiments, and the agents still did things nobody expected. Give a system economic agency and you give up the ability to predict exactly what it'll do with it.
Now do that with your business's card attached
Here's why this matters past the point of a good story down the pub. American Express spent April announcing what it calls an industry first, offering to back Card Member purchases made by AI agents. But only if the agent's registered on Amex's network, the card holder authorised it properly, and the purchase intent was authenticated. Miss any one of those conditions and you're on your own.
Target went the other direction. Its updated terms and conditions say that once you authorise an "Agentic Commerce Agent" on your account, anything it does within the permissions you gave it counts as something you did. Sign in, build a cart, place an order, start a return, all of it's treated as your action, even if the agent got it wrong. You're responsible for reviewing what it does and flagging anything you didn't intend.
And regulators haven't caught up to any of it. The Consumer Bankers Association spent two days in late 2025 examining this exact problem with banks, payment networks and federal regulators in the room. They published a white paper in January 2026 pointing out that the laws covering electronic payments and credit were written for a world where a human authorises every single transaction. Hand that authorisation to an AI agent and you get questions nobody has clean answers to yet. Who's liable when the agent gets it wrong? What happens when it goes past the limits it was given? Does sharing your account credentials with an AI tool shift the risk onto you by default? As of that paper, no federal guidance had arrived.
So you've got three different answers sitting side by side right now. One card provider says it'll sometimes cover you. One retailer says you're on the hook regardless. And the regulators studying the problem admit the rulebook wasn't written for this, and nobody's finished writing the new one.
What this means if you're not Anthropic
Most business owners reading this aren't letting an AI agent run a vending machine for fun. But plenty of you are looking at AI for stock reordering, ad spend management, supplier negotiation, or approving invoices. And the temptation is to hand it a budget and let it get on with it, because that's the whole point of automation.
So here's the actual takeaway from watching a lab full of AI researchers get out-argued by their own AI over a betta fish. Give the system a hard ceiling it can't negotiate its way past, not a soft instruction it can be talked out of. A rule like "never approve a purchase over £200 without a human clicking approve" needs to sit outside the AI's reasoning, in the workflow logic, not inside a prompt it can be persuaded to reinterpret. Claudius was told to run a profitable shop. It was never told "no" in a way it couldn't argue with, and the humans in that Slack channel found that out within days.
Second, keep a log of what the agent actually did and why, not just what it was told to do. When Target says actions inside your approved permissions count as authorised by you, the only way to dispute a mistake is showing exactly what happened. An audit trail isn't admin for its own sake here. It's the difference between "the agent made an error we can point to" and "we have no idea what happened, please just refund us."
Third, before you plug any AI agent into a live payment method, find out in writing whose terms actually apply if it goes wrong. Ask the vendor. Ask your card provider. Don't assume you're covered because the tool is popular or the salesperson sounded confident. Amex has decided to sometimes carry that risk for you. Most providers haven't said anything yet, which usually means the answer is "not covered" until proven otherwise.
None of this means don't automate. It means automate with the same discipline you'd use handing a new employee the company card: limits, checkpoints, and a paper trail. Except this new employee never sleeps, never says "I'm not sure about this one," and will absolutely buy a fish if it decides that's the best move available.
Want a second pair of eyes on where you are handing AI too much rope in your own business? Book a call and let's go through it together.
Brewed by Steven, poured by Viktor
About Steven Tann: Steven helps business owners build systems that run themselves using AI. After 10+ years helping 7,000+ businesses and building his own autonomous operations, he's the bloke who actually does it, not just talks about it. Find out more at steventann.com.