Quick one before we start: AI Build Night is on tonight, 6pm London. Come along.
Thirty-nine people got the same email from me on 15 October. It was a thank-you for a Build Night session. The session was on 30 September.
That's a bit awkward. A thank-you for a session two weeks old, landing out of nowhere with a stale date on it.
So this is a post about how I did that to my own list. And about what I changed so it can't happen the same way twice.
What actually happened
Every Wednesday after Build Night, a job pulls the Zoom attendance list, writes up a summary, and sends each person a follow-up with the replay. Nobody touches it. That's the point of the thing.
Each week the job gets its own little runner file. It holds the session date, the summary and the list of who attended. Old runners stay in the folder afterwards. They're handy as a reference.
On the 15th I wanted to test a change to the email. The quickest way to test it was to copy last fortnight's runner and tweak it. I know. I can hear you.
That old runner had no guard on it. No dry-run default, no "are you sure". It did exactly what it was built to do. It looked up the registrants, matched the attendees from the 30 September session, and sent the emails. For real.
The machine did nothing wrong. It followed the file I handed it. I handed it a loaded one and called it a test.
The boring truth about most automation mistakes
Look, everyone worries about AI going rogue. In my experience, the more common failure is a perfectly obedient system doing exactly what an old instruction says, at exactly the wrong time.
And the code was fine. The process around it was sloppy. Three things were wrong:
- Sending was the default. A runner that sends unless you tell it not to is a runner waiting for a bad Wednesday.
- Old runners were left lying around. A finished job's script is a live round. It should be locked away or retired.
- My test path and my real path were the same path. I had no separate way to rehearse.
So if you run any kind of scheduled sending, bulk emails, invoices, review requests, texts, check yours against that list. I'd bet you've got at least one.
What I changed
I rebuilt the flow straight away. Here's how it works now.
The Wednesday night run doesn't send anything. After the session it pulls the Zoom attendance, builds that week's runner from a clean template, and runs it in dry-run mode only. Then it posts a message in the Build Night channel saying "prepared for this many registrants, sends automatically once the replay is live".
A run without the explicit send flag does nothing. It prints HELD and stops. If I run it by accident, nothing leaves the building. That's now the correct behaviour, not a bug.
A second job waits for the replay. It checks every twenty minutes from Thursday to Saturday. Once the recording or the YouTube link exists for that session date, it runs the prepared runner with the send flag. Exactly once. It writes a little state file so it can't fire twice.
If the runner is missing, it shouts. One alert to me, not silence.
And there's a bonus. The email now goes out when the replay is actually ready, instead of Wednesday night before the recording exists. Registrants get the link when the link works. That one is better for the reader, and I only got there because I'd made a mess.
The part that was already working
It wasn't all bad news. Earlier that week I'd changed what the email links to. Every registrant now gets their own replay page, not a generic link. It plays the video, and it tells me who opened it, who pressed play and how far they got.
So when I look at the Thursday numbers, I'm not guessing whether people watched. I can see it. That's the sort of thing an autonomous system is good at. It does the dull bookkeeping and hands me the answer.
But good tracking doesn't help when the thing being tracked shouldn't have gone out at all. Measuring a mistake more accurately is still a mistake.
The rule I wrote down
One line, now at the top of that job's notes: never copy an old runner to test anything. Test only with dry-run on the template-derived file.
That sounds like a small thing. But it's the whole fix. The system had no memory of the mistake until I wrote it down where the next run reads first. Every automation I run now has a notes file that the next run reads before it does anything. A rule that lives only in my head is a rule the machine will break on a Wednesday night.
Why I'm telling you
I could have said nothing. Thirty-nine people got a duplicate and most would have shrugged. But I run these systems in public on purpose. Selling AI ops without showing the bits that go wrong is just a brochure.
And the lesson isn't really about email. It applies to anything where a script can touch a customer. Invoice runs. Review requests. Text reminders. Follow-up sequences. Each of them has an old, finished version sitting in a folder somewhere, ready to fire if somebody points it at a live list.
Ask yourself one question about each of your sends. If I ran the oldest version of this by accident right now, who would hear about it? If the answer is "customers", you've got work to do.
What to take from this
If you've got anything that sends to real people on a schedule, do three things this week:
- Make sending opt-in. Default to a dry run that shows you what would go out, and make real sends take a deliberate flag.
- Retire finished jobs. Move last month's one-off scripts out of the way, or strip the send step out of them.
- Give yourself a rehearsal lane. Test sends should go to you and only you. Never to a copied list.
Do that and the worst Wednesday you'll have is a message that says "prepared, waiting for the replay".
I'd rather tell you about the cock-up than pretend my systems never have one. The mistake cost me a slightly red face with 39 people. And now there's a proper guard rail where a loose script used to be.
Tonight's AI Build Night. 6pm London, live on Zoom, me building one of these systems from scratch while you watch. No slides, no rehearsed demo, and when it breaks on camera you get to see how I unbreak it. That's usually the useful bit. Grab a seat.
Brewed by Steven, poured by Viktor
About Steven Tann: Steven helps business owners build systems that run themselves using AI. After 10+ years helping 7,000+ businesses and building his own autonomous operations, he's the bloke who actually does it, not just talks about it. Find out more at steventann.com.