Quick one before we start: Build Night is on tonight, 6pm London. We're doing "Build Your Own Humanizer", the exact system this post is about. Details here.
I built a tool that scores my content for how much it sounds like AI. Every blog post, every LinkedIn comment, every Gmail draft goes through it before it goes out. If it doesn't hit 85 out of 100, it goes back and gets rewritten.
Last week I ran a deliberately terrible paragraph through it. Every corporate buzzword I've spent months teaching the checker to hate, stacked into one sentence on purpose, the kind of line that should set off every alarm I've built.
It scored 95 out of 100.
That's not a near miss. So the checker was telling me a sentence built entirely from the words it's meant to catch was basically clean. If I hadn't happened to run that exact test before tonight, I'd have kept trusting a gate that was, sort of, waving slop straight through as well.
Why I built this thing in the first place
Back in July a reader on LinkedIn called out a blog post for reading like obvious AI. Fair call, it did. So I built a checker that does two things before anything with my name on it gets published:
- Detection. Does the text avoid the words and phrases that give AI away. There's a list. It's long.
- Voice. Does it actually sound like me. Contractions, the way I start sentences with "So" or "Look", short punchy lines, the odd "mate" and "sorted".
It's pure Python. No model call, no cost, just word lists and pattern matching counting things. That's the whole appeal, it's cheap enough to run on everything, and it can't be talked into flattering me the way an AI reviewing its own homework can.
Where it broke
Tonight's Build Night is us pulling this thing apart with the room, so this week I went back through it looking for edge cases before I stand it up on a screen in front of people. Good move, because I found two.
The word list only caught the exact word. One of the banned verbs on my list, the fancy word people use instead of just saying "use", was banned in its plain form but not in the form ending in "-ing". The rule was written to match one specific shape of the word and nothing else, so the moment someone typed the gerund instead of the base verb, it walked straight past a filter built specifically to catch it. Same problem hit half a dozen other verbs on the list. The checker was set up to fail on tense.
The banned phrases needed to match exactly. If a phrase in my list was worded one way and the draft had it worded almost the same way but not word for word, it didn't match. AI writing doesn't repeat itself character for character, it riffs on the same handful of shapes with small variations. A filter that only catches the exact phrase catches almost nothing in the wild.
Put those two gaps together and you get exactly what I saw. A paragraph built entirely out of AI's greatest hits sailed through at 95 because half the hits were spelled slightly differently to how the filter expected them.
The fix, and why it matters more than the bug
I rewrote the word matching to catch the family of a word, not just one exact spelling, so every tense and form of a banned verb gets caught by one rule instead of three you have to remember to write separately. And I changed the phrase matching to catch near misses, not just character for character copies.
Same slop paragraph, same checker, after the fix: 30 out of 100.
That's the number that should worry anyone using an AI checker they didn't build themselves, mine included until last week. A 65 point swing on identical text, caused entirely by how the checker was written rather than anything about the content. If you're trusting a score to decide what goes out under your name, the score is only as good as the pattern matching behind it, and pattern matching is very easy to get subtly wrong in a way that looks fine until you go looking for it.
I regression tested it after the fix against six months of already published posts to make sure I hadn't introduced the opposite problem, real Steven sentences getting falsely flagged. No false positives. The gate now catches what it was always meant to catch. It just took a genuinely awful sentence, written on purpose, to prove it wasn't.
The bit that's actually useful to you
You don't need my exact checker. What you need is the two ideas behind it, because they apply to anything you build to catch a pattern automatically.
Test your filter with something that should obviously fail. Not a normal draft, a deliberately bad one built entirely from the thing you're trying to catch. If your filter passes the worst possible example, it's not working, whatever score it's showing you.
Word lists need to catch the whole family of a word, not just one spelling of it. If you're banning a word in its base form, you're also banning it in every tense and ending someone might actually type, and if you don't write the rule that way you've banned the word in the one form nobody actually uses.
Same goes for spam filters, invoice matching rules, lead scoring, anything where you've built a bit of automation that's supposed to catch a category of thing. Run something through it that should obviously trip the alarm. If it doesn't, you've got a gap you don't know about yet, and gaps like that don't announce themselves. They just let things through until someone happens to check.
Tonight we're doing this live. I'll show the checker scoring the same slop paragraph before and after the fix, then everyone in the room builds their own voice profile from a call recording or a batch of their own emails, so their content sounds like them instead of like everyone else using the same three AI models. Grab a seat and come watch it break in real time.
Brewed by Steven, poured by Viktor
About Steven Tann: Steven helps business owners build systems that run themselves using AI. After 10+ years helping 7,000+ businesses and building his own autonomous operations, he's the bloke who actually does it, not just talks about it. Find out more at steventann.com.