It Worked in the Demo. Make Them Run It Again. Twenty Times.

I've walked into three companies as interim CTO where the same conversation happened in the first week. The founder shows me a recording of a board demo. An AI feature ingests something messy, thinks for a moment, produces exactly the right answer. Everyone on the call claps. Then the founder tells me customers are complaining and nobody knows why.

Nothing broke. The feature never worked in the first place. It worked once, in front of an audience, and everyone in the room treated that single run as proof.

IBM Research published something last week that puts a name on this. Their write-up on the inconsistency gap describes what happens when you take an agent that scored well on a task and simply run it again, and again, on the exact same task. The success rate doesn't hold. It scatters. The number you reported was a sample of one, and the distribution behind it is much wider and much lower than that sample suggested.

I haven't used their tooling. I don't need to, because I've been on the receiving end of this gap for years, and so has every engineer who has ever shipped something probabilistic to a lot of users.

Why the demo lies to you specifically

Traditional software is boring in a very useful way. If a login form works on Tuesday with a given input, it works on Wednesday with that input. One successful run is genuinely decent evidence. You learned that instinct from twenty years of using computers, and it is now actively working against you.

An AI feature is a dice roll with good odds. Sometimes very good odds. But the agency demoing it to you gets to choose which roll you watch. They don't even have to be dishonest about it. They ran it eight times while preparing, four were rough, they fixed a prompt, the ninth was beautiful, and that's the one on the screen. Everyone involved genuinely believes the feature works.

Then you ship, and reality runs it ten thousand times a day with inputs nobody rehearsed.

At Lucky Day we got to 10M monthly active users, and that number changes how you think about percentages forever. A 5% failure rate is not a footnote in a status update. On that kind of traffic it is tens of thousands of broken sessions every single day, which is a support inbox, an app store rating, and a churn number that your investors will ask about before you've finished diagnosing it. I have watched a "minor edge case" turn into the only thing the entire company talked about for two weeks.

The cruel part is that a 95% success rate looks perfect in a demo. You'd have to sit through twenty runs to see the bad one. Nobody sits through twenty runs.

So sit through twenty runs.

The one thing you can demand without being technical

Here is my honest read of the position most of my clients are in. You raised money, you have between one and fifteen people, and you're paying an agency or your first engineering hire to build something you cannot personally evaluate. You can't read the code. Reviewing the architecture diagram is theater. Asking "did you write tests?" gets you a yes, and you have no way to interrogate the answer.

But you can count to twenty.

Make this your acceptance criterion, in plain language, in writing:

Run it twenty times on twenty realistic inputs, in front of me, without touching anything in between. Show me every failure. Tell me what the success rate was.

That's it. No jargon required. You don't need to know what temperature or a retry loop or an eval harness is. You're asking for a count, and the count is either produced or it isn't.

What makes this so effective is not the twenty runs themselves. It's what happens in the two weeks before the meeting. A competent team, once they know you're going to ask, starts running it twenty times on their own. They find the failures first. They build the retries, the validation, the fallback path, the "I'm not sure, let a human look at this" branch. The demand for twenty runs is how you force engineering discipline into a relationship where you have no technical leverage.

And a team that pushes back on this is telling you something for free. I have never met a good engineer who was offended by being asked to show their failure rate. I've met plenty who were relieved someone finally cared.

Put it in three places

In the contract or SOW. Not "the feature will work." Write down the number. This feature will succeed on at least 18 of 20 representative inputs, demonstrated live, with failures documented. Pick your own threshold, but pick one. A milestone you can't measure is a milestone you'll pay for twice.

In every milestone review. Same drill, every time, not just at the end. If the success rate drops between milestone two and milestone four, you want to know in week six, not after launch. This is a huge part of what I actually do during a fractional CTO sprint: I'm the person in the room asking for the twenty runs so the founder doesn't have to be the bad guy.

In your first engineer's trial project. This one matters most. Give a candidate a small, real, messy task. Don't grade the code. Ask them to tell you how often it works and show you where it breaks. The ones who come back with "it's 17 out of 20, here are the three failures, two of them are the same root cause and here's my fix" are the ones you hire. The ones who come back with a single flawless demo and no numbers will build you exactly the product I keep getting hired to rescue.

The mindset shift

Stop asking whether an AI feature works. That question has no answer. Start asking how often it works, and what happens on the runs where it doesn't.

Every AI feature you ship has a failure rate. You either know the number and designed for it, or you don't know the number and your customers are about to discover it for you. There is no third option, and the demo you're about to watch is not evidence of anything.

If you're mid-build with an agency right now and nobody has ever shown you a failure rate, that's worth thirty minutes of conversation. Bring the demo. We'll run it again.