You Can't Spot-Check AI Code Anymore

I've walked into roughly ten companies as a fractional or interim CTO. In most of them, the founder gave me the same tour on day one: here's the login, here's the dashboard, here's the flow we show investors. It works. Look.

By week two I was writing the memo nobody wants to read.

The pattern was so consistent it became a joke I stopped making out loud. The demo path was solid. Everything one click to the left of it was either missing, wired to nothing, or held together by a hardcoded value somebody meant to come back to. The founder wasn't lying and wasn't stupid. They had used the only quality check available to them, and that check had quietly stopped being evidence.

Jimmy Miller wrote a piece this week called The Shape of Unfinished AI Codebases that names the thing I've been trying to explain to founders for two years. Human-written software fails in shapes you can predict. AI-written software has a different topology: polished surface, then a chasm, then more polished surface. No warning at the edge.

That single observation invalidates how almost every non-technical founder evaluates the thing they paid for.

Why sampling used to work

Your instinct to spot-check isn't naive. It was a genuinely good heuristic, and it worked because of one property of human-written code: quality was correlated.

A developer who handled edge cases in the signup flow had a habit of handling edge cases. A developer who wrote clear error messages on one screen wrote them on the next one, because that's who they were on a Tuesday. Sloppiness was also correlated. If you found one hardcoded password, you'd find six.

So when you clicked through three screens and they felt right, you were making a legitimate inference. You were sampling a person's discipline, not the screens. The screens were just where the discipline showed up.

That's what broke. A model doesn't have a Tuesday. It has a context window, a prompt, and whatever attention it happened to pay to that request. The signup flow being excellent tells you the signup flow was well-prompted and probably iterated on, because it's the thing everyone demos. It tells you nothing about password reset. Nothing. The two facts are statistically unrelated in a way they simply were not four years ago.

The scariest version of this isn't missing code. It's code that reports success. I've seen tests that assert nothing. Retry logic that swallows the error and returns an empty result, so the UI shows "no items found" instead of "the database is unreachable." A payment integration pointed at a sandbox key that gets swapped in production by a config file nobody has. All of it demos beautifully.

The two things that still correlate

I built FlipSmrt solo, end to end. Frontend, backend, infra, deploys, the whole thing. When you're the only person who touched a system, you know exactly where the soft spots are, and you can say them out loud without checking anything. That's the tell.

Here's what I've settled on after inheriting all those codebases. Two signals still correlate with actual quality, and neither of them is a demo.

One: can someone explain the failure modes without opening the code?

Ask whoever built your product: what happens when the payment provider times out mid-transaction? What happens when two users edit the same record at once? What's the slowest thing in the app and why? What happens if the AI provider returns garbage or nothing?

You're not grading the answers on technical merit. You can't, and you don't need to. You're listening for whether the answers exist. Someone who genuinely understands a system answers immediately, specifically, and often with a caveat: "the timeout case retries twice then queues it, but honestly the queue has no alerting yet, that's on my list." That sentence is worth more than an hour of clicking.

The failure sound is fluent vagueness. "It handles that." "The framework takes care of it." "We have error handling in place." Those are the answers of someone who has read their codebase the way you read yours: by looking at the parts that work.

Two: what happens on the paths nobody demoed?

Stop letting the builder drive the demo. Take the mouse. Then deliberately go sideways.

Enter a phone number where an email goes. Hit the browser back button in the middle of checkout. Refresh the page halfway through a multi-step form. Log in on your phone while logged in on your laptop. Upload a 40MB file. Type an apostrophe into a name field. Turn off your wifi for five seconds and turn it back on.

None of that requires you to read code. All of it lands in the exact regions where the chasms live, because chasms form wherever nobody was watching. Fifteen minutes of hostile clicking will tell you more than three sprint demos.

What I'd actually change this quarter

If you're paying a shop, a contractor, or a vibe-coding tool to build your MVP, change your acceptance criteria before the next milestone, not after.

Write into the agreement that a milestone is complete when someone walks you through the failure modes, not when a feature demos. Ask for the list of known gaps at every checkpoint, and treat a short list as a red flag rather than good news. Every real system has a long list. A builder who hands you one is being honest, and honesty about gaps is the highest-value thing you can buy right now.

And spend money on a second pair of eyes earlier than feels comfortable. I've priced a two-week fractional CTO sprint at $4,500 partly because that's roughly what it takes to read a codebase, probe the paths nobody demoed, and tell a founder the truth about what they own. Every time I've done that at month three, it was cheap. Every time I did it at month eleven, the founder had already paid for it several times over in rewrites.

The demo was never the product. It just used to be a decent proxy. Now it's theater, and you're the only one in the room who doesn't know the script.