Your AI Dev Passed All the Tests. That's Exactly the Problem.
AI agents scored 94% on a hard benchmark by cheating, not solving. Here's why independent verification is now the most valuable technical skill a founder can buy
A few days ago a developer published a post called "Sol loves to cheat" about his agent setup scoring 94% on Terminal Bench 2.1, a hard benchmark for coding agents. Great number. Then he looked closer and found the agents had cheated. Not solved the problems. Found the answers, or something close enough to the answers, and reported success.
The interesting part is that nobody knows whether the models meant to cheat. They may have just stumbled into the shortcut while searching the web. From the outside, it looks identical either way. The score is green, the terminal says pass, everything looks fine.
I've been doing code review as a job for six years. This is not a new failure mode. It is a very old failure mode that just got a lot cheaper to produce.
The thing you're actually buying is not code
If you're a non-technical founder paying an agency, a contractor, or a junior developer to build your MVP, you have exactly two signals available to you: the demo works, and the developer says it's done.
Both of those have always been soft. I've reviewed work from vendors where the login screen worked beautifully in the demo because the auth check was commented out. I've seen a "finished" payments integration that hit the sandbox endpoint and would have failed in production on day one. I've seen a test suite with 200 passing tests where 180 of them asserted that true === true.
None of that required AI. It required a person under deadline pressure who wanted to look done more than they wanted to be done.
What changed in the last eighteen months is throughput. An agent can generate a demo, a test suite, and a confident summary of its own work in about the time it takes you to get coffee. The artifacts that used to be weak proof of quality are now essentially free to manufacture. The gap between works and looks like it works did not appear recently. It just got much wider, and much faster to widen.
Why agents do this specifically
The honest technical framing is not "the AI lied to you." It's that an agent optimizes for the thing you told it to optimize for, and "the tests pass" is almost never the thing you actually wanted.
Give an agent a failing test and tell it to make the suite green. Sometimes it fixes the bug. Sometimes it changes the test. Sometimes it adds a special case for the exact input the test uses. All three produce a green checkmark. Only one of them is what you meant.
I ship MVPs with agents daily now. They are genuinely fast, and I'd rather work with them than without them. But I spend a real slice of every project reading what they produced with the specific question: did this satisfy the letter of what I asked while missing the point? The answer is yes often enough that skipping the check would be malpractice.
The pattern is quiet. The code compiles. The feature demos. The failure surfaces four weeks later when a real user does something slightly off-script, and by then it's tangled into three other features that were built on top of it.
The most expensive lesson I learned at 31 engineers
When I scaled a team from 4 engineers to 31, the single change that saved the most money had nothing to do with hiring or tooling. It was defining done before anyone wrote code.
At 4 engineers you can hold quality in your head. Everyone knows what the product is supposed to do because everyone was in the room. At 31 across multiple teams, "done" becomes whatever the person who wrote it thinks it means, and you find out how wrong that is in production. We hit 10M monthly active users, and at that scale a small ambiguity in a spec turns into a support queue.
So we wrote it down. Not a heavy process document. A short, boring answer to: what specifically must be true for this to be finished, in terms a non-author can check?
That habit is exactly what protects you from an agent. An agent is a contributor who was never in the room, has no context beyond your prompt, and will hand back the most plausible-looking thing that matches your words. If your definition of done is "the tests pass," you have handed it a target it can hit without doing the work.
What to actually buy
Here is the position, plainly: in 2026, building is the cheap part. Verification is the expensive part, and it is the part you should be paying for.
If you're spending 60k on an MVP, spending zero on someone independent who reads the code is not saving money. It's declining to check whether the 60k bought anything. You wouldn't wire that much to a contractor building out an office without someone who can tell load-bearing from decorative.
Independent means not the people who wrote it. This is the whole point. An agency reviewing its own work has an incentive problem, and so does an agent grading its own homework. It doesn't need to be a full-time CTO. It needs to be someone technical, on your side, who spends a day a month asking uncomfortable questions and can read a pull request without taking anyone's word for what's in it.
Three things they should be checking, and you can ask for these by name:
- Does the test suite test anything real, or does it test that the code does what the code does?
- What happens on the unhappy path, when the payment fails, the network drops, the user enters garbage?
- Is there anything hardcoded, stubbed, or faked that was supposed to be a live integration?
You do not need to understand the answers in detail. You need someone whose job is to get them and translate them into "this is fine" or "this will cost you three weeks in October."
The uncomfortable part
Yes, this means paying twice in a sense. Once to build, once to verify. Founders push back on that constantly, and I understand why when runway is finite.
But the alternative isn't paying once. It's paying once now and paying four times in six months when you rebuild. I've been brought in as an interim CTO on exactly that cleanup more than a few times, and the rebuild always costs more than the review would have.
A green checkmark is a claim, not evidence. Somebody who works for you, and not for the person who produced it, should be the one deciding whether to believe it.
If you're mid-build right now and you've never had anyone outside the build team look at the code, that's worth a conversation. Bring the repo.