Same Model. 62.7% or 99.9%. The Difference Was Code Somebody Had to Write.
The same model scored 62.7% and 99.9% on the same benchmark. The gap was software someone wrote, and that same gap sits between your AI demo and your product
I have had the same meeting at least five times in the last two years. A founder shows me a demo they saw from a vendor, or a demo their own team built, and asks why the production version is embarrassing. The model is the same. The prompt is roughly the same. The output is garbage maybe one time in four.
The answer is always the same, and it is never the model.
Last week gave me the cleanest possible example. OpenAI claimed an AGI milestone off a 99.9% score on ARC-AGI-3. When the benchmark's own maintainers ran what appears to be the same model through their standard software, it scored 62.7%. Same weights. Same questions. A 37-point spread.
The difference was the harness. The code wrapped around the model that feeds it the problem, manages its state, lets it retry, checks its work, and decides when it is done.
The harness is the product
I am not going to relitigate whether that counts as AGI. I do not care, and neither should you. What matters for anyone deciding where to put money this quarter is the shape of that number.
Thirty-seven points of reliability came from software. Not from a better model, not from a bigger context window, not from a pricing tier. From engineers writing code around an API call.
Everyone's mental model of AI products is still "the model does the work, the app is a thin skin on top." That was roughly true in 2023 when the app really was a text box and a system prompt. It is not true now. The model is the cheapest, most commoditized, most replaceable part of what you are building. The expensive part is everything that turns a probabilistic guess into something you can put in front of a paying customer without a human checking it first.
I have shipped MVPs on top of these models for founders. The prompt is usually a day. The harness is usually most of the build.
What is actually in there
When I say harness, here is the concrete work, described in decisions rather than jargon:
- Validation. The model returns something. Code checks whether it is the shape you asked for, whether the numbers add up, whether the citation it invented actually exists in the source document. When it fails, the system tries again with the failure as feedback rather than shrugging and shipping it.
- State. Multi-step tasks need memory of what has already been tried, what worked, and what is off the table. Left to itself, a model will happily repeat the same failed step nine times.
- Retries and fallbacks. What happens when the API is slow, rate-limited, or down. What happens on the third failure. Whether the user sees an error or silently gets a worse answer.
- Stopping rules. When to declare the task done, when to escalate to a human, when to refuse.
- Tool access. What the model is allowed to touch, in what order, and what it must never touch without a human saying yes.
None of that is model capability. All of it is billable engineering. And it is the difference between the demo that worked in the room and the thing that worked in production, every time I have had to explain that gap as an interim CTO.
What this changes about buying
Here is the part that should change how you behave this week.
When a vendor demos an AI product to you, you are looking at their harness. You are not looking at a capability you get by calling the same API yourself, and you are not looking at something that transfers to your use case for free.
That cuts two ways, and both matter.
If you are considering buying, the demo is more meaningful than you thought, because it is evidence of engineering, not just evidence of a good model. But ask what happens in the 5% of cases the demo did not show you. Ask what the system does when it is wrong. A vendor with a real harness has a sharp answer, usually something like "it flags it and routes to a human." A vendor without one changes the subject to their model partner.
If you are considering waiting for the next model release to fix your reliability problem, stop. That is the expensive mistake. The 37-point gap in the ARC results did not come from a weaker model. It came from an absence of software. GPT-6 or whatever comes next will not ship with your validation logic in it, because your validation logic encodes your business rules, your risk tolerance, and what your customers will tolerate being wrong about. Nobody else can write that.
And if you are considering building, price the harness, not the prompt. I have watched founders scope an AI feature by asking "can the model do this?", get a yes from a two-hour prototype, and then discover the actual project was the six weeks of plumbing that makes the yes hold up on a Tuesday afternoon with real customer data. That is why our fixed-scope MVP is $14,500 and not a weekend. The prompt was never the work.
The uncomfortable version
The counter-argument is real and I will give it its due: models are getting better fast, and some things that needed a hundred lines of scaffolding eighteen months ago now just work. That is true. The harness shrinks over time.
It does not go to zero, because part of the harness is not compensating for model weakness. It is encoding what you consider a correct answer. That is a business definition, not a technical one, and no lab is going to guess it for you.
So the question I would ask about any AI product you are about to buy, build, or bet a quarter on is not "which model is it using." It is "who wrote the code that catches it when it is wrong, and how do I know that code exists."
If nobody can answer that, you are buying the 62.7%.
If you want a second opinion on a vendor demo or a build estimate before you sign, grab 30 minutes. I will tell you which part of the pitch is the model and which part is the work.