Nobody's Agent Tool Is Magic. It Just Prices Differently.
A 21-way test of coding agents shows the tooling wrapped around the model barely changes whether work gets done, but can multiply the bill. Here's how founders push back
Three times in the last year, a founder has forwarded me a proposal from a dev shop and asked me to translate one line. The line was always some variation of "premium agent tooling and orchestration." The number next to it was always big enough to notice and vague enough to be hard to argue with.
Each founder asked me the same question, in slightly different words: is this real, or am I being upsold?
Now there's evidence you can point at. The team at Arena ran an evaluation of 21 model-harness pairs across seven models and three harnesses. A harness is just the software wrapped around the model that gives it a terminal, file access, retries, and a loop to work in. Their finding: which harness you pick has little effect on whether the task gets done. It has a large effect on what the task costs. A simple harness stayed competitive with the fancy ones.
That is not a small result. It means a big chunk of what gets sold to you as capability is actually a spending decision that somebody else is making on your behalf, with your money.
Why this keeps getting mispriced
I ship MVPs with coding agents every week. The pattern I keep seeing is that two builds of a near-identical scope can land within a few days of each other on the calendar and be separated by a multiple on the bill. Not because one produced better software. Because one setup burned through far more tokens getting to the same place, retrying, re-reading, re-planning, and fanning out into subtasks that nobody asked for.
The output looked the same. The invoice did not.
This is uncomfortable for the people selling it, so it doesn't get said out loud. The agent tooling market is moving fast enough that "we use the newest thing" reads as competence. It's easier to sell a stack than to sell judgment. And an engineer who has spent two weekends tuning their agent setup genuinely believes that setup is why their work is good. Sometimes it is. Usually the setup is downstream of the fact that they knew what to build.
I made this mistake as a hiring manager for years. Scaling from 4 engineers to 31, I interviewed a lot of people who could talk fluently about their tools and much less fluently about their decisions. Tool fluency is easy to perform and easy to mistake for skill, especially if you're the non-technical person in the room. I hired a few of those people. The ones who worked out were the ones who could also tell me what a tool cost them and when they'd stopped using it.
"Which tools do you use" is a bad question
If you're a founder with no engineering leadership, you're looking for a proxy. Something you can ask that gives you signal without needing to evaluate code. Tooling feels like a good proxy because the answers are checkable and the vocabulary sounds technical.
It's a bad proxy for two reasons.
First, the answer doesn't predict outcomes. That's the whole point of the Arena result. If success rate barely moves across harnesses, then knowing which harness someone uses tells you almost nothing about whether they'll finish your product.
Second, asking the question signals that you can be sold to on tooling. Once a vendor knows that's the axis you evaluate on, they will compete on that axis, and the premium stack becomes a line item forever.
Better questions, and none of them require you to understand any of this:
- What did the last three projects like mine cost you to run, not to bill?
- When the agent goes down a wrong path, how do you notice, and how fast?
- What part of this build are you not going to use an agent for, and why?
- If your tooling costs doubled next month, what would you change?
That last one is my favorite. Someone with judgment has an answer ready, because they've already thought about where the spend goes. Someone who adopted a stack because it's what the good engineers on Twitter use will tell you it's a cost of doing business.
The honest counter-argument
Harness choice does matter for some things. Permissions, audit trails, what the agent is allowed to touch, whether a human has to approve before something reaches production. If your vendor is paying for tooling because it gives them a controlled path to your live systems, that is a real reason and worth the money.
But notice what that is. It's a governance decision, not a capability one. The pitch should be "this is how we stop the agent from doing something expensive to your database," not "this is how we build better software." If a proposal can't tell you which one it's selling, it's selling neither.
What I'd do this quarter
Ask for the tooling cost as a separate, visible line. Not bundled into a rate. If a vendor won't break it out, that's your answer about who's absorbing the variance.
Then ask what happens to that line if they switched to something simpler. If the honest answer is "the work would still ship, it would just be less pleasant for us," you now know exactly what you're funding. That might still be fine. Developer comfort is worth something, and unhappy engineers ship slower. But you should be choosing to pay for it, not discovering it inside a total.
For reference, our fixed-scope MVP is $14,500, and that number doesn't move based on which agent we're running that month. Absorbing tooling variance is part of the job. Passing it to you and calling it capability is not.
The tool isn't the moat. It never was. The person deciding what to build with it is the only expensive part, and that's the part you should be interviewing.
If you've got a proposal in your inbox with a line you can't read, send it over. Translating those is most of what a fractional CTO does in the first week anyway.