Same Model, Triple the Right Answers: Your AI Isn't Dumb, Your Retrieval Is
The same model went from 26.7% to 86% correct on document questions by changing how it searches. When your AI misreads your own contracts, fix the retrieval
Three times in the last eighteen months, a founder has shown me a quote for "training a custom AI model on our documents." Every time, the actual problem was that their product read the first three paragraphs it found and guessed.
That is not a model problem. That is a search problem. And the gap between those two diagnoses is the difference between a two-week fix and a project that eats a quarter of your runway.
Mistral put a number on it last week. They shipped an agentic search setup that gives a model five operations instead of one: it can search, open a document, navigate inside it, read a section, and grep for a specific string. So instead of accepting whatever text got pulled in on the first try, the model can go look, follow a reference, and check itself before answering. On FinanceBench, a benchmark of questions about real financial filings, correctness went from 26.7% to 86%. Same model. Different plumbing.
Their tests, their harness, so discount it accordingly. But I have watched this exact delta in production, and 26.7% to 86% is not out of line with what I have seen when a team stops doing one-shot retrieval.
What one-shot retrieval actually does to your product
Most document AI features built in the last two years work like this. You chop your PDFs into chunks of a few hundred words, store them, and when a user asks a question, the system grabs the five or ten chunks that look most similar to the question and hands them to the model. The model answers from those chunks and nothing else.
That works fine when the answer lives in one paragraph and the question uses the same words the document uses.
It falls apart the moment your documents behave like real documents. A lease that defines "Premises" on page 2 and then uses the term forty times. A financial filing where the number is in a table and the caveat is in a footnote eleven pages later. An immigration policy manual that says "see subsection (c)" and subsection (c) is what actually matters.
I spent a stretch as fractional CTO on an immigration AI product where the entire value proposition was answering questions from long government PDFs. Get it wrong and you are not annoying a user, you are giving someone bad advice about their status. Same shape of problem on the proptech side: leases, disclosures, HOA packets, inspection reports. The product lived or died on whether it could find the one clause that mattered.
In both cases, the wrong answers were almost never the model failing to reason. They were the model reasoning perfectly over the wrong three paragraphs. It never saw the clause. It could not have known it existed.
Why fine-tuning does not fix this, and gets sold anyway
Fine-tuning means taking a general model and further training it on your data so it behaves more like you want. It is genuinely useful for teaching a model a format, a tone, a classification scheme, or a narrow task it keeps fumbling.
It is close to useless for making a model know facts that live in your documents.
Two reasons. First, fine-tuning bakes patterns into weights, not a lookup table. You cannot reliably ask a fine-tuned model "what is the termination notice period in the Riverside lease" and trust the answer, because it has absorbed a statistical impression of your corpus, not an index of it. Second, and this is the one that kills it commercially, your documents change. New leases every week, new filings every quarter, policy updates whenever the government feels like it. Retraining on every change is a treadmill you will still be running on in year three.
So why does it keep getting quoted? Because it sounds like the serious answer. "We're training our own model" is a sentence that plays well in a board update. "We rewrote how the system searches" sounds like maintenance. One of those is a defensible line item and the other is the one that actually moves your accuracy number.
I have watched a founder get a fine-tuning quote that cost more than building their entire product would have, for a problem that was one architecture change deep.
The diagnostic you can run without an engineer
Next time someone tells you the model needs to be bigger, custom, or trained, ask three questions.
When the system gets an answer wrong, can you see which parts of the document it looked at? If nobody can answer that, you do not have a model problem, you have an observability problem, and no amount of training will help because you cannot tell if it improved.
If I hand-pasted the right section of the document into ChatGPT with the same question, would it get it right? Run this yourself, ten times, on your ten worst failures. If the model nails eight or nine of them when it is handed the correct text, your model is fine. Your retrieval is broken. Stop the fine-tuning conversation immediately.
Can the system look twice? Ask whether the thing does a single search and answers, or whether it can search, read, realize it is missing something, and go back. If it is one-shot, that is your ceiling, and it is a low one.
Those three questions cost you an afternoon and can save you a quarter.
The honest counter-argument
There are real cases where a bigger or tuned model is the answer. If your task is genuinely hard reasoning over text the model already has in front of it, better retrieval will not save you. If you need a specific structured output format, tuning beats prompt gymnastics. If you are operating at volume where a smaller tuned model is meaningfully cheaper per call than a frontier one, that is a legitimate cost play.
None of those describe the founder whose product answers questions about contracts and gets a third of them wrong.
What I would do with your next two weeks
Instrument the failures so you can see what the system read. Add a second pass so the model can go back and look again. Make it cite the section it used, which forces it to actually find one. Test on your fifty worst real questions, not a synthetic set.
That is a sprint, not a hire. It is the cheapest line on your AI roadmap and it is almost always the one with the largest delta.
If someone has quoted you for training a custom model on your documents and you want a second opinion before you sign, book thirty minutes. I will tell you honestly if the quote is right. Usually it is not.