RAG or Fine-Tuning? Which One Your Business Actually Needs
A client asked us last year to fine-tune a model on their company documentation. They had been told this was how you make an AI assistant that knows about your business. They had a budget approved and a timeline in mind.
We talked them out of it, and the thing they actually needed took a fraction of the time and money.
The confusion is understandable, because the two techniques get discussed as if they were competing answers to one question. They are not. Retrieval and fine-tuning solve different problems, and picking the wrong one is expensive in a way that is not obvious until months later.
Here is the distinction in one line, before the detail: retrieval changes what the model knows, fine-tuning changes how the model behaves.
What retrieval actually does
Retrieval augmented generation — usually shortened to RAG — works by finding relevant information at the moment a question is asked, and handing it to the model along with the question.
The model itself is unchanged. You are not teaching it anything. You are giving it the right page open in front of it before it answers.
Mechanically, you split your documents into chunks, store them in a way that lets you search by meaning rather than exact words, and at question time you fetch the few most relevant pieces and include them in the prompt with an instruction to answer only from that material.
The properties that matter commercially:
- Update a document and the system knows the new answer immediately
- You can show the user which source an answer came from
- Wrong answers are usually traceable to a retrieval failure you can inspect
- Setup is measured in days, not weeks
What fine-tuning actually does
Fine-tuning continues training a model on examples of the behaviour you want, adjusting its weights so that behaviour becomes its default.
It is genuinely good at a specific set of things: producing output in a consistent structure, adopting a particular voice or house style, handling a specialised domain vocabulary, and performing a narrow task more reliably and with a shorter prompt.
What it is not good at is knowledge. If you fine-tune on your documentation and then update that documentation, the model does not know. You have baked a snapshot into the weights. Changing it means training again.
This is the mistake we see most: teams fine-tune to make a model "know about our company", then discover they have built something that is confidently out of date and expensive to correct.
The rule that decides it in most cases
Ask one question: is the thing I want the model to handle knowledge, or behaviour?
If the answer changes when a document changes — product details, policies, pricing, inventory, support articles, internal procedures — that is knowledge, and you want retrieval.
If the answer is about how the output should look or sound regardless of the facts — always return this JSON shape, always use our terminology, always follow this triage format — that is behaviour, and fine-tuning may earn its cost.
Most business use cases we see are knowledge problems wearing a behaviour costume. Customer support assistants, internal search, document question-answering, sales enablement tools: all retrieval.
Try the boring option first
Before either, try the simplest thing: a well-written prompt with the relevant context included, using a capable model.
An enormous share of the AI features we are asked to build work acceptably with careful prompting alone. Modern models follow instructions well, and a page of clear guidance often achieves what teams assume requires training.
The sensible escalation is: prompting, then retrieval, then fine-tuning — moving to the next step only when the current one demonstrably fails. Each step up costs more and constrains you more.
Anthropic's guidance on reducing hallucinations is a good practical starting point for the first two steps, and covers most of what teams need before they consider training anything.
What each one costs
Rough shapes rather than quotes, because the variance is wide.
Retrieval costs you an ingestion pipeline, a place to store and search the embeddings, and slightly more per request because you are sending extra context. Building a support assistant grounded in your content typically lands in the $2,500 to $7,000 range for us, with running costs of $30 to $400 per month for most small and mid-size products.
Fine-tuning costs dataset preparation, which is the real expense and the part everyone underestimates. You need hundreds to thousands of high-quality examples of the behaviour you want, and someone has to produce and check them. Then training, then evaluation, then repeating the whole cycle when the base model you built on is superseded.
That last point deserves emphasis. Base models improve every few months. A fine-tune is attached to a particular one, and moving to a newer, better model means doing the work again.
If you are trying to scope this
The decision usually gets easier once someone has seen both approaches applied to a comparable problem, which is not something a blog post can substitute for.
If you are weighing an AI feature and want a straight answer about which approach fits — including the case where careful prompting is enough and you should spend the budget elsewhere — we are happy to walk through it with you.
When you genuinely want both
The two are not mutually exclusive, and the combination is occasionally the right answer.
A realistic example: a legal document assistant. Fine-tuning teaches it to structure its output as a specific clause-by-clause analysis with consistent terminology. Retrieval supplies the actual current regulations and the specific contract being reviewed. Behaviour from training, knowledge from retrieval.
The sequence matters. Build retrieval first, ship it, and see whether the output quality is genuinely limited by behaviour rather than knowledge. If it is, you now have real usage data, which also happens to be the raw material for a good fine-tuning dataset.
How to know whether it is working
Whichever path you take, decide in advance how you will judge it. Teams that skip this end up arguing about whether the output "feels" better.
Build a small evaluation set — fifty to a hundred real questions with known good answers — before you build the feature. Run it against every change. Track two things: how often the answer is correct, and how often the system wrongly claims confidence about something it should have declined.
For retrieval specifically, separate the two failure modes. If the answer is wrong because the right document was never fetched, that is a retrieval problem and no amount of model work fixes it. If the right document was fetched and the answer still went astray, that is a prompting or model problem. Teams routinely spend weeks on the second when the first was the cause.
The part everyone skips: preparing the content
If you go the retrieval route, the quality of your answers is determined less by the model than by the state of your documents. This is unglamorous and it is where most of the real work lives.
Three things matter more than people expect.
Chunking. How you split documents decides what can be retrieved. Split too small and you lose the context that made a passage meaningful. Split too large and you dilute the relevant sentence among four paragraphs of unrelated material. Splitting along the document's own structure — sections and headings — beats splitting at a fixed character count almost every time.
Contradictions. Most companies have two documents that disagree, usually because one was never updated. A human reader resolves this by knowing which is current. A retrieval system does not, and will confidently present whichever it happened to fetch. Finding and reconciling these is a content project, not an engineering one.
Things that should not be retrievable. Internal notes, draft pricing, half-finished policies. If it is in the index, it can surface in an answer. Decide what is in scope deliberately rather than pointing the ingestion pipeline at a shared drive and hoping.
Teams that treat this as a content exercise get good results quickly. Teams that treat it as purely technical spend weeks tuning retrieval parameters to compensate for messy source material. Google's guidance on retrieval augmented generation covers the mechanics well, but the content problem is yours to solve.
A short decision path
If you want this as a sequence to follow:
- Write a clear prompt with the needed context inline. Test it honestly against real questions.
- If it fails because the model lacks your specific information, add retrieval.
- If it fails because the output format or tone is inconsistent, try a stricter prompt and structured output constraints first.
- Only if that still fails, and you have hundreds of good examples and a stable base model, consider fine-tuning.
- Measure at every step against the same evaluation set.
Most teams will stop at step two and be entirely happy. That is not a lesser outcome — it is the cheapest system that solves the problem, which is the goal.
Choose the one you can change later
The real argument for retrieval over fine-tuning in most business contexts is not accuracy or cost. It is reversibility.
A retrieval system can be updated by editing a document. It can be pointed at a better model next quarter without redoing anything. It can show its sources, which makes it auditable when someone asks why it said what it said.
A fine-tune is a commitment: to a base model, to a dataset, and to a retraining cycle you will own indefinitely. Sometimes that commitment is worth it. Far more often, the thing people wanted when they said fine-tuning was simply a model that knows about their business — and that is a retrieval problem with a much shorter path to working.
If you are planning an AI feature and want help deciding between retrieval and fine-tuning before committing a budget, book a free consultation call or email info@playxoft.com. We will tell you which one your case needs, including when the answer is neither.
