When a project needs a language model to work with company-specific material, two options come up: retrieval-augmented generation and fine-tuning. They are discussed as alternatives on a spectrum of cost and sophistication, which is the wrong frame and leads to a predictable mistake.

They do different jobs. Retrieval supplies knowledge. Fine-tuning shapes behaviour. Almost every decision follows from that one sentence.

What each actually does

Retrieval-augmented generation leaves the model untouched. When a question arrives, the system searches your documents, selects the passages most likely to be relevant, and includes them in the prompt with an instruction to answer from them. The model reads your material at the moment it answers.

Fine-tuning changes the model itself, continuing its training on examples you supply until its default behaviour shifts towards them. The knowledge is not stored as retrievable text — it is absorbed into the model's weights as a tendency.

That difference in mechanism is why they fail differently. A retrieval system that cannot find a document says so. A fine-tuned model that was trained on outdated material states the outdated thing with complete confidence, because it has no notion of where it learned anything.

Choose retrieval when the answer is in a document

This covers most business use cases: answering from a knowledge base, searching policies and contracts, support built on existing documentation, anything where a person would otherwise go and look something up.

Retrieval wins here for reasons that are practical rather than theoretical.

  • Currency. Update the document and the next answer reflects it. No retraining, no release.
  • Attribution. The system can show which passage it used, so an answer can be checked in seconds. In most organisations this is the difference between a tool that is trusted and one that is abandoned.
  • Access control. Retrieval can respect permissions at query time, returning only documents this user may see. A fine-tuned model cannot un-learn something for one user.
  • Reversibility. Removing a document removes its influence immediately, which matters when a customer exercises a deletion right.

Choose fine-tuning when the problem is form, not fact

Fine-tuning is the right tool when you need consistent behaviour that is hard to specify in instructions: a house tone across thousands of generated messages, a strict output format the model keeps drifting away from, or a classification into your own categories with your own edge cases.

It also helps where a prompt would otherwise have to be enormous. If reliable output requires two pages of instructions and eight examples on every request, a fine-tuned model can encode that, which reduces token cost and latency on every call — a real saving at volume.

What it does not do is make the model know things. A model fine-tuned on your product documentation learns to sound like your documentation while still inventing specifics, which is the most expensive way to discover this distinction.

The order to try things in

Work up this list and stop at the first step that is good enough. Most projects stop well before the end.

  • Prompting. A carefully written prompt with a few worked examples solves more than people expect, costs nothing to change, and takes an afternoon.
  • Retrieval. Add grounding in your own documents. This is where the large majority of business use cases land, and where they stay.
  • Both, refined. Improve the retrieval — better chunking, better search, reranking the results. Retrieval quality is almost always the bottleneck, not the model.
  • Fine-tuning on top. With retrieval working and a real dataset of good outputs in hand, tune for form and consistency.

Reaching step four first, which is the common instinct, means paying for training data and a tuned model to solve a problem that grounding would have solved.

What fine-tuning costs after the invoice

The training run is the cheap part. The commitments are ongoing.

You now own a model version. When the base model is superseded — which happens on a cadence of months — the tuning must be repeated on the new one, which means keeping the training set current and the process repeatable.

You also need enough examples, and they must be good. A few hundred high-quality, consistent examples beat several thousand assembled carelessly, and assembling them is human work that is rarely in the estimate.

And evaluation becomes mandatory rather than advisable. Without a test set you cannot tell whether a tuned model improved anything, and the failure mode is subtle: better on the cases you looked at, worse on a category you did not.

The short answer

If the question is "what does our documentation say about X", you need retrieval. If the question is "make every reply sound like us and always return this exact structure", you may need fine-tuning. If it is both, build retrieval first and add tuning once it is working.

And if someone proposes fine-tuning so the model will "know about our business", that is the sentence to stop at. It is the one thing fine-tuning reliably fails to deliver.