RAG vs fine-tuning comes down to one question: does the model lack knowledge, or does it behave the wrong way? Retrieval-augmented generation (RAG) fetches the right documents at the moment someone asks, so answers use your current data. Fine-tuning trains the model on examples, so it answers in a consistent format or style. For most business AI assistants, RAG is the place to start. This post explains why, what each costs, and when to combine them.
What is the difference between RAG and fine-tuning?
RAG keeps your data outside the model. Documents are split into chunks and turned into embeddings, which are numbers that capture meaning. When a user asks a question, the system finds the most similar chunks and passes them to the model with the question. The term comes from a 2020 paper by Lewis and colleagues, which described models that "combine pre-trained parametric and non-parametric memory."
Fine-tuning changes the model itself. You train it on many example inputs and the outputs you want. The model learns a pattern: a format, a tone, a classification.
OpenAI's accuracy guide puts it simply. Use RAG when the model's knowledge is missing, out of date or proprietary. Use fine-tuning when it knows enough but acts inconsistently. RAG targets accuracy of content. Fine-tuning targets consistency of behaviour.
| RAG | Fine-tuning | |
|---|---|---|
| Fixes | Missing or stale knowledge | Inconsistent format, tone or task behaviour |
| Your data lives | In your own store, outside the model | Inside the trained weights |
| Updating facts | Re-index the changed documents | Retrain on new examples |
| Can cite sources | Yes, it knows which chunk it used | Not reliably |
| Main effort | Ingestion, chunking, search quality | Collecting and labelling examples |
Which one keeps answers up to date?
RAG, by a wide margin. If a price list, policy or product sheet changes, you update the index and the next answer uses it. A fine-tuned model only knows what was in its training examples. Teaching it a new fact means training again.
This matters for business assistants, where most questions are about things that change: stock, opening hours, contract terms, the latest version of a procedure.
What do they cost in effort?
RAG effort goes into the data pipeline. You need connectors to each source, sensible chunking, an embedding model, a vector store and a way to test whether the right chunks come back. Note that some model providers don't make embeddings themselves. Anthropic's documentation, for example, says it does not offer its own embedding model and points to Voyage AI.
Fine-tuning effort goes into examples. You need a dataset of inputs and correct outputs, and someone to judge what correct means. Methods such as LoRA cut the cost by training a small set of added weights while the original model stays frozen. Availability also varies by provider. As of November 2026, OpenAI's model optimisation guide says it is winding down its fine-tuning platform, and new users can no longer start training jobs.
Before either, try prompt engineering. OpenAI's guide notes that a well-written prompt with examples may be all you need.
What about privacy?
With RAG, your documents stay in a store you control. You can delete a record, apply access rules per user and keep data in the EEA. You can also filter what each person is allowed to retrieve, so a sales assistant never sees payroll files.
With fine-tuning, training data shapes the model's weights. Removing one customer's data later is not as simple as deleting a row. For personal data under the GDPR, that is a strong reason to keep facts in retrieval and use fine-tuning, if at all, for style and format.
When should you combine them?
OpenAI describes the two as "additive, not exclusive". A fine-tuned model can follow your output format with a shorter prompt, while RAG supplies the facts. That suits high-volume, narrow tasks, such as classifying support tickets and then answering from a knowledge base.
The same guide also warns against combining them by default. In one of its own case studies, adding RAG to a fine-tuned model lowered accuracy. Start with one method, measure, and add the other only when a test shows it helps.
How we use RAG in practice
Munin is our AI knowledge base and customer service platform. It ingests endpoints, databases, CSVs and PDFs into a vector database, and workflow agents use that knowledge to answer questions and act. Because the knowledge sits outside the model, a changed document or a new source changes the answers without retraining anything.
That is the pattern we recommend for most companies building their first assistant. Get retrieval right, test it on real questions, and only then ask whether fine-tuning would add anything.
If you're planning an AI assistant on your own data, our software work shows how we build them, and you can tell us what it should answer.