Comparison

RAG vs fine-tuning

Written with AI from the sources below and checked against our facts sheet. Last checked 7 Nov 2026. How this site is made · Spot an error?

Part of: EU AI Act

The one-line rule

RAG changes what a model can see: it retrieves your documents at query time and leaves the weights alone. Fine-tuning changes what the model is: it retrains the weights on your examples.

Side by side

	RAG (retrieval-augmented generation)	Fine-tuning
What changes	The prompt: retrieved passages are added at query time	The model's weights, through more training
Knowledge freshness	Current as the index; update a document and answers change	Frozen at training time; new facts need retraining
Best for	Changing knowledge, cited answers, per-user permissions	Stable style, format or specialist behavior at high volume
Cost profile	Index and retrieval cost on every query	Training cost up front; cheaper calls if a smaller model results
Removing content	Delete it from the index	Hard; often means retraining
Citing sources	Natural: it can point to the passage used	Not built in
Personal data risk	Retrieval of files a user should not see	Memorized training data reappearing in outputs
Security risks	Indirect prompt injection, poisoned or stale documents, weak access controls	Data poisoning, weakened safety behavior
EU AI Act role risk	Low by itself; the purpose decides	Substantial modification or repurposing can make you the provider

When each applies

Use RAG when the facts change often or answers must cite a current document. The term comes from a 2020 research paper that paired a language model with a searchable document index. Think of it as an open-book exam. The model looks things up before answering, so a withdrawn policy stops influencing answers once it leaves the index.

The book becomes your risk. Govern the corpus like production data, with owners, quality checks, permissions and freshness rules. Text hidden in a retrieved document can smuggle in instructions, which is indirect prompt injection. The OWASP Top 10 for LLM Applications ranks prompt injection first. It also lists vector and embedding weaknesses, a risk specific to retrieval systems.

Use fine-tuning when you need stable behavior: a house style, a rigid format or specialist vocabulary at scale. Fine-tuning brings data governance straight in. Do you have the rights to the training data? Is it representative? Could the model memorize personal data and repeat it? Fine-tuning can also weaken safety behavior the base model learned, so re-test before release.

Watch your role under the EU AI Act. A substantial modification is a change the provider's conformity assessment did not foresee. It affects compliance or alters the assessed intended purpose (Art. 3(23)). If you substantially modify a high-risk system, or repurpose a system into high-risk, Art. 25 makes you its provider. For general-purpose models, the Commission's GPAI guidelines set an indicative threshold. A modifier using more than a third of the original training compute becomes the provider of the modified model.

Worked example

Kestrel Mutual, a fictional insurer, wants two assistants built on the same general-purpose model.

The policy assistant answers staff questions about policy wording. The wording changes monthly, answers must cite the clause, and some documents are restricted to underwriters. Kestrel chooses RAG with permission-aware retrieval. It assigns an owner to the document library, removes superseded versions the day they change and tests for indirect prompt injection.

The claims summarizer turns long adjuster notes into a fixed eight-field summary, about two million times a year. The format never changes and cost per call matters most. Kestrel fine-tunes a smaller model on 20,000 past summaries. First it confirms its right to use the notes for training and removes names and policy numbers. It tests for memorized personal data and re-runs its safety tests.

Neither assistant makes decisions about customers, so neither is high-risk under the Act's Annex III. If Kestrel later used the fine-tuned model to price life or health insurance, it would cross into Annex III point 5(c). Kestrel would then be the provider of a high-risk system. See provider vs deployer.

The exam trap

The classic trap is "RAG retrains the model so it permanently learns your documents." It does not. RAG leaves the weights alone and retrieves at query time.

The tells:

Changing facts, citations, same-day removal or per-user access point to RAG.
Stable style, fixed format and huge volume point to fine-tuning.
Memorized personal data and weakened safety behavior are fine-tuning risks.
Poisoned documents and indirect prompt injection are RAG risks.
A system prompt steers behavior but is never a security boundary.
Giving the model tools to act is a third step: agentic AI, with its own controls.

Drill the adaptation ladder in lesson IV.A.4.

Frequently asked questions

Is RAG better than fine-tuning?

Neither is better in general. RAG suits knowledge that changes often, answers that must cite sources and per-user access rules. Fine-tuning suits stable behavior, such as a house style or a fixed output format at high volume. Many teams start with prompts and RAG because they are cheaper and easier to reverse.

Can you combine RAG and fine-tuning?

Yes. A common pattern fine-tunes a model for tone, format or domain vocabulary, then uses RAG to supply current facts at query time. The combination carries both sets of risks. You govern the training data for the fine-tune and the document corpus for retrieval.

Does fine-tuning a model make you a provider under the EU AI Act?

It can. A deployer that substantially modifies a high-risk system, or changes a system's purpose so it becomes high-risk, becomes its provider under Art. 25. For general-purpose AI models, the Commission's guidelines set an indicative line. A modifier using more than a third of the original training compute becomes the provider of the modified model.

Does RAG stop AI hallucinations?

It reduces them by grounding answers in retrieved text, but it does not eliminate them. The model can still misread a passage, and a wrong, stale or poisoned document produces a confident wrong answer. Citations help users check the source. Test retrieval quality and answer faithfulness, not just the model.

Sources

Spot an error? Tell us