Prompting, RAG or Fine-Tuning: A Decision Guide
Which technique first? A decision guide for prompting, RAG and fine-tuning: what each changes, what it costs to maintain and how to decide with an evaluation.
Topics:RAG and retrieval
Illustrative scenario: a support assistant keeps quoting the wrong refund period. A team member proposes fine-tuning a model on past tickets. Before anyone books GPU time, someone checks the assistant's input and finds the current policy was never in it. The model is not misbehaving; it was never shown the facts. No amount of training on old tickets would have fixed that, and a single retrieval step would have. Choosing the right technique starts with diagnosing the actual gap.
TL;DR
Use prompting first: it is the cheapest and fastest way to change behaviour. Add retrieval (RAG) when the model needs knowledge it does not have, such as your documents, current facts or private data. Consider fine-tuning when you need consistent behaviour, format or style, or a smaller model to do a narrow task well, and when prompting and retrieval have been tried and measured. These are not exclusive and are often combined. Decide with an evaluation set, not a hunch, and count the long-term cost of maintaining each.
What each technique actually changes
The three act on different things. Knowing which lever you are pulling prevents using a training run to fix a missing document.
Prompting
- Changes the instructions and examples in the request
- Takes minutes to try and is easy to revise
- Cannot teach facts the model does not have
- Limited by what fits in the context
RAG (retrieval-augmented generation)
- Fetches relevant documents and supplies them in the request
- Gives access to current, private or large bodies of knowledge, with citations
- Needs ingestion, indexing and retrieval quality work
- Facts update when documents update, with no retraining
Fine-tuning
- Trains the model further on examples, changing its behaviour
- Can improve consistency of style, format or a narrow task
- Not a reliable way to add changing facts
- Needs data, training, evaluation and retraining when requirements change
For the basics, see the chapters on prompt engineering, why RAG and fine-tuning fundamentals.
Diagnose before you choose
Start from the symptom, not the technique. The same bad answer can have different causes, and each points to a different fix.
If
The answer is wrong because it lacks a fact or uses outdated information
Choose
Add retrieval
Because
The model needs knowledge it was not given
If
The model ignores instructions or answers in the wrong shape
Choose
Improve the prompt, examples and output format first
Because
It is often a clarity problem, cheap to fix
If
The facts are right but the style, format or behaviour is inconsistent
Choose
Consider fine-tuning, after prompting is exhausted
Because
You need a changed behaviour, not new knowledge
If
A large model works but is too slow or costly for a narrow task
Choose
Consider fine-tuning a smaller model
Because
Training can help a smaller model do one job well
If
The task needs both current facts and a consistent format
Choose
Combine retrieval with prompting, and fine-tune only if still needed
Because
The techniques complement each other
The good retrieval, bad answers challenge is a useful reminder that even after you add retrieval, the failure can sit in the pipeline, which is why you measure rather than assume.
A sensible order
Step 1: Clarify the task and write a test set
Real examples with expected outputs, so every option is judged the same way
Step 2: Improve the prompt
Clear instructions, examples and an explicit output format
Step 3: Add retrieval for missing knowledge
Ground answers in your documents, and measure retrieval quality
Step 4: Reconsider the model and context
A stronger model, or better context selection, may close the gap
Step 5: Fine-tune if a gap remains
For consistent behaviour or a smaller, cheaper model, with data you can defend
The order reflects cost and reversibility, not a rule. If your problem is clearly about private documents, start with retrieval. If you are building a high-volume, narrow classifier, fine-tuning a small model may be where you end up. The point is to reach each step knowing what the previous one could not do.
What each option costs over time
The first build is the visible cost. The ongoing cost is what decides.
Prompting
Good: Prompts are versioned and tested with the evaluation set, and edits are reviewed
Red flag: Prompts are edited ad hoc and changes are not tested
RAG
Good: Ingestion is automated, the index stays fresh and retrieval quality is measured
Red flag: The index goes stale and nobody notices until answers are wrong
Fine-tuning
Good: Training data is curated, the tuned model is versioned and there is a plan to retrain
Red flag: The tuned model ages as requirements change, with no one owning it
Combined
Good: Each layer has an owner and a test, so failures can be located
Red flag: A tangle of prompt, index and tuned model nobody can debug
Fine-tuning in particular creates a model you must keep current. When your policy, products or tone change, a prompt edit or a document update takes effect immediately, while a fine-tuned model needs new data and another round of training and evaluation. Prompts and retrieval keep knowledge where you can edit it; fine-tuning bakes behaviour into the model.
Decide with an evaluation, not an opinion
The most reliable way to choose is to measure. Build a test set of real inputs with the outputs you want, then score each option on it, with the same metrics and the same judge.
By opinion
- "Fine-tuning will make it smarter about our business"
- No shared test set, so each option is judged by impressions
- The costly option is chosen because it sounds serious
By evidence
- A test set of real cases with expected results
- Each option scored on the same metrics, including cost and latency
- The simplest option that meets the bar is chosen
LLM evaluation fundamentals explains how to build the set and calibrate a judge. Include cost and latency in the comparison: a fine-tuned small model that matches a large one on your task can be cheaper to run, which is something only a measurement will show. Report what you measure with its limits rather than a single score.
Combining the techniques
Real systems often use more than one. A common pattern is a well-prompted model, retrieval for knowledge and, where measurement shows a gap in format or consistency, a fine-tuned model for that narrow role. The production RAG architecture guide covers the retrieval side in depth, and context engineering versus prompt engineering covers how to arrange what the model sees. Keep each layer separately testable so you can tell which one failed when something goes wrong.
When fine-tuning is the right answer
It is the right tool in specific situations: you need highly consistent output in a particular format or style that prompting does not achieve reliably, you want a smaller, cheaper model to do one task well, or you have a stable task with good training data. Even then, check your provider's current documentation on what fine-tuning supports and what it costs, and plan for the data quality, evaluation and retraining it requires. If you can solve the problem with a better prompt or retrieval, you avoid a lasting maintenance burden.
Frequently asked questions
Should I use prompting, RAG or fine-tuning? Start with prompting, add RAG when the model needs knowledge it lacks such as your documents or current facts, and consider fine-tuning for consistent behaviour, format or a smaller model on a narrow task, after the first two have been tried and measured. They are often combined.
Does fine-tuning teach a model new facts? It is not a reliable way to add facts, especially ones that change. Retrieval supplies current knowledge at request time, while fine-tuning changes how the model behaves.
When is RAG better than fine-tuning? When the problem is missing, private or changing knowledge. With RAG, updating a document updates the answers, with no retraining, and answers can cite sources.
When is fine-tuning better than RAG? When you need consistent behaviour, style or output format, or a smaller cheaper model for a narrow stable task, and you have good training data. Retrieval does not change how a model behaves.
Can I combine all three? Yes. A typical setup is a well-written prompt, retrieval for knowledge and, if measurement shows a remaining gap, a fine-tuned model for a specific role. Keep each layer testable on its own.
How do I decide between them? Build a test set of real cases, score each option on the same metrics including cost and latency, and choose the simplest option that meets your bar. Opinions about which approach is more advanced are not evidence.
What are the hidden costs of fine-tuning? Curating training data, running and evaluating training, versioning the tuned model and retraining when requirements change. A prompt edit or a document update takes effect immediately, while a tuned model needs another training cycle.
Related
RAG and retrieval
Your next step
Reading is the start. Turn it into skill: read the chapter, practise on something real, check the numbers, then follow a path.
More on RAG and retrieval
- The Enterprise AI Engineering BlueprintA reference architecture for enterprise AI: gateway, orchestration, RAG, MCP tools, identity, guardrails, evaluation, observability and governance.
- Production RAG Architecture: A Complete GuideThe full production RAG architecture: ingestion, chunking, embeddings, hybrid search, reranking, generation, evaluation and observability, with code.
- Why Good Retrieval Still Produces Bad AnswersDebug RAG failures with a runnable Python exercise that separates retrieval misses, dropped evidence, unsupported claims and incorrect refusals.
Discuss this article
Got a question, a counter-example or a correction? Bring it to the community.
Prefer a feed reader? Follow by RSS.