II · THE IDEA · ARTIFICIAL INTELLIGENCE
Fine-Tune, Prompt, or Retrieve
▶ Listen · narrated
The same task can be solved three ways, but only one of them will stay cheap enough to run at scale and stay current enough to trust next year.
At a glance
- Prompting
- Put the information in the context window. Zero training cost, pay per token each time.
- Retrieval
- Fetch relevant documents at inference time, then prompt with them. Adds latency and a search system.
- Fine-tuning
- Update the model's weights on new examples. Upfront cost, then fixed inference cost regardless of knowledge volume.
Imagine you need to teach someone a fact. You can tell them every time they ask, which costs nothing upfront but gets expensive if they ask a thousand times. You can write it in a notebook they carry, but if the notebook is huge they will spend all their time flipping pages. Or you can teach them once so they remember it, which takes effort at the start but then they know it forever — except if the fact changes, you have to reteach them from scratch. Prompting is telling them each time. Retrieval is the notebook with an index so they can find the right page quickly. Fine-tuning is teaching them to remember. The best choice depends on how often the fact changes, how often they need it, and whether they need to show their working.
Prompting places knowledge in the context window as part of the input. The model processes it as tokens, attending to it the same way it attends to the user's query, and generates a response. The cost is the input token count multiplied by the per-token price, paid on every request. The method is stateless: the model retains nothing between requests, so the knowledge must be included each time. Retrieval-augmented generation separates knowledge storage from the model. A query triggers a search against an external index — typically a vector database using dense embeddings, though keyword search and hybrid methods are also used — and the top-k results are inserted into the prompt. The model then generates conditioned on the retrieved passages. Latency is the sum of retrieval time and generation time, and accuracy depends on both retrieval precision (did we fetch the right documents?) and the model's ability to synthesise them. Fine-tuning updates the model's weights through continued training on a task-specific dataset. The knowledge is encoded in the parameters, so inference cost is independent of knowledge volume, but updating requires retraining. The training cost scales with model size, dataset size, and the number of gradient steps. Fine-tuning can also cause the model to overfit to the training distribution or to forget behaviours learned during pre-training, so practitioners typically validate on held-out tasks and use techniques like low-rank adaptation to limit the scope of weight changes.
Look closer
Prompting is free to start but expensive to scale
If your knowledge fits in ten thousand tokens and you query it once, prompting costs almost nothing. If you query it a million times, you have paid for those ten thousand tokens a million times over. The cost is linear with usage and with the size of the knowledge you include. The knowledge also displaces conversation history or other context, which matters when the window is already tight.
Fine-tuning inverts the cost structure
You pay once, upfront, to train the model on new examples — typically hundreds to tens of thousands of dollars depending on model size and the number of training steps. After that, inference costs nothing extra for the knowledge itself, because it is encoded in the weights. But updating the knowledge means training again from scratch or from a checkpoint, and the model cannot tell you where its answers came from.
Retrieval adds a search problem and a latency penalty
Before the model runs, a separate system searches a database or document store for relevant passages, then inserts them into the prompt. This keeps the context current and lets you cite sources, but it adds the cost and complexity of maintaining a search index, the latency of the retrieval step itself, and a new failure mode: the search may return irrelevant or incomplete results that mislead the model more than prompting nothing would have.
The story
The question is not which method is best. The question is which costs you can afford and which failure modes you can tolerate.
Start with prompting if the knowledge is small, changes frequently, and must be auditable. A product catalogue, a policy document, a user's recent conversation history — anything under a few thousand tokens that you can simply paste into the context window. The model reads it every time, so it is always current, and you can see exactly what it was given. The cost is transparent: you pay for the input tokens and the output tokens, and nothing else. The failure mode is also transparent: if the model ignores a detail or misreads it, you can inspect the prompt and see why.
The method breaks down when the knowledge is large or the query volume is high. If you need to include a hundred-page manual in every request, you will spend more on input tokens than the answer is worth. If you run ten thousand queries a day against the same static knowledge base, you are paying to re-read it ten thousand times. Prompting scales badly with both dimensions.
Retrieval-augmented generation solves the scale problem by adding a search layer. You store the knowledge in a database or vector index, retrieve only the few most relevant passages for each query, and prompt the model with those. The model never sees the full knowledge base, so input costs stay manageable even when the underlying corpus is enormous. The knowledge can be updated independently of the model — you change the database, not the weights — and you can return citations showing which documents informed each answer.
The trade-off is complexity and latency. You now maintain two systems: the model and the search infrastructure. The retrieval step adds tens to hundreds of milliseconds before the model even starts generating, and the quality of the final answer depends on the quality of the search. If the retrieval system returns the wrong passages, the model has no way to know. It will answer confidently based on incomplete or misleading evidence. This failure mode is harder to debug than a bad prompt, because the retrieval logic is often opaque and the ranking heuristics are tuned separately from the model.
Fine-tuning makes sense when the knowledge is stable, the behaviour you want is not just factual but stylistic or structural, and you will use the model enough to amortise the training cost. Teaching a model to follow a house style, to generate code in a particular framework, to answer in a specific format — these are tasks where fine-tuning often wins. The knowledge is baked into the weights, so inference is fast and the cost per query does not grow with the amount of knowledge. You also avoid the retrieval latency and the risk of search failure.
But fine-tuning is the least flexible option. Updating the knowledge means running another training job, which takes hours to days and costs hundreds to tens of thousands of dollars depending on scale. The model cannot cite its sources, and if it generates something wrong, you cannot easily trace it back to a specific training example. Fine-tuning also risks overfitting: if the training set is small or unrepresentative, the model may learn the examples too literally and fail to generalise. The method works best when you have thousands of high-quality examples and the task is well-defined.
The decision tree is not complicated. If the knowledge fits in the context window and you query it infrequently, prompt. If the knowledge is large or changes often and you need citations, retrieve. If the knowledge is stable and you need consistent behaviour at high volume, fine-tune. Most mistakes come from fine-tuning too early, before the task is well-understood, or from prompting at scale without noticing the cost has become unsustainable.
Why it mattered then
The choice became urgent in 2020, when GPT-3 demonstrated that a single large model could perform many tasks through prompting alone, without task-specific training. The paper introducing it, Language Models are Few-Shot Learners, showed that including a handful of examples in the prompt — few-shot learning — could match or exceed the performance of models fine-tuned on thousands of examples. This was surprising. Fine-tuning had been the default method for adapting models to new tasks, and the assumption was that you needed to update the weights to get good results. GPT-3 suggested that a large enough model, given the right context, could generalise without weight updates at all. The implication was that prompting might replace fine-tuning for many tasks, which would lower the barrier to using large models and shift the engineering effort from training pipelines to prompt design. Retrieval-augmented generation emerged around the same time as a way to give models access to knowledge bases too large to fit in the context window, combining the flexibility of prompting with the scale of external storage.
Why it matters now
The choice matters more now because the cost structure has become clearer and the trade-offs are no longer hypothetical. Prompting at scale can cost more than fine-tuning, especially with long contexts and high query volumes. Retrieval has matured into production systems with well-understood latency and accuracy profiles. Fine-tuning has become cheaper and faster, with tools that let you update a model in hours rather than days, but it still locks you into a fixed version of the knowledge. The decision is also more consequential because models are being used for tasks where the wrong method creates compounding costs or unacceptable failure modes. A customer service system that re-reads a static knowledge base in every prompt is burning money. A legal research tool that cannot cite its sources is unusable. A code generation model that drifts from the current API because the training data is six months old is worse than one that retrieves fresh documentation. The methods are not interchangeable, and choosing badly can make a system uneconomical or unreliable before it even launches.
The surprising detail
Fine-tuning can make a model worse at tasks it was previously good at, a phenomenon sometimes called catastrophic forgetting. If you fine-tune on a narrow dataset, the model's weights shift to fit the new examples, and in doing so they can drift away from the broader knowledge learned during pre-training. A model fine-tuned to answer questions about a specific product may become worse at general reasoning or at answering questions about other products. The effect is not uniform — some tasks are more fragile than others — but it is common enough that practitioners often fine-tune from a checkpoint and validate on held-out tasks to check for regression. Prompting and retrieval do not have this problem, because they leave the weights untouched.
Remember this
Prompt when the knowledge is small and changes often. Retrieve when it is large and you need citations. Fine-tune when it is stable and you need speed at volume.
Test yourself
You are building a system that answers questions about a company's internal policies. The policies change every few months, the query volume is moderate, and users need to know which policy document each answer came from. Which method should you choose, and what is the failure mode you are most likely to encounter?
Retrieval-augmented generation. The policies are too dynamic for fine-tuning, because you would need to retrain every few months, and they are probably too large to fit comfortably in every prompt if you have dozens of policy documents. Retrieval lets you update the knowledge base independently, and you can return citations by tracking which documents were retrieved. The failure mode to watch for is retrieval quality: if the search system returns irrelevant or incomplete passages, the model will answer confidently based on the wrong evidence, and the user will trust a citation that does not actually support the answer. You will need to monitor retrieval precision and tune the ranking heuristics, which is a different skill set from tuning prompts or training models.
Go deeper
- Retrieval-Augmented Generation for Large Language Models: A Survey · arXiv · Yunfan Gao et al. · 2023-12-18
- Language Models are Few-Shot Learners · arXiv · Tom B. Brown et al. · 2020-05-28
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.