II · THE IDEA · ARTIFICIAL INTELLIGENCE
Retrieval-Augmented Generation
▶ Listen · narrated
A model trained last year cannot know this morning's news, but it can read it if you supply it. The difficulty is choosing what to supply.
At a glance
- What it is
- A system that retrieves documents, then prompts a model with them to generate an answer
- Two-stage architecture
- Retriever finds candidate passages; generator reads them and produces the response
- Common retrieval method
- Embed query and documents as vectors, return the nearest neighbours by cosine similarity
- Main failure mode
- The retriever returns plausible-looking documents that do not contain the answer
Imagine you are taking an exam, but it is open-book: you can bring in any notes you like. The catch is that someone else chooses which notes you get, based on a guess about which ones will help. If they guess well, you have everything you need. If they guess poorly, you are answering from irrelevant pages, and no amount of cleverness will substitute for having the right page in front of you.
Retrieval-augmented generation works the same way. The model is the exam-taker, the documents are the notes, and the retriever is the person choosing which notes to provide. The model never sees the full library — it only sees the few documents the retriever selected. If the retriever chose well, the model can produce an accurate answer by reading and synthesising what it was given. If the retriever chose poorly, the model is stuck. It might fabricate an answer, or it might answer a different question that the documents do support. The weak point is not the model's reading ability; it is the retriever's guessing ability.
The system has two components. The retriever takes a query and a document collection, embeds both using a shared encoder, and returns the top-k documents by cosine similarity in the embedding space. These documents are then serialised into the prompt, typically with delimiters, and passed to the generator along with the original query. The generator is a standard autoregressive language model, usually instruction-tuned, and it produces a response by conditioning on the concatenated input.
Retrieval is most commonly implemented as approximate nearest-neighbour search over dense vectors, using libraries like FAISS or purpose-built vector databases. The embedding model is often a bi-encoder like SBERT or a model from the same family as the generator but much smaller. Some systems use a cross-encoder to re-rank the top-k results, trading latency for precision. Hybrid retrieval combines dense vectors with BM25 or another sparse method, then merges the result lists.
The generator has no architectural awareness that some of its input came from retrieval. It cannot request more documents, and it has no explicit signal about which parts of the context are retrieved versus user-provided. Some implementations prepend instructions like "Answer using only the following documents", but this is prompt engineering, not a model capability. The model's behaviour is shaped entirely by its instruction-tuning, and models not specifically tuned for this task may ignore the retrieved documents or blend them with memorised training data.
Failure modes are asymmetric. If retrieval succeeds, generation is usually reliable: the model is good at extracting and synthesising information from text it can see. If retrieval fails, generation quality collapses, because the model has no source material and will often fabricate a plausible-sounding answer rather than refusing. Evaluation therefore focuses heavily on retrieval: measuring recall at k, comparing embedding models, tuning the retrieval threshold. The generator is largely treated as a black box that works if given the right inputs.
Look closer
The context window is the bridge between stages
Once the retriever has chosen its documents, they are inserted into the prompt verbatim, usually with light formatting to mark boundaries. The generator then behaves exactly as it would with any other prompt: it has no special signal that this text came from retrieval, and no way to go back and request different documents if the ones it received are unhelpful. The quality of the final answer is therefore bounded by the quality of the retrieval step, and no amount of prompting the generator can recover from a retriever that returned the wrong passages.
Retrieval is usually dense vector search
The query and every document are each passed through an embedding model, producing a vector for each. The system then returns the documents whose vectors have the highest cosine similarity to the query vector. This works well when the query and the answer use similar vocabulary, but struggles when they do not — a question about "car accidents" may miss a document that discusses "vehicle collisions" unless the embedding model has learned to place those phrases near each other. Older systems used keyword search; hybrid systems use both.
The retrieved set is fixed before generation begins
In the original formulation and most deployed systems, retrieval happens once. The model generates its answer from whatever that single retrieval step returned, with no opportunity to notice a gap and request more documents. Some experimental systems allow the model to emit a special token that triggers another retrieval pass, but this remains uncommon in production. The generator is therefore answering under constraint: it must work with the documents it was given, even if it could frame a better query itself.
The story
A language model's knowledge is fixed at training time. It cannot learn new facts without new training, and even if you retrain it daily, anything that happened this morning remains invisible. Retrieval-augmented generation solves this by sidestepping the problem: instead of encoding facts in the model's weights, you store them in documents and fetch the relevant ones at runtime.
The architecture has two stages. First, a retriever takes your query and searches a collection of documents, returning the handful it judges most relevant. Second, a generator — the language model — reads those documents and produces an answer. The documents are simply inserted into the prompt, so from the model's perspective this is ordinary text generation. It has no idea it is participating in a retrieval system.
The retriever is usually a vector search system. Each document in the collection has been passed through an embedding model, producing a vector that represents its meaning. When a query arrives, it too is embedded, and the system returns the documents whose vectors sit closest to the query vector in that high-dimensional space. This is dense retrieval: every word contributes to the vector, and similarity is geometric rather than lexical.
The appeal is immediacy. You can add this morning's documents to the collection and they become available instantly, with no retraining. You can also use it with private documents that were never part of any training corpus — internal reports, customer records, anything you can convert to text. The model reads them the same way it reads anything else.
The original paper, published by Lewis and colleagues in 2020, demonstrated the approach on open-domain question answering. Given a question like "What year did the Titanic sink?", the system retrieved passages from Wikipedia, inserted them into the prompt, and generated an answer. It outperformed models that relied solely on memorised training data, particularly on questions about less common facts.
But the results also revealed the central difficulty. When retrieval succeeded — when the returned documents actually contained the answer — generation was reliable. When retrieval failed, the model had nothing to work with. It would sometimes fabricate an answer anyway, because that is what a language model does when prompted with a question. The problem was not the generator. The problem was choosing the right five passages from five million candidates.
That remains the primary engineering challenge. Embedding models have improved, hybrid retrieval systems combine vector search with keyword matching, and some implementations re-rank the initial results with a second, slower model. But the task is intrinsically hard. A user's query is often short, ambiguous, or phrased in vocabulary that does not appear in the document that would answer it. The retriever must bridge that gap with no further guidance, and it must do so in milliseconds.
The architecture also exposes a mismatch in capability. The generator is typically a frontier model, trained on trillions of tokens and capable of nuanced reasoning. The retriever is a much smaller model, often trained on a narrower task, and it makes a single irrevocable decision before the generator ever runs. The system's overall performance is therefore limited by its weakest component, and that component runs first.
Why it mattered then
The 2020 paper arrived at a moment when language models were growing rapidly in capability but also in the cost and difficulty of updating them. GPT-3 had been released earlier that year, and it was already clear that training such models from scratch was becoming impractical for most research groups. Retrieval-augmented generation offered a way to extend a model's usefulness without retraining: you could give it access to new information by changing the document collection, not the weights. It also addressed a weakness that had become more visible as models improved. Larger models memorised more of their training data, but that made them better at reciting facts they had seen, not at reasoning about facts they had not. A model trained in 2019 would confidently give wrong answers about 2020 events, and no amount of prompting could teach it otherwise. Retrieval turned the model into a reader rather than a rememberer, which was both more flexible and more auditable: you could see exactly which documents it had used.
Why it matters now
Most commercial systems that claim to let a model "know about your documents" are using some form of retrieval-augmented generation. The document collection might be called a knowledge base, a vector store, or a retrieval index, but the architecture is the same: embed the documents, store the vectors, search them at runtime, and insert the results into the context window. It has become the standard solution for giving models access to private, recent, or frequently updated information. A customer support system can retrieve the latest version of a policy document. A research assistant can search this week's preprints. A legal tool can pull relevant case law without encoding every judgment in the model's weights. But the retrieval problem has not been solved, only made more visible. As context windows have grown, the cost of a bad retrieval has increased: you are now filling 128,000 tokens with documents that may not contain the answer, and the model will generate confidently from them anyway. Some practitioners are returning to hybrid methods, combining vector search with keyword filters, metadata tags, and even letting the model generate multiple queries to try different phrasings. The engineering effort has shifted from the generator, which is largely a solved problem, to the retriever, which remains stubbornly difficult.
The surprising detail
The generator cannot tell you that the retriever failed. If the returned documents do not contain the answer, the model has no mechanism to signal that gap — it was not trained to say "the documents you gave me are not helpful", because during training it simply received text and learned to continue it. Some systems now add explicit instructions in the prompt: "If the documents do not contain enough information, say so rather than guessing." This works inconsistently. The model's prior training created a strong expectation that questions receive answers, and overriding that expectation with a single prompt instruction is unreliable. The result is a system that fails silently, producing a confident answer from irrelevant sources.
Remember this
The model only reads what the retriever fetched. If the right document is not in that set, no amount of prompting will recover it.
Test yourself
You have built a retrieval-augmented system. A user asks a question and the generator produces a wrong answer, citing a document that was retrieved but does not support the claim. Name two architecturally distinct points where you could intervene, and explain what each intervention would actually change about the system's behaviour.
First, you could improve the retriever: change the embedding model, add keyword filters, retrieve more documents, or re-rank them with a second model. This changes which documents enter the context window, giving the generator better source material. Second, you could change the generator's instructions: add examples of refusing to answer, prompt it to quote directly from the documents, or ask it to assess whether the retrieved passages actually contain the answer before generating. This changes how the generator uses the documents it receives, though it cannot retrieve documents it was not given. A third option, architecturally different again, is to add a verification step after generation: pass the answer and the source documents to another model and ask whether the answer is supported. This does not prevent the error but can catch it before the user sees it. Each intervention addresses a different failure mode, and production systems often use all three.
Go deeper
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks · arXiv · Patrick Lewis et al. · 2020-05-22
- Retrieval-Augmented Generation for Large Language Models: A Survey · arXiv · Yunfan Gao et al. · 2023-12-18
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.