II · THE IDEA · ARTIFICIAL INTELLIGENCE
Chunking and Reranking
▶ Listen · narrated
Most RAG failures happen after retrieval succeeds. The documents came back fast, but the model never saw the sentence that mattered.
At a glance
- Chunking
- Splitting documents into pieces small enough to fit in a prompt
- Reranking
- Scoring retrieved chunks with a second model trained to judge relevance
- Cross-encoder
- A model that sees query and candidate together, not as separate embeddings
- Typical improvement
- Reranking often adds more accuracy than changing the embedding model
Imagine you are looking for a specific recipe in a large cookbook, but you can only photocopy five pages to take home. You flip through quickly and mark fifty pages that might have it. That is the retrieval step—fast but imprecise. Then you read those fifty pages more carefully, checking whether each one actually answers your question, and you pick the best five. That is reranking. The first pass is quick because you are just skimming. The second pass is slower but accurate because you are reading properly. A RAG system works the same way: retrieve a lot of candidates fast, then judge them carefully before showing them to the model.
A RAG system splits documents into chunks and embeds each chunk into a vector space using a bi-encoder—a model that encodes text independently. At query time, the query is embedded with the same model and the retrieval system returns the top k chunks by cosine similarity. This is fast because chunk embeddings are precomputed and the search is a nearest-neighbour problem, but the query and chunk were encoded separately, so the similarity score is based on whether their independent summaries are close in vector space, not on whether the chunk actually answers the query.
A cross-encoder reranker takes the query and each retrieved chunk, concatenates them with a separator token, and runs them through a model—typically BERT or a similar architecture—that was trained to output a relevance score. The model sees both inputs simultaneously, so it can perform token-level attention between the query and the chunk. This allows it to notice lexical overlap, semantic relationships, and whether the chunk contains the specific information the query is asking for, rather than just whether the two texts are topically similar.
The cross-encoder is far more accurate but cannot be precomputed and must run on every candidate, so it is used as a second stage. The bi-encoder retrieves hundreds or thousands of candidates in milliseconds, the cross-encoder reranks the top twenty or fifty in tens of milliseconds, and the top five or ten proceed to the generation model. The two-stage architecture trades off speed and accuracy: the first stage is fast and recalls relevant documents, the second stage is slow and ranks them correctly.
Look closer
Chunk size is a forced trade-off with no good answer
Small chunks—two or three sentences—preserve precision. The retrieval system can point at exactly the paragraph that answers the query. But context vanishes: a pronoun refers to something two paragraphs earlier, now in a different chunk, and the model has no way to resolve it. Large chunks—whole sections, a thousand words—keep context intact but dilute the signal. The embedding represents an average of many ideas, so retrieval becomes vague. Most systems settle on 256 to 512 tokens per chunk and accept that both problems remain, just smaller.
First-stage retrieval is fast because it ignores half the problem
The initial retrieval step—usually vector similarity between query and document embeddings—encodes them separately. The query becomes a vector, each chunk becomes a vector, and you measure distance. This is fast enough to run over millions of chunks because the chunk embeddings are precomputed and the search is a geometric problem. But the query and the chunk never see each other during encoding. The model that made the query embedding had no idea what documents existed, and the chunk embeddings were created long before this query arrived. Relevance is being guessed from two independent summaries.
A cross-encoder sees both at once, and that costs everything
Reranking with a cross-encoder means taking the query and a candidate chunk, concatenating them, and running them together through a model—often BERT or a variant—that was trained specifically to output a relevance score. Because it sees both sides simultaneously, it can notice that the query asks about 'the second event' and the chunk mentions two events in sequence, or that a word is being used in the wrong sense. This is much more accurate. It is also far too slow to run on a million chunks, which is why reranking happens after retrieval has already narrowed the set down to ten or twenty candidates.
The story
Retrieval-augmented generation was described by Lewis and others in 2020 as a way to give a language model access to information it was not trained on, without retraining it. The idea is to retrieve relevant documents from an external corpus and include them in the prompt. The model then generates an answer conditioning on both the query and the retrieved text.
Two decisions dominate whether this works: how you split the corpus into pieces, and how you decide which pieces to show the model.
Chunking is the first problem. A retrieval system does not fetch whole books. It fetches fragments, because the model's context window is limited and because smaller units make retrieval more precise. But documents do not come pre-segmented into the right size. You must choose whether to split on paragraphs, sentences, fixed token counts, or semantic boundaries detected by another model. Each choice has consequences.
Split too small and you lose context. A chunk containing 'He resigned the following year' is useless if 'he' was identified three paragraphs earlier. Split too large and the retrieval signal diffuses. An embedding of a thousand-word section represents an average of many topics, so a query about one specific sentence may not pull that section to the top of the results. The chunk also consumes more of the context window, meaning fewer chunks fit in the final prompt.
Most systems settle somewhere between 256 and 512 tokens per chunk. This is not because that range solves the problem. It is because it makes both failure modes tolerable.
The second problem is ranking. A vector database returns chunks sorted by embedding similarity—cosine distance between the query embedding and each chunk embedding. This is fast because the embeddings are precomputed and the search is geometric. But the query and the chunks were encoded independently. The model that embedded the query never saw the document corpus. The model that embedded the chunks never saw this query. You are measuring relevance by comparing two summaries made in isolation.
Nogueira and others demonstrated in 2019 that a cross-encoder reranker—a model that sees the query and the candidate chunk concatenated together and outputs a relevance score—substantially outperforms embedding similarity. The improvement is not small. In their experiments on passage ranking, the cross-encoder improved the top result accuracy by double-digit percentage points compared to embedding retrieval alone.
The reason is straightforward. When the model sees both the query and the chunk at the same time, it can notice things that embedding similarity cannot. It can see that the query asks about 'the second event' and the chunk describes two events in chronological order. It can see that the word 'bank' in the query refers to a financial institution and the word 'bank' in the chunk refers to a riverbank. Embedding similarity sees two vectors, both of which activate weakly for 'bank', and calls that a match.
The cost is speed. A cross-encoder must process every candidate, and processing means a full forward pass through a model that is often comparable in size to the embedding model. You cannot precompute anything. This is why reranking is a second stage. The embedding search narrows a million chunks down to fifty. The cross-encoder reranks those fifty and you take the top five or ten for the final prompt.
The architecture of most RAG systems follows this two-stage pattern: retrieve fast and coarsely with embeddings, then rerank the top candidates with a model that actually looks at the query and the chunk together. The performance difference between a system that skips reranking and one that includes it is often larger than the difference between two embedding models. It is the highest-return change available in most retrieval pipelines, and it is frequently omitted because it adds latency and complexity.
Chunking and reranking are both compromises. There is no chunk size that preserves all context and all precision. There is no reranker fast enough to run on the entire corpus. The system works when the compromises are chosen carefully and when the two stages are actually doing different jobs: one finds candidates quickly, the other judges them accurately.
Why it mattered then
The RAG paper in 2020 arrived at a moment when language models were becoming capable enough to generate fluent long-form text but were still limited by what they had seen during training. Fine-tuning on new information was expensive and slow. Retrieval offered a way to give a model access to new documents, or to a private corpus, without retraining. The idea was not entirely new—information retrieval and question answering had been combined before—but the scale and fluency of the generation models made the approach practical in a way it had not been previously. The cross-encoder reranking work by Nogueira and others came slightly earlier, in 2019, and was focused on search rather than generation. The problem it addressed was that first-stage retrieval systems, even good ones, often put the best result in position three or seven rather than position one. Reranking with BERT improved top-result accuracy substantially. When RAG systems began to appear, the reranking approach transferred directly: the model generates from the top few chunks, so getting the ranking right in that small set matters more than getting the ranking right across the entire corpus.
Why it matters now
Retrieval-augmented generation is now one of the most common ways to deploy a language model in a setting where it needs access to specific information—internal documentation, legal databases, customer support histories, scientific papers. The model itself is general, and the retrieval layer makes it specific. Chunking remains a problem without a satisfying solution. Context windows have grown, which reduces the penalty for large chunks, but does not eliminate the trade-off. Systems that retrieve overlapping chunks, or that retrieve a chunk and then fetch its neighbours, add complexity and often help. Semantic chunking—using a model to detect topic boundaries—sometimes improves results, but adds a preprocessing step and is not clearly better than fixed-size splits in all settings. Reranking is underused relative to its impact. Many production RAG systems retrieve with embeddings and send the top results directly to the model, skipping the reranking step entirely. The reason is usually latency. A cross-encoder adds tens or hundreds of milliseconds, and that is enough to make a system feel slower. But the accuracy gain is large, and in most applications a slightly slower correct answer is worth more than a fast wrong one. The highest-return change available in a RAG pipeline that does not rerank is to add reranking.
The surprising detail
The embedding models used for retrieval and the cross-encoders used for reranking are often trained on the same data and have similar architectures, but they cannot be used interchangeably. An embedding model is trained to map text to a vector such that similar texts have similar vectors. A cross-encoder is trained to take two texts and output a score. You cannot extract embeddings from a cross-encoder in a useful way, because it never learns to represent a document independently—it only learns to judge a pair. This means you cannot precompute anything with a cross-encoder, which is why it must remain a second stage. The two models are solving related but distinct problems, and the architecture of each is a direct consequence of which problem it is solving.
Remember this
Chunking and reranking are where most RAG systems fail or succeed. Retrieval speed matters less than whether the right sentence reaches the model.
Test yourself
A RAG system retrieves ten chunks and sends them all to the model. Adding a cross-encoder reranker improves accuracy substantially. Explain why, even though the same ten chunks are still being retrieved.
The embedding retrieval returns ten chunks, but their order is based on vector similarity computed from independent encodings. The model generates from all ten, but it is more likely to rely on chunks that appear earlier in the prompt, and it may stop reading carefully after the first few. The cross-encoder reranks those ten by actually looking at the query and each chunk together, so it can move the truly relevant chunk from position seven to position one. The model then sees the best chunk first, when it is paying the most attention, and the answer improves even though the set of chunks has not changed. The order matters because the model's attention and generation are not uniform across the entire context.
Go deeper
- Passage Re-ranking with BERT · arXiv · Rodrigo Nogueira et al. · 2019-01-13
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks · arXiv · Patrick Lewis et al. · 2020-05-22
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.