II · THE IDEA · ARTIFICIAL INTELLIGENCE
Speculative Decoding
▶ Listen · narrated
A large model spends most of its time waiting: one token, one forward pass, repeat. Speculative decoding fills that waiting time with guesses from a smaller model, checked in bulk.
At a glance
- What it is
- A small model drafts K tokens; the large model verifies them all in one parallel pass
- Speed gain
- Up to 2–3× faster when the draft is frequently correct, exactly break-even when it is not
- Output guarantee
- Identical distribution to standard sampling from the large model alone
- Draft model
- Typically 10–100× smaller; same tokeniser required
Imagine you are editing a student's essay. Normally, you read one sentence, decide if it is acceptable, then move to the next. Speculative decoding is like having the student write five sentences at once while you are still reading the first. When you finish, you check all five in one sitting. If they are all good, you have saved time. If sentence three is wrong, you cross it out, rewrite it yourself, and hand the essay back for the student to continue from there. The student is fast but sometimes wrong. You are slow but always right. Together, you finish faster than you would alone, as long as the student is right often enough that you are not constantly rewriting everything.
Speculative decoding interleaves a small draft model Mₛ with a large target model Mₗ. At each step, Mₛ autoregressively generates K tokens {t₁, t₂, ..., tₖ}. Then Mₗ runs a single forward pass with all K tokens as input, computing logits qᵢ at each position i. For each position, the method compares qᵢ to the draft model's distribution pᵢ that produced tᵢ. The token tᵢ is accepted with probability min(1, qᵢ(tᵢ) / pᵢ(tᵢ)), and if rejected, a new token is sampled from the adjusted distribution max(0, qᵢ - pᵢ), normalised. This rejection sampling scheme ensures that the marginal distribution of each token matches Mₗ exactly. The speedup is K × acceptance_rate / (1 + cost_ratio), where cost_ratio is the draft cost relative to one large-model pass. If acceptance is high and the draft is cheap, this approaches K×. If acceptance is low, it approaches 1× (break-even). The draft and target must share a tokeniser, because acceptance probabilities are computed over token ids. The method requires no training, no architecture changes, and no approximation: it is a pure sampling algorithm that trades draft computation for reduced large-model calls.
Look closer
The large model checks the entire draft in one forward pass
Standard generation is serial: produce token one, feed it back, produce token two. Speculative decoding breaks this. The draft model proposes tokens one through K. The large model then runs a single forward pass with all K tokens as input, computing logits at every position in parallel. At each position, it compares its own distribution to what the draft model predicted. If they agree closely enough—sampled using a specific acceptance rule—the token is kept. The moment one is rejected, generation stops, resamples from the large model at that position, and starts a new draft from there.
The acceptance rule preserves the target distribution exactly
You might expect approximate output: the draft is wrong sometimes, so surely the final text differs slightly from what the large model would have produced alone. It does not. Both papers prove that their acceptance sampling schemes produce tokens from exactly the large model's distribution. The draft serves only to guess where the large model will go; when the guess is wrong, the method rejects it and resamples correctly. The result is mathematically identical to standard sampling, just faster when the draft is accurate.
The draft model must use the same tokeniser and vocabulary
The large model is checking token ids proposed by the draft. If the two models have different tokenisers, an id means different things to each, and the acceptance logic breaks. This rules out pairing models trained independently unless their vocabularies happen to align. In practice, the draft is often a smaller model from the same family, distilled from the large one, or trained with the same tokeniser from the start. The requirement is strict: same vocabulary, same ids.
The story
Language model generation is slow for a structural reason. Each token depends on all the tokens before it, so the model must run once per token, and each run must wait for the previous one to finish. You cannot generate token five until you have token four. This serialisation is why a model with a hundred billion parameters, capable of enormous parallel computation, still produces text at a few dozen tokens per second.
Speculative decoding attacks this bottleneck by doing two things at different speeds. A small draft model—perhaps a hundred times smaller—generates K tokens quickly. It is guessing what the large model would say. Then the large model wakes up and checks all K guesses in a single forward pass, because checking does not require serial generation: you can compute the logits for positions one through K in parallel if you already have the tokens.
The checking step uses a sampling rule that ensures the final output matches exactly what the large model would have produced on its own. If the draft token at position three is plausible under the large model's distribution, it is accepted. If not, it is rejected, the large model samples a replacement, and drafting resumes from that point. The acceptance probability is calibrated so that over many runs, the frequency of each token matches what you would see from the large model operating alone.
The speed gain depends entirely on how often the draft is correct. If the small model guesses well—common in fluent, predictable text—you accept several tokens per large-model pass, and generation accelerates by a factor of two or three. If the draft is poor, you reject every token, resample from the large model, and pay the cost of running both models for no gain. The method is therefore break-even in the worst case: it cannot be slower than standard generation, because a rejected draft simply returns you to the normal one-token-per-pass regime.
The choice of draft model matters. It must be fast enough that generating K tokens takes less time than one forward pass of the large model, or the overhead dominates. It must also be accurate enough that acceptances are frequent. A model distilled from the target or trained on similar data usually works well. The tokeniser must match exactly, because the large model is checking token ids, not text: if id 5432 means different subwords to the two models, the acceptance logic compares distributions over incompatible vocabularies and the output becomes undefined.
Both the Leviathan and Chen papers prove that their sampling schemes are unbiased: the final token distribution is exactly that of the large model. This is not approximate. The draft accelerates generation without changing what is generated, which means speculative decoding is a pure optimisation, not a modelling choice. You can enable it or disable it without retraining, retuning, or expecting different output quality.
Why it mattered then
The papers appeared in 2023, when inference cost was becoming the dominant expense for deployed models. Training a large model is expensive once; running it is expensive every time someone uses it. Speculative decoding offered a way to reduce that recurring cost without changing the model itself, which mattered for organisations serving millions of requests daily. The method also required no new architecture, no fine-tuning, and no approximation—properties that made it unusually easy to adopt. If you already had a small model from the same family, you could implement speculative decoding in a few hundred lines and see immediate gains on predictable tasks. The proof of distributional equivalence was particularly important in 2023, because approximate methods were common and often mistrusted: users worried that faster inference meant worse output. Speculative decoding removed that trade-off entirely.
Why it matters now
Speculative decoding is now implemented in several production inference engines, including vLLM and TensorRT-LLM, and it is a standard option for serving open-weight models. The method remains most effective in settings where the draft model can predict the large model accurately—customer service bots, code completion in familiar languages, and other domains with constrained, repetitive output. It is less useful for creative or highly unpredictable generation, where the draft is wrong too often. The technique has also inspired related work: some systems now use multiple draft models of different sizes, or draft models fine-tuned specifically to mimic a target. The core insight—that verification can be parallelised even when generation cannot—has become a standard tool in the effort to make large models practical to run.
The surprising detail
The method guarantees that output is statistically identical to the large model alone, but the wall-clock time to generate that output is random. If the draft happens to guess well for a particular prompt, you see a 3× speedup. If it guesses poorly, you see none. Two identical requests can therefore take very different amounts of time, which complicates latency guarantees in production systems. Some implementations now track draft acceptance rates per request type and disable speculative decoding dynamically when it stops helping, treating it as an adaptive optimisation rather than a fixed feature.
Remember this
Free speed when the draft is right, exact break-even when it is wrong, and output identical either way.
Test yourself
You have a large model and a draft model. The draft generates five tokens per large-model forward pass. On a particular task, the draft's first token is accepted 80% of the time, but subsequent tokens are accepted only 20% of the time. Should you use speculative decoding here, and if so, what draft length?
You should use it, but with a short draft—probably K=1 or K=2. Speculative decoding is break-even at worst, so there is no penalty for enabling it. However, if only the first token is usually accepted, generating five tokens per draft wastes computation: you run the draft model four extra times for tokens that will be rejected. A draft length of one or two matches the actual acceptance pattern, minimising wasted draft work while still capturing the 80% acceptance on the first token. The optimal K depends on the ratio of draft cost to large-model cost and the acceptance rate at each position, and it is often lower than you might expect.
Go deeper
- Fast Inference from Transformers via Speculative Decoding · arXiv · Yaniv Leviathan et al. · 2022-11-30
- Accelerating Large Language Model Decoding with Speculative Sampling · arXiv · Charlie Chen et al. · 2023-02-02
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.