II · THE IDEA · ARTIFICIAL INTELLIGENCE
Speculative Decoding for Faster Inference
▶ Listen · narrated
Autoregressive decoding is stubbornly serial: each token waits for the last. Speculative decoding keeps the output distribution identical while letting hardware do more useful work per step.
At a glance
- Core idea
- A cheap draft model proposes tokens; the target verifies several at once
- Serial bottleneck
- Ordinary decoding needs one full forward pass per output token
- Parallel check
- The target scores a whole draft sequence in a single forward pass
- Distribution
- Accept/reject rules keep samples identical to sampling the target alone
- Speedup source
- Accepted drafts turn idle memory bandwidth into useful tokens
Ordinary generation is like writing a sentence one word at a time with a slow expert who must reread the whole page before each new word. Speculative decoding gives the expert a fast junior assistant. The junior scribbles the next few words. The expert then checks that whole scribble in one go. Words the expert endorses stay; at the first disagreement the expert crosses out the rest and writes the correct word instead. Because the expert’s check follows strict probability rules, the finished sentence is statistically the same as if the expert had written every word alone — only the waiting time changes. When the junior is often roughly right, several words land per expert check and generation feels snappier.
Let q denote the draft model’s conditional distributions and p the target’s. The draft autoregressively samples tokens y1…yγ. A single target forward pass yields p(yi | prefix, y1…y{i-1}) at each drafted position. Walking left to right, accept yi with probability min(1, p(yi)/q(yi)). On the first rejection at position i, sample from the normalised residual max(0, p − q) at that step and discard y{i+1}…yγ. If all γ tokens are accepted, sample one further token from p at the next position. The procedure is a lossless acceleration of sampling from p: the law of the committed sequence matches ordinary ancestral sampling from the target. Throughput gain equals the expected number of tokens committed per target pass, which rises with draft–target KL agreement and falls when rejections occur early. Draft cost must stay low enough that serial drafting does not erase the parallel verification win. Shared tokeniser and related training data help agreement; they are engineering choices, not part of the proof.
Look closer
Proposal is cheap; verification is shared
The draft model runs autoregressively for a short horizon — a handful of tokens — because it is small enough that those serial steps are inexpensive. The target model then consumes the whole draft prefix in one forward pass, producing logits at every drafted position. The expensive compute is amortised across several candidate tokens rather than spent on one.
Accept and reject are not a simple match
A drafted token is not kept merely because the target would have ranked it highly. Acceptance depends on a probability ratio between draft and target at that step. When a token is rejected, a replacement is drawn from an adjusted distribution so that the overall sample is exactly what pure target sampling would have produced. The method is exact, not approximate.
The draft need only be good enough often
Throughput rises when long prefixes survive verification. A draft that roughly tracks the target — same tokenizer, similar training mixture, smaller width or depth — tends to agree on easy continuation tokens and diverge on harder choices. Even moderate agreement yields several accepted tokens per target pass; perfect agreement is unnecessary.
The story
Standard transformer decoding is memory-bound and serial. To emit the next token you load the full model weights, run one forward pass, sample, append, and repeat. The arithmetic intensity is low: a great deal of data movement for a single integer out. Latency per token is therefore dominated by how fast the hardware can stream parameters, not by how clever the sampler is.
Speculative decoding attacks that structure without changing what the large model is allowed to say. A smaller draft model first generates a short candidate sequence — a handful of tokens — conditioned on the same prompt and the tokens already committed. Those candidates are guesses. They are cheap guesses, because the draft is far smaller than the target.
The target model then runs once over the prompt plus the entire draft. In that single pass it produces, at every drafted position, the distribution it would have used had it been decoding alone. With those distributions in hand, an accept/reject procedure walks the draft from left to right. Each drafted token is accepted with a probability derived from the ratio of target to draft probability at that step. If it is accepted, the procedure continues to the next drafted token. If it is rejected, a token is sampled from a corrected residual distribution, the remainder of the draft is discarded, and the round ends. If the whole draft is accepted, one extra token can be sampled from the target at the position just beyond the draft, so the round still makes progress.
Two closely related lines of work established the idea. Leviathan, Kalman and Matias framed speculative decoding as a way to accelerate transformer inference while proving that the output distribution matches ordinary sampling from the target. Chen and colleagues described speculative sampling in a similar spirit for large language models, again with an exactness guarantee. In both cases the point is not to approximate the large model, but to rearrange work so that its forward passes cover more accepted tokens.
The speedup is statistical rather than fixed. On easy stretches — common phrases, boilerplate, predictable syntax — the draft and target often agree, and several tokens commit per target pass. On harder stretches the draft is rejected earlier and the method falls back toward ordinary one-token steps. The wall-clock gain therefore depends on draft quality, draft length, and how well the hardware overlaps the draft's serial work with the target's heavier compute. The mathematical contract stays the same throughout: every committed token is distributed exactly as if the target had been sampled alone.
Why it mattered then
By late 2022 and early 2023, large language models were already expensive to serve. Decoding cost scaled with output length, and each new token demanded a full pass over multi-billion-parameter weights. Hardware sat under-utilised on the arithmetic side while memory bandwidth did the real limiting. Speculative decoding offered a structural answer rather than another distillation or quantisation trade-off: keep the large model as the source of truth, but stop asking it to spend an entire forward pass on a single token when a cheaper colleague can propose a plausible run of candidates. The acceptance proofs mattered as much as the speedups. Serving stacks could adopt the technique without arguing that quality had quietly shifted.
Why it matters now
Inference cost and latency still dominate the practical life of large models. Context windows have grown, tool-using agents emit long traces, and interactive applications remain sensitive to tokens per second. Speculative decoding fits cleanly beside other systems work — batching, paged attention, quantisation — because it does not require retraining the target or accepting a different distribution. Wherever a smaller aligned draft is available, or can be trained cheaply, the same accept/reject skeleton turns spare draft compute into higher target throughput. The idea also clarifies a broader design lesson: when the bottleneck is serial memory traffic, parallelism must be introduced in the proposal and verification pattern, not only inside a single matmul.
The surprising detail
The method can look like it is “trusting” a small model, yet a rejected draft never leaks into the output distribution. The residual sampling step is doing careful probability bookkeeping so that the target remains the sole author of what users see. Speed comes from how often the draft is right, not from lowering the standard of proof when it is wrong.
What is disputed
Reported speedups vary with draft length, draft–target agreement, batch size and hardware memory bandwidth. The papers establish the algorithm and the distributional guarantee; they do not fix a universal factor by which every deployment will accelerate.
Remember this
A small model drafts ahead; a large model verifies many positions in one pass; accept/reject keeps the sample identical to target-only decoding.
Test yourself
If the draft model is often wrong, speculative decoding still preserves the target model’s output distribution. What, then, actually determines whether you see a wall-clock speedup?
The average number of draft tokens accepted per target forward pass. High agreement yields several committed tokens per expensive pass and higher throughput; frequent early rejections waste draft work and fall back toward ordinary serial decoding. Exactness is guaranteed either way; speed is not.
Go deeper
- [2211.17192] Fast Inference from Transformers via Speculative Decoding · arxiv.org
- [2302.01318] Accelerating Large Language Model Decoding with Speculative Sampling · arxiv.org
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.