II · THE IDEA · ARTIFICIAL INTELLIGENCE
Speculative Decoding for Faster LLM Inference
▶ Listen · narrated
Autoregressive generation is serial by design. Speculative decoding keeps the large model’s distribution intact while letting a cheaper drafter spend most of the serial steps.
At a glance
- Core idea
- Small draft model proposes tokens; large target verifies several at once
- Output
- Statistically identical to sampling from the target alone
- Win condition
- Draft agrees often enough that one target pass yields multiple tokens
- Cost shape
- Serial work shifts to the cheap model; target work becomes more parallel
- Variant
- Staged drafting adds intermediate draft stages before the target
Think of a junior colleague who types several words ahead while a senior editor reads the line once and ticks off every word they would have allowed. Words that pass stay; at the first doubtful word the editor crosses out the rest and writes their own choice, then the junior starts ahead again.
Speculative decoding works like that with two language models. A small, fast draft model predicts a short run of tokens. The large target model scores that whole run in a single forward pass instead of one pass per token. An acceptance rule keeps the longest prefix that is consistent with sampling from the target. If a token fails the test, it and everything after it are dropped, and the target supplies the next token itself.
You still get text as if only the large model had been sampled. The saving appears when the draft is often right enough that one expensive check yields several committed tokens.
Speculative decoding couples a draft model q and a target model p over the same vocabulary. The draft samples a block γ = (y1,…,yk) autoregressively from the current prefix. The target then evaluates p(yi | prefix, y<i) for all i in one forward pass (with appropriate causal masking / KV handling).
For each i in order, accept yi with probability min(1, p(yi)/q(yi)). On the first rejection, sample the replacement token from the normalised residual max(0, p − q), discard the remainder of the draft block, and repeat. If the whole block is accepted, one further token may be sampled from p at the block’s end. The procedure yields draws distributed exactly as ancestral sampling from p, while the number of target forward passes scales with the number of blocks (and rejections), not with every token.
Wall-clock gain requires E[accepted length] high enough to amortise draft cost and the verify pass. Draft–target mismatch, domain shift, and too-large k all shrink acceptance. Staged speculative decoding inserts intermediate models between q and p so that cheap stages extend proposals and stronger stages raise survival until the final parallel verify; the target-side acceptance rule remains the correctness boundary. Limitations include draft overhead under low agreement, implementation complexity around KV caches and batched verify, and no relief when the bottleneck is not serial target decode.
Look closer
Verification is one forward pass, not a replay
The draft model runs autoregressively for a short stretch and emits a candidate continuation. Those candidates are then fed to the target in a single forward pass that scores every position at once. Acceptance is decided token by token from the two models’ probabilities; rejected tokens and everything after them are discarded, and the target’s own sample fills the first rejection. The expensive model still does real work, but it no longer pays a full serial step for every accepted token.
Acceptance is not greedy agreement
A draft token is not kept simply because both models’ top choices match. The procedure compares the draft and target probabilities and uses a rejection rule that can accept a draft token even when the target would sometimes prefer another, provided the draft did not over-propose relative to the target. When a token is rejected, a fresh sample is drawn from an adjusted target distribution. That detail is what makes the overall sequence distribution match ordinary target sampling.
Staging changes where the draft work sits
Staged speculative decoding inserts one or more intermediate draft stages rather than a single small model proposing straight to the large target. Earlier stages propose aggressively and cheaply; later stages filter or refine before the target’s verification pass. The observable effect is a pipeline of increasing model capacity, still ending in a parallel check that preserves the target distribution when the acceptance rule is applied correctly.
The story
Language-model inference is awkward for hardware. Each new token depends on the one before it, so the large model normally runs a full forward pass, emits one token, then does it again. Bandwidth and launch overhead dominate; arithmetic units sit under-used. Speculative decoding attacks that serial bottleneck without changing what the large model is allowed to say.
The arrangement uses two models that share a vocabulary. A small, fast draft model generates a short block of candidate tokens in the usual autoregressive way. Those candidates are then presented to the large target model together, so one target forward pass produces logits at every candidate position. From those logits, an acceptance test decides how much of the draft prefix to keep.
The test is the subtle part. It is not a simple equality check on argmax tokens. At each position the procedure looks at the probability the draft assigned to its chosen token and the probability the target assigns to the same token. If the target is at least as confident, the token is accepted and the algorithm moves on. If the draft overshot, the token is rejected with a probability that corrects for the mismatch, and generation resumes from a sample drawn from a residual target distribution at that position. Everything after a rejection is thrown away. The net effect is that the sequences you observe are distributed exactly as if you had sampled from the target alone, token by token.
Speed comes from acceptance length. When the draft and target tend to agree, a single expensive forward pass can commit several tokens. When they disagree early, you fall back toward ordinary one-token steps, plus the wasted draft work. So the method is most attractive when a much smaller model still shadows the large one well enough on the domain you care about—shared pretraining, distillation, or a draft specialised to the same chat template all help.
Staged speculative decoding, described in later work, inserts intermediate draft capacity between the tiniest proposer and the final target. Cheap stages stretch the speculative horizon; stronger intermediate stages raise the chance that candidates survive until the target’s parallel check. The acceptance logic at the target remains the guarantor of correctness: staging is an engineering choice about where serial draft cost and parallel verify cost sit, not a change to the sampling contract.
Nothing in the method requires modifying the target’s weights. The draft can be swapped, resized, or restaged without retraining the model whose outputs you actually want. That separation is why speculative decoding shows up as an inference-time system technique rather than a modelling trick: the contract is the target distribution; the draft is only a proposal engine whose mistakes are caught in bulk.
Why it mattered then
By late 2022, large transformers were already expensive to serve token-by-token. The speculative decoding paper formalised a way to keep exact target sampling while turning the target’s work into verification of a short draft block in one pass. That mattered because it separated quality (which model you sample from) from wall-clock structure (how many serial target steps you pay). Staged variants followed as practitioners looked for higher acceptance without making the drafter as costly as the target. The moment was less about a new capability than about making existing large models cheaper to run under the same sampling semantics.
Why it matters now
Serving stacks still spend most of their life in autoregressive decode. Contexts are longer, batching helps prefills more than single-stream decode, and users notice latency per token. Speculative and staged speculative decoding remain practical levers precisely because they do not ask you to accept approximate outputs: if the acceptance rule is implemented correctly, the target distribution is unchanged. Draft choice, stage depth, and speculative length are knobs operators can turn per model family and workload, which is why the pattern keeps reappearing in production inference systems.
The surprising detail
The large model can accept a draft token it would not have chosen as its own top prediction. Acceptance depends on probability mass, not on matching argmax. A draft that is merely “plausible enough” under the target can survive, and the residual sampling step on rejection is what restores exactness. The speed-up is therefore not a measure of how often two models agree on a single best token; it is a measure of how much probability mass they share along the draft path.
What is disputed
Published speed-ups depend on draft quality, speculative length, hardware batching, and how closely draft and target distributions match the workload. The exact wall-clock gain is empirical; the distribution-preserving claim holds only when the acceptance and residual-sampling rule are implemented as specified.
Remember this
A cheap drafter proposes; the target verifies several positions in one pass; accepted prefixes match ordinary target sampling.
Test yourself
If the draft model is almost never aligned with the target, what happens to latency and to the distribution of the final text, and why?
Latency tends to get worse or stay flat: you still pay for draft steps and for target verification, but acceptance lengths collapse toward one (or zero) useful tokens per target pass, so the extra work buys little parallel commit. The distribution of the final text, however, remains that of the target alone, because rejections discard the bad draft tail and residual sampling fills from the target’s adjusted distribution. Speculative decoding degrades toward expensive ordinary sampling rather than toward wrong sampling.
Go deeper
- [2211.17192] Fast Inference from Transformers via Speculative Decoding · arxiv.org
- [2308.04623] Accelerating LLM Inference with Staged Speculative Decoding · arxiv.org
Image: Original diagram, The Daily Triptych. Licence: Original work. Source.