Speculative decoding

Draft fast, verify in bulk

Sources: Fast Inference from Transformers via Speculative Decoding — Leviathan et al., 2023; Accelerating Large Language Model Decoding with Speculative Sampling — Chen et al., 2023

The previous sections improve batching and cache management. They do not change the usual rule that decode produces one token per forward pass through the target model. Speculative decodingspeculative decodingA small draft model proposes K tokens; the big target model verifies them all in one pass. Net effect: more tokens per target-model step.See in glossary → challenges that rule directly. By using a draft model to guess multiple future tokens and verifying them in one target-model pass, it can produce multiple target-distributed tokens per costly target step. The speedup is workload-dependent, and the exact rejection-sampling variant preserves the target distribution.

It is one of those rare ideas that feels too good to be true and turns out to actually be true.

The asymmetry it exploits

Recall from §11/§13 that small-batch decode is memory-bound: much of the time is spent reading weights rather than doing arithmetic. A target verification pass for 8 proposed tokens can therefore cost much less than eight separate decode steps, though it is not free: its cost grows with the number of proposals, context length, batch size, and kernel behavior.

So if we could somehow propose 8 candidate future tokens and check whether the target model would have generated each of them, we could verify all 8 in a single target-model forward pass at almost the same cost as decoding 1 token. That’s the entire premise.

The protocol

Two models cooperate:

• Target model: the big one you actually want to sample from (Llama-3-70B, say).
• Draft model: a much smaller, faster model that approximates the target (e.g. Llama-3-8B, or a custom-trained 1B model).

Each iteration:

1. Draft proposes K tokens. Run the draft model autoregressively for K steps starting from the current context. This produces K candidate tokens with their probabilities.

2. Target verifies. Run the target model once on the proposed continuation, using the current context and cache. The target produces the conditional logits needed to score the K proposed positions in parallel, effectively a short prefill.

3. Accept / reject. Walk through the K candidates left to right. At each position, compare the draft’s probability for that token to the target’s probability. Accept stochastically with probability $\min(1, p_{\text{target}} / p_{\text{draft}})$. The first time you reject, stop. (Importantly: when you reject, the already-computed target logits also let you sample a correction from a corrected residual distribution, so the iteration produces at least one new token without another target pass.)

4. Add the accepted prefix + the corrective token to the context. Repeat.

Step 3 is the magic step. Done correctly (the “rejection sampling” version), the output distribution is provably identical to plain sampling from the target. This is not an approximation to the target distribution, though it can still change latency, compute cost, and engineering complexity.

What determines speedup

Two factors:

1. Acceptance rate. If the draft model is well-aligned with the target (i.e. it tends to agree with what the target would have generated) the accepted prefix is long and you get many tokens per target step. Acceptance varies substantially by target, draft method, prompt, sampling settings, and proposal length. The higher the acceptance, the bigger the potential win.

2. Draft cost. Running the draft model has its own forward-pass cost. If that cost is large relative to a target step, the potential gain disappears. A smaller or specialized draft can help, but the useful size ratio depends on hardware, batching, and acceptance rate.

The net speedup is roughly:

Try it

The widget below simulates the loop. Move the sliders for K, acceptance rate, and draft cost to see how speedup changes.

A few things worth seeing:

• At K = 4, acceptance = 70%, draft cost = 10%, the simplified model shows a meaningful speedup. Real performance also depends on target verification cost and batching.
• Push K too high and you may reject most of the back of the chain; the marginal token is often less likely to be accepted, and draft cost adds up. There is a workload-specific sweet spot.
• Push acceptance below ~25% and you actually slow down: the draft costs more than it saves.

Draft model designs

The drafts are where most of the recent research lives:

• n-gram drafts: no neural model at all; commonly use repeated token sequences in the prompt to propose a continuation. Their overhead is small and their acceptance is workload-dependent.

• Smaller-model drafts: a smaller version of the same architecture. Easy to set up, but their acceptance depends on how closely their distribution matches the target.

• EAGLEEAGLEA draft-model architecture that predicts feature vectors of the target model, achieving high acceptance rates.See in glossary → / EAGLE-2 / EAGLE-3: a family of methods that train a draft component using target-model hidden-state information and can use tree-structured branching. They can achieve high acceptance, but no one method is best for every workload.

• MedusaMedusaAdds multiple parallel “medusa heads” onto the base model to propose several future tokens at once — no separate draft model.See in glossary →: bolt several “Medusa heads” onto the target model itself, each predicting a future token in parallel. No separate draft model required. Slightly lower acceptance than EAGLE but no two-model coordination.

• Token-level tree drafts propose not one chain of K tokens but a tree: many candidate continuations at once, verified together. Boosts the expected accepted-prefix length at moderate extra cost.

vLLM supports multiple speculative-decoding methods; the available draft types and APIs are version-dependent.

Where speculative decoding fits in the bigger picture

The trick changes the balance between the three serving metrics (§11):

• Throughput: improves substantially under speculation, since you push more tokens through the same target-model step.
• ITL: improves (each new token arrives faster on average).
• TTFT: speculation does not directly reduce the first target token, because it begins after prefill; it can still affect TTFT indirectly through shared serving resources.

It also interacts with batching. At very high batch sizes the target-model step can already be compute-bound, leaving less spare capacity to amortize and shrinking speculative gains. Serving systems must balance the token budget used for verification against ordinary decode work; the exact scheduling policy is implementation-dependent.

We have now covered every major optimization in modern single-GPU inference. The remaining sections look at what happens when one GPU isn’t enough.