Speculative decoding

Guess several steps ahead, then check all at once

Ordinary decode generates one token at a time, each step a separate, sequential, memory-bound pass through the full model — covered in prefill and decode and memory-bound vs compute-bound. Speculative decoding changes the schedule of this work without changing what the model is allowed to say: a small, fast draft model proposes several candidate next tokens at once, guessing ahead of where the full target model has actually computed to. The target model then checks all of those proposed tokens in a single parallel pass, similar in character to prefill, rather than generating each one through its own separate sequential step.

Why verification is exact, not a shortcut on quality

This is not an approximation technique. For each proposed token, the target model computes what it would actually have generated at that position on its own, and compares it against the draft model’s guess. A correct guess is accepted and the computation moves on to checking the next one; an incorrect guess is rejected, and the target model’s own correct token is used instead, with every proposal after that point discarded and the process restarting from there. The final sequence of tokens produced is identical, in distribution, to what ordinary step-by-step decoding would have produced — the technique only changes how the computation is scheduled, never what the model is permitted to output.

What to watch for

A reading-group walkthrough of the seminal speculative decoding papers, along with Medusa and EAGLE’s specific improvements on the basic idea — covering the draft-target mechanism and the named algorithms this lesson touches on in real depth. It runs over ninety minutes as a genuine reading-group session rather than a tight talk; treat it as a reference to dip into at a relevant point rather than something to watch start to finish.

Why this actually saves time

The saving comes from exploiting the exact asymmetry covered in memory-bound vs compute-bound: a sequential decode step pays the full cost of moving the target model’s weights through memory for just one token’s worth of useful output. Verifying several proposed tokens at once pays a similar cost in memory movement but produces several tokens’ worth of output if the draft model’s guesses hold up, converting several expensive sequential passes into one pass that does more useful work per unit of memory traffic. This only pays off when the draft model is cheap enough that proposing tokens barely costs anything and accurate enough that most of its guesses survive verification — a draft model that guesses poorly gains nothing, since every rejected proposal is wasted computation on top of what ordinary decoding would have cost anyway.

Named variants worth recognising

Medusa and EAGLE are specific, more sophisticated approaches to generating the draft proposals, each improving on simpler draft-model setups in different ways — EAGLE in particular has gone through several iterations (EAGLE, EAGLE-2, EAGLE-3) each refining how the draft predictions are produced. Recognising these names is useful context for navigating this fast-moving area; the underlying draft-then-verify mechanism covered above is what every one of them builds on.