Speculative decoding speedup
A small draft model proposes k tokens and the large model verifies them all in one pass. The expected yield depends almost entirely on the acceptance rate, so a draft that agrees often beats a draft that is merely fast.
- The probability the target model accepts a drafted token.
- How many tokens the draft model proposes per step.