Memory-bound token latency

upshard

t_{\text{token}} \approx \frac{M_{\text{weights}} + M_{\text{KV}}}{\mathrm{BW}}

When generating one token at a time, every weight has to be read from memory and barely any arithmetic is done with it. The floor on time per token is therefore set by bandwidth, not by FLOPs — which is the whole argument for batching.

Symbols

$M_{\text{weights}}$: Bytes of model weights read per token.
$M_{\text{KV}}$: Bytes of KV cache read per token.
$\mathrm{BW}$: Achievable memory bandwidth, in bytes per second.

Save your progress

Your account

Delete your account?

Memory-bound token latency

Keyboard shortcuts