KV-cache size

upshard

M_{\text{KV}} = 2 \cdot L \cdot H \cdot d_h \cdot S \cdot B \cdot b

Every generated token must keep its keys and values so later tokens can attend to them. The cache grows linearly with sequence length and batch size, and at long contexts it, not the weights, is what fills the GPU.

Symbols

$L$: The number of layers.
$H$: The number of key/value heads.
$d_h$: The dimension of each head.
$S$: The sequence length so far.
$B$: The batch size.
$b$: Bytes per value — two for fp16.

Save your progress

Your account

Delete your account?

KV-cache size

Keyboard shortcuts