Model memory footprint

upshard

M_{\text{weights}} = N \cdot b

Parameters times bytes per parameter. At fp16 a 7-billion-parameter model needs about 14 GB before any activations or cache; at 4-bit it needs under 4. Training needs several times more, because the optimiser keeps its own copies.

Symbols

$N$: The number of parameters.
$b$: Bytes per parameter — 4 at fp32, 2 at fp16, 0.5 at int4.

Save your progress

Your account

Delete your account?

Model memory footprint

Keyboard shortcuts