Chinchilla scaling law

upshard

L(N, D) = E + \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}}

Loss falls as a power law in both parameters and training tokens, with an irreducible floor. Fitting it showed the models of the day were badly undertrained — roughly twenty tokens per parameter is the compute-optimal balance.

Symbols

$N$: The number of parameters.
$D$: The number of training tokens.
$E$: The irreducible loss, the entropy of the data itself.
$\alpha$: The exponent governing returns to scale.

Save your progress

Your account

Delete your account?

Chinchilla scaling law

Keyboard shortcuts