Layer normalization

upshard

\mathrm{LN}(\mathbf{x}) = \gamma \odot \frac{\mathbf{x} - \mu}{\sqrt{\sigma^2 + \epsilon}} + \beta

Standardises each example across its own features, then lets the model rescale and shift the result. Because it never looks across the batch, it behaves identically at batch size one — which is why transformers use it and not batch norm.

Symbols

$\mu$: The mean over the feature dimension of this example.
$\sigma^2$: The variance over the same dimension.
$\gamma$: A learned per-feature scale.
$\beta$: A learned per-feature shift.

Save your progress

Your account

Delete your account?

Layer normalization

Keyboard shortcuts