Batch normalization

upshard

\mathrm{BN}(x_i) = \gamma \frac{x_i - \mu_{\mathcal{B}}}{\sqrt{\sigma_{\mathcal{B}}^2 + \epsilon}} + \beta

Standardises each feature across the batch rather than within an example. It stabilises and speeds up convolutional training, at the cost of a train/test mismatch and a dependence on batch size that transformers refuse to accept.

Symbols

$\mu_{\mathcal{B}}$: The feature's mean across the current batch.
$\sigma_{\mathcal{B}}^2$: Its variance across the batch.
$\epsilon$: A small constant for numerical stability.

Save your progress

Your account

Delete your account?

Batch normalization

Keyboard shortcuts