Momentum

upshard

v_{t+1} = \beta v_t + \nabla_{\theta} L, \qquad \theta_{t+1} = \theta_t - \eta v_{t+1}

Keeps a running average of past gradients so the step has inertia. Consistent directions build up speed while oscillating ones cancel out, which is what gets you along a narrow valley instead of bouncing across it.

Symbols

$v$: The velocity, an exponentially weighted average of gradients.
$\beta$: The decay rate, usually around 0.9.

Save your progress

Your account

Delete your account?

Momentum

Keyboard shortcuts