Gradient descent

upshard

\theta_{t+1} = \theta_t - \eta \nabla_{\theta} L(\theta_t)

Step downhill: subtract a scaled gradient from the parameters and repeat. The entire difficulty is in the step size — too large and it diverges, too small and it never arrives.

Symbols

$\theta$: The parameters being optimised.
$\eta$: The learning rate, the step size.
$\nabla_{\theta} L$: The gradient of the loss with respect to the parameters.

Save your progress

Your account

Delete your account?

Gradient descent

Keyboard shortcuts