Gradient descent
The cost function, the gradient as the direction of steepest ascent, and learning as repeatedly stepping the other way.
This lesson is part of a preview roadmap.
The engine around it is real — progress, notes and bookmarks all work — but the video has not been through the sourcing and review protocol a shipped lesson goes through. See the roadmap for what that means.
Gradient descent
The player loads from youtube-nocookie.com when you press play. Nothing reaches YouTube before that.
What you'll learn
- Training is minimising a cost function over the weights, not over the input data.
- The gradient points in the direction of steepest increase, so learning steps along its negative.
- The step size is a choice with consequences — too large overshoots, too small never arrives.
From a fixed network to a learning one
The previous node left a network full of arbitrary numbers. This one adds the only two things needed to improve them: a number that says how wrong the network currently is, and a rule for which way to nudge every weight to make that number smaller.
What to watch for
The cost surface is drawn in two dimensions because that is what fits on a screen; the real one has as many dimensions as the network has parameters. The intuition that survives the jump is local and modest — you cannot see the landscape, you can only feel which way is downhill from where you stand.
Where this sits
Gradient descent says step along the negative gradient. It does not say how to compute that gradient for a network with millions of weights. That is backpropagation, and it is the next node.
Check yourself
What exactly are the variables being optimised?
The weights and biases. The training examples are fixed; the cost is a function of the parameters alone.
Why does anyone settle for a local minimum?
Because in very high dimensions the local minima that gradient descent actually reaches tend to be good enough, and finding a global one is not tractable.
Go deeper
The maths underneath
Notes
Notes are saved to your account.
Everything else on this page works without one — progress and bookmarks keep themselves in this browser and come with you when you sign in. Notes are the one thing worth not trusting to a browser you might clear.
Your notes could not be loaded.