Backpropagation
How a single training example votes on every weight in the network, and why the votes are averaged rather than applied.
This lesson is part of a preview roadmap.
The engine around it is real — progress, notes and bookmarks all work — but the video has not been through the sourcing and review protocol a shipped lesson goes through. See the roadmap for what that means.
Backpropagation
The player loads from youtube-nocookie.com when you press play. Nothing reaches YouTube before that.
What you'll learn
- Backpropagation computes the gradient; gradient descent is what then uses it.
- Each training example proposes a change to every weight, and the batch's proposals are averaged.
- Influence flows backwards — a weight matters in proportion to the activation feeding it and the error above it.
The algorithm, before the notation
This node is the intuition pass, and it is worth taking it on its own terms before meeting the symbols in the next one. The claim is simple: for one training example you can say, for every weight in the network, whether it should go up or down and roughly how strongly it wants to.
What to watch for
The framing of each example “voting” on each weight is doing real work. It explains why the gradient is an average, why a batch of examples that disagree produces a small step, and why a class the network rarely sees exerts little pull on the weights it would need.
Where this sits
The next node writes the same algorithm in the language of partial derivatives. Nothing new happens there — it is this argument, made precise enough to implement.
Check yourself
What is the difference between backpropagation and gradient descent?
Backpropagation is the algorithm that computes the gradient efficiently. Gradient descent is the rule that takes a step once you have it.
Why use mini-batches rather than the whole training set?
Because a noisy gradient from a small sample is far cheaper per step and, in practice, converges faster than the exact gradient computed rarely.
Go deeper
The maths underneath
Notes
Notes are saved to your account.
Everything else on this page works without one — progress and bookmarks keep themselves in this browser and come with you when you sign in. Notes are the one thing worth not trusting to a browser you might clear.
Your notes could not be loaded.