Backpropagation calculus
The same algorithm in symbols — partial derivatives, the chain rule, and where each term in the expression comes from.
This lesson is part of a preview roadmap.
The engine around it is real — progress, notes and bookmarks all work — but the video has not been through the sourcing and review protocol a shipped lesson goes through. See the roadmap for what that means.
Backpropagation calculus
The player loads from youtube-nocookie.com when you press play. Nothing reaches YouTube before that.
What you'll learn
- Every term in the backward pass is a chain-rule factor with a concrete meaning in the network.
- The derivative with respect to a weight is the activation feeding it times the error signal above it.
- The same expression generalises from one unit per layer to many by replacing numbers with indices.
The notation is the point
The previous node argued that influence flows backwards. This one writes that argument down. It is worth watching with a pen, because the expression for a single weight’s derivative is short enough to reconstruct from memory once, and reconstructing it once is what makes the general case readable.
What to watch for
Follow where each factor comes from rather than the final formula. One factor is the activation feeding the weight, one is the slope of the nonlinearity, and one is the error arriving from above. Every deep-learning framework’s autograd is bookkeeping over exactly these three.
Where this sits
That completes the training story for an ordinary feedforward network. The last group swaps the architecture for the one language models actually use, and the training story carries over unchanged.
Check yourself
Where does the activation of the previous layer appear in the weight's derivative, and why?
As a multiplying factor. A weight attached to a near-zero activation has almost no effect on the output, so its derivative is small.
What does moving from one neuron per layer to many change?
Only the bookkeeping. Each quantity gains an index and the sums run over the units of the next layer; the chain rule is unchanged.
Go deeper
The maths underneath
Notes
Notes are saved to your account.
Everything else on this page works without one — progress and bookmarks keep themselves in this browser and come with you when you sign in. Notes are the one thing worth not trusting to a browser you might clear.
Your notes could not be loaded.