Backpropagation

The problem: whose fault was the error?

A network with many layers and millions of weights produces one final error at its output. Training requires knowing how much each individual weight, scattered across every layer, contributed to that error — information needed to know which direction and how much to adjust each one. Checking this one weight at a time, by nudging it slightly and re-running the entire network to see what changes, would be prohibitively slow for anything beyond a tiny network. Backpropagation is the algorithm that computes this efficiently for every weight at once.

The chain rule, applied layer by layer

The mathematical tool underneath backpropagation is the chain rule: if a change in one quantity affects a second quantity, which in turn affects a third, the chain rule lets you compute how the first quantity ultimately affects the third by multiplying the individual effects along the chain. A neural network is exactly this kind of chain, stretched across many layers — a weight in an early layer affects that layer’s output, which affects the next layer’s output, and so on until it finally affects the error at the very end. Backpropagation applies the chain rule systematically, working backward from the output layer toward the input layer, computing at each layer how the error depends on that layer’s weights, using quantities already computed for the layer just after it. This backward, layer-by-layer reuse is what makes the whole computation efficient rather than requiring a separate pass for every individual weight.

What to watch for

This video builds the intuition first, before any formula — error propagating backward through the network, and which weights and biases matter most to that error — continuing directly from the same series this group’s basics lesson started. It assumes the neuron, weight, bias and layer vocabulary that earlier lesson already covers.

It computes the gradient; it does not update anything itself

A point worth being precise about: backpropagation is not a separate learning algorithm standing apart from everything earlier in this roadmap. It computes the gradient — exactly the same kind of quantity gradient descent already uses to fit a linear regression, here applied across every weight in a much larger network at once. Once that gradient is known for every weight, the actual update — move each weight a small step in the direction that reduces the error — is performed by gradient descent, covered properly in training neural networks, exactly the same optimisation idea introduced all the way back at the very start of this roadmap’s supervised-models group.

Why the activation function choice from the last lesson matters here

The error signal computed by backpropagation has to travel backward through every layer’s activation function on its way from the output to the input. At each layer, the signal gets multiplied by that activation function’s local gradient — which is precisely why the previous lesson’s coverage of vanishing gradients matters directly here: an activation function whose gradient shrinks toward zero for large inputs causes the backward error signal to shrink at every layer it passes through, and by the time it reaches an early layer in a genuinely deep network, it can become too small to produce a meaningful weight update there at all. Backpropagation is the mechanism carrying the signal backward; whether that signal survives the trip intact depends directly on the activation functions the signal has to pass through along the way.