What is a neural network?
Layers, weights, biases and activations — the whole forward pass, drawn on one handwritten digit.
This lesson is part of a preview roadmap.
The engine around it is real — progress, notes and bookmarks all work — but the video has not been through the sourcing and review protocol a shipped lesson goes through. See the roadmap for what that means.
What is a neural network?
The player loads from youtube-nocookie.com when you press play. Nothing reaches YouTube before that.
What you'll learn
- A layer is a matrix multiplication, a bias, and a nonlinearity, applied in that order.
- Weights are what the network learns; the architecture is what you choose before it learns anything.
- Without the nonlinearity, any stack of layers collapses into a single layer.
The architecture, before any learning
This node is deliberately about the forward pass only. A network that has learned nothing still computes something, and understanding what it computes — a chain of matrix multiplications with a squashing function between them — makes the training story in the next nodes much shorter.
What to watch for
The digit-recognition example assigns a hopeful interpretation to the hidden layers: this one finds edges, that one assembles loops. The video is careful to flag that real trained networks mostly do not organise themselves so tidily. Hold both ideas at once — the clean story explains the architecture, and its failure is what later interpretability work is about.
Where this sits
Everything so far has been fixed. The next node introduces the cost function and the idea of nudging weights downhill, which is where a network stops being a formula and starts being a thing that learns.
Check yourself
What would happen to a ten-layer network if you removed every activation function?
It would become mathematically identical to a one-layer network, because a composition of linear maps is itself linear.
What is the bias term for?
It shifts the threshold at which a unit starts responding, so a unit is not forced to pass through the origin.
Go deeper
The maths underneath
Notes
Notes are saved to your account.
Everything else on this page works without one — progress and bookmarks keep themselves in this browser and come with you when you sign in. Notes are the one thing worth not trusting to a browser you might clear.
Your notes could not be loaded.