Neural network basics

The model this roadmap has been building toward

Everything in this roadmap’s classical-models groups fit a relatively simple, explicit mathematical form to data — a line, a tree of thresholds, a margin between classes. A neural network is a different kind of model: instead of committing in advance to one specific mathematical shape, it is built from many extremely simple computational units, arranged in layers, whose combined behaviour can approximate an enormous range of possible relationships between inputs and outputs — provided it has the right structure and is trained correctly, both of which the rest of this group covers.

A single neuron is nothing new

A single neuron’s computation is almost exactly what linear regression already does: take every input, multiply each by its own learned weight, sum the results together with a learned bias term added on top. If this were the entire story, a neural network would be no more powerful than linear regression, no matter how many neurons were used — which is exactly the problem the next idea solves.

What to watch for

The video builds the full picture layer by layer, from a single neuron up to a complete multi-layer network, using handwritten-digit recognition as a running example throughout — watching weights and biases accumulate into something that can actually recognise a digit is the clearest way to see how these simple pieces compound into something considerably more capable. It is the first of a four-chapter series this group’s later lessons continue directly.

Non-linearity is what makes stacking worth anything

Composing one linear function with another linear function produces, after the algebra is worked through, just another linear function — stacking ten purely linear layers collapses mathematically into something no more expressive than a single one. This is why an activation function — a non-linear function such as sigmoid or ReLU, covered properly in the next lesson — is applied after every layer’s weighted sum, before the result passes to the next layer. Inserting that non-linear step between layers is precisely what lets a multi-layer network represent relationships a single linear layer structurally cannot, no matter how its weights are set.

The universal approximation idea

This combination — enough layers, enough neurons per layer, and a non-linear activation between each one — gives a neural network a property no model earlier in this roadmap has: with enough capacity, it can approximate essentially any reasonably well-behaved function connecting inputs to outputs, a result usually called the universal approximation property. This does not mean a network automatically finds that approximation in practice — training it correctly, covered in training neural networks, and giving it a sensible architecture are both still required — but it explains why this family of models can represent patterns, from recognising images to generating coherent text, that no single linear or even tree-based model, however large, is structurally capable of representing at all.

Weights and biases are learned, not designed

Just as a linear regression’s coefficients are fit by minimising a cost function rather than chosen by a person, a neural network’s weights and biases — easily millions or billions of individual numbers in a large network — are discovered the same way: by defining a cost function measuring how wrong the network’s current predictions are, and adjusting every weight and bias to reduce that cost. The scale is vastly larger than anything earlier in this roadmap, but the underlying principle, optimising parameters against a cost function, is exactly the same idea introduced all the way back in linear regression. How that optimisation actually happens across so many parameters at once is the subject of the next two lessons.