Training neural networks

The loop underneath every neural network

Training a neural network, regardless of its size or architecture, repeats one loop: a forward pass computes the network’s current predictions for a batch of training examples. The loss measures how wrong those predictions are, exactly as covered in evaluation metrics. The backward pass — backpropagation, covered in the previous lesson — computes the gradient of that loss with respect to every weight in the network. The optimizer step then uses that gradient to actually update every weight, moving each one a small distance in the direction that reduces the loss. This cycle repeats, over and over, across the entire training dataset, for as many epochs — full passes through the data — as it takes for performance to stop meaningfully improving.

What to watch for

The video introduces the cost function and gradient descent as the mechanism turning a forward pass’s measured error into an actual weight update — the conceptual heart of this entire loop, presented as the natural continuation of the same series this group’s earlier lessons have followed. It does not cover batch size, epochs, specific optimizers, batch normalization or dropout; the rest of this lesson’s written material covers that practical layer directly.

Batches: trading accuracy for speed

Computing the gradient using the entire training set for every single update is accurate, but prohibitively slow once a dataset is large — one update would require a full pass through all the data. Using a smaller batch of training examples for each update gives a noisier estimate of the true gradient, but allows updates to happen far more frequently for the same amount of total computation. Batch size is itself a setting that trades off these two concerns — a larger batch gives a more accurate but more expensive gradient estimate per update; a smaller batch is noisier but allows many more updates within the same time budget. Training on batches of a size smaller than the full dataset but larger than a single example is usually called mini-batch gradient descent, and it is the practical default nearly everywhere.

Optimizers: smarter than a plain fixed step

Plain gradient descent moves every weight by the gradient scaled by a fixed learning rate. In practice, several refinements consistently outperform this plain version. Momentum accumulates a running average of recent gradients, which smooths out the update and helps push through small bumps and flat regions in the loss landscape that would otherwise slow plain gradient descent down considerably. RMSprop and Adam go further, adapting the effective learning rate separately for every individual weight based on the recent history of its own gradients — a weight whose gradient has been consistently small gets a relatively larger effective step, and one whose gradient has been large and erratic gets a relatively smaller one. Adam combines the ideas behind momentum and RMSprop together and is, in practice, the most common default optimizer across modern neural network training.

Batch normalization: keeping every layer’s inputs well-behaved

As training proceeds and earlier layers’ weights change, the distribution of values flowing into a later layer can shift considerably from batch to batch — a problem that compounds as networks get deeper. Batch normalization addresses this directly by normalising each layer’s inputs within every batch, keeping them in a consistent, well-behaved range throughout training. This generally makes training both faster and more stable, and it has become close to a standard component in many modern network architectures.

Dropout: fighting overfitting by removing redundancy on purpose

Dropout addresses overfitting specifically by randomly disabling a fraction of neurons — commonly somewhere around twenty to fifty percent — during each training pass, forcing the remaining active neurons to produce a sensible result without being able to rely on any specific set of other neurons always being present to help. This prevents the network from developing an over-reliance on any small group of neurons, pushing it instead toward representations that are more robust and distributed, which tends to generalise to new data considerably better. Dropout is active only during training; once training is complete, every neuron is used during actual prediction.