Recurrent neural networks

Data that has an order, not just a position

Every architecture earlier in this roadmap processes its input as a fixed, complete whole — a row of features, or a whole image at once. Some data is fundamentally different: a sentence, a time series of sensor readings, a sequence of user actions, where the order genuinely matters and the sequence can be of varying, unpredictable length. A recurrent neural network (RNN) is built specifically for this kind of data, processing it one element at a time while carrying information forward from everything seen so far.

The hidden state: memory carried forward

At each step, an RNN takes the current input together with a hidden state — a vector summarising everything processed up to that point — and produces an updated hidden state, which then carries forward into the next step. This is the mechanism that gives the network any notion of context: by the time it processes the fifth word of a sentence, its hidden state theoretically carries some compressed trace of the first four. The same small set of weights is reused at every single time step, regardless of how long the sequence is, which is what lets an RNN handle sequences of varying length without needing a different architecture for each possible length.

Why plain RNNs struggle with long sequences

Training an RNN means backpropagating error backward through time, across every step of the sequence, in a process directly analogous to backpropagating through the layers of a deep feedforward network, covered in backpropagation. Over a long enough sequence, this is mechanically equivalent to a very deep network, and it suffers the same vanishing gradient problem covered in activation functions: the error signal shrinks as it propagates backward across many time steps, which means a plain RNN struggles to learn dependencies between elements that are far apart in a sequence — a word at the very start of a long passage has essentially no influence on predictions made many steps later, not because the information is irrelevant, but because the gradient carrying that influence backward has vanished by the time it gets there.

What to watch for

This video covers both halves of this lesson’s scope in one connected explanation — why plain RNNs fail on long sequences, built on the same vanishing-gradient intuition from earlier in this roadmap, and then LSTM’s gates built up step by step as the direct fix. Covering both in one place is genuinely useful here, since the mechanism and its motivation make far more sense told together than split across separate videos.

LSTM and GRU: gates that protect the gradient

LSTM (long short-term memory) networks address the vanishing gradient problem directly by adding learned gates at each step: a forget gate decides what to discard from the carried-forward state, an input gate decides what new information to add, and an output gate decides what to actually expose as the current output. These gates create a more direct path for gradients to flow backward across many time steps without the same shrinkage a plain RNN suffers, which lets an LSTM learn dependencies spanning much longer sequences. GRU (gated recurrent unit) is a simpler variant using fewer, more combined gates, generally achieving comparable performance to LSTM with somewhat less computation.

Where attention eventually replaced this approach

RNNs, and LSTMs especially, were the dominant approach to sequence modelling for years. The hidden state carried between steps is, however, a bottleneck — the entire history of a long sequence has to be compressed into one fixed-size vector, no matter how much information that sequence actually contains. The next lesson in this roadmap, attention and transformers, covers the architectural idea that eventually replaced this hidden-state bottleneck for most large-scale sequence modelling — letting a model look directly at any earlier part of a sequence rather than relying on a single compressed summary carried step by step.