Attention and transformers

The bottleneck attention was built to remove

A recurrent network, covered in the previous lesson, carries its entire view of a sequence forward in one fixed-size hidden state, updated step by step. For a long sequence, this becomes a genuine bottleneck — an enormous amount of information about everything seen so far has to be compressed into a vector of fixed size, and inevitably some of it gets lost, particularly information from early in a long sequence by the time the network reaches the end. Attention was introduced specifically to remove this bottleneck: at each step, rather than relying solely on the compressed hidden state, the model is allowed to look directly back at any earlier position in the sequence and decide, dynamically, which of those earlier positions matter most for producing the current output.

What to watch for

This is the shortest, most purely conceptual option available, which is deliberately the right choice here: this lesson is meant to be a brief bridge, not a deep technical dive, since the full mechanics behind how modern language models actually use attention belong to Upshard’s separate AI Engineer roadmap rather than to this one. Under six minutes, it is light on the historical narrative of why attention mattered against the RNN bottleneck it replaced — the deeper links cover that history and the fuller mechanics for anyone who wants to go further right now.

The transformer: attention, and nothing sequential at all

A transformer takes this idea further than simply adding attention alongside recurrence — it removes recurrence entirely, building the whole architecture out of attention layers instead. Because there is no step that depends on the step before it, every position in a sequence can be processed simultaneously rather than one at a time, which is what makes transformers far more parallelizable, and therefore dramatically faster to train on modern GPU hardware, than an RNN processing the same sequence step by step ever could be.

Why this lesson stays brief on purpose

A full treatment of how attention actually computes which positions to attend to — queries, keys and values, the scaling factor, multiple attention heads, how position information gets encoded since attention itself has no inherent sense of sequence order — is a substantial topic in its own right, and it is exactly the territory Upshard’s separate AI Engineer roadmap covers in depth, since that mechanism is the foundation underneath every large language model covered there. This lesson’s job is narrower and more modest: establishing why attention mattered as an architectural idea, in contrast to the RNN approach this roadmap just covered, rather than teaching the full mechanics of how a transformer is built. Readers who want the complete mechanical treatment — the formulas behind scaled dot-product attention and positional encoding included — will find it there.

Where this leaves the sequence-modelling story

RNNs, with their sequential hidden state, and transformers, built entirely on parallel attention, represent two genuinely different architectural answers to the same underlying problem: how should a model handle data where order matters? Transformers have displaced RNNs for most large-scale sequence modelling today, particularly in language, largely because of this parallelism advantage during training at the enormous scale modern models operate at — but the RNN’s core idea, carrying a running summary forward through a sequence, remains a genuinely useful and considerably simpler tool for smaller-scale sequential problems where that full architectural complexity is not warranted.