What is a transformer?
Embeddings, the residual stream, and the shape of the computation that turns a prompt into a next-token distribution.
This lesson is part of a preview roadmap.
The engine around it is real — progress, notes and bookmarks all work — but the video has not been through the sourcing and review protocol a shipped lesson goes through. See the roadmap for what that means.
What is a transformer?
The player loads from youtube-nocookie.com when you press play. Nothing reaches YouTube before that.
What you'll learn
- Tokens become vectors, and those vectors are repeatedly updated as they pass through the model.
- Directions in embedding space carry meaning, which is why vector arithmetic on them is sometimes sensible.
- The final step is a softmax over the vocabulary, and temperature is a knob on that softmax.
A different architecture, the same training story
Nothing in the previous group is discarded here. A transformer is still a stack of matrix multiplications trained by gradient descent with backpropagation. What changes is the shape of the stack and, crucially, that each position’s vector can read from every other position’s.
What to watch for
The embedding-space tour in the first half is the part that pays off later. If you accept that a direction in that space can mean something — and that the residual stream is a running total of such directions — then attention in the next node has somewhere to write its answer to.
Where this sits
The final node opens the one mechanism this video treats as a black box. It is the last node of the preview, and it is also where the formula library and the roadmap meet: scaled dot-product attention is a page in both.
Check yourself
What does the model actually output?
A probability distribution over the whole vocabulary for the next token. Everything else — sampling, temperature, top-p — operates on that distribution.
What does raising the temperature do?
It flattens the distribution before sampling, making unlikely tokens relatively more likely. Lowering it sharpens towards the argmax.
Go deeper
The maths underneath
Notes
Notes are saved to your account.
Everything else on this page works without one — progress and bookmarks keep themselves in this browser and come with you when you sign in. Notes are the one thing worth not trusting to a browser you might clear.
Your notes could not be loaded.