Attention
Queries, keys and values, scaled dot-product attention, and why dividing by the square root of the dimension is not cosmetic.
This lesson is part of a preview roadmap.
The engine around it is real — progress, notes and bookmarks all work — but the video has not been through the sourcing and review protocol a shipped lesson goes through. See the roadmap for what that means.
Attention
The player loads from youtube-nocookie.com when you press play. Nothing reaches YouTube before that.
What you'll learn
- Every token forms a query, a key and a value, each a learned linear map of its current vector.
- Attention weights are a softmax over query–key dot products, scaled by the square root of the head dimension.
- Masking is what keeps a language model from reading tokens it has not generated yet.
The mechanism, in one pass
Attention is three linear maps and a weighted average, and almost all of the difficulty is in keeping track of which vector is playing which role. The query–key dot product is the same operation from the second node of this roadmap, now asked a specific question: how relevant is that token to this one?
What to watch for
The scaling factor gets a few seconds in the video and deserves more. It is the difference between a softmax that distributes attention and one that collapses onto a single token and stops producing useful gradients — a small constant doing load-bearing work.
Where this sits
This is the last node of the preview. It is also the clearest illustration of why the formula library exists alongside the roadmaps: the expression behind this lesson has its own page, with its symbols named, and lessons and formulas link to each other from both sides.
Check yourself
Why divide by the square root of the head dimension before the softmax?
To keep the variance of the dot products from growing with dimension. Without it the softmax saturates and the gradients through it vanish.
What is the value vector for, as distinct from the key?
The key decides how much attention a token receives; the value is what actually gets added to the attending token's vector.
Go deeper
The maths underneath
Notes
Notes are saved to your account.
Everything else on this page works without one — progress and bookmarks keep themselves in this browser and come with you when you sign in. Notes are the one thing worth not trusting to a browser you might clear.
Your notes could not be loaded.