Scaled dot-product attention

upshard

\mathrm{Attention}(Q, K, V) = \mathrm{softmax}\!\left(\frac{QK^{\top}}{\sqrt{d_k}}\right) V

Every token proposes a query, every token advertises a key, and the match between them decides whose value gets read. Dividing by the square root of the key dimension keeps the dot products from growing with width and flattening the softmax gradient.

Symbols

$Q$: The queries — what each position is looking for.
$K$: The keys — what each position offers.
$V$: The values — what actually gets mixed and returned.
$d_k$: The key dimension, whose square root rescales the scores.

Save your progress

Your account

Delete your account?

Scaled dot-product attention

Keyboard shortcuts