Sinusoidal positional encoding

Deep learning · Formula · Appears in 1 lesson

PE(pos,2i)=sin⁡ ⁣(pos100002i/d)PE_{(pos, 2i)} = \sin\!\left(\frac{pos}{10000^{2i/d}}\right)

Attention is order-blind, so position has to be added to the input explicitly. Using sinusoids at geometrically spaced frequencies means relative offsets show up as fixed rotations, and positions longer than any seen in training still produce valid codes.

Symbols

pospos
The index of the token in the sequence.
ii
The index of the dimension within the encoding.
dd
The model's embedding dimension.
Esc
↑↓ navigate↵ open? shortcuts

Keyboard shortcuts

Press Esc to close

Anywhere

⌘K
Search and commands
/
Search
?
This sheet
Esc
Close

Reading a lesson

J
Next lesson
K
Previous lesson
B
Bookmark