Sinusoidal positional encoding
Attention is order-blind, so position has to be added to the input explicitly. Using sinusoids at geometrically spaced frequencies means relative offsets show up as fixed rotations, and positions longer than any seen in training still produce valid codes.
- The index of the token in the sequence.
- The index of the dimension within the encoding.
- The model's embedding dimension.