Singular value decomposition
Factors any matrix at all into a rotation, a stretch along axes, and another rotation. Truncating the stretch to its largest few values gives the best low-rank approximation there is, which is the basis of compression and of LoRA.
- Orthogonal matrix whose columns are the output directions.
- Diagonal matrix of singular values, largest first.
- Orthogonal matrix whose columns are the input directions.