SVD — singular value decomposition

$U$ and $V$ are orthogonal matrices, whose columns are perpendicular unit-length directions; each acts as a rotation (possibly with a mirror flip). $\Sigma$ (capital sigma) is diagonal: zero except on its diagonal, which holds the singular values $\sigma_1 \ge \sigma_2 \ge \dots \ge 0$. Each one says how strongly $W$ stretches along one direction.

The Eckart–Young–Mirsky theorem gives the payoff: keeping only the $k$ largest singular values and their directions gives the best possible approximation of $W$ with rank $k$. That result justifies low-rank compression and motivates LoRA. LoRA adds its own bet, which the theorem does not prove but which holds well in practice: that the change fine-tuning makes to a weight matrix is close to low-rank.