LoRA — low-rank adaptation

$B \in \mathbb{R}^{d \times r}$ and $A \in \mathbb{R}^{r \times k}$, with $r$ much smaller than $d$ (written $r \ll d$). Their product $BA$ has the same $d \times k$ shape as $W_0$ but needs only $r(d + k)$ numbers. The factor $\alpha/r$ scales the correction so that changing $r$ does not require retuning the learning rate.

$B$ starts at all zeros, so $\Delta W = BA = 0$ at the first step and training begins from exactly the pretrained model. After training, $BA$ can be added into $W_0$ once, so running the model costs nothing extra.