{"id":2407,"date":"2025-03-14T07:02:20","date_gmt":"2025-03-14T07:02:20","guid":{"rendered":"https:\/\/mailitics.com\/index.php\/2025\/03\/14\/are-you-still-using-lora-to-fine-tune-your-llm\/"},"modified":"2025-03-14T07:02:20","modified_gmt":"2025-03-14T07:02:20","slug":"are-you-still-using-lora-to-fine-tune-your-llm","status":"publish","type":"post","link":"https:\/\/mailitics.com\/index.php\/2025\/03\/14\/are-you-still-using-lora-to-fine-tune-your-llm\/","title":{"rendered":"Are You Still Using LoRA to Fine-Tune Your LLM?"},"content":{"rendered":"

Are You Still Using LoRA to Fine-Tune Your LLM?
\n \t

\n
<\/BR>
\n
\n \t

\n
<\/BR><\/p>\n

\n

LoRA (Low Rank Adaptation \u2013 arxiv.org\/abs\/2106.09685<\/a>) is a popular technique for fine-tuning Large Language Models (LLMs) on the cheap. But 2024 has seen an explosion of new parameter-efficient fine-tuning techniques, an alphabet soup of LoRA alternatives: SVF, SVFT, MiLoRA, PiSSA, LoRA-XS \"\ud83e\udd2f\"\u2026 And most are based on a matrix technique I like a lot: the SVD (Singular Value Decomposition). Let\u2019s dive in.<\/p>\n

LoRA<\/h2>\n

The original Lora<\/a> insight is that fine-tuning all the weights of a model is overkill. Instead, LoRA freezes the model and only trains a small pair of low-rank \u201cadapter\u201d matrices. See the illustrations below (where W is any matrix of weights in a transformer LLM).<\/p>\n

\"\"<\/figure>\n

This saves memory and compute cycles since far fewer gradients have to be computed and stored. For example, here is a Gemma 8B model<\/a> fine-tuned to speak like a pirate using LoRA: only 22M parameters are trainable, 8.5B parameters remain frozen.<\/p>\n

\"\"<\/figure>\n

LoRA is very popular. It has even made it as a single-line API into mainstream ML frameworks like Keras:<\/p>\n

gemma.backbone.enable_lora(rank=8)<\/code><\/pre>\n
But is LoRA the best? Researchers have been trying hard to improve on the formula. Indeed, there are many ways of selecting smaller \u201cadapter\u201d matrices. And since most of them make clever use of the singular value decomposition (SVD) of a matrix, let\u2019s pause for a bit of Math<\/a>.<\/p>\n
SVD: the simple math<\/h2>\n
The SVD is a great tool for understanding the structure of matrices. The technique splits a matrix into three: W = USVT<\/sup> where U and V are orthogonal (i.e., base changes), and S is the diagonal matrix of sorted singular values. This decomposition always exists.<\/p>\n
\"\"<\/figure>\n
In \u201ctextbook\u201d SVD, U and V are square, while S is a rectangle with singular values on the diagonal and a tail of zeros. In practice, you can work with a square S and a rectangular U or V \u2013 see the picture \u2013 the chopped-off pieces are just multiplications by zero. This \u201ceconomy-sized\u201d SVD is what is used in common libraries, for example, numpy.linalg.svd<\/a>.<\/p>\n
So how can we use this to more efficiently select the weights to train? Let\u2019s quickly go through five recent SVD-based low-rank fine-tuning techniques, with commented illustrations.<\/p>\n
SVF<\/h2>\n
The simplest alternative to LoRA is to use the SVD on the model\u2019s weight matrices and then fine-tune the singular values directly. Oddly, this is the most recent technique, called SVF, published in the Transformers\u00b2 paper (arxiv.org\/abs\/2501.06252v2<\/a>).<\/p>\n
\"\"<\/figure>\n
SVF is much more economical in parameters than LoRA. And as a bonus, it makes tuned models composable. For more info on that, see my Transformers\u00b2 explainer here,<\/a> but composing two SVF fine-tuned models is just an addition:<\/p>\n
\"\"<\/figure>\n
SVFT<\/h2>\n
Should you need more trainable parameters, the SVFT paper (arxiv.org\/abs\/2405.19597<\/a>) explores multiple ways of doing that, starting by adding more trainable weights on the diagonal.<\/p>\n
\"\"<\/figure>\n
It also evaluates multiple alternatives like spreading them randomly through the \u201cM\u201d matrix.<\/p>\n
\"\"<\/figure>\n
More importantly, the SVFT paper confirms that having more trainable values than just the diagonal is useful. See their fine-tuning results below.<\/p>\n
\"\"<\/figure>\n
Next come several techniques that split singular values in two sets, \u201clarge\u201d and \u201csmall\u201d. But before we proceed, let\u2019s pause for a bit more SVD math.<\/p>\n
More SVD math<\/h2>\n
The SVD is usually seen as a decomposition into three matrices W=USVT<\/sup> but it can also be thought of as a weighted sum of many rank-1 matrices, weighted by the singular values:<\/p>\n
\"\"<\/figure>\n
Should you want to prove it, express individual matrix elements Wjk<\/sub> using the USVT<\/sup><\/strong> form and the formula for matrix multiplication on one hand, the
\u03a3 si<\/sub>ui<\/sub>vi<\/sub>T<\/sup><\/strong> form, on the other, simplify using the fact that S is diagonal and notice that it\u2019s the same thing.<\/p>\n
In this representation, it\u2019s easy to see that you can split the sum in two. And as you can always sort the singular values, you can make this a split between \u201clarge\u201d and \u201csmall\u201d singular values.<\/p>\n
Going back to the tree-matrix form W=USV<\/strong>T<\/sup><\/strong>, this is what the split looks like:<\/p>\n
\"\"<\/figure>\n
Based on this formula, two papers have\u00a0explored what happens if you tune only the large singular values or only the small ones, PiSSA and MiLoRA.<\/p>\n
PiSSA<\/h2>\n
PiSSA (Principal Singular values and Singular Vectors Adaptation, arxiv.org\/abs\/2404.02948<\/a>) claims that you should only tune the large principal values. The mechanism is illustrated below:<\/p>\n
\"\"<\/figure>\n
From the paper: \u201cPiSSA is designed to approximate full finetuning by adapting the principal singular components, which are believed to capture the essence of the weight matrices. In contrast, MiLoRA aims to adapt to new tasks while maximally retaining the base model\u2019s knowledge.\u201d<\/p>\n
The PiSSA paper also has an interesting finding: full fine-tuning is prone to over-fitting. You might get better results in the absolute with a low-rank fine-tuning technique.<\/p>\n
\"\"<\/figure>\n
MiLoRA<\/h2>\n
MiLoRA (Minor singular component LoRA arxiv.org\/abs\/2406.09044<\/a>), on the other hand, claims that you should only tune the small principal values. It uses a similar mechanism to PiSSA:<\/p>\n
\"\"<\/figure>\n
Surprisingly, MiLoRA seems to have the upper hand, at least when tuning on math datasets which are probably fairly aligned with the original pre-training. Arguably, PiSSA should be better for bending the behavior of the LLM further from its pre-training.<\/p>\n
\"\"<\/figure>\n
LoRA-XS<\/h2>\n
Finally, I\u2019d like to mention LoRA-XS (arxiv.org\/abs\/2405.17604<\/a>). Very similar to PiSSA but slightly different mechanism. It also shows good results with significantly fewer params than LoRA.<\/p>\n
\"\"<\/figure>\n
The paper offers a mathematical explanation of why this setup is \u201cideal\u2019 under two conditions:<\/p>\n