Recently, I finished reading LoRA: Low-Rank Adaptation of Large Language Models to understand how large language models can be adapted to specific tasks with fewer trainable parameters. What interested me most was the separation between the knowledge already stored in a pre-trained model and the changes needed for a new task.
Why full fine-tuning gets expensive
Full fine-tuning updates all of a model’s parameters. For a large model, keeping a separate version for every task quickly becomes expensive in both training memory and storage.
LoRA keeps the pre-trained weights frozen and learns a much smaller set of task-specific parameters. This makes it possible to specialize a model without storing another complete copy of its weights for each task.
Learning a low-rank update
The central idea is to represent a weight update using two smaller matrices, A and B. For a frozen weight matrix W, the adapted weight is:
W′ = W + (α / r) BA
Here, r is the adapter rank and α controls the update’s scale. Only A and B are trained. For a matrix with d rows and k columns, this uses r(d + k) trainable parameters instead of dk. When r is small, the difference is substantial.
The paper’s hypothesis is that the changes needed for adaptation can have a low intrinsic rank. That does not require the original model’s weight matrices themselves to be low-rank.
One base model, many tasks
The storage example that stood out to me was GPT-3 175B. At 16-bit precision, its weights take roughly 350GB. In the paper’s example, keeping one shared model and 100 small LoRA adapters takes about 354GB, compared with 35TB for 100 full model copies. The adapter size depends on the rank and which matrices are adapted, so this is an example rather than a fixed cost.
This separation makes task-specific adapters easier to store and swap. For a single task, the learned update can also be merged into the base weights for inference.
What the results tell me
On benchmarks including GLUE, WikiSQL, and SAMSum, the paper reports results competitive with full fine-tuning while training far fewer parameters. These results make LoRA an appealing starting point for task specialization, though they do not guarantee the same outcome for every model or dataset.
The practical benefit is especially clear when I think about maintaining several specialized versions of one model. I would want to compare their task performance alongside their training memory and storage requirements.
Questions I want to explore
I’m interested in experimenting with small, task-specific adapters and measuring how much capacity a task actually needs. How does changing the rank affect quality, and which weight matrices are worth adapting?
My next reading is Thinking Machines Lab’s LoRA Without Regret, which examines the conditions under which LoRA can match full fine-tuning. I want to use it to better understand how rank, learning rate, and other hyperparameters affect the comparison.

最近,我读完了论文 LoRA: Low-Rank Adaptation of Large Language Models,想了解如何用更少的可训练参数,让大语言模型适应特定任务。 最吸引我的是,它把预训练模型已经掌握的知识与新任务所需的改动分开了。
全量微调为什么成本高
全量微调会更新模型的所有参数。 对于大模型,为每个任务保留一个独立版本,很快就会带来高昂的训练显存和存储成本。
LoRA 冻结预训练权重,只学习一小部分任务专属参数。 这样就能让模型适应新任务,而不必为每个任务再存储一份完整的模型权重。
学习低秩更新
核心思路是用两个较小的矩阵 A 和 B 表示权重更新。 对于冻结的权重矩阵 W,适配后的权重为:
W′ = W + (α / r) BA
其中,r 是 adapter 的秩,α 控制更新的缩放比例。 训练时只更新 A 和 B。 对于一个 d 行、k 列的矩阵,可训练参数量由 dk 变成了 r(d + k)。 当 r 较小时,两者的差距非常明显。
论文的假设是,任务适配所需的权重变化可以具有较低的内在秩。 这并不要求原模型的权重矩阵本身也是低秩的。
一个基础模型,多个任务
让我印象最深的是 GPT-3 175B 的存储示例。 使用 16 位精度时,它的权重约占 350GB。 在论文的示例中,一个共享模型加上 100 个小型 LoRA adapters 约需 354GB,而 100 份完整模型则需要 35TB。 adapter 的大小取决于秩以及适配哪些矩阵,因此这只是一个示例,并非固定成本。
这种分离让任务专属 adapters 更容易存储和切换。 针对单一任务,也可以把学到的更新合并到基础权重中进行推理。
如何理解这些结果
在 GLUE、WikiSQL 和 SAMSum 等 benchmark 上,论文报告的 LoRA 结果可以与全量微调相竞争,同时训练的参数少得多。 这些结果让我觉得 LoRA 很适合作为任务适配的起点,但它们并不保证每个模型、每个数据集上都能得到相同结论。
当我考虑维护同一个模型的多个专用版本时,这种方法的实际价值尤其明显。 我希望在比较任务表现的同时,也记录训练显存和存储需求。
我想继续探索的问题
我想尝试训练小型的任务专属 adapters,观察不同任务到底需要多大的容量。 改变秩会如何影响效果,又有哪些权重矩阵值得适配?
接下来,我准备阅读 Thinking Machines Lab 的 LoRA Without Regret,这篇文章研究了 LoRA 在什么条件下可以达到全量微调的效果。 我希望借此更好地理解秩、学习率和其他超参数如何影响两者的比较。
