The 10,000× smaller fine-tune that still needs the whole model

Fine-tuning means continuing to train a pretrained model for a new job. For a giant language model, the usual version changes every weight and stores another giant checkpoint for every job. LoRA asks a sharper question: what if the useful change is much simpler than the model it changes?

The paper's answer is two skinny matrices per adapted layer. It reported competitive or better results while training far fewer numbers. That is a real saving—but “far fewer trainable numbers” is not the same claim as “a far smaller model.”

A dense layer multiplies an input by a weight matrix W. Full fine-tuning learns an equally shaped update ΔW. LoRA freezes W and writes that update as BA, where the inner dimension—the rank—is deliberately small. Rank is the number of independent directions the update can express.

If W is 4,096 by 4,096, a dense update contains 16.8 million numbers. At rank 8, A and B contain only 65,536: 256× fewer. Our tested toy update has rank 1, exactly the limit imposed by its one-column bottleneck.

During training, the layer computes Wx + (α/r)BAx. The paper initializes A randomly and B to zero, so the adapter initially changes nothing. After training, multiplication distributes: merge (α/r)BA into W, then run the ordinary dense layer.

def merge_lora(weight, a, b, alpha):
    rank = len(a)
    update = matmul(b, a)
    scale = alpha / rank
    return [[weight[i][j] + scale * update[i][j]
             for j in range(len(weight[0]))]
            for i in range(len(weight))]

The three versions are faithful translations of the tested core. Tests verify that the separate and merged paths produce identical outputs. This is why LoRA can add no inference latency—provided the adapter is merged before serving.

Here is the sentence a skim can miss. For rank 4 on GPT-3's query and value projections, the paper shrinks each task checkpoint from 350 GB to 35 MB—then immediately notes that deployment still needs the 350 GB base model. LoRA compresses the task-specific delta, not the knowledge already stored in the base.

On WikiSQL and MultiNLI, adapting query and value projections at rank 1 was already competitive with ranks up to 64. Yet the authors explicitly warn that a small rank should not work for every task or dataset. Their example is adaptation to a new language, which may demand something closer to changing the whole model.

• Stress rank under domain and language shifts rather than nearby tasks.
• Measure end-to-end memory, including activations and the resident base.
• Compare merged latency with dynamic multi-adapter batching.

To see the query and value projections LoRA modifies, poke at the interactive transformer page. The same factorized update also makes sense on any linear layer.