LoRA Fine-Tuning: Rank, Alpha and Which Modules to Target

LoRA freezes the base model and trains two small matrices alongside each weight it wants to adapt. The product of those matrices is added to the frozen weight. Because the update is low-rank, the number of trainable parameters collapses from billions to millions — and the memory you save is not in the weights, it is in the gradients and optimiser state.

What it actually changes

For a weight matrix W of shape d × k, LoRA learns B (d × r) and A (r × k) and computes:

y = Wx + (alpha / r) · B A x

Only A and B receive gradients. W stays untouched, which is why you can keep several adapters for different tasks and swap them at load time instead of storing a full copy of the model per task.

That property — one frozen base, many small adapters — is the real reason LoRA won. The memory saving is a bonus.

Rank: how much capacity you are buying

Rank r is the inner dimension of the update. It caps how complex a correction the adapter can express.

Rank	Typical use
4–8	A narrow format or style change, plenty of data
16	The default for almost everything
32	Complex behavioural change, or a large diverse dataset
64–128	Rarely justified. Measure before you pay for it

Higher rank does not monotonicly improve quality. On small datasets it mostly buys overfitting: you give the adapter enough capacity to memorise the training examples instead of learning the pattern.

The practical test: run rank 16 and rank 32 on the same data, evaluate both on your held-out set. If 32 wins by less than your noise band, ship 16 and keep the memory.

Alpha: the scale on the update

alpha divides into the update, so the effective learning rate of the adapter scales with alpha / r. The convention alpha = 2r keeps that ratio constant as you change rank, which is why so many configs pair 16 with 32.

If you change rank without changing alpha, you also change the effective learning rate — and then you are comparing two things at once. That is the most common mistake in LoRA ablation runs.

Which modules to target

The original LoRA paper experimented on attention projections only (q_proj, v_proj). That advice is still repeated and it is outdated for instruction tuning.

Target all linear projections:

target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
                  "gate_proj", "up_proj", "down_proj"]

The MLP layers hold roughly two thirds of a modern transformer’s parameters. Adapting only attention leaves most of the model’s capacity untouched, and the quality difference on instruction-following tasks is measurable — several points on most evals, at a modest memory cost, because the adapters are still a fraction of a percent of the model.

Skip the embedding and LM head unless your task genuinely needs new tokens. Adapting them adds parameters proportional to the vocabulary and destabilises training.

The memory you actually save

People expect LoRA to shrink the model. It does not — the base weights are still resident at full precision.

For an 8B model at rank 16:

	Full fine-tune	LoRA bf16
Weights	16.1 GB	16.1 GB
Gradients	16.1 GB	0.10 GB
Optimiser	~80 GB	0.64 GB
Total	~112 GB	~18 GB

The saving is entirely in gradients and optimiser state. To cut the weights too, you need QLoRA, which stores them in 4-bit.

Merging and serving

merge_and_unload() fuses the adapter into the base weights, producing a normal model with no inference overhead. Do this for production: an unmerged adapter costs a small amount of extra compute per forward pass, and most serving stacks handle merged weights more efficiently.

You can also serve adapters unmerged and hot-swap them per request — useful when you have many tenants or tasks sharing one base model.

A defensible starting config

LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    target_modules=["q_proj","k_proj","v_proj","o_proj",
                    "gate_proj","up_proj","down_proj"],
    bias="none",
    task_type="CAUSAL_LM",
)

Then set the learning rate around 2e-4, one to three epochs, and gradient checkpointing on. Generate the full config sized to your card with the config generator.