Why Your Weights Stay Frozen: LoRA Explained with HuggingFace PEFT

LoRA freezes all original model weights and injects two tiny trainable matrices per layer, cutting the trainable parameter count by 10–100x while matching full fine-tune quality — and you can merge the adapter back in for zero inference overhead.

What you'll be able to do after this:

• Understand the W + BA rank decomposition that lets you adapt a model by training 0.04% of its parameters
• Configure LoraConfig — rank, target modules, lora_alpha — for any transformer with HuggingFace PEFT
• Merge, save, and share a LoRA adapter without touching the base model weights

Full fine-tuning a 7B model requires updating and storing gradients for billions of parameters — pushing VRAM past 80GB. LoRA solves this with one elegant insight: the effective rank of the useful weight update is low. You don't need to update every element of W; you can approximate the update as the product of two small matrices.

The math (no linear algebra course needed)

Every transformer layer has a weight matrix W (shape d × d). Full fine-tuning modifies W directly: W' = W + ΔW. LoRA approximates the update instead:

ΔW ≈ B × A    where  B ∈ ℝ^(d × r)  and  A ∈ ℝ^(r × d)

When r = 8 and d = 4096 (a typical attention layer in a 7B model): full ΔW has 16.7M parameters; LoRA has only 2 × 4096 × 8 = 65,536 — a 256x reduction. During training, W stays frozen; only A and B change. At inference, you merge: W_eff = W + BA, adding zero latency.

Walk-through with HuggingFace PEFT

from transformers import AutoModelForCausalLM
from peft import LoraConfig, get_peft_model, TaskType
import torch

# Load base model in bfloat16 to save ~50% VRAM vs float32
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3-8B",
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

config = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=8,                                   # rank: higher = more params, more capacity
    lora_alpha=32,                         # scaling: alpha/r is the effective multiplier
    target_modules=["q_proj", "v_proj"],   # which projection matrices to adapt
    lora_dropout=0.05,
    bias="none",
)

model = get_peft_model(model, config)
model.print_trainable_parameters()
# trainable params: 3,407,872 || all params: 8,033,669,120 || trainable%: 0.04

Train with any standard Trainer loop — only A and B accumulate gradients. After training:

# Option A: save just the adapter (~16 MB, shareable)
model.save_pretrained("my-lora-adapter/")

# Option B: merge into base model (zero inference overhead)
merged = model.merge_and_unload()
merged.save_pretrained("my-merged-model/")

Choosing rank r

r	Use case
4	Small datasets; minor behavioral tweaks
8	Community default; most instruction fine-tunes
16–32	Large behavioral shifts; domain adaptation
>64	Rarely helps; use more data instead

lora_alpha tip: set it to 2× your rank (alpha=16 with r=8) as a starting point. The effective scale applied to ΔW is alpha/r, so keeping this ratio stable as you tune r gives consistent behavior.

Sources: LoRA (Low-Rank Adaptation) — HuggingFace LLM Course Ch 11.4 · LoRA conceptual guide — HuggingFace PEFT docs · Chapter 7: LLM Finetuning Explained: LoRA, PEFT & When to Fine-Tune — YouTube