Fine-tuning a 7B model requires updating all 7 billion weights: expensive and slow. LoRA learns two tiny matrices whose product approximates the full weight update. Same quality, a fraction of the cost.
Sign in to save progress
Toggle between full fine-tuning and LoRA to see how the weight update changes.
Full fine-tuning updates every weight in W (4096×4096 = 16.7M params per layer).
W (8×8 scaled)
Update all 64 values
Real: 16,777,216 per layer
7B model: 7 billion total
Drag the slider to see how rank r controls trainable parameters and memory usage.
Rank r = 16
d = 4,096 (fixed)
LoRA params
131.1K
Full layer
16.78M
Param savings
99.22%
LoRA mem (fp32)
0.50 MB
2 × 4096 × 16 = 131,072 params
Low-rank fine-tuning achieves near full-fine-tuning quality while training a fraction of parameters.
Positions are illustrative, based on the LoRA (2021) and QLoRA (2023) papers: low-rank tuning typically lands within a point or two of full fine-tuning and sometimes beats it. Exact numbers depend on the task, model, and rank.
LoRA adapters are inserted into specific weight matrices inside each transformer block.
LoRA enabled
Multi-Head Attention
Q
Q projection
K
K projection
V
V projection
Feed-Forward Network
Up
Up-projection
Down
Down-projection
Numbers are for a 7B parameter model on a typical fine-tuning workload.
| Method | Trainable Params | VRAM (7B model) | Quality |
|---|---|---|---|
| Full Fine-Tune | 100% | ~80 GB | Baseline |
| LoRA (r=16) | ~0.2% | ~16 GB | Near parity |
| QLoRA (r=16) | ~0.2% | ~8 GB | Near parity |
| Adapters | ~0.5% | ~20 GB | Within 1-2% (illustrative) |
Key Insight
QLoRA enabled fine-tuning LLaMA 65B on a single 48 GB GPU, where full 16-bit fine-tuning needs over 780 GB. This democratized LLM fine-tuning for researchers and small teams. Note that the original Alpaca and Vicuna were full fine-tunes; it was community reproductions like Alpaca-LoRA, and the QLoRA paper's own Guanaco models, that showed adapters can reach comparable quality at a fraction of the cost. The underlying reason it works: weight updates during fine-tuning tend to be inherently low-rank, so most task-specific information lives in a small subspace.