LoRA (Low-Rank Adaptation)
A parameter-efficient fine-tuning method that trains a tiny fraction of model weights.
Simple explanation
LoRA is a way to fine-tune large models cheaply. Instead of updating all billions of parameters, it trains small 'adapter' matrices (0.1% of the model). The adapters are small (MB not GB) and can be hot-swapped.
Engineering explanation
LoRA freezes the base model and trains low-rank decomposition matrices (A, B) injected into specific layers (q_proj, v_proj, etc.). Rank (r) controls capacity — r=16 is typical. QLoRA quantises the base model to 4-bit, enabling fine-tuning of a 70B model on a single GPU. Adapters are small and composable — serve multiple from one base.
Example
Fine-tune Llama 70B with QLoRA on 1 GPU → adapter is 50MB → serve base + adapter → swap adapters without reloading base.
When to use it
Most fine-tuning. Full fine-tuning only for 100k+ examples and serious compute budget.
Common misconception
LoRA is NOT 'worse' than full fine-tuning. For most tasks, LoRA matches full FT quality at 1% of the cost. Full FT is rarely worth it.