Quantization
Reducing model precision (e.g., 16-bit to 4-bit) to save memory and enable local inference.
Simple explanation
Quantization shrinks a model by using fewer bits per parameter. A 16-bit 70B model needs 140GB of RAM; a 4-bit quantised version needs ~35GB — fitting on one GPU.
Engineering explanation
Quantization reduces parameter precision: FP16 (16-bit) → INT8 (8-bit) → INT4 (4-bit). Methods: GPTQ, AWQ, GGUF (for CPU/Mac). Trade-off: smaller/faster but slight quality loss. QLoRA uses 4-bit quantisation of the base model during LoRA fine-tuning. Production local serving (Ollama, vLLM) uses quantised models.
Example
Llama 3.1 70B: FP16 = 140GB VRAM (2× A100). INT4 = 35GB (1× A100 or Mac with 64GB RAM).
When to use it
Local inference, self-hosting, fine-tuning on limited GPUs, cost reduction.
Common misconception
Quantization does NOT destroy quality. 4-bit quantised models retain 95%+ of full-precision quality for most tasks. The trade-off is almost always worth it.