Big models are big because every parameter costs bytes. Quantization stores those parameters in fewer bits. Squeeze a 70B model onto a gaming GPU, then measure exactly what the compression does to the weights.
An LLM's memory footprint is dominated by its weights: memory = parameter count × bytes per parameter. A 70B model at full 32-bit precision needs 280 GB before it computes a single token. Store each weight in fewer bits and the same model shrinks proportionally. (Parameters are learned weights, not vocabulary entries; see Tokenization for what tokens are, or start with What is an LLM if this is new territory.)
Weight precision
Half precision. The common serving baseline.
8B params × 2 bytes/param
16.0GB
Weights only, with 1 GB = 109 bytes. The KV cache, activations, and framework overhead all add more on top of this number.
Fits on (weights alone)
Quantization maps continuous weights onto a small grid of integer levels. This lab runs real symmetric uniform quantization on a deterministic sample of 2,000 bell-shaped weights. Every number below is computed from that sample:
scale = max|w| / (2^(b-1) - 1) q = clamp(round(w / scale), -(2^(b-1) - 1), +(2^(b-1) - 1)) w_hat = q × scale (error = w - w_hat)
Target precision
Representable levels
255
Step size (scale)
5.4e-4
RMS error
1.6e-4
Error / weight RMS
0.8%
Weight distribution: original vs 8-bit reconstruction
2,000 deterministic bell-shaped weights. After quantization every weight snaps to one of 255 levels spaced 5.4e-4 apart, so the gold bars collapse onto a grid as the bit width drops.
Quantization error
Original minus reconstructed, plotted on the same axis range as the weights above. At 8 bits the error is a needle at zero; at 2 bits it spreads across a large share of the weight range.
At 8-bit: RMS error 1.6e-4, which is 0.8% of the weight RMS (0.020). Largest single-weight error: 2.7e-4. 0.8% of all weights snap to exactly zero, negligible.
| Bit width | Levels | Step size | RMS error | vs 8-bit |
|---|---|---|---|---|
| 8-bit (selected) | 255 | 5.4e-4 | 1.6e-4 | 1.0× |
| 4-bit | 15 | 9.7e-3 | 2.8e-3 | 18× |
| 3-bit | 7 | 0.023 | 6.6e-3 | 42× |
| 2-bit | 3 | 0.068 | 0.018 | 113× |
PTQ vs QAT
Post-training quantization (PTQ) compresses a model after training, often using a small calibration set, with no gradient updates. Quantization-aware training (QAT) simulates the low-precision grid during training so the weights learn to live on it. PTQ is cheap and is the usual route for LLM deployment; QAT costs training compute but tends to hold up better at very low bit widths.
The outlier problem
Naive int8 quantization worked for smaller models but struggled as LLMs scaled, because activations (not the weights themselves) develop rare, large-magnitude outlier features. A single shared scale must stretch to cover those outliers, which crushes the resolution left for everything else. Mixed-precision schemes such as LLM.int8() route the outlier dimensions through higher precision while the bulk of the computation runs in int8.
QLoRA
QLoRA (Dettmers et al., 2023) fine-tunes on top of a frozen base model stored in 4-bit NF4, a data type designed for normally distributed weights. Gradients flow through the frozen 4-bit weights into small trainable LoRA adapters kept in higher precision, which makes fine-tuning very large models feasible on a single GPU. See LoRA & Adapters for the adapter math, and Fine-tuning vs Prompting for when to fine-tune at all.
GGUF and llama.cpp
GGUF is the model file format of the llama.cpp ecosystem: a single file carrying quantized weights at many bit widths plus metadata, built for running LLMs on ordinary CPUs and consumer GPUs.
How much quality do you lose?
It depends on the model, the task, the bit width, and the method, and the answer shifts with every generation of techniques. As a rough pattern reported across the literature: 8-bit weight quantization is usually close to lossless, 4-bit is often acceptable, and below 4 bits degradation grows quickly, exactly the shape the error lab above shows. There is no universal percentage; measure on your own task before you ship.