Quantization
A technique that reduces model weights from 16-bit or 32-bit floats to smaller representations (8-bit, 4-bit, or lower), cutting memory use and inference cost by 2–8× with minimal accuracy loss.
Quantization is the most important inference-cost optimization in 2026. A 70B model in 16-bit (FP16) requires ~140GB of GPU memory; quantized to 4-bit (INT4), it fits in 35GB and runs at twice the speed. Quality degradation is typically under 1% on most benchmarks for 8-bit; 2–5% for 4-bit.
For agent builders running open-weight models locally or on rented GPUs, quantization is the difference between "fits on consumer hardware" and "requires a data-center GPU." Tools like llama.cpp, vLLM, and TensorRT-LLM all support multiple quantization schemes out of the box.
Trade-offs: 8-bit is nearly lossless and the safe default; 4-bit is the cost-quality sweet spot for most use cases; 2-bit and lower require model-specific calibration and have higher accuracy loss.
Frequently asked
Does quantization make models worse?+
8-bit: barely noticeable for most tasks. 4-bit: 2–5% accuracy loss on benchmarks but often unnoticeable in practice. 2-bit and lower: real accuracy gaps, only worth it when memory is the binding constraint.
Should I quantize for production?+
Yes if you self-host. Quantized models run faster and cost less. For frontier API users (Claude, GPT-5), the vendor handles this and you do not need to think about it.