Quantization

Level 1

Making the AI smaller so it can run on regular computers and phones instead of needing a giant supercomputer.

Level 2

A technique to shrink AI models by using less precise numbers — like rounding 3.14159 to 3.1. The model gets smaller and faster with only a small drop in quality.

Level 3

A model compression technique that reduces the numerical precision of weights and activations (e.g., from 32-bit to 4-bit), decreasing memory usage and speeding up inference.

Level 4

Mapping continuous-valued model parameters to a discrete set of lower-precision values (FP16, INT8, INT4), trading representational fidelity for reduced memory footprint and increased throughput.

Level 5

Post-training or quantization-aware reduction of weight and activation precision — using techniques like GPTQ, AWQ, and SqueezeLLM to navigate the Pareto frontier between model quality and hardware efficiency across diverse accelerator architectures.

Sources

• NIST Trustworthy and Responsible AI Resource Center: Glossary — National Institute of Standards and Technology. Accessed 2026-07-20.