Knowledge Distillation

Level 1

Making a small AI learn from a big AI — like a little kid learning from a wise teacher to become smart without getting as big.

Level 2

A technique where a large, powerful AI model teaches a smaller model to give similar answers — so you get good results on cheaper hardware.

Level 3

A training technique where a smaller "student" model learns to mimic the outputs of a larger "teacher" model, achieving competitive performance with fewer parameters.

Level 4

A model compression technique where soft probability distributions from a teacher model provide richer training signal than hard labels, enabling the student to approximate the teacher's performance at a fraction of the compute.

Level 5

Transfer of learned representations from a high-capacity teacher to a compact student by matching soft logits (Hinton distillation), intermediate features, or attention patterns — trading capacity for inference efficiency while preserving most of the teacher's generalization.

Sources

• NIST Trustworthy and Responsible AI Resource Center: Glossary — National Institute of Standards and Technology. Accessed 2026-07-20.