What is model distillation in deep learning?

Model distillation is a technique in deep learning where a smaller, simpler model (the “student”) is trained to replicate the behavior of a larger, more complex model (the “teacher”). The goal is to transfer the knowledge captured by the teacher model into a more efficient form without significant loss in performance. This process is particularly useful for deploying models in resource-constrained environments, such as mobile devices or edge computing systems, where the computational cost or memory footprint of the original model might be prohibitive.

The core idea involves training the student model using not just the original training data but also the outputs of the teacher model. Instead of relying solely on hard labels (e.g., class indices in classification tasks), the student learns from the teacher’s "soft targets"—the probability distributions the teacher generates over possible classes. For example, in an image classification task, the teacher might assign a high probability to the correct class (e.g., “cat”) but also smaller probabilities to semantically related classes (e.g., “dog” or “tiger”). These nuanced outputs provide richer guidance than hard labels, helping the student model generalize better. A common implementation uses a loss function that combines the student’s error relative to the true labels and its divergence from the teacher’s soft predictions. Techniques like temperature scaling—adjusting the softmax function’s sharpness—are often applied to make the teacher’s output distributions more informative during training.

A practical example is compressing a large language model like BERT into a smaller variant (e.g., DistilBERT). The student model mimics the teacher’s predictions on tasks like text classification while using fewer layers and parameters. Similarly, in computer vision, a ResNet-50 model’s knowledge can be distilled into a lightweight MobileNet for faster inference on mobile devices. The benefits include reduced latency, lower memory usage, and easier deployment, though there’s often a trade-off between size and accuracy. By focusing on the teacher’s learned patterns rather than raw data alone, distillation enables efficient models that retain much of the original’s capability.