知识蒸馏

Training a smaller "student" model on the outputs of a larger "teacher" model.

The student imitates the teacher's answers and distributions, approaching its quality at a fraction of the cost. Many "mini" and "nano" model variants are produced this way.