AIoT and DSA Seminar | Knowledge Distillation: Towards Efficient and Compact Neural Networks   

Deep neural networks usually have a large number of parameters, which makes them powerful in representation learning, but also limits their deployment in real-world application. Knowledge distillation, which aims to transfer the knowledge from an over-parameterized teacher model to an efficient student model, has become one of the most popular method for neural network compression and acceleration. In this report, I will introduce my works on knowledge distillation in the last five years, which mainly focus on two aspects: the fundamental problems in knowledge distillation, and how to apply knowledge distillation to more challenging tasks. Besides, I will introduce the challenge and opportunities of AI model compression in the decade of large models.