A Functional Perspective on Knowledge Distillation in Neural Networks

ArXi:2510.12615v2 Announce Type: replace-cross Knowledge distillation is considered a compression mechanism when judged on the resulting student's accuracy and loss, yet its functional impact is poorly understood. We quantify the compression capacity of knowledge distillation and the resulting knowledge transfer from a functional perspective, decoupling compression from architectural reduction to provide an improved understanding of knowledge distillation.