AI RESEARCH
Knowledge Distillation Must Account for What It Loses
arXiv CS.LG
•
ArXi:2604.25110v1 Announce Type: new This position paper argues that knowledge distillation must account for what it loses: student models should be judged not only by retained task scores, but by whether they preserve the teacher capabilities that make those scores reliable. This matters because distillation is increasingly used to turn large, often frontier models into deployable systems, yet headline metrics can hide losses in uncertainty, boundary behavior, process reliability, on-policy stability, grounding, privacy, safety, and diversity.