Exploring Knowledge Purification in Multi-Teacher Knowledge Distillation for LLMs

ArXi:2602.01064v2 Announce Type: replace Knowledge distillation has emerged as a pivotal technique for transferring knowledge from stronger large language models (LLMs) to smaller, efficient models. However, traditional distillation approaches face challenges related to knowledge conflicts and high resource demands, particularly when leveraging multiple teacher models. In this paper, we