Diversity-Aware Reverse Kullback-Leibler Divergence for Large Language Model Distillation

ArXi:2604.00223v1 Announce Type: cross Reverse Kullback-Leibler (RKL) divergence has recently emerged as the preferred objective for large language model (LLM) distillation, consistently outperforming forward KL (FKL), particularly in regimes with large vocabularies and significant teacher-student capacity mismatch, where RKL focuses learning on dominant modes rather than enforcing dense alignment. However, RKL