AI RESEARCH
UniSD: Towards a Unified Self-Distillation Framework for Large Language Models
arXiv CS.LG
•
ArXi:2605.06597v1 Announce Type: cross Self-distillation (SD) offers a promising path for adapting large language models (LLMs) without relying on stronger external teachers. However, SD in autoregressive LLMs remains challenging because self-generated trajectories are free-form, correctness is task-dependent, and plausible rationales can still provide unstable or unreliable supervision. Existing methods mainly examine isolated design choices, leaving their effectiveness, roles, and interactions unclear.