GR-SAP: Generative Replay for Safety Alignment Preservation during Fine-Tuning

ArXi:2603.10243v1 Announce Type: new Recent studies show that the safety alignment of large language models (LLMs) can be easily compromised even by seemingly non-adversarial fine-tuning. To preserve safety alignment during fine-tuning, a widely used strategy is to jointly optimize safety and task objectives by mixing in the original alignment data, which is typically inaccessible even for open-weight LLMs.