Few-Shot Truly Benign DPO Attack for Jailbreaking LLMs

ArXi:2605.10998v1 Announce Type: cross Fine-tuning APIs make frontier LLMs easy to customize, but they can also weaken safety alignment during fine-tuning. While prior work shows that benign supervised fine-tuning (SFT) can reduce refusal behavior, deployed fine-tuning pipelines increasingly preference-based objectives, whose safety risks remain less understood. We show that Direct Preference Optimization (DPO)