Preserve and Personalize: Personalized Text-to-Image Diffusion Models without Distributional Drift

ArXi:2505.19519v3 Announce Type: replace Personalizing text-to-image diffusion models involves integrating novel visual concepts from a small set of reference images while retaining the model's original generative capabilities. However, this process often leads to overfitting, where the model ignores the user's prompt and merely replicates the reference images. We attribute this issue to a fundamental misalignment between the true goals of personalization, which are subject fidelity and text alignment, and the