AI RESEARCH
Is the Modality Gap a Bug or a Feature? A Robustness Perspective
arXiv CS.CV
•
ArXi:2603.29080v1 Announce Type: new Many modern multi-modal models (e.g. CLIP) seek an embedding space in which the two modalities are aligned. Somewhat surprisingly, almost all existing models show a strong modality gap: the distribution of images is well-separated from the distribution of texts in the shared embedding space. Despite a series of recent papers on this topic, it is still not clear why this gap exists nor whether closing the gap in post-processing will lead to better performance on downstream tasks.