Beyond Cross-Modal Alignment: Measuring and Leveraging Modality Gap in Vision-Language Models

ArXi:2502.14888v4 Announce Type: replace The success of vision-language models is primarily attributed to effective alignment across modalities such as vision and language. However, modality gaps persist in existing alignment algorithms and appear necessary for human perception as evidenced by modality-specific phenomena like visual texture and linguistic tone. These observations motivate us to computationally measure and leverage modality gaps to improve downstream tasks. We first