AI RESEARCH
When Vision-Language Models Judge Without Seeing: Exposing Informativeness Bias
arXiv CS.AI
•
ArXi:2604.17768v1 Announce Type: new The reliability of VLM-as-a-Judge is critical for the automatic evaluation of vision-language models (VLMs). Despite recent progress, our analysis reveals that VLM-as-a-Judge often pays limited attention to the image when making decisions. Instead, they often blindly favor the informative answer, even when they can recognize it conflicts with the image content. We call this problem informativeness bias, which significantly undermines judge reliability.