AI RESEARCH
MM-JudgeBias: A Benchmark for Evaluating Compositional Biases in MLLM-as-a-Judge
arXiv CS.CL
•
ArXi:2604.18164v1 Announce Type: new Multimodal Large Language Models (MLLMs) have been increasingly used as automatic evaluators-a paradigm known as MLLM-as-a-Judge. However, their reliability and vulnerabilities to biases remain underexplored. We find that many MLLM judges fail to reliably integrate key visual or textual cues, yielding unreliable evaluations when evidence is missing or mismatched, and exhibiting instability under semantically irrelevant perturbations. To address this, we systematically define Compositional Bias in MLLM-as-a-Judge systems and