Measuring Social Bias in Vision-Language Models with Face-Only Counterfactuals from Real Photos

ArXi:2601.06931v2 Announce Type: replace-cross Vision-Language Models (VLMs) are increasingly deployed in socially consequential settings, raising concerns about social bias driven by graphic cues. A central challenge in measuring such social bias is attribution under visual confounding: real-world images entangle race and gender with correlated factors such as background and clothing, obscuring attribution. We propose a \textbf{face-only counterfactual evaluation paradigm} that isolates graphic effects while preserving real-image realism.