AI RESEARCH
JudgeSense: A Benchmark for Prompt Sensitivity in LLM-as-a-Judge Systems
arXiv CS.CL
•
ArXi:2604.23478v1 Announce Type: new Large language models are increasingly deployed as automated judges for evaluating other models, yet the stability of their verdicts under semantically equivalent prompt paraphrases remains unmeasured. We Evaluating nine judge models on 494 validated paraphrase pairs, we find that coherence is the only task where judges meaningfully differ, with JSS ranging from 0.389 to 0.992. On factuality, all judges cluster near JSS about 0.63, driven by a polarity-inverted prompt artifact; after correction, factuality JSS rises to about 0.9.