The Silent Vote: Improving Zero-Shot LLM Reliability by Aggregating Semantic Neighborhoods

ArXi:2605.09739v1 Announce Type: cross Large Language Models are increasingly used as zero-shot classifiers in complex reasoning tasks. However, standard constrained decoding suffers from a phenomenon we define as Renormalization Bias. When a model is restricted to a small set of target labels, the standard softmax operation discards the probability mass assigned to semantic synonyms in the original distribution. This loss of information, which we call the Silent Vote, results in artificial overconfidence and poor calibration.