Lost in the Prompt Order: Revealing the Limitations of Causal Attention in Language Models

ArXi:2601.14152v2 Announce Type: replace-cross Large language models exhibit surprising sensitivity to the structure of the prompt, but the mechanisms underlying this sensitivity remain poorly understood. In this work, we conduct an in-depth investigation on a striking case: in multiple-choice question answering, placing context before the questions and options (CQO) outperforms the reverse order (QOC) by over 14%p, consistently over a wide range of models and datasets.