AI RESEARCH
On the Proper Treatment of Units in Surprisal Theory
arXiv CS.CL
•
ArXi:2604.28147v1 Announce Type: new Surprisal theory links human processing effort to the predictability of an upcoming linguistic unit, but empirical work often leaves the notion of a unit underspecified. In practice, experimental stimuli are segmented into linguistically motivated units (e.g., words), while pretrained language models assign probability mass to a fixed token alphabet that typically does not align with those units.