AI RESEARCH
DMAP: A Distribution Map for Text
arXiv CS.LG
•
ArXi:2602.11871v2 Announce Type: replace-cross Large Language Models (LLMs) are a powerful tool for statistical text analysis, with derived sequences of next-token probability distributions offering a wealth of information. Extracting this signal typically relies on metrics such as perplexity, which do not adequately account for context; how one should interpret a given next-token probability is dependent on the number of reasonable choices encoded by the shape of the conditional distribution.