Cram Less to Fit More: Training Data Pruning Improves Memorization of Facts

ArXi:2604.08519v1 Announce Type: new Large language models (LLMs) can struggle to memorize factual knowledge in their parameters, often leading to hallucinations and poor performance on knowledge-intensive tasks. In this paper, we formalize fact memorization from an information-theoretic perspective and study how