AI RESEARCH
FLEXITOKENS: Flexible Tokenization for Evolving Language Models
arXiv CS.CL
•
ArXi:2507.12720v4 Announce Type: replace Adapting language models to new data distributions by simple finetuning is challenging. This is due to the rigidity of their subword tokenizers, which typically remain unchanged during adaptation. This inflexibility often leads to inefficient tokenization, causing overfragmentation of text in out-of-distribution domains, unseen languages, or scripts. In this work, we develop byte-level LMs with learnable tokenizers to make tokenization adaptive.