SozKZ: Training Efficient Small Language Models for Kazakh from Scratch

ArXi:2603.20854v1 Announce Type: cross Kazakh, a Turkic language spoken by over 22M people, remains underserved by existing multilingual language models, which allocate minimal capacity to low-resource languages and employ tokenizers ill-suited to agglutinative morphology. We present SozKZ, a family of Llama-architecture language models (50M-600M parameters) trained entirely from scratch on 9B tokens of Kazakh text with a dedicated 50K BPE tokenizer.