AI RESEARCH
MegaTrain: Full Precision Training of 100B+ Parameter Large Language Models on a Single GPU
arXiv CS.CL
•
ArXi:2604.05091v1 Announce Type: new We present MegaTrain, a memory-centric system that efficiently trains 100B+ parameter large language models at full precision on a single GPU. Unlike traditional GPU-centric systems, MegaTrain s parameters and optimizer states in host memory (CPU memory) and treats GPUs as transient compute engines. For each layer, we stream parameters in and compute gradients out, minimizing persistent device state. To battle the CPU-GPU bandwidth bottleneck, we adopt two key optimizations. 1) We.