AI RESEARCH
Mixture of Heterogeneous Grouped Experts for Language Modeling
arXiv CS.LG
•
ArXi:2604.23108v1 Announce Type: cross Large Language Models (LLMs) based on Mixture-of-Experts (MoE) are pivotal in industrial applications for their ability to scale performance efficiently. However, standard MoEs enforce uniform expert sizes,creating a rigidity that fails to align computational costs with varying token-level complexity.