AI RESEARCH
Holistic Scaling Laws for Optimal Mixture-of-Experts Architecture Optimization
arXiv CS.LG
•
ArXi:2603.21862v1 Announce Type: new Scaling laws for Large Language Models govern macroscopic resource allocation, yet translating them into precise Mixture-of-Experts (MoE) architectural configurations remains an open problem due to the combinatorially vast design space. Existing MoE scaling studies are constrained by experimental budgets to either augment scaling formulas with extra MoE variables, risking unreliable fits, or fix all non-MoE factors, ignoring global interactions. We propose a reusable framework for holistic MoE architectural optimization that bridges this gap.