AI RESEARCH
Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling
arXiv CS.CL
•
ArXi:2604.25578v1 Announce Type: new We present Marco-MoE, a suite of fully open multilingual sparse Mixture-of-Experts (MoE) models. Marco-MoE features a highly sparse design in which only around 5\% of the total parameters are activated per input token. This extreme sparsity, combined with upcycling from dense models, enables efficient pre-