AI RESEARCH
Federation of Experts: Communication Efficient Distributed Inference for Large Language Models
arXiv CS.LG
•
ArXi:2605.06206v1 Announce Type: new Mixture of experts has emerged as the primary mechanism for making Large Language Models (LLMs) computationally efficient. However, in distributed settings, communicating token embeddings between experts is a significant bottleneck. We present the novel Federation of Experts (FoE) architecture. FoE restructures the MoE block of a transformer layer into multiple MoE clusters. Each cluster is responsible for only one of the KV heads and expert parallelism is applied between those experts.