Run Qwen3.5 flagship model with 397 billion parameters at 5 – 9 tok/s on a $2,100 desktop! Two $500 GPUs, 32GB RAM, one NVMe drive. Uses Q4_K_M quants

Introducing FOMOE: Fast Opportunistic Mixture Of Experts (pronounced fomo). The problem: Large Mixture of Experts (MoEs) need a lot of memory for weights (hundreds of GBs), which are typically stored in flash memory (eg NVMe). During inference, only a small fraction of these weights are needed, however you don't know which ones ahead of time. This makes inference completely impractical on consumer hardware since flash latencies are too high for random access patterns. The solution: make most expert weight reads unnecessary...