NVIDIA Just Released a 120B Model That Runs Like a 12B

Towards AI
AI Hardware

And the Architecture Behind It Is Genuinely Weird