When are we getting consumer inference chips?

r/LocalLLaMA
Generative AI Open Source AI

Dumb question but I genuinely don't get it. Billions of $ poured into AI startups the last few years and nobody has shipped a consumer chip with a model built in? Like a $200 stick that runs Llama 3 at reading speed, 30W, plug into your desktop, done. Taalas is kinda doing this but only aimed at datacenters. Why tho? Today's OS models are already good enough for 90% of what most people actually need and will still be for years. The "model will be obsolete before the chip tapes out" argument feels weaker every month.