3-Part Series: LLM Latency in Production (Part 1)

Substack.com. Part 1 - Model-Level Speed: Make the Model Fast on the GPU If you’re shipping LLMs to production, your first performance bottleneck isn’t serving logic or network overhead-it’s the raw arithmetic happening inside the GPU. Most teams waste weeks tuning their batching logic before realizing their model baseline is 3-4x slower than it should be. This part is about fixing that baseline. Why LLM Inference Is Memory-Bandwidth Bound (Especially in Decode) The fundamental misconception: LLMs are not always compute-bound.