Why LLM Inference Slows Down with Longer Contexts

A systems-level view of how long contexts shift LLM inference from compute-bound to memory-bound You send a prompt to an LLM, and at first everything feels fast. Short prompts return almost instantly, and even moderately long inputs do not seem to cause any noticeable delay. The system appears stable, predictable, almost indifferent to the amount of text you provide. But this does not scale the way you might expect. As the prompt grows longer, latency does increase. But importantly, the system itself starts behaving differently.