Those of you who use LLMs have probably seen this: sometimes they code like a senior engineer, and other times they seem to forget even basic syntax. Research suggests that this is not hallucination.

So what is actually happening inside the “brain” of an AI when you give it a problem that exceeds its capabilities? A recent study has uncovered an extremely intriguing mechanism inside large language models: as the difficulty of a problem increases, the internal information flows within an LLM shift from a distributed state to a locally clustered one. The authors summarize this phenomenon in one short phrase: the out-of-distribution the input, the sparser the representation becomes.