Tokens: The Atom of Everything in Large Language Models

Series: An Engineer’s Explorations in LLM Concepts “It started with a simple question: What exactly is a token? Three hours later, I had filled six browser tabs, youtube videos and had somehow ended up reading about some 5-6 concepts of LLM.” If you’ve worked with any LLM API - OpenAI, Anthropic, Gemini - you’ve seen the word token everywhere. In the docs, in the pricing, in the error messages.