How token counters actually work in 2026, and when to trust them
Dev.to AI
•
Generative AI
NLP
Most "free token counter" tools in your bookmarks are not running the model's tokenizer. They're running a character-ratio estimate and labeling the output "tokens". For OpenAI's GPT family the official tokenizer is open and easy to ship in a browser. For Claude, Gemini, and most others it isn't. Here's what that means for your context-window math. Up-front disclosure on this one: the tool I link to below is one I built.