Tokenizer-Aware Markdown Chunking That Doesn't Shred Tables
Dev.to AI
•
Generative AI
NLP
AI Tools
Book: RAG Pocket Guide: Retrieval, Chunking, and Reranking Patterns for Production My project: Hermes IDE | GitHub - an IDE for developers who ship with Claude Code and other AI coding tools Me: xgabriel.com | GitHub Picture a engineer asking a doc bot why a webhook retry is failing. The retriever returns half of a markdown table: the header row, two body rows, then a hard cut at exactly 512 tokens. The remaining four rows, including the one that lists 429 as the retry trigger, go to the next chunk. The generator answers confidently and wrong. A better embedder will not save you.