The LLM already knows git better than your retrieval pipeline

We shipped DiffMem a few months ago. Git-backed memory for AI agents, Markdown files, version history as context. It did well. We kept building it in production on Annabelle, our AI companion running on WhatsApp. Then I started actually looking at the retrieval layer and realized it was kind of a lie. Under the hood we had sentence-transformers doing cosine similarity scoring, rank-bm25 for keyword search, a two-pass LLM pipeline to distill queries and synthesize results, and scikit-learn and numpy just sitting there as collateral damage.