How to Build a RAG Evaluation Framework That Catches Real Problems
Dev.to AI
•
Generative AI
AI Research
Six months into running a production RAG system, I had a problem: my users kept complaining about wrong answers, but my evaluation metrics looked fine. Retrieval accuracy: 87%. User satisfaction: 82%. Everything looked good on paper. Then I sat with users for a week and watched them interact with the system. The real problems were invisible to my metrics.