Model Switching in Production: How We Evaluated LLMs for a Conversational Chatbot

Picture by ChatGPT When we first built our conversational chatbot, we assumed the model we chose would stick around for a while. Maybe we would upgrade it someday, but it didn’t feel like something we needed to actively plan for. Then the AI landscape started changing every few months. New models were released at a rapid pace. Benchmarks shifted. Pricing changed. Some models became faster, others slower. At the same time, older models were retired quickly than before. What once felt stable started to feel temporary.