We benchmarked TranslateGemma against 5 other LLMs on subtitle translation across 6 languages. At first glance the numbers told a clean story, but then human QA added a chapter. [D]

We evaluated six models on English subtitle translation into Spanish, Japanese, Korean, Thai, Chinese Simplified, and Chinese Traditional - 167 segments per language pair, scored with two reference-free QE metrics. Models tested: TranslateGemma-12b claude-sonnet-4-6 deepseek-v3.2 gemini-3.1-flash-lite-preview gpt-5.4-mini gpt-5.4-nano Scoring We used MetricX-24 (lower = better) and COMETKiwi (higher = better) - both reference-free QE metrics.