The Great Compression: How AI Model Distillation Is Rewriting the Rules of the Industry

How knowledge transfers from large teacher models to smaller student models through distillation techniques. Image: GaussianWaves The Free Lunch That Came With a Bill In October 2019, a team at Hugging Face, a French-American AI startup, published a model called DistilBERT. It retained 97% of its parent model’s performance on language-understanding benchmarks while running 60% faster and carrying 40% fewer parameters. “The numbers looked like a free lunch.