Reproducing Chinchilla Scaling on a Budget

Training a 70B parameter model costs millions of dollars. Scaling laws exist so you don't have to guess how to spend that budget. Here's what I learned reproducing them on a free GPU. Introduction Scaling laws are basically rules that tell us how model performance improves as you increase quantities such as model size, dataset size, and compute...