[New Optimizer] 🌹 Rose: low VRAM, easy to use, great results, Apache 2.0

Hello, World! I have finally publicly released a new PyTorch optimizer I've been researching and developing for the last couple of years. It's named "Rose" in memory of my mother, who loved to hear about my discoveries and progress with AI. Without going into the technical details (which you can read about in the GitHub repo), here are some of its benefits: It's stateless, which means it uses less memory than even AdamW8bit. If it weren't for working memory, its memory use would be as low as plain vanilla SGD ( without momentum.