AI RESEARCH
Universally Empowering Zeroth-Order Optimization via Adaptive Layer-wise Sampling
arXiv CS.LG
•
ArXi:2604.18264v1 Announce Type: new Zeroth-Order optimization presents a promising memory-efficient paradigm for fine-tuning Large Language Models by relying solely on forward passes. However, its practical adoption is severely constrained by slow wall-clock convergence and high estimation variance. In this work, we dissect the runtime characteristics of ZO algorithms and identify a critical system bottleneck where the generation of perturbations and parameter updates accounts for over 40% of the