AI RESEARCH
Privacy Amplification in Differentially Private Zeroth-Order Optimization with Hidden States
arXiv CS.LG
•
ArXi:2506.00158v2 Announce Type: replace Zeroth-order optimization has emerged as a promising approach for fine-tuning large language models under differential privacy (DP) and memory constraints. While privacy amplification by iteration (PABI) provides convergent DP bounds for first-order methods, establishing similar guarantees for zeroth-order methods remains an open problem. First-order PABI analysis relies on the fact that gradients are perturbed with isotropic noise, allowing privacy bounds to be iteratively tracked via shifted R\'enyi divergence.