AI RESEARCH
A Self-Attentive Meta-Optimizer with Group-Adaptive Learning Rates and Weight Decay
arXiv CS.LG
•
ArXi:2605.04055v1 Announce Type: new Adaptive optimizers like AdamW apply uniform hyperparameters across all parameter groups, ignoring heterogeneous optimization dynamics across layers and modules. We address this limitation by proposing MetaAdamW - a new optimizer that integrates a self-attention mechanism to dynamically modulate per-group learning rates and weight decay. The modulation factors are produced by a lightweight Transformer encoder that operates on statistical features (gradient norms, momentum norms, correlations) extracted from each parameter group.