AI RESEARCH
Randomized Advantage Transformation (RAT): Computing Natural Policy Gradients via Direct Backpropagation
arXiv CS.LG
•
ArXi:2605.18591v1 Announce Type: new Natural policy gradients improve optimization by accounting for the geometry of distribution space, but their practical use is limited by the cost of estimating and inverting the Fisher matrix. We present Randomized Advantage Transformation (RAT), a method for estimating Tikhono-regularized natural policy gradients via direct backpropagation. By applying the Woodbury formula, we reformulate the regularized natural policy gradients as vanilla policy gradients with a transformed advantage.