AI RESEARCH
Bi-Level Policy Optimization with Nystr\"om Hypergradients
arXiv CS.LG
•
ArXi:2505.11714v2 Announce Type: replace The dependency of the actor on the critic in actor-critic (AC) reinforcement learning means that AC can be characterized as a bilevel optimization (BLO) problem, also called a Stackelberg game. This characterization motivates two modifications to vanilla AC algorithms. First, the critic's update should be nested to learn a best response to the actor's policy. Second, the actor should update according to a hypergradient that takes changes in the critic's behavior into account.