Shape of Memory: a Geometric Analysis of Machine Unlearning in Second-Order Optimizers

ArXi:2604.23046v1 Announce Type: new We argue that current definitions of machine unlearning are underspecified for second-order optimizers. We compare first-order and second-order learners for their ability to handle the data deletion task with varying degrees of eigendecomposition to mimic the loss model memory. While both first and second-order methods realign with the ideal counterfactul in terms of performance and gradient, the second-order optimizer shows significant volatility in the optimizer state.