AI RESEARCH
Feature Starvation as Geometric Instability in Sparse Autoencoders
arXiv CS.LG
•
ArXi:2605.05341v1 Announce Type: new Sparse autoencoders (SAEs) are used to disentangle the dense, polysemantic internal representations of large language models (LLMs) into interpretable, monosemantic concepts. However, standard $\ell_1$-regularized SAEs suffer from feature starvation (dead neurons) and shrinkage bias, often requiring computationally expensive heuristic resampling and nondifferentiable hard-masking methods to bypass these challenges.