StelLA: Subspace Learning in Low-rank Adaptation using Stiefel Manifold

ArXi:2510.01938v2 Announce Type: replace Low-rank adaptation (LoRA) has been widely adopted as a parameter-efficient technique for fine-tuning large-scale pre-trained models. However, it still lags behind full fine-tuning in performance, partly due to its insufficient exploitation of the geometric structure underlying low-rank manifolds. In this paper, we propose a geometry-aware extension of LoRA that uses a three-factor decomposition $U\! SV^\top