Rethinking Parameter Sharing for LLM Fine-Tuning with Multiple LoRAs

ArXi:2509.25414v2 Announce Type: replace Large language models are often adapted using parameter-efficient techniques such as Low-Rank Adaptation (LoRA), formulated as $y = W_0x + BAx$, where $W_0$ is the pre-trained parameters and $x$ is the input to the adapted layer. While multi-adapter extensions often employ multiple LoRAs, prior studies suggest that the inner $A$ matrices are highly similar during