AI RESEARCH
LoRA vs. Full Fine-Tuning: A Theoretical Perspective
arXiv CS.LG
•
ArXi:2605.19018v1 Announce Type: new Fine-tuning adapts a pre-trained model to downstream tasks using a small amount of labeled data. Low-Rank Adaptation (LoRA) is an efficient fine-tuning method that reduces memory and computation costs while often achieving performance close to full fine-tuning. Despite its widespread use, the theoretical behavior of LoRA is not yet well understood. In this paper, we study LoRA in a simple linear regression setting and compare its excess risk with that of full fine-tuning.