AI RESEARCH
Low-Rank Adapters Initialization via Gradient Surgery for Continual Learning
arXiv CS.LG
•
ArXi:2605.12752v1 Announce Type: new LoRA is widely adopted for continual fine-tuning of Large Language Models due to its parameter efficiency, modularity across tasks, and compatibility with replay strategies. However, LoRA-based continual learning remains vulnerable to catastrophic forgetting, whose severity depends on how successive task gradients interact: when consecutive task gradients conflict, standard adapter initializations channel updates into subspaces that overwrite previously learned directions.