AI RESEARCH
Understanding Catastrophic Forgetting In LoRA via Mean-Field Attention Dynamics
arXiv CS.LG
•
ArXi:2402.15415v2 Announce Type: replace Low-Rank Adaptation (LoRA) is the dominant parameter-efficient fine-tuning method due to its favorable compute-performance trade-off, yet it suffers from catastrophic forgetting. We study forgetting through a tractable _mean-field self-attention_ toy model, where tokens evolve as an interacting particle system and LoRA acts as a low-rank perturbation. Using tools from partial differential equations and dynamical systems, we characterize regimes suggesting a phase transition between forgetting and non-forgetting behavior.