ORBIT: Preserving Foundational Language Capabilities in GenRetrieval via Origin-Regulated Merging

ArXi:2605.12419v1 Announce Type: cross Despite the rapid advancements in large language model (LLM) development, fine-tuning them for specific tasks often results in the catastrophic forgetting of their general, language-based reasoning abilities. This work investigates and addresses this challenge in the context of the Generative Retrieval (GenRetrieval) task. During GenRetrieval fine-tuning, we find this forgetting occurs rapidly and correlates with the distance between the fine-tuned and original model parameters.