AI RESEARCH
Beyond Sequential Distance: Inter-Modal Distance Invariant Position Encoding
arXiv CS.CV
•
ArXi:2603.10863v1 Announce Type: new Despite the remarkable capabilities of Multimodal Large Language Models (MLLMs), they still suffer from visual fading in long-context scenarios. Specifically, the attention to visual tokens diminishes as the text sequence lengthens, leading to text generation detached from visual constraints. We attribute this degradation to the inherent inductive bias of Multimodal RoPE, which penalizes inter-modal attention as the distance between visual and text tokens increases.