AI RESEARCH
Short window attention enables long-term memorization
arXiv CS.LG
•
ArXi:2509.24552v3 Announce Type: replace Recent works show that hybrid architectures combining local sliding window attention layers and global attention layers outperform either of these architectures taken separately. However, the impact of the window length and the interplay between local layers and global layers remain under-studied. In this work, we first analyze the interaction between short and long term memory by considering SWAX: a hybrid architecture consisting of sliding-window attention and xLSTM linear RNN layers.