Saccade Attention Networks: Using Transfer Learning of Attention to Reduce Network Sizes

ArXi:2604.16485v1 Announce Type: cross One of the limitations of transformer networks is the sequence length due to the quadratic nature of the attention matrix. Classical self attention uses the entire sequence length, however, the actual attention being used is sparse. Humans use a form of sparse attention when analyzing an image or scene called saccades. Focusing on key features greatly reduces computation time.