AI RESEARCH
LOOPE: Learnable Optimal Patch Order in Positional Embeddings for Vision Transformers
arXiv CS.AI
•
ArXi:2504.14386v2 Announce Type: replace-cross Positional embeddings (PE) play a crucial role in Vision Transformers (ViTs) by providing spatial information otherwise lost due to the permutation invariant nature of self attention. While absolute positional embeddings (APE) have shown theoretical advantages over relative positional embeddings (RPE), particularly due to the ability of sinusoidal functions to preserve spatial inductive biases like monotonicity and shift invariance, a fundamental challenge arises when mapping a 2D grid to a 1D sequence.