AI RESEARCH
Positional LSH: Binary Block Matrix Approximation for Attention with Linear Biases
arXiv CS.LG
•
ArXi:2605.09472v1 Announce Type: new Positional encoding in transformers is commonly implemented through positional embeddings, attention masks, or bias terms, but formal connections between these mechanisms remain limited. We study attention with positional bias through the lens of locality-sensitive hashing (LSH), focusing on Attention with Linear Biases (ALiBi). We show that the ALiBi bias matrix is the expectation of contiguous block-diagonal binary masks induced by a ``positional LSH'' scheme.