Multi-modal, multi-scale representation learning for satellite imagery analysis just needs a good ALiBi

ArXi:2604.10347v1 Announce Type: new Vision foundation models have been shown to be effective at processing satellite imagery into representations fit for downstream tasks, however, creating models which operate over multiple spatial resolutions and modes is challenging. This paper presents Scale-ALiBi, a linear bias transformer attention mechanism with a spatial encoding bias to relationships between image patches at different ground sample distance scales.