AI RESEARCH
Do VLMs Need Vision Transformers? Evaluating State Space Models as Vision Encoders
arXiv CS.LG
•
ArXi:2603.19209v1 Announce Type: cross Large vision--language models (VLMs) often use a frozen vision backbone, whose image features are mapped into a large language model through a lightweight connector. While transformer-based encoders are the standard visual backbone, we ask whether state space model (SSM) vision backbones can be a strong alternative. We systematically evaluate SSM vision backbones for VLMs in a controlled setting. Under matched ImageNet-1K initialization, the SSM backbone achieves the strongest overall performance across both VQA and grounding/localization.