AI RESEARCH
See, Symbolize, Act: Grounding VLMs with Spatial Representations for Better Gameplay
arXiv CS.AI
•
ArXi:2603.11601v1 Announce Type: new Vision-Language Models (VLMs) excel at describing visual scenes, yet struggle to translate perception into precise, grounded actions. We investigate whether providing VLMs with both the visual frame and the symbolic representation of the scene can improve their performance in interactive environments. We evaluate three state-of-the-art VLMs across Atari games, VizDoom, and AI2-THOR, comparing frame-only, frame with self-extracted symbols, frame with ground-truth symbols, and symbol-only pipelines.