AI RESEARCH
ClickAIXR: On-Device Multimodal Vision-Language Interaction with Real-World Objects in Extended Reality
arXiv CS.CV
•
ArXi:2604.04905v1 Announce Type: new We present ClickAIXR, a novel on-device framework for multimodal vision-language interaction with objects in extended reality (XR). Unlike prior systems that rely on cloud-based AI (e.g., ChatGPT) or gaze-based selection (e.g., GazePointAR), ClickAIXR integrates an on-device vision-language model (VLM) with a controller-based object selection paradigm, enabling users to precisely click on real-world objects in XR. Once selected, the object image is processed locally by the VLM to answer natural language questions through both text and speech.