AI RESEARCH
Are GUI Agents Focused Enough? Automated Distraction via Semantic-level UI Element Injection
arXiv CS.CL
•
ArXi:2604.07831v1 Announce Type: cross Existing red-teaming studies on GUI agents have important limitations. Adversarial perturbations typically require white-box access, which is unavailable for commercial systems, while prompt injection is increasingly mitigated by stronger safety alignment. To study robustness under a practical threat model, we propose Semantic-level UI Element Injection, a red-teaming setting that overlays safety-aligned and harmless UI elements onto screenshots to misdirect the agent's visual grounding.