SmartBench: Evaluating LLMs in Smart Homes with Anomalous Device States and Behavioral Contexts

ArXi:2603.06636v1 Announce Type: new Due to the strong context-awareness capabilities nstrated by large language models (LLMs), recent research has begun exploring their integration into smart home assistants to help users manage and adjust their living environments. While LLMs have been shown to effectively understand user needs and provide appropriate responses, most existing studies primarily focus on interpreting and executing user behaviors or instructions. However, a critical function of smart home assistants is the ability to detect when the home environment is in an anomalous state.