Credit: VentureBeat made with Midjourney

A comprehensive new survey from Microsoft researchers and academic partners reveals that artificial intelligence agents powered by large language models (LLMs) are becoming increasingly capable of controlling graphical user interfaces (GUIs), potentially changing how humans interact with software.

The technology essentially gives AI systems the ability to see and manipulate computer interfaces just like humans do — clicking buttons, filling out forms, and navigating between applications. Rather than requiring users to learn complex software commands, these “GUI agents” can interpret natural language requests and automatically execute the necessary actions.

“These agents represent a paradigm shift, enabling users to perform intricate, multi-step tasks through simple conversational commands,” the researchers write. “Their applications span across web navigation, mobile app interactions, and desktop automation, offering a transformative user experience that revolutionizes how individuals interact with software.”

Think of it as having a highly skilled executive assistant who can operate any software program on your behalf. You simply tell the assistant what you want to accomplish, and they handle all the technical details of making it happen.