HATS: Hardness-Aware Trajectory Synthesis for GUI Agents
AI 摘要
HATS提出一种硬度感知轨迹合成框架,提升GUI智能体在语义模糊场景下的泛化能力。
主要贡献
- 提出硬度感知的轨迹合成框架HATS
- 设计硬度驱动的探索模块,寻找有信息量的交互
- 设计对齐引导的优化模块,校验和修复指令-执行对齐
方法论
通过硬度驱动探索和对齐引导优化两个模块的闭环反馈,提升智能体对语义模糊动作的理解和执行能力。
原文摘要
Graphical user interface (GUI) agents powered by large vision-language models (VLMs) have shown remarkable potential in automating digital tasks, highlighting the need for high-quality trajectory data to support effective agent training. Yet existing trajectory synthesis pipelines often yield agents that fail to generalize beyond simple interactions. We identify this limitation as stemming from the neglect of semantically ambiguous actions, whose meanings are context-dependent, sequentially dependent, or visually ambiguous. Such actions are crucial for real-world robustness but are under-represented and poorly processed in current datasets, leading to semantic misalignment between task instructions and execution. To address these issues, we propose HATS, a Hardness-Aware Trajectory Synthesis framework designed to mitigate the impact of semantic ambiguity. We define hardness as the degree of semantic ambiguity associated with an action and develop two complementary modules: (1) hardness-driven exploration, which guides data collection toward ambiguous yet informative interactions, and (2) alignment-guided refinement, which iteratively validates and repairs instruction-execution alignment. The two modules operate in a closed loop: exploration supplies refinement with challenging trajectories, while refinement feedback updates the hardness signal to guide future exploration. Extensive experiments show that agents trained with HATS consistently outperform state-of-the-art baselines across benchmark GUI environments.