Strategic Navigation or Stochastic Search? How Agents and Humans Reason Over Document Collections
AI 摘要
该论文提出了MADQA基准,用于评估多模态Agent在文档理解中的策略推理能力,揭示了现有Agent依赖暴力搜索的问题。
主要贡献
- 提出了MADQA基准数据集
- 设计了评估Agent推理能力的评估协议
- 揭示了现有Agent在文档理解任务中依赖暴力搜索的现象
方法论
构建包含人类编写问题的文档数据集,并设计新的评估协议,通过准确率-效率的权衡来评估Agent的推理能力。
原文摘要
Multimodal agents offer a promising path to automating complex document-intensive workflows. Yet, a critical question remains: do these agents demonstrate genuine strategic reasoning, or merely stochastic trial-and-error search? To address this, we introduce MADQA, a benchmark of 2,250 human-authored questions grounded in 800 heterogeneous PDF documents. Guided by Classical Test Theory, we design it to maximize discriminative power across varying levels of agentic abilities. To evaluate agentic behaviour, we introduce a novel evaluation protocol measuring the accuracy-effort trade-off. Using this framework, we show that while the best agents can match human searchers in raw accuracy, they succeed on largely different questions and rely on brute-force search to compensate for weak strategic planning. They fail to close the nearly 20% gap to oracle performance, persisting in unproductive loops. We release the dataset and evaluation harness to help facilitate the transition from brute-force retrieval to calibrated, efficient reasoning.