LLM Reasoning 相关度: 9/10

LUDOBENCH: Evaluating LLM Behavioural Decision-Making Through Spot-Based Board Game Scenarios in Ludo

Ojas Jain, Dhruv Kumar
arXiv: 2604.05681v1 发布: 2026-04-07 更新: 2026-04-07

AI 摘要

LudoBench通过飞行棋场景评估LLM在不确定性下的战略推理能力。

主要贡献

  • 构建了LudoBench飞行棋基准数据集
  • 开发了四种类型的飞行棋AI智能体
  • 揭示了LLM在策略选择上的缺陷与prompt敏感性

方法论

手工设计飞行棋场景,通过对比LLM与不同AI智能体的决策,分析其策略选择和推理能力。

原文摘要

We introduce LudoBench, a benchmark for evaluating LLM strategic reasoning in Ludo, a stochastic multi-agent board game whose dice mechanics, piece capture, safe-square navigation, and home-path progression introduce meaningful planning complexity. LudoBench comprises 480 handcrafted spot scenarios across 12 behaviorally distinct decision categories, each isolating a specific strategic choice. We additionally contribute a fully functional 4-player Ludo simulator supporting Random, Heuristic, Game-Theory, and LLM agents. The game-theory agent uses Expectiminimax search with depth-limited lookahead to provide a principled strategic ceiling beyond greedy heuristics. Evaluating six models spanning four model families, we find that all models agree with the game-theory baseline only 40-46% of the time. Models split into distinct behavioral archetypes: finishers that complete pieces but neglect development, and builders that develop but never finish. Each archetype captures only half of the game theory strategy. Models also display measurable behavioral shifts under history-conditioned grudge framing on identical board states, revealing prompt-sensitivity as a key vulnerability. LudoBench provides a lightweight and interpretable framework for benchmarking LLM strategic reasoning under uncertainty. All code, the spot dataset (480 entries) and model outputs are available at https://anonymous.4open.science/r/LudoBench-5CBF/

标签

LLM 战略推理 棋类游戏 基准测试

arXiv 分类

cs.AI cs.CL cs.GT cs.LG cs.MA