Multimodal Learning 相关度: 9/10

Generate, Analyze, and Refine: Training-Free Sound Source Localization via MLLM Meta-Reasoning

Subin Park, Jung Uk Kim
arXiv: 2604.06824v1 发布: 2026-04-08 更新: 2026-04-08

AI 摘要

提出了一种基于MLLM的训练自由的声音源定位框架GAR,通过生成、分析和精炼实现精确声音定位。

主要贡献

  • 提出了一种新的基于MLLM的训练自由的声音源定位框架GAR
  • 利用MLLM的推理能力进行音频-视觉一致性分析
  • 引入自适应门控机制防止不必要的调整

方法论

GAR流程包括生成初始边界框和音频分类,分析音频-视觉一致性,以及通过自适应门控进行精炼。

原文摘要

Sound source localization task aims to identify the locations of sound-emitting objects by leveraging correlations between audio and visual modalities. Most existing SSL methods rely on contrastive learning-based feature matching, but lack explicit reasoning and verification, limiting their effectiveness in complex acoustic scenes. Inspired by human meta-cognitive processes, we propose a training-free SSL framework that exploits the intrinsic reasoning capabilities of Multimodal Large Language Models (MLLMs). Our Generation-Analysis-Refinement (GAR) pipeline consists of three stages: Generation produces initial bounding boxes and audio classifications; Analysis quantifies Audio-Visual Consistency via open-set role tagging and anchor voting; and Refinement applies adaptive gating to prevent unnecessary adjustments. Extensive experiments on single-source and multi-source benchmarks demonstrate competitive performance. The source code is available at https://github.com/VisualAIKHU/GAR-SSL.

标签

MLLM Sound Source Localization Multimodal Learning Reasoning

arXiv 分类

cs.CV