FINER: MLLMs Hallucinate under Fine-grained Negative Queries
AI 摘要
针对MLLM在细粒度负查询下产生幻觉的问题，提出了FINER基准和FINER-Tuning方法。
主要贡献
- 提出了FINER基准，用于评估MLLM在细粒度负查询下的幻觉问题
- 分析了MLLM在多种场景下的幻觉现象
- 提出了FINER-Tuning方法，通过DPO进行微调，有效缓解了幻觉问题
方法论
通过构建细粒度负查询数据集FINER，分析MLLM幻觉问题，并使用DPO在FINER启发的数据上进行微调（FINER-Tuning）。
原文摘要
Multimodal large language models (MLLMs) struggle with hallucinations, particularly with fine-grained queries, a challenge underrepresented by existing benchmarks that focus on coarse image-related questions. We introduce FIne-grained NEgative queRies (FINER), alongside two benchmarks: FINER-CompreCap and FINER-DOCCI. Using FINER, we analyze hallucinations across four settings: multi-object, multi-attribute, multi-relation, and ``what'' questions. Our benchmarks reveal that MLLMs hallucinate when fine-grained mismatches co-occur with genuinely present elements in the image. To address this, we propose FINER-Tuning, leveraging Direct Preference Optimization (DPO) on FINER-inspired data. Finetuning four frontier MLLMs with FINER-Tuning yields up to 24.2\% gains (InternVL3.5-14B) on hallucinations from our benchmarks, while simultaneously improving performance on eight existing hallucination suites, and enhancing general multimodal capabilities across six benchmarks. Code, benchmark, and models are available at \href{https://explainableml.github.io/finer-project/}{https://explainableml.github.io/finer-project/}.