Multimodal Learning 相关度: 9/10

MMEmb-R1: Reasoning-Enhanced Multimodal Embedding with Pair-Aware Selection and Adaptive Control

Yuchi Wang, Haiyang Yu, Weikang Bian, Jiefeng Long, Xiao Liang, Chao Feng, Hongsheng Li
arXiv: 2604.06156v1 发布: 2026-04-07 更新: 2026-04-07

AI 摘要

提出MMEmb-R1,通过自适应推理提升多模态嵌入效果,降低推理开销并优化延迟。

主要贡献

  • 提出pair-aware推理选择,通过反事实干预选择有效推理路径
  • 引入强化学习,选择性地调用推理,减少计算开销
  • 在MMEB-V2基准上取得了新的state-of-the-art

方法论

将推理视为潜在变量,利用强化学习自适应选择是否进行推理,提升多模态嵌入。

原文摘要

MLLMs have been successfully applied to multimodal embedding tasks, yet their generative reasoning capabilities remain underutilized. Directly incorporating chain-of-thought reasoning into embedding learning introduces two fundamental challenges. First, structural misalignment between instance-level reasoning and pairwise contrastive supervision may lead to shortcut behavior, where the model merely learns the superficial format of reasoning. Second, reasoning is not universally beneficial for embedding tasks. Enforcing reasoning for all inputs may introduce unnecessary computation and latency, and can even obscure salient semantic signals for simple cases. To address these issues, we propose MMEmb-R1, an adaptive reasoning-based multimodal embedding framework. We formulate reasoning as a latent variable and introduce pair-aware reasoning selection that employs counterfactual intervention to identify reasoning paths beneficial for query-target alignment. Furthermore, we adopt reinforcement learning to selectively invoke reasoning only when necessary. Experiments on the MMEB-V2 benchmark demonstrate that our model achieves a score of 71.2 with only 4B parameters, establishing a new state-of-the-art while significantly reducing reasoning overhead and inference latency.

标签

多模态学习 推理 嵌入 强化学习

arXiv 分类

cs.CV cs.AI cs.CL