Multimodal Learning 相关度: 9/10

Enhance-then-Balance Modality Collaboration for Robust Multimodal Sentiment Analysis

Kang He, Yuzhe Ding, Xinrong Wang, Fei Li, Chong Teng, Donghong Ji
arXiv: 2604.12518v1 发布: 2026-04-14 更新: 2026-04-14

AI 摘要

EBMC模型通过增强弱模态、平衡模态贡献和信任度蒸馏,提升多模态情感分析的性能和鲁棒性。

主要贡献

  • 提出Enhance-then-Balance Modality Collaboration (EBMC)框架
  • 设计语义解耦和跨模态增强策略以增强弱模态
  • 引入能量引导的模态协调机制,实现隐式梯度重平衡

方法论

EBMC通过增强弱模态表示、能量引导平衡模态贡献、实例感知模态信任度蒸馏来实现更鲁棒的多模态情感分析。

原文摘要

Multimodal sentiment analysis (MSA) integrates heterogeneous text, audio, and visual signals to infer human emotions. While recent approaches leverage cross-modal complementarity, they often struggle to fully utilize weaker modalities. In practice, dominant modalities tend to overshadow non-verbal ones, inducing modality competition and limiting overall contributions. This imbalance degrades fusion performance and robustness under noisy or missing modalities. To address this, we propose a novel model, Enhance-then-Balance Modality Collaboration framework (EBMC). EBMC improves representation quality via semantic disentanglement and cross-modal enhancement, strengthening weaker modalities. To prevent dominant modalities from overwhelming others, an Energy-guided Modality Coordination mechanism achieves implicit gradient rebalancing via a differentiable equilibrium objective. Furthermore, Instance-aware Modality Trust Distillation estimates sample-level reliability to adaptively modulate fusion weights, ensuring robustness. Extensive experiments demonstrate that EBMC achieves state-of-the-art or competitive results and maintains strong performance under missing-modality settings.

标签

多模态情感分析 模态增强 模态平衡 鲁棒性

arXiv 分类

cs.CL