Multimodal Learning 相关度: 9/10

Evaluation Before Generation: A Paradigm for Robust Multimodal Sentiment Analysis with Missing Modalities

Rongfei Chen, Tingting Zhang, Xiaoyu Shen, Wei Zhang
arXiv: 2604.05558v1 发布: 2026-04-07 更新: 2026-04-07

AI 摘要

针对多模态情感分析中模态缺失问题,提出基于Prompt的缺失模态自适应框架,提升模型鲁棒性。

主要贡献

  • 提出缺失模态评估器,避免低质量数据插补
  • 设计模态不变Prompt解耦模块,捕捉局部相关性
  • 引入动态Prompt权重模块,抑制缺失模态的干扰
  • 构建多层Prompt动态连接模块,增强全局一致性

方法论

通过Prompt学习,结合预训练模型和伪标签,动态评估缺失模态重要性,并解耦Prompt,动态加权,最终连接全局Prompt。

原文摘要

The missing modality problem poses a fundamental challenge in multimodal sentiment analysis, significantly degrading model accuracy and generalization in real world scenarios. Existing approaches primarily improve robustness through prompt learning and pre trained models. However, two limitations remain. First, the necessity of generating missing modalities lacks rigorous evaluation. Second, the structural dependencies among multimodal prompts and their global coherence are insufficiently explored. To address these issues, a Prompt based Missing Modality Adaptation framework is proposed. A Missing Modality Evaluator is introduced at the input stage to dynamically assess the importance of missing modalities using pretrained models and pseudo labels, thereby avoiding low quality data imputation. Building on this, a Modality invariant Prompt Disentanglement module decomposes shared prompts into modality specific private prompts to capture intrinsic local correlations and improve representation quality. In addition, a Dynamic Prompt Weighting module computes mutual information based weights from cross attention outputs to adaptively suppress interference from missing modalities. To enhance global consistency, a Multi level Prompt Dynamic Connection module integrates shared prompts with self attention outputs through residual connections, leveraging global prompt priors to strengthen key guidance features. Extensive experiments on three public benchmarks, including CMU MOSI, CMU MOSEI, and CH SIMS, demonstrate that the proposed framework achieves state of the art performance and stable results under diverse missing modality settings. The implementation is available at https://github.com/rongfei-chen/ProMMA

标签

多模态情感分析 模态缺失 Prompt学习 鲁棒性

arXiv 分类

cs.CV