Multimodal Learning 相关度: 9/10

COMPASS: Complete Multimodal Fusion via Proxy Tokens and Shared Spaces for Ubiquitous Sensing

Hao Wang, Yanyu Qian, Pengcheng Weng, Zixuan Xia, William Dan, Yangxin Xu, Fei Wang
arXiv: 2604.02056v1 发布: 2026-04-02 更新: 2026-04-02

AI 摘要

COMPASS提出了一种基于代理令牌和共享空间的多模态融合框架,有效解决了模态缺失问题。

主要贡献

  • 提出了一种基于代理令牌的模态缺失融合框架COMPASS
  • 使用pairwise源到目标的生成器在共享潜在空间中合成代理令牌
  • 结合代理对齐、共享空间正则化和per-proxy判别监督,提高代理令牌的质量

方法论

COMPASS为每个缺失模态合成代理令牌,确保融合头接收固定输入的结构,并通过多种方式提升代理令牌的质量。

原文摘要

Missing modalities remain a major challenge for multimodal sensing, because most existing methods adapt the fusion process to the observed subset by dropping absent branches, using subset-specific fusion, or reconstructing missing features. As a result, the fusion head often receives an input structure different from the one seen during training, leading to incomplete fusion and degraded cross-modal interaction. We propose COMPASS, a missing-modality fusion framework built on the principle of fusion completeness: the fusion head always receives a fixed N-slot multimodal input, with one token per modality slot. For each missing modality, COMPASS synthesizes a target-specific proxy token from the observed modalities using pairwise source-to-target generators in a shared latent space, and aggregates them into a single replacement token. To make these proxies both representation-compatible and task-informative, we combine proxy alignment, shared-space regularization, and per-proxy discriminative supervision. Experiments on XRF55, MM-Fi, and OctoNet under diverse single- and multiple-missing settings show that COMPASS outperforms prior methods on the large majority of scenarios. Our results suggest that preserving a modality-complete fusion interface is a simple and effective design principle for robust multimodal sensing.

标签

多模态学习 模态缺失 代理令牌 融合

arXiv 分类

cs.CV