Multimodal Learning 相关度: 8/10

BrainVista: Modeling Naturalistic Brain Dynamics as Multimodal Next-Token Prediction

Xuanhua Yin, Runkai Zhao, Lina Yao, Weidong Cai
arXiv: 2602.04512v1 发布: 2026-02-04 更新: 2026-02-04

AI 摘要

BrainVista通过多模态自回归框架模拟自然状态下大脑的动态预测,实现先进的fMRI编码。

主要贡献

  • 提出BrainVista多模态自回归框架
  • 引入Network-wise Tokenizers和Spatial Mixer Head
  • 提出Stimulus-to-Brain (S2B)掩码机制

方法论

构建多模态自回归模型,解耦系统动态,捕捉网络间信息流,同步感觉刺激与血流动力学信号。

原文摘要

Naturalistic fMRI characterizes the brain as a dynamic predictive engine driven by continuous sensory streams. However, modeling the causal forward evolution in realistic neural simulation is impeded by the timescale mismatch between multimodal inputs and the complex topology of cortical networks. To address these challenges, we introduce BrainVista, a multimodal autoregressive framework designed to model the causal evolution of brain states. BrainVista incorporates Network-wise Tokenizers to disentangle system-specific dynamics and a Spatial Mixer Head that captures inter-network information flow without compromising functional boundaries. Furthermore, we propose a novel Stimulus-to-Brain (S2B) masking mechanism to synchronize high-frequency sensory stimuli with hemodynamically filtered signals, enabling strict, history-only causal conditioning. We validate our framework on Algonauts 2025, CineBrain, and HAD, achieving state-of-the-art fMRI encoding performance. In long-horizon rollout settings, our model yields substantial improvements over baselines, increasing pattern correlation by 36.0\% and 33.3\% on relative to the strongest baseline Algonauts 2025 and CineBrain, respectively.

标签

fMRI multimodal learning autoregressive model brain dynamics

arXiv 分类

q-bio.NC cs.AI