Towards Fine-grained Temporal Perception: Post-Training Large Audio-Language Models with Audio-Side Time Prompt
AI 摘要
论文提出Audio-Side Time Prompt和TimePro-RL框架,提升LALM在音频时间感知任务中的性能。
主要贡献
- 提出Audio-Side Time Prompt
- 构建TimePro-RL框架,结合SFT和RL
- 实验验证了TimePro-RL在多个音频时间任务上的有效性
方法论
将时间戳编码为嵌入,插入音频特征序列,用以提示模型,再使用RL优化时间对齐性能。
原文摘要
Large Audio-Language Models (LALMs) enable general audio understanding and demonstrate remarkable performance across various audio tasks. However, these models still face challenges in temporal perception (e.g., inferring event onset and offset), leading to limited utility in fine-grained scenarios. To address this issue, we propose Audio-Side Time Prompt and leverage Reinforcement Learning (RL) to develop the TimePro-RL framework for fine-grained temporal perception. Specifically, we encode timestamps as embeddings and interleave them within the audio feature sequence as temporal coordinates to prompt the model. Furthermore, we introduce RL following Supervised Fine-Tuning (SFT) to directly optimize temporal alignment performance. Experiments demonstrate that TimePro-RL achieves significant performance gains across a range of audio temporal tasks, such as audio grounding, sound event detection, and dense audio captioning, validating its robust effectiveness.