MemoSight: Unifying Context Compression and Multi Token Prediction for Reasoning Acceleration
AI 摘要
MemoSight通过压缩上下文和多token预测，加速LLM的CoT推理，降低内存占用。
主要贡献
- 提出MemoSight框架，统一上下文压缩和多token预测
- 采用特殊token和位置布局实现高效压缩和预测
- 实验证明MemoSight能有效降低内存占用和加速推理
方法论
MemoSight通过特殊token和对应的位置布局，同时进行上下文压缩和多token预测，优化KV cache。
原文摘要
While Chain-of-thought (CoT) reasoning enables LLMs to solve challenging reasoning problems, as KV cache grows linearly with the number of generated tokens, CoT reasoning faces scaling issues in terms of speed and memory usage. In this work, we propose MemoSight (Memory-Foresight-based reasoning), a unified framework that integrates both context compression and multi-token prediction to mitigate the efficiency issues while maintaining CoT reasoning performance. Our framework adopts the same minimalist design for both context compression and multi-token prediction via special tokens and their corresponding position layout tailored to each token type. Comprehensive experiments on four reasoning benchmarks demonstrate that MemoSight reduces the KV cache footprint by up to 66% and accelerates inference by 1.56x, while outperforming existing CoT compression methods.