Optimal Brain Decomposition for Accurate LLM Low-Rank Approximation
AI 摘要
提出OBD-LLM，利用二阶 Hessian 信息进行LLM的低秩分解，显著提升分解效果。
主要贡献
- 提出基于二阶 Hessian 信息的 OBD-LLM 分解方法
- 理论证明了分解需要考虑输入和输出信息
- 实现了比 SVD-LLM 更好的低秩分解效果
方法论
通过 Hessian 矩阵的 Kronecker 分解，推导出考虑输入和输出信息的双向白化权重矩阵分解方案，得到闭式解。
原文摘要
Low-rank decomposition has emerged as an important problem in Large Language Model (LLM) fine-tuning and inference. Through Singular Value Decomposition (SVD), the weight matrix can be factorized into low-rank spaces optimally. Previously, a common practice was to decompose the weight in the activation-whitened space, and then achieve satisfying results. In this work, we propose Optimal Brain Decomposition LLM (OBD-LLM), which studies the decomposition problem in the model space by utilizing second-order Hessian information. Through a rigorous Kronecker-factorization of the Hessian, we show that the decomposition needs to consider both input and output information of the layer, and achieves much better decomposition results compared to input only method. Our loss-aware decomposition method involves a bi-directional whitening on the weight matrix. As a result, OBD-LLM is a closed-form solution for the optimal decomposition of weights in the language model. Remarkably, we achieve ~20-40\% better results than previous state-of-the-art decomposition methods, the SVD-LLM.