Dobi-SVD : Differentiable SVD for
LLM Compression
and Some New Perspectives

Qinsi Wang1*, Jinghan Ke2*, Masayoshi Tomizuka2, Yiran Chen1, Kurt Keutzer2, Chenfeng Xu2✝
1Duke University, 2University of California, Berkeley
* denotes equal contribution. Order is decided by coin flip.
✝ denotes corresponding author.
ICLR 2025
  Paper   Code (Coming Soon)
×

Introduction

Dobi-SVD is a novel large-model compression solution
on low-cost computation devices!

We provide a new LLM-compression solution via SVD, unlocking new possibilities for LLM compression beyond quantization and pruning. We point out that the optimal use of SVD lies in truncating activations, rather than merely using activations as an optimization distance. Building on this principle, we address three critical challenges in SVD-based LLM compression: including (1) How can we determine the optimal activation truncation position for each weight matrix in LLMs? (2) How can we efficiently reconstruct the weight matrices based on truncated activations? (3) How can we address the inherent "injection" nature that results in the information loss of the SVD? We propose Dobi-SVD, which establishes a new, principled approach to SVD-based LLM compression! Get ready for a cool website presentation : )

NOTE: click on the icons      in the page to get more information.



Layoffs
click
Layoffs
click


Overview

Overview framework of Dobi-SVD
click
Overview framework of Dobi-SVD: 1-3: Differentiable Truncation Position Training. By applying parameter renormalization for continuous rank ratio selection and using Taylor expansion to prevent gradient explosion, our method enables robust and adaptive optimization of truncation positions. 4: Weight Update. Using IPCA, we sequentially extract and optimally update weight matrix features. 5: Remapping. We resolve a long-overlooked limitation of traditional SVD-based compression through remapping, fully unlocking SVD's potential for data compression.


Overview framework of Dobi-SVD
click



Demo :   LLM   &   VLM
click

LLM

Model
LLaMa-2-7B

Device
NVIDIA TITAN Xp 12G

Compression Ratio
0.4

Task
Language question-answering

@o_l_bom



BibTeX

@misc{wang2025dobisvddifferentiablesvdllm,
                title={Dobi-SVD: Differentiable SVD for LLM Compression and Some New Perspectives}, 
                author={Qinsi Wang and Jinghan Ke and Masayoshi Tomizuka and Yiran Chen and Kurt Keutzer and Chenfeng Xu},
                year={2025},
                eprint={2502.02723},
                archivePrefix={arXiv},
                primaryClass={cs.LG},
                url={https://arxiv.org/abs/2502.02723}, 
          }