The standard SVD aims to minimize the reconstruction error, however, this objective does not always align with the objective of preserving task performance. In this paper, we propose a novel method called RankDyna, which conducts low-rank estimation throughout the fine-tuning process. Unlike prior approaches that statically assign importance to individual parameters, RankDyna dynamically captures the importance changes associated with singular groups. Experimental results show that RankDyna enables dynamical parameter allocation across layers by monitoring their importance variations, leading to improved compression outcomes. Meanwhile, RankDyna also significantly reduces the computational overhead compared to methods that require two rounds of fine-tuning.

https://aclanthology.org/2023.findings-emnlp.621.pdf