Optimizing low-rank adaptation with decomposed matrices and adaptive rank allocation

doi:10.1007/s11704-024-40317-w

Front. Comput. Sci.

2025, Vol. 19

Issue (5) : 195337 https://doi.org/10.1007/s11704-024-40317-w

Artificial Intelligence

Optimizing low-rank adaptation with decomposed matrices and adaptive rank allocation

Dacao ZHANG¹, Fan YANG¹, Kun ZHANG¹(

), Xin LI², Si WEI², Richang HONG¹, Meng WANG¹

¹. School of Computer Science and Information Engineering, Hefei University of Technology, Hefei 230601, China
². Artificial Intelligence Research Institute, iFLYTEK Company Ltd., Hefei 230088, China

Download: PDF(382 KB) HTML
Export: BibTeX | EndNote | Reference Manager | ProCite | RefWorks

Corresponding Author(s): Kun ZHANG

Just Accepted Date: 11 September 2024 Issue Date: 15 October 2024

Cite this article:

Dacao ZHANG,Fan YANG,Kun ZHANG, et al. Optimizing low-rank adaptation with decomposed matrices and adaptive rank allocation[J]. Front. Comput. Sci., 2025, 19(5): 195337.

URL:

https://academic.hep.com.cn/fcs/EN/10.1007/s11704-024-40317-w
https://academic.hep.com.cn/fcs/EN/Y2025/V19/I5/195337

Fig.1 The overall diagram of our proposed method. (a) The structure of matrix decomposition method; (b) the diagram of the task-specific rank allocation method

Model & method	Params/per task	MNLI	SST-2	MRPC	CoLA	QNLI	QQP	RTE	STS-B	AVG
Single-task training
$R o B l a r g e (L o R A r = 8)$	3M	89.71	95.87	90.67	66.52	94.67	91.28	84.48	91.62	88.10
$R o B l a r g e (O u r s)$	3.01M	90.36	95.99	90.93	70.02	94.53	91.30	85.56	92.32	88.87
Multi-task training
$R o B b a s e (L o R A r = 8)$	0.24M	84.39	93.46	88.48	63.16	90.68	87.12	75.81	?	83.30
$R o B b a s e (L o R A r = 16)$	0.48M	84.79	94.50	88.97	60.32	91.12	88.12	75.81	?	83.38
$R o B b a s e (O u r s)$	0.48M	85.40	93.92	88.73	64.18	91.27	88.33	76.17	?	84.00

Tab.1 The results of our methods on GLUE tasks

Tab.2 The results of our decomposition method compared with DyLoRA [5] and AdaLoRA [4]

Fig.2 Visualization of the last layer task embeddings

1	Wang A, Singh A, Michael J, Hill F, Levy O, Bowman S. GLUE: a multi-task benchmark and analysis platform for natural language understanding. In: Proceedings of 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP. 2018, 353−355
2	L, Chen L, Wu K, Zhang R, Hong D, Lian Z, Zhang J, Zhou M Wang . Improving recommendation fairness via data augmentation. In: Proceedings of the ACM Web Conference 2023. 2023, 1012−1020
3	Hu E J, Shen Y, Wallis P, Allen-Zhu Z, Li Y, Wang S, Wang L, Chen W. LoRA: low-rank adaptation of large language models. In: Proceedings of the 10th International Conference on Learning Representations. 2021, 1−26
4	Zhang Q, Chen M, Bukharin A, He P, Cheng Y, Chen W, Zhao T. Adaptive budget allocation for parameter-efficient fine-tuning. In: Proceedings of the 11th International Conference on Learning Representations. 2023, 1−17
5	M, Valipour M, Rezagholizadeh I, Kobyzev A Ghodsi . DyLoRA: parameter-efficient tuning of pre-trained models using dynamic search-free low-rank adaptation. In: Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics. 2023, 3274−3287
6	Y, Wang Y, Lin X, Zeng G Zhang . MultiLoRA: democratizing LoRA for better multi-task learning. 2023, arXiv preprint arXiv: 2311.11501

[1]

Download

Viewed

Full text

Abstract

Cited

Shared

Discussed