A survey on LoRA of large language models

doi:10.1007/s11704-024-40663-9

Front. Comput. Sci.

2025, Vol. 19

Issue (7) : 197605 https://doi.org/10.1007/s11704-024-40663-9

Information Systems

A survey on LoRA of large language models

Yuren MAO^1,², Yuhang GE¹, Yijiang FAN¹, Wenyi XU¹, Yu MI¹, Zhonghao HU¹, Yunjun GAO^1,²(

)

¹. School of Software Technology, Zhejiang University, Ningbo 315000, China
². College of Computer Science and Technology, Zhejiang University, Hangzhou 310007, China

Download: PDF(2136 KB) HTML
Export: BibTeX | EndNote | Reference Manager | ProCite | RefWorks

Abstract

Low-Rank Adaptation (LoRA), which updates the dense neural network layers with pluggable low-rank matrices, is one of the best performed parameter efficient fine-tuning paradigms. Furthermore, it has significant advantages in cross-task generalization and privacy-preserving. Hence, LoRA has gained much attention recently, and the number of related literature demonstrates exponential growth. It is necessary to conduct a comprehensive overview of the current progress on LoRA. This survey categorizes and reviews the progress from the perspectives of (1) downstream adaptation improving variants that improve LoRA’s performance on downstream tasks; (2) cross-task generalization methods that mix multiple LoRA plugins to achieve cross-task generalization; (3) efficiency-improving methods that boost the computation-efficiency of LoRA; (4) data privacy-preserving methods that use LoRA in federated learning; (5) application. Besides, this survey also discusses the future directions in this field.

Keywords low-rank adaptation LoRA large language models LLMs

Corresponding Author(s): Yunjun GAO

Just Accepted Date: 14 August 2024 Issue Date: 16 October 2024

Cite this article:

Yuren MAO,Yuhang GE,Yijiang FAN, et al. A survey on LoRA of large language models[J]. Front. Comput. Sci., 2025, 19(7): 197605.

URL:

https://academic.hep.com.cn/fcs/EN/10.1007/s11704-024-40663-9
https://academic.hep.com.cn/fcs/EN/Y2025/V19/I7/197605

Fig.1 The taxonomy of this paper

Fig.2 An illustration of full fine-tuning (a), LoRA (b) and its variants for improving downstream adaptation, which includes breaking the low-rank bottleneck (c) and dynamic rank allocation (d)

Tab.1 Performance of LoRA and its variants for RoBERTa-base model on the GLUE benchmark. We report Matthew’s correlation for CoLA, Pearson correlation for STS-B, and accuracy for the other datasets. The results are reported according to the results reported in literature [9,32,45,89,90]

Fig.3 An illustration of LoRA mixture methods

Fig.4 An illustration of efficiency improving methods

Fig.5 An illustration of LoRA for federated learning

1	J, Devlin M W, Chang K, Lee K Toutanova . BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT. 2019, 4171−4186
2	A, Chowdhery S, Narang J, Devlin M, Bosma G, Mishra A, Roberts P, Barham H W, Chung C, Sutton S, Gehrmann P, Schuh K, Shi S, Tsvyashchenko J, Maynez A, Rao P, Barnes Y, Tay N, Shazeer V, Prabhakaran E, Reif N, Du B, Hutchinson R, Pope J, Bradbury J, Austin M, Isard G, Gur-Ari P, Yin T, Duke A, Levskaya S, Ghemawat S, Dev H, Michalewski X, Garcia V, Misra K, Robinson L, Fedus D, Zhou D, Ippolito D, Luan H, Lim B, Zoph A, Spiridonov R, Sepassi D, Dohan S, Agrawal M, Omernick A M, Dai T S, Pillai M, Pellat A, Lewkowycz E, Moreira R, Child O, Polozov K, Lee Z, Zhou X, Wang B, Saeta M, Diaz O, Firat M, Catasta J, Wei K, Meier-Hellstern D, Eck J, Dean S, Petrov N Fiedel . PaLM: scaling language modeling with pathways. The Journal of Machine Learning Research, 2023, 24( 1): 240
3	Y, Chen S, Qian H, Tang X, Lai Z, Liu S, Han J Jia . LongLoRA: efficient fine-tuning of long-context large language models. In: Proceedings of the 12th International Conference on Learning Representations. 2024
4	R, Pan X, Liu S, Diao R, Pi J, Zhang C, Han T Zhang . LISA: layerwise importance sampling for memory-efficient large language model fine-tuning. 2024, arXiv preprint arXiv: 2403.17919
5	N, Ding Y, Qin G, Yang F, Wei Z, Yang Y, Su S, Hu Y, Chen C M, Chan W, Chen J, Yi W, Zhao X, Wang Z, Liu H T, Zheng J, Chen Y, Liu J, Tang J, Li M Sun . Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, 2023, 5( 3): 220–235
6	N, Houlsby A, Giurgiu S, Jastrzebski B, Morrone Laroussilhe Q, de A, Gesmundo M, Attariyan S Gelly . Parameter-efficient transfer learning for NLP. In: Proceedings of the 36th International Conference on Machine Learning. 2019, 2790−2799
7	Lester B, Al-Rfou R, Constant N. The power of scale for parameter-efficient prompt tuning. In: Proceedings of 2021 Conference on Empirical Methods in Natural Language Processing. 2021, 3045−3059
8	E B, Zaken Y, Goldberg S Ravfogel . BitFit: simple parameter-efficient fine-tuning for transformer-based masked language-models. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 2022, 1−9
9	E J, Hu Y, Shen P, Wallis Z, Allen-Zhu Y, Li S, Wang L, Wang W Chen . LoRA: low-rank adaptation of large language models. In: Proceedings of the 10th International Conference on Learning Representations. 2022
10	W X, Zhao K, Zhou J, Li T, Tang X, Wang Y, Hou Y, Min B, Zhang J, Zhang Z, Dong Y, Du C, Yang Y, Chen Z, Chen J, Jiang R, Ren Y, Li X, Tang Z, Liu P, Liu J Y, Nie J R Wen . A survey of large language models. 2023, arXiv preprint arXiv: 2303.18223
11	Z, Han C, Gao J, Liu J, Zhang S Q Zhang . Parameter-efficient fine-tuning for large models: a comprehensive survey. 2024, arXiv preprint arXiv: 2403.14608
12	S, Malladi A, Wettig D, Yu D, Chen S Arora . A kernel-based view of language model fine-tuning. In: Proceedings of the 40th International Conference on Machine Learning. 2023, 23610−23641
13	H, Koubbi M, Boussard L Hernandez . The impact of LoRA on the emergence of clusters in transformers. 2024, arXiv preprint arXiv: 2402.15415
14	U, Jang J D, Lee E K Ryu . LoRA training in the NTK regime has no spurious local minima. 2024, arXiv preprint arXiv: 2402.11867
15	J, Zhu K, Greenewald K, Nadjahi Ocáriz Borde H S, de R B, Gabrielsson L, Choshen M, Ghassemi M, Yurochkin J Solomon . Asymmetry in low-rank adapters of foundation models. 2024, arXiv preprint arXiv: 2402.16842
16	Y, Zeng K Lee . The expressive power of low-rank adaptation. In: Proceedings of the 12th International Conference on Learning Representations. 2024
17	V, Lialin S, Muckatira N, Shivagunde A Rumshisky . ReLoRA: high-rank training through low-rank updates. In: Proceedings of the 12th International Conference on Learning Representations. 2024
18	T, Jiang S, Huang S, Luo Z, Zhang H, Huang F, Wei W, Deng F, Sun Q, Zhang D, Wang F Zhuang . MoRA: high-rank updating for parameter-efficient fine-tuning. 2024, arXiv preprint arXiv: 2405.12130
19	M, Huh B, Cheung J, Bernstein P, Isola P Agrawal . Training neural networks from scratch with parallel low-rank adapters. 2024, arXiv preprint arXiv: 2402.16828
20	Y S, Liang W J Li . InfLoRA: interference-free low-rank adaptation for continual learning. 2024, arXiv preprint arXiv: 2404.00228
21	H, Zhao B, Ni H, Wang J, Fan F, Zhu Y, Wang Y, Chen G, Meng Z Zhang . Continual forgetting for pre-trained vision models. 2024, arXiv preprint arXiv: 2403.11530
22	W, Ren X, Li L, Wang T, Zhao W Qin . Analyzing and reducing catastrophic forgetting in parameter efficient tuning. 2024, arXiv preprint arXiv: 2402.18865
23	H Zhang . SinkLoRA: enhanced efficiency and chat capabilities for long-context large language models. 2024, arXiv preprint arXiv: 2406.05678
24	W, Xia C, Qin E Hazan . Chain of LoRA: efficient fine-tuning of language models via residual learning. 2024, arXiv preprint arXiv: 2401.04151
25	P, Ren C, Shi S, Wu M, Zhang Z, Ren Rijke M, de Z, Chen J Pei . MELoRA: mini-ensemble low-rank adapters for parameter-efficient fine-tuning. 2024, arXiv preprint arXiv: 2402.17263
26	Y, Hao Y, Cao L Mou . Flora: low-rank adapters are secretly gradient compressors. 2024, arXiv preprint arXiv: 2402.03293
27	B, Zi X, Qi L, Wang J, Wang K F, Wong L Zhang . Delta-LoRA: fine-tuning high-rank parameters with the delta of low-rank matrices. 2023, arXiv preprint arXiv: 2309.02411
28	Q, Zhang M, Chen A, Bukharin P, He Y, Cheng W, Chen T Zhao . Adaptive budget allocation for parameter-efficient fine-tuning. In: Proceedings of the 11th International Conference on Learning Representations. 2023
29	Y, Hu Y, Xie T, Wang M, Chen Z Pan . Structure-aware low-rank adaptation for parameter-efficient fine-tuning. Mathematics, 2023, 11( 20): 4317
30	F, Zhang L, Li J, Chen Z, Jiang B, Wang Y Qian . IncreLoRA: incremental parameter allocation method for parameter-efficient fine-tuning. 2023, arXiv preprint arXiv: 2308.12043
31	Y, Mao K, Huang C, Guan G, Bao F, Mo J Xu . DoRA: enhancing parameter-efficient fine-tuning with dynamic rank distribution. 2024, arXiv preprint arXiv: 2405.17357
32	R, Zhang R, Qiang S A, Somayajula P Xie . AutoLoRA: automatically tuning matrix ranks in low-rank adaptation based on meta learning. 2024, arXiv preprint arXiv: 2403.09113
33	Ding N, Lv X, Wang Q, Chen Y, Zhou B, Liu Z, Sun M. Sparse low-rank adaptation of pre-trained language models. In: Proceedings of 2023 Conference on Empirical Methods in Natural Language Processing. 2023, 4133−4145
34	Z, Liu J, Lyn W, Zhu X, Tian Y Graham . ALoRA: allocating low-rank adaptation for fine-tuning large language models. 2024, arXiv preprint arXiv: 2403.16187
35	M, Valipour M, Rezagholizadeh I, Kobyzev A Ghodsi . DyLoRA: parameter-efficient tuning of pre-trained models using dynamic search-free low-rank adaptation. In: Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics. 2023, 3274−3287
36	S, Hayou N, Ghosh B Yu . The impact of initialization on LoRA finetuning dynamics. 2024, arXiv preprint arXiv: 2406.08447
37	F, Meng Z, Wang M Zhang . PiSSA: principal singular values and singular vectors adaptation of large language models. 2024, arXiv preprint arXiv: 2404.02948
38	H, Wang Z, Xiao Y, Li S, Wang G, Chen Y Chen . MiLoRA: harnessing minor singular components for parameter-efficient LLM finetuning. 2024, arXiv preprint arXiv: 2406.09044
39	F, Zhang M Pilanci . Riemannian preconditioned LoRA for fine-tuning foundation models. 2024, arXiv preprint arXiv: 2402.02347
40	S, Hayou N, Ghosh B Yu . LoRA+: efficient low rank adaptation of large models. 2024, arXiv preprint arXiv: 2402.12354
41	S, Shi S, Huang M, Song Z, Li Z, Zhang H, Huang F, Wei W, Deng F, Sun Q Zhang . ResLoRA: identity residual mapping in low-rank adaption. 2024, arXiv preprint arXiv: 2402.18039
42	Z, Wen J, Zhang Y Fang . SIBO: a simple booster for parameter-efficient fine-tuning. 2024, arXiv preprint arXiv: 2402.11896
43	F, Jin Y, Liu Y Tan . Derivative-free optimization for low-rank adaptation in large language models. 2024, arXiv preprint arXiv: 2403.01754
44	S Y, Liu C Y, Wang H, Yin P, Molchanov Y C F, Wang K T, Cheng M H Chen . DoRA: weight-decomposed low-rank adaptation. 2024, arXiv preprint arXiv: 2402.09353
45	R, Qiang R, Zhang P Xie . BiLoRA: a bi-level optimization framework for overfitting-resilient low-rank adaptation of large pre-trained models. 2024, arXiv preprint arXiv: 2403.13037
46	Y, Lin X, Ma X, Chu Y, Jin Z, Yang Y, Wang H Mei . LoRA dropout as a sparsity regularizer for overfitting control. 2024, arXiv preprint arXiv: 2404.09610
47	S, Wang L, Chen J, Jiang B, Xue L, Kong C Wu . LoRA meets dropout under a unified framework. 2024, arXiv preprint arXiv: 2403.00812
48	A X, Yang M, Robeyns X, Wang L Aitchison . Bayesian low-rank adaptation for large language models. In: Proceedings of the 12th International Conference on Learning Representations. 2024
49	Qi Z, Tan X, Shi S, Qu C, Xu Y, Qi Y. PILLOW: enhancing efficient instruction fine-tuning via prompt matching. In: Proceedings of 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track. 2023, 471−482
50	L, Zhang J, Wu D, Zhou G Xu . STAR: constraint LoRA with dynamic active learning for data-efficient fine-tuning of large language models. 2024, arXiv preprint arXiv: 2403.01165
51	X, Wang L, Aitchison M Rudolph . LoRA ensembles for large language model fine-tuning. 2023, arXiv preprint arXiv: 2310.00035
52	Z, Zhao L, Gan G, Wang W, Zhou H, Yang K, Kuang F Wu . LoraRetriever: input-aware LoRA retrieval and composition for mixed tasks in the wild. 2024, arXiv preprint arXiv: 2402.09997
53	J S, Smith P, Cascante-Bonilla A, Arbelle D, Kim R, Panda D, Cox D, Yang Z, Kira R, Feris L Karlinsky . ConStruct-VL: data-free continual structured VL concepts learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023, 14994−15004
54	Y, Sun M, Li Y, Cao K, Wang W, Wang X, Zeng R Zhao . To be or not to be? An exploration of continuously controllable prompt engineering. 2023, arXiv preprint arXiv: 2311.09773
55	J, Zhang S, Chen J, Liu J He . Composing parameter-efficient modules with arithmetic operations. 2023, arXiv preprint arXiv: 2306.14870
56	R, Chitale A, Vaidya A, Kane A Ghotkar . Task arithmetic with LoRA for continual learning. 2023, arXiv preprint arXiv: 2311.02428
57	J Belofsky . Token-Level Adaptation of LoRA adapters for downstream task generalization. In: Proceedings of the 6th Artificial Intelligence and Cloud Computing Conference. 2023, 168−172
58	W, Jiang B, Lin H, Shi Y, Zhang Z, Li J T Kwok . Effective and parameter-efficient reusing fine-tuned models. 2023, arXiv preprint arXiv: 2310.01886
59	Asadi N, Beitollahi M, Khalil Y, Li Y, Zhang G, Chen X. Does combining parameter-efficient modules improve few-shot transfer accuracy? 2024, arXiv preprint arXiv: 2402.15414
60	C, Huang Q, Liu B Y, Lin T, Pang C, Du M Lin . LoraHub: efficient cross-task generalization via dynamic LoRA composition. 2023, arXiv preprint arXiv: 2307.13269
61	P, Yadav L, Choshen C, Raffel M Bansal . ComPEFT: compression for communicating parameter efficient updates via sparsification and quantization. 2023, arXiv preprint arXiv: 2311.13171
62	A, Tang L, Shen Y, Luo Y, Zhan H, Hu B, Du Y, Chen D Tao . Parameter-efficient multi-task model fusion with partial linearization. In: Proceedings of the 12th International Conference on Learning Representations. 2024
63	Y, Shen Z, Xu Q, Wang Y, Cheng W, Yin L Huang . Multimodal instruction tuning with conditional mixture of LoRA. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024, 637−648
64	E L, Buehler M J Buehler . X-LoRA: mixture of low-rank adapter experts, a flexible framework for large language models with applications in protein mechanics and molecular design. APL Machine Learning, 2024, 2( 2): 026119
65	S, Yang M A, Ali C L, Wang L, Hu D Wang . MoRAL: MoE augmented LoRA for LLMs’ lifelong learning. 2024, arXiv preprint arXiv: 2402.11260
66	S, Dou E, Zhou Y, Liu S, Gao J, Zhao W, Shen Y, Zhou Z, Xi X, Wang X, Fan S, Pu J, Zhu R, Zheng T, Gui Q, Zhang X Huang . LoRAMoE: alleviate world knowledge forgetting in large language models via MoE-style plugin. 2023, arXiv preprint arXiv: 2312.09979
67	Y, Gou Z, Liu K, Chen L, Hong H, Xu A, Li D Y, Yeung J T, Kwok Y Zhang . Mixture of cluster-conditional LoRA experts for vision-language instruction tuning. 2023, arXiv preprint arXiv: 2312.12379
68	Q, Liu X, Wu X, Zhao Y, Zhu D, Xu F, Tian Y Zheng . When MOE meets LLMs: parameter efficient fine-tuning for multi-task medical applications. In: Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2024, 1104−1114
69	Feng W, Hao C, Zhang Y, Han Y, Wang H. Mixture-of-LoRAs: an efficient multitask tuning method for large language models. In: Proceedings of 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation. 2024, 11371−11380
70	Y, Wang Y, Lin X, Zeng G Zhang . MultiLoRA: democratizing LoRA for better multi-task learning. 2023, arXiv preprint arXiv: 2311.11501
71	Y, Yang P T, Jiang Q, Hou H, Zhang J, Chen B Li . Multi-task dense prediction via mixture of low-rank experts. 2024, arXiv preprint arXiv: 2403.17749
72	A, Agiza M, Neseem S Reda . MTLoRA: low-rank adaptation approach for efficient multi-task learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2024, 16196−16205
73	C, Gao K, Chen J, Rao B, Sun R, Liu D, Peng Y, Zhang X, Guo J, Yang V S Subrahmanian . Higher layers need more LoRA experts. 2024, arXiv preprint arXiv: 2402.08562
74	S, Chen Z, Jie L Ma . LLaVA-MoLE: sparse mixture of LoRA experts for mitigating data conflicts in instruction finetuning MLLMs. 2024, arXiv preprint arXiv: 2401.16160
75	Y, Zhu N, Wichers C C, Lin X, Wang T, Chen L, Shu H, Lu C, Liu L, Luo J, Chen L Meng . SiRA: sparse mixture of low rank adaptation. 2023, arXiv preprint arXiv: 2311.09179
76	Z, Chen Z, Wang Z, Wang H, Liu Z, Yin S, Liu L, Sheng W, Ouyang Y, Qiao J Shao . Octavius: mitigating task interference in MLLMs via MoE. 2023, arXiv preprint arXiv: 2311.02684
77	Y, Wen S Chaudhuri . Batched low-rank adaptation of foundation models. In: Proceedings of the Twelfth International Conference on Learning Representations. 2024
78	Wu T, Wang J, Zhao Z, Wong N. Mixture-of-Subspaces in Low-Rank Adaptation. 2024, arXiv preprint arXiv:2406.11909
79	Y, Wu Y, Xiang S, Huo Y, Gong P Liang . LoRA-SP: streamlined partial parameter adaptation for resource efficient fine-tuning of large language models. In: Proceedings of the 3rd International Conference on Algorithms, Microchips, and Network Applications. 2024, 131711Z
80	L, Zhang L, Zhang S, Shi X, Chu B Li . LoRA-FA: memory-efficient low-rank adaptation for large language models fine-tuning. 2023, arXiv preprint arXiv: 2308.03303
81	Z, Liu S, Kundu A, Li J, Wan L, Jiang P A Beerel . AFLoRA: adaptive freezing of low rank adaptation in parameter efficient fine-tuning of large models. 2024, arXiv preprint arXiv: 2403.13269
82	S, Woo B, Park B, Kim M, Jo S, Kwon D, Jeon D Lee . DropBP: accelerating fine-tuning of large language models by dropping backward propagation. 2024, arXiv preprint arXiv: 2402.17812
83	K, Bałazy M, Banaei K, Aberer J Tabor . LoRA-XS: low-rank adaptation with extremely small number of parameters. 2024, arXiv preprint arXiv: 2405.17604
84	H, Zhou X, Lu W, Xu C, Zhu T, Zhao M Yang . LoRA-drop: efficient LoRA parameter pruning based on output evaluation. 2024, arXiv preprint arXiv: 2402.07721
85	M, Zhang H, Chen C, Shen Z, Yang L, Ou X, Yu B Zhuang . LoRAPrune: structured pruning meets low-rank parameter-efficient fine-tuning. In: Proceedings of the Findings of the Association for Computational Linguistics. 2024, 3013−3026
86	T, Chen T, Ding B, Yadav I, Zharkov L Liang . LoRAShear: efficient large language model structured pruning and knowledge recovery. 2023, arXiv preprint arXiv: 2310.18356
87	Y, Zhu X, Yang Y, Wu W Zhang . Parameter-efficient fine-tuning with layer pruning on free-text sequence-to-sequence modeling. 2023, arXiv preprint arXiv: 2305.08285
88	D J, Kopiczko T, Blankevoort Y M Asano . VeRA: vector-based random matrix adaptation. In: Proceedings of the 12th International Conference on Learning Representations. 2024
89	Y, Li S, Han S Ji . VB-LoRA: extreme parameter efficient fine-tuning with vector banks. 2024, arXiv preprint arXiv: 2405.15179
90	Z, Gao Q, Wang A, Chen Z, Liu B, Wu L, Chen J Li . Parameter-efficient fine-tuning with discrete Fourier transform. 2024, arXiv preprint arXiv: 2405.03003
91	T, Dettmers A, Pagnoni A, Holtzman L Zettlemoyer . QLORA: efficient finetuning of quantized LLMs. In: Proceedings of the 37th International Conference on Neural Information Processing Systems. 2023
92	Y, Xu L, Xie X, Gu X, Chen H, Chang H, Zhang Z, Chen X, Zhang Q Tian . QA-LoRA: quantization-aware low-rank adaptation of large language models. In: Proceedings of the 12th International Conference on Learning Representations. 2024
93	Y, Li Y, Yu C, Liang P, He N, Karampatziakis W, Chen T Zhao . LoftQ: LoRA-fine-tuning-aware quantization for large language models. In: Proceedings of the 12th International Conference on Learning Representations. 2024
94	B, Liao C, Herold S, Khadivi C Monz . ApiQ: finetuning of 2-bit quantized large language model. 2024, arXiv preprint arXiv: 2402.05147
95	H, Jeon Y, Kim J J Kim . L4Q: parameter efficient quantization-aware training on large language models via LoRA-wise LSQ. 2024, arXiv preprint arXiv: 2402.04902
96	Z, Ye D, Li J, Tian T, Lan J, Zuo L, Duan H, Lu Y, Jiang J, Sha K, Zhang M Tang . ASPEN: high-throughput LoRA fine-tuning of large language models with a single GPU. 2023, arXiv preprint arXiv: 2312.02515
97	L, Chen Z, Ye Y, Wu D, Zhuo L, Ceze A Krishnamurthy . Punica: multi-tenant LoRA serving. In: Proceedings of the Seventh Annual Conference on Machine Learning and Systems. 2024, 1−13
98	Y, Sheng S, Cao D, Li C, Hooper N, Lee S, Yang C, Chou B, Zhu L, Zheng K, Keutzer J E, Gonzalez I Stoica . S-LoRA: serving thousands of concurrent LoRA adapters. 2023, arXiv preprint arXiv: 2311.03285
99	S, Li H, Lu T, Wu M, Yu Q, Weng X, Chen Y, Shan B, Yuan W Wang . CaraServe: CPU-assisted and rank-aware LoRA serving for generative LLM inference. 2024, arXiv preprint arXiv: 2401.11240
100	S, Babakniya A R, Elkordy Y H, Ezzeldin Q, Liu K B, Song M, El-Khamy S Avestimehr . SLoRA: federated parameter efficient fine-tuning of language models. 2023, arXiv preprint arXiv: 2308.06522
101	Y, Yan S, Tang Z, Shi Q Yang . FeDeRA: efficient fine-tuning of language models in federated learning leveraging weight decomposition. 2024, arXiv preprint arXiv: 2404.18848
102	Sun Y, Li Z, Li Y, Ding B. Improving LoRA in privacy-preserving federated learning. In: Proceedings of the 12th International Conference on Learning Representations. 2024
103	P, Wu K, Li T, Wang F Wang . FedMS: federated learning with mixture of sparsely activated foundations models. 2023, arXiv preprint arXiv: 2312.15926
104	J, Bai D, Chen B, Qian L, Yao Y Li . Federated fine-tuning of large language models under heterogeneous language tasks and client resources. 2024, arXiv preprint arXiv: 2402.11505
105	Y J, Cho L, Liu Z, Xu A, Fahrezi M, Barnes G Joshi . Heterogeneous LoRA for federated fine-tuning of on-device foundation models. In: Proceedings of the International Workshop on Federated Learning in the Age of Foundation Models in Conjunction with NeurIPS. 2023
106	L, Yi H, Yu G, Wang X, Liu X Li . pFedLoRA: model-heterogeneous personalized federated learning with LoRA tuning. 2023, arXiv preprint arXiv: 2310.13283
107	W, Huang Y, Wang A, Cheng A, Zhou C, Yu L Wang . A fast, performant, secure distributed training framework for large language model. 2024, arXiv preprint arXiv: 2401.09796
108	Y, Wang Y, Lin X, Zeng G Zhang . PrivateLoRA for efficient privacy preserving LLM. 2023, arXiv preprint arXiv: 2311.14030
109	Y, Zhang M, Wang Y, Wu P, Tiwari Q, Li B, Wang J Qin . DialogueLLM: context and emotion knowledge-tuned large language models for emotion recognition in conversations. 2024, arXiv preprint arXiv: 2310.11374
110	Z, Li X, Li Y, Liu H, Xie J, Li F L, Wang Q, Li X Zhong . Label supervised LLaMA finetuning. 2023, arXiv preprint arXiv: 2310.01208
111	T, Bornheim N, Grieger P G, Blaneck S Bialonski . Speaker attribution in German parliamentary debates with QLoRA-adapted large language models. 2024, arXiv preprint arXiv: 2309.09902
112	L, Xue D, Zhang Y, Dong J Tang . AutoRE: document-level relation extraction with large language models. 2024, arXiv preprint arXiv: 2403.14888
113	D M, Alves N M, Guerreiro J, Alves J, Pombal R, Rei Souza J G C, de P, Colombo A F T Martins . Steering large language models for machine translation with finetuning and in-context learning. In: Proceedings of the Findings of the Association for Computational Linguistics. 2023, 11127−11148
114	J, Zheng H, Hong X, Wang J, Su Y, Liang S Wu . Fine-tuning large language models for domain-specific machine translation. 2024, arXiv preprint arXiv: 2402.15061
115	V, Mujadia A, Urlana Y, Bhaskar P A, Pavani K, Shravya P, Krishnamurthy D M Sharma . Assessing translation capabilities of large language models involving English and Indian languages. 2023, arXiv preprint arXiv: 2311.09216
116	Y, Zhang J, Wang L C, Yu D, Xu X Zhang . Personalized LoRA for human-centered text understanding. In: Proceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence. 2024, 19588−19596
117	Y, Liu C, An X Qiu . Y-tuning: an efficient tuning paradigm for large-scale pre-trained models via label representation learning. Frontiers of Computer Science, 2024, 18( 4): 184320
118	S, Liu J, Keung Z, Yang F, Liu Q, Zhou Y Liao . Delving into parameter-efficient fine-tuning in code change learning: an empirical study. In: Proceedings of the IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). 2024, 465−476
119	Y, Guo X, Gao B Jiang . An empirical study on JIT defect prediction based on BERT-style model. 2024, arXiv preprint arXiv: 2403.11158
120	S, Ayupov N Chirkova . Parameter-efficient finetuning of transformers for source code. 2022, arXiv preprint arXiv: 2212.05901
121	A, Silva S, Fang M Monperrus . RepairLLaMA: efficient representations and fine-tuned adapters for program repair. 2023, arXiv preprint arXiv: 2312.15698
122	R, Roberson G, Kaki A Trivedi . Analyzing the effectiveness of large language models on text-to-SQL synthesis. 2024, arXiv preprint arXiv: 2401.12379
123	J, Pan A, Sadé J, Kim E, Soriano G, Sole S Flamant . SteloCoder: a decoder-only LLM for multi-language to python code translation. 2023, arXiv preprint arXiv: 2310.15539
124	H, Sidahmed S, Phatale A, Hutcheson Z, Lin Z, Chen Z, Yu J, Jin R, Komarytsia C, Ahlheim Y, Zhu S, Chaudhary B, Li S, Ganesh B, Byrne J, Hoffmann H, Mansoor W, Li A, Rastogi L Dixon . PERL: parameter efficient reinforcement learning from human feedback. 2024, arXiv preprint arXiv: 2403.10704
125	M, Santacroce Y, Lu H, Yu Y, Li Y Shen . Efficient RLHF: reducing the memory usage of PPO. 2023, arXiv preprint arXiv: 2309.00754
126	S, Sun D, Gupta M Iyyer . Exploring the impact of low-rank adaptation on the performance, efficiency, and regularization of RLHF. 2023, arXiv preprint arXiv: 2309.09055
127	S Quan . DMoERM: recipes of mixture-of-experts for effective reward modeling. 2024, arXiv preprint arXiv: 2403.01197
128	S, Zhang Z, Chen S, Chen Y, Shen Z, Sun C Gan . Improving reinforcement learning from human feedback with efficient reward model ensemble. 2024, arXiv preprint arXiv: 2401.16635
129	Y, Zhai H, Zhang Y, Lei Y, Yu K, Xu D, Feng B, Ding H Wang . Uncertainty-penalized reinforcement learning from human feedback with diverse reward LoRA ensembles. 2023, arXiv preprint arXiv: 2401.00243
130	A X, Yang M, Robeyns T, Coste Z, Shi J, Wang H, Bou-Ammar L Aitchison . Bayesian reward models for LLM alignment. 2024, arXiv preprint arXiv: 2402.13210
131	Daxberger E, Kristiadi A, Immer A, Eschenhagen R, Bauer M, Hennig P. Laplace redux-effortless bayesian deep learning. Advances in Neural Information Processing Systems. 2021
132	H, Tran Z, Yang Z, Yao H Yu . BioInstruct: instruction tuning of large language models for biomedical natural language processing. 2023, arXiv preprint arXiv: 2310.19975
133	A P, Gema P, Minervini L, Daines T, Hope B Alex . Parameter-efficient fine-tuning of LLaMA for the clinical domain. 2023, arXiv preprint arXiv: 2307.03042
134	A, Toma P R, Lawler J, Ba R G, Krishnan B B, Rubin B Wang . Clinical camel: an open-source expert-level medical language model with dialogue-based knowledge encoding. 2023, arXiv preprint arXiv: 2305.12031
135	K, Suri P, Mishra S, Saha A Singh . Suryakiran at MEDIQA-Sum 2023: leveraging LoRA for clinical dialogue summarization. In: Proceedings of the Working Notes of the Conference and Labs of the Evaluation Forum. 2023, 1720−1735
136	Y, Ji Z, Yu Y Wang . Assertion detection large language model in-context learning LoRA fine-tuning. 2024, arXiv preprint arXiv: 2401.17602
137	Wang R, Duan Y, Lam C, Chen J, Xu J, Chen H, Liu X, Pang P C I, Tan T. IvyGPT: InteractiVe Chinese pathway language model in medical domain. In: Proceedings of the 3rd CAAI International Conference on Artificial Intelligence. 2024, 378−382
138	A, Bhatti S, Parmar S Lee . SM70: a large language model for medical devices. 2023, arXiv preprint arXiv: 2312.06974
139	T, Konstantinidis G, Iacovides M, Xu T G, Constantinides D Mandic . FinLlama: financial sentiment classification for algorithmic trading applications. 2024, arXiv preprint arXiv: 2403.12285
140	B M Pavlyshenko . Financial news analytics using fine-tuned llama 2 GPT model. 2023, arXiv preprint arXiv: 2308.13032
141	X Y, Liu G, Wang H, Yang D Zha . FinGPT: democratizing internet-scale data for financial large language models. 2023, arXiv preprint arXiv: 2307.10485
142	J, Li Y, Lei Y, Bian D, Cheng Z, Ding C Jiang . RA-CFGPT: Chinese financial assistant with retrieval-augmented large language model. Frontiers of Computer Science, 2024, 18( 5): 185350
143	X, Zhou Z, Sun G Li . DB-GPT: large language model meets database. Data Science and Engineering, 2024, 9( 1): 102–111
144	S Li . DiffStyler: diffusion-based localized image style transfer. 2024, arXiv preprint arXiv: 2403.18461
145	Y, Frenkel Y, Vinker A, Shamir D Cohen-Or . Implicit style-content separation using B-LoRA. 2024, arXiv preprint arXiv: 2403.14572
146	Y, Liu C, Yu L, Shang Y, He Z, Wu X, Wang C, Xu H, Xie W, Wang Y, Zhao L, Zhu C, Cheng W, Chen Y, Yao W, Zhou J, Xu Q, Wang Y, Chen X, Xie B Sun . FaceChain: a playground for human-centric artificial intelligence generated content. 2023, arXiv preprint arXiv: 2308.14256
147	Q, Liao G, Xia Z Wang . Calliffusion: Chinese calligraphy generation and style transfer with diffusion modeling. 2023, arXiv preprint arXiv: 2305.19124
148	S, Shrestha V S S, Sripada A Venkataramanan . Style transfer to Calvin and Hobbes comics using stable diffusion. 2023, arXiv preprint arXiv: 2312.03993
149	L, Li H, Zeng C, Yang H, Jia D Xu . Block-wise LoRA: revisiting fine-grained LoRA for effective personalization and stylization in text-to-image generation. 2024, arXiv preprint arXiv: 2403.07500
150	Z, Kong Y, Zhang T, Yang T, Wang K, Zhang B, Wu G, Chen W, Liu W Luo . OMG: occlusion-friendly personalized multi-concept generation in diffusion models. 2024, arXiv preprint arXiv: 2403.10983
151	J, Shi H Hua . Space narrative: generating images and 3D scenes of Chinese garden from text using deep learning. In: Proceedings of the xArch-Creativity in the Age of Digital Reproduction Symposium. 2024, 236−243
152	Z, Jin Z Song . Generating coherent comic with rich story using ChatGPT and stable diffusion. 2023, arXiv preprint arXiv: 2305.11067
153	H, Wang X, Xiang Y, Fan J H Xue . Customizing 360-degree panoramas through text-to-image diffusion models. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2024, 4921−4931
154	J, Guo X, Xu Y, Pu Z, Ni C, Wang M, Vasu S, Song G, Huang H Shi . Smooth diffusion: crafting smooth latent spaces in diffusion models. 2023, arXiv preprint arXiv: 2312.04410
155	J, Cheng P, Xie X, Xia J, Li J, Wu Y, Ren H, Li X, Xiao M, Zheng L Fu . ResAdapter: domain consistent resolution adapter for diffusion models. 2024, arXiv preprint arXiv: 2403.02084
156	J S, Smith Y C, Hsu Z, Kira Y, Shen H Jin . Continual diffusion with STAMINA: STack-and-mask INcremental adapters. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024, 1744−1754
157	J, Sun D, Fu Y, Hu S, Wang R, Rassin D C, Juan D, Alon C, Herrmann Steenkiste S, van R, Krishna C Rashtchian . Dreamsync: aligning text-to-image generation with image understanding feedback. In: Proceedings of the Synthetic Data for Computer Vision Workshop@ CVPR 2024. 2023
158	Z, Wang X, Wang L, Xie Z, Qi Y, Shan W, Wang P Luo . StyleAdapter: a single-pass LoRA-free model for stylized image generation. 2023, arXiv preprint arXiv: 2309.01770
159	Y, Gu X, Wang J Z, Wu Y, Shi Y, Chen Z, Fan W, Xiao R, Zhao S, Chang W, Wu Y, Ge Y, Shan M Z Shou . Mix-of-show: decentralized low-rank adaptation for multi-concept customization of diffusion models. In: Proceedings of the 37th International Conference on Neural Information Processing Systems. 2023
160	S, Luo Y, Tan S, Patil D, Gu Platen P, von A, Passos L, Huang J, Li H Zhao . LCM-LoRA: a universal stable-diffusion acceleration module. 2023, arXiv preprint arXiv: 2311.05556
161	P A Golnari . LoRA-enhanced distillation on guided diffusion models. 2023, arXiv preprint arXiv: 2312.06899
162	Y, Ren Y, Zhou J, Yang J, Shi D, Liu F, Liu M, Kwon A Shrivastava . Customize-A-video: one-shot motion customization of text-to-video diffusion models. 2024, arXiv preprint arXiv: 2402.14780
163	Y, Deng R, Wang Y, Zhang Y W, Tai C K Tang . DragVideo: interactive drag-style video editing. 2023, arXiv preprint arXiv: 2312.02216
164	S, Yang Y, Zhou Z, Liu C C Loy . Rerender A video: zero-shot text-guided video-to-video translation. In: Proceedings of the SIGGRAPH Asia 2023 Conference Papers. 2023, 95
165	A Khandelwal . InFusion: inject and attention fusion for multi concept zero-shot text-based video editing. In: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops. 2023, 3009−3018
166	A, Blattmann T, Dockhorn S, Kulal D, Mendelevitch M, Kilian D, Lorenz Y, Levi Z, English V, Voleti A, Letts V, Jampani R Rombach . Stable video diffusion: scaling latent video diffusion models to large datasets. 2023, arXiv preprint arXiv: 2311.15127
167	Guo Y, Yang C, Rao A, Liang Z, Wang Y, Qiao Y, Agrawala M, Lin D, Dai B. AnimateDiff: animate your personalized text-to-image diffusion models without specific tuning. In: Proceedings of the 12th International Conference on Learning Representations. 2024
168	T, Huang Y, Zeng Z, Zhang W, Xu H, Xu S, Xu R W H, Lau W Zuo . DreamControl: control-based text-to-3D generation with 3D self-prior. 2023, arXiv preprint arXiv: 2312.06439
169	Y, Ma Y, Fan J, Ji H, Wang X, Sun G, Jiang A, Shu R Ji . X-dreamer: creating high-quality 3D content by bridging the domain gap between text-to-2D and text-to-3D generation. 2023, arXiv preprint arXiv: 2312.00085
170	K, Yu J, Liu M, Feng M, Cui X Xie . Boosting3D: high-fidelity image-to-3D by boosting 2D diffusion prior to 3D prior with progressive learning. 2023, arXiv preprint arXiv: 2311.13617
171	S, Yoo K, Kim V G, Kim M Sung . As-plausible-as-possible: plausibility-aware mesh deformation using 2D diffusion priors. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024, 4315−4324
172	Y, Zhang Q, Xu L Zhang . DragTex: generative point-based texture editing on 3D mesh. 2024, arXiv preprint arXiv: 2403.02217
173	H, Ding J, Gao Y, Yuan Q Wang . SamLP: a customized segment anything model for license plate detection. 2024, arXiv preprint arXiv: 2401.06374
174	Z, Ye L, Lovell A, Faramarzi J Ninic . SAM-based instance segmentation models for the automation of structural damage detection. 2024, arXiv preprint arXiv: 2401.15266
175	S, Na Y, Guo F, Jiang H, Ma J Huang . Segment any cell: a SAM-based auto-prompting fine-tuning framework for nuclei segmentation. 2024, arXiv preprint arXiv: 2401.13220
176	X, Chen C, Wang H, Ning S, Li M Shen . SAM-OCTA: prompting segment-anything for OCTA image segmentation. 2023, arXiv preprint arXiv: 2310.07183
177	W, Feng L, Zhu L Yu . Cheap lunch for medical image segmentation by fine-tuning SAM on few exemplars. 2023, arXiv preprint arXiv: 2308.14133
178	K, Zhang D Liu . Customized segment anything model for medical image segmentation. 2023, arXiv preprint arXiv: 2304.13785
179	A, Wang M, Islam M, Xu Y, Zhang H Ren . SAM meets robotic surgery: an empirical study on generalization, robustness and adaptation. In: Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention. 2023, 234−244
180	L, Lin H, Fan Z, Zhang Y, Wang Y, Xu H Ling . Tracking meets LoRA: faster training, larger model, stronger performance. 2024, arXiv preprint arXiv: 2403.05231
181	C, Kong H, Li S Wang . Enhancing general face forgery detection via vision transformer with low-rank adaptation. In: Proceedings of the 6th International Conference on Multimedia Information Processing and Retrieval. 2023, 102−107
182	Z, Chen H, Huang A, Andrusenko O, Hrinchuk K C, Puvvada J, Li S, Ghosh J, Balam B Ginsburg . SALM: speech-augmented language model with in-context learning for speech recognition and translation. In: Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2024, 13521−13525
183	X, Dong P, Zhang Y, Zang Y, Cao B, Wang L, Ouyang X, Wei S, Zhang H, Duan M, Cao W, Zhang Y, Li H, Yan Y, Gao X, Zhang W, Li J, Li K, Chen C, He X, Zhang Y, Qiao D, Lin J Wang . InternLM-XComposer2: mastering free-form text-image composition and comprehension in vision-language large model. 2024, arXiv preprint arXiv: 2401.16420
184	Q, Ye H, Xu G, Xu J, Ye M, Yan Y, Zhou J, Wang A, Hu P, Shi Y, Shi C, Li Y, Xu H, Chen J, Tian Q, Qian J, Zhang F, Huang J Zhou . mPLUG-Owl: modularization empowers large language models with multimodality. 2023, arXiv preprint arXiv: 2304.14178
185	B K, Lee B, Park C W, Kim Y M Ro . CoLLaVO: crayon large language and vision mOdel. 2024, arXiv preprint arXiv: 2402.11248
186	J H, Yeo S, Han M, Kim Y M Ro . Where visual speech meets language: VSP-LLM framework for efficient and context-aware visual speech processing. 2024, arXiv preprint arXiv: 2402.15151
187	Liu Z, Li S, Luo Y, Fei H, Cao Y, Kawaguchi K, Wang X, Chua T S. MolCA: molecular graph-language modeling with cross-modal projector and uni-modal adapter. In: Proceedings of 2023 Conference on Empirical Methods in Natural Language Processing. 2023, 15623−15638
188	Y, Ren Y, Chen S, Liu B, Wang H, Yu Z Cui . TPLLM: a traffic prediction framework based on pretrained large language models. 2024, arXiv preprint arXiv: 2403.02221
189	A, Aghajanyan S, Gupta L Zettlemoyer . Intrinsic dimensionality explains the effectiveness of language model fine-tuning. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. 2021, 7319−7328
190	V, Fomenko H, Yu J, Lee S, Hsieh W Chen . A note on LoRA. 2024, arXiv preprint arXiv: 2404.05086
191	D, Bershatsky D, Cherniuk T, Daulbaev A, Mikhalev I Oseledets . LoTR: low tensor rank weight adaptation. 2024, arXiv preprint arXiv: 2402.01376
192	A, Edalati M, Tahaei I, Kobyzev V P, Nia J J, Clark M Rezagholizadeh . KronA: parameter efficient tuning with kronecker adapter. 2022, arXiv preprint arXiv: 2212.10650
193	He X, Li C, Zhang P, Yang J, Wang X E. Parameter-efficient model adaptation for vision transformers. In: Proceedings of the 37th AAAI Conference on Artificial Intelligence. 2023, 817−825
194	Zhao Z, Gan L, Wang G, Hu Y, Shen T, Yang H, Kuang K, Wu F. Retrieval-augmented mixture of lora experts for uploadable machine learning. 2024 , arXiv preprint arXiv:2406.16989.
195	R K, Mahabadi J, Henderson S Ruder . COMPACTER: efficient low-rank hypercomplex adapter layers. In: Proceedings of the 35th International Conference on Neural Information Processing Systems. 2021, 79
196	B, Liao Y, Meng C Monz . Parameter-efficient fine-tuning without introducing new latency. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. 2023, 4242−4260
197	D, Hendrycks C, Burns S, Basart A, Zou M, Mazeika D, Song J Steinhardt . Measuring massive multitask language understanding. In: Proceedings of the 9th International Conference on Learning Representations. 2021
198	J, He C, Zhou X, Ma T, Berg-Kirkpatrick G Neubig . Towards a unified view of parameter-efficient transfer learning. In: Proceedings of the 10th International Conference on Learning Representations. 2022
199	B, Geshkovski C, Letrouit Y, Polyanskiy P Rigollet . A mathematical perspective on transformers. 2023, arXiv preprint arXiv: 2312.10794
200	B, Geshkovski C, Letrouit Y, Polyanskiy P Rigollet . The emergence of clusters in self-attention dynamics. In: Proceedings of the 37th International Conference on Neural Information Processing Systems. 2023
201	M E, Sander P, Ablin M, Blondel G Peyré . Sinkformers: transformers with doubly stochastic attention. In: Proceedings of the 25th International Conference on Artificial Intelligence and Statistics. 2022, 3515−3530
202	A, Jacot F, Gabriel C Hongler . Neural tangent kernel: convergence and generalization in neural networks. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems. 2018, 8580−8589
203	H, Touvron L, Martin K, Stone P, Albert A, Almahairi Y, Babaei N, Bashlykov S, Batra P, Bhargava S, Bhosale D, Bikel L, Blecher Ferrer C, Canton M, Chen G, Cucurull D, Esiobu J, Fernandes J, Fu W, Fu B, Fuller C, Gao V, Goswami N, Goyal A, Hartshorn S, Hosseini R, Hou H, Inan M, Kardas V, Kerkez M, Khabsa I, Kloumann A, Korenev P S, Koura M A, Lachaux T, Lavril J, Lee D, Liskovich Y, Lu Y, Mao X, Martinet T, Mihaylov P, Mishra I, Molybog Y, Nie A, Poulton J, Reizenstein R, Rungta K, Saladi A, Schelten R, Silva E M, Smith R, Subramanian X E, Tan B, Tang R, Taylor A, Williams J X, Kuan P, Xu Z, Yan I, Zarov Y, Zhang A, Fan M, Kambadur S, Narang A, Rodriguez R, Stojnic S, Edunov T Scialom . Llama 2: open foundation and fine-tuned chat models. 2023, arXiv preprint arXiv: 2307.09288
204	Chang Y, Chang Y, Wu Y. Bias-Aware Low-Rank Adaptation: Mitigating Catastrophic Inheritance of Large Language Models. 2024 , arXiv preprint arXiv:2408.04556
205	J, Zhao Z, Zhang B, Chen Z, Wang A, Anandkumar Y Tian . Galore: memory-efficient LLM training by gradient low-rank projection. 2024, arXiv preprint arXiv: 2403.03507
206	D, Biderman J G, Ortiz J, Portes M, Paul P, Greengard C, Jennings D, King S, Havens V, Chiley J, Frankle C, Blakeney J P Cunningham . LoRA learns less and forgets less. 2024, arXiv preprint arXiv: 2405.09673
207	A, Han J, Li W, Huang M, Hong A, Takeda P, Jawanpuria B Mishra . SLTrain: a sparse plus low-rank approach for parameter and memory efficient pretraining. 2024, arXiv preprint arXiv: 2406.02214
208	Y, Sui M, Yin Y, Gong J, Xiao H, Phan B Yuan . ELRT: efficient low-rank training for compact convolutional neural networks. 2024, arXiv preprint arXiv: 2401.10341
209	X, Meng D, Dai W, Luo Z, Yang S, Wu X, Wang P, Wang Q, Dong L, Chen Z Sui . PeriodicLoRA: breaking the low-rank bottleneck in LoRA optimization. 2024, arXiv preprint arXiv: 2402.16141
210	M, Frank P Wolfe . An algorithm for quadratic programming. Naval Research Logistics Quarterly, 1956, 3( 1-2): 95–110
211	Rajabzadeh H, Valipour M, Zhu T, Tahaei M, Kwon HJ, Ghodsi A, Chen B, Rezagholizadeh M. Qdylora: Quantized dynamic low-rank adaptation for efficient large language model tuning. 2024 , arXiv preprint arXiv:2402.10462
212	T, Elsken J H, Metzen F Hutter . Neural architecture search: a survey. The Journal of Machine Learning Research, 2019, 20( 1): 1997–2017
213	Y, Liu M, Ott N, Goyal J, Du M, Joshi D, Chen O, Levy M, Lewis L, Zettlemoyer V Stoyanov . RoBERTa: a robustly optimized BERT pretraining approach. 2019, arXiv preprint arXiv: 1907.11692
214	Wang A, Singh A, Michael J, Hill F, Levy O, Bowman S R. GLUE: a multi-task benchmark and analysis platform for natural language understanding. In: Proceedings of 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP. 2018, 353−355
215	Renduchintala A, Konuk T, Kuchaiev O. Tied-LoRA: enhancing parameter efficiency of LoRA with weight tying. In: Proceedings of 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2024, 8694−8705
216	N, Hansen A Ostermeier . Adapting arbitrary normal mutation distributions in evolution strategies: the covariance matrix adaptation. In: Proceedings of the IEEE International Conference on Evolutionary Computation. 1996, 312−317
217	M, Ye X, Fang B, Du P C, Yuen D Tao . Heterogeneous federated learning: state-of-the-art and research challenges. ACM Computing Surveys, 2024, 56( 3): 79
218	X Y, Liu R, Zhu D, Zha J, Gao S, Zhong M, White M Qiu . Differentially private low-rank adaptation of large language model using federated learning. 2023, arXiv preprint arXiv: 2312.17493

[1]	Xiao MA, Shen-Yi ZHAO, Zhao-Heng YIN, Wu-Jun LI. Clustered Reinforcement Learning[J]. Front. Comput. Sci., 2025, 19(4): 194313-.
[2]	Jingyu LIU, Shi CHEN, Li SHEN. A comprehensive survey on graph neural network accelerators[J]. Front. Comput. Sci., 2025, 19(2): 192104-.
[3]	Peng YANG, Qi YANG, Ke TANG, Xin YAO. Parallel exploration via negatively correlated search[J]. Front. Comput. Sci., 2021, 15(5): 155333-.
[4]	Huafeng YU, Yue MA, Thierry GAUTIER, Loïc BESNARD, Jean-Pierre TALPIN, Paul Le GUERNIC, Yves SOREL. Exploring system architectures in AADL via Polychrony and SynDEx[J]. Front. Comput. Sci., 2013, 7(5): 627-649.
[5]	Amit Kumar SINGH, Akash KUMAR, Jigang WU, Thambipillai SRIKANTHAN. CADSE: communication aware design space exploration for efficient run-time MPSoC management[J]. Front Comput Sci, 2013, 7(3): 416-430.
[6]	Sertan GIRGIN, Jérémie MARY, Philippe PREUX, Olivier NICOL. Managing advertising campaigns—an approximate planning approach[J]. Front Comput Sci, 2012, 6(2): 209-229.
[7]	Weimin WANG, Jingchun ZHANG, Cong CAO, Tao HOU, Yue LIU, Keji CHEN. An efficient approach to representing and mining knowledge from Qing court medical records[J]. Front Comput Sci Chin, 2011, 5(4): 395-404.

Viewed

Full text

Abstract

Cited

Shared

Discussed