Please wait a minute...
Shopping Cart Email List Chinese
Advanced Search Fig/Tab
Frontiers of Computer Science

ISSN 2095-2228

ISSN 2095-2236(Online)

CN 10-1014/TP

Postal Subscription Code 80-970

2018 Impact Factor: 1.129

Featured Articles  |  Most Downloaded  |  Most Reading
Online Submission Subscribe to Our RSS Content Feed
Front. Comput. Sci.    2025, Vol. 19 Issue (1) : 191202    https://doi.org/10.1007/s11704-023-3256-9
Software
Code context-based reviewer recommendation
Dawei YUAN1, Xiao PENG1, Zijie CHEN1, Tao ZHANG1(), Ruijia LEI2
1. School of Computer Science and Engineering, Macau University of Science and Technology, Macao 999078, China
2. Faculty of Science, University of Amsterdam, Amsterdam WB 1018, Netherlands
 Download: PDF(2502 KB)   HTML
 Export: BibTeX | EndNote | Reference Manager | ProCite | RefWorks
Abstract

Code review is a critical process in software development, contributing to the overall quality of the product by identifying errors early. A key aspect of this process is the selection of appropriate reviewers to scrutinize changes made to source code. However, in large-scale open-source projects, selecting the most suitable reviewers for a specific change can be a challenging task. To address this, we introduce the Code Context Based Reviewer Recommendation (CCB-RR), a model that leverages information from changesets to recommend the most suitable reviewers. The model takes into consideration the paths of modified files and the context derived from the changesets, including their titles and descriptions. Additionally, CCB-RR employs KeyBERT to extract the most relevant keywords and compare the semantic similarity across changesets. The model integrates the paths of modified files, keyword information, and the context of code changes to form a comprehensive picture of the changeset. We conducted extensive experiments on four open-source projects, demonstrating the effectiveness of CCB-RR. The model achieved a Top-1 accuracy of 60%, 55%, 51%, and 45% on the Android, OpenStack, QT, and LibreOffice projects respectively. For Mean Reciprocal Rank (MRR), CCB achieved 71%, 62%, 52%, and 68% on the same projects respectively, thereby highlighting its potential for practical application in code reviewer recommendation.
Keywords code reviewer recommendation      code context      pull request     
Corresponding Author(s): Tao ZHANG   
Just Accepted Date: 18 October 2023   Issue Date: 12 March 2024
 Cite this article:   
Dawei YUAN,Xiao PENG,Zijie CHEN, et al. Code context-based reviewer recommendation[J]. Front. Comput. Sci., 2025, 19(1): 191202.
 URL:  
https://academic.hep.com.cn/fcs/EN/10.1007/s11704-023-3256-9
https://academic.hep.com.cn/fcs/EN/Y2025/V19/I1/191202
Fig.1  The flowchart illustrating the modern code review process
Fig.2  An example of a pull request
Fig.3  An overview of CCB-RR
Statistical type Android OpenStack Qt LibreOffice
Reviews 5126 6586 23810 6523
Code-reviewers 94 82 202 63
Files 26840 16953 78401 35273
Average Reviewers 1.06 1.44 1.07 1.01
Average Files 8.26 6.04 10.64 11.14
Tab.1  Statistics of project reviews, reviewers, and modified files
Project Method Top 1 Top 3 Top 5 Top 10 MRR
Android RS 0.37 0.55 0.59 0.66 0.46
REV 0.40 0.64 0.68 0.81 0.55
WRC 0.43 0.70 0.68 0.77 0.55
TIE 0.52 0.76 0.82 0.87 0.65
TNE 0.40 0.65 0.70 0.75 0.60
RevRec 0.53 0.42 0.34 0.29 0.64
CCB 0.60 0.80 0.85 0.83 0.71
OpenStack RS 0.28 0.61 0.73 0.77 0.40
REV 0.32 0.61 0.73 0.82 0.50
WRC 0.36 0.63 0.70 0.86 0.44
TIE 0.38 0.68 0.78 0.86 0.55
TNE 0.30 0.60 0.70 0.75 0.50
RevRec 0.54 0.46 0.38 0.31 0.58
CCB 0.55 0.72 0.81 0.87 0.62
QT RS 0.33 0.57 0.67 0.63 0.45
REV 0.26 0.33 0.39 0.64 0.26
WRC 0.33 0.57 0.64 0.62 0.36
TIE 0.25 0.40 0.47 0.57 0.36
TNE 0.27 0.55 0.65 0.70 0.50
RevRec 0.44 0.40 0.36 0.29 0.49
CCB 0.51 0.61 0.73 0.77 0.52
LibreOffice RS 0.15 0.28 0.37 0.66 0.45
REV 0.19 0.36 0.46 0.69 0.35
WRC 0.27 0.43 0.52 0.72 0.50
TIE 0.71 0.86 0.88 0.91 0.79
TNE 0.25 0.45 0.55 0.65 0.47
RevRec ? ? ? ? ?
CCB 0.45 0.59 0.72 0.80 0.68
Tab.2  Top-k Prediction accuracies (Top 1, 3, 5, 10) and MRR results achieved by each method for each system. These methods contain Recommendation System (RS), REVFINDER (REV), Weighted Reviewer Classification (WRC), Text Information Extraction [1] (TIE), Topic-based Navigation Engine [27] (TNE), Reviewer Recommendation [33] (RevRec), and Code Context Based Reviewer Recommendation (CCB) represent different models
Project RS REV WRC TIE TNE RevRec CCB
Android 0.35 0.37 0.36 0.38 0.33 0.31 0.42
OpenStack 0.32 0.33 0.35 0.40 0.30 0.29 0.43
QT 0.34 0.35 0.40 0.39 0.32 0.30 0.44
LibreOffice 0.39 0.37 0.36 0.35 0.34 0.32 0.41
Average. 0.35 0.36 0.37 0.38 0.33 0.31 0.42
Tab.3  FaR metric comparison for various models. Recommendation System (RS), REVFINDER (REV), Weighted Reviewer Classification (WRC), Text Information Extraction [1] (TIE), Topic-based Navigation Engine [27] (TNE), Reviewer Recommendation [33] (RevRec), and Code Context Based Reviewer Recommendation (CCB) represent different models
Project Top-1 Top-3 Top-5 Top-10 MRR
I II III F I II III F I II III F I II III F I II III F
Android 0.44 0.45 0.27 0.60 0.72 0.76 0.67 0.80 0.79 0.80 0.75 0.85 0.84 0.86 0.86 0.83 0.59 0.61 0.47 0.71
OpenStack 0.38 0.34 0.33 0.55 0.70 0.76 0.60 0.72 0.81 0.80 0.75 0.81 0.91 0.88 0.88 0.87 0.67 0.61 0.57 0.62
Qt 0.34 0.28 0.30 0.51 0.71 0.53 0.56 0.61 0.78 0.64 0.64 0.73 0.86 0.74 0.74 0.77 0.51 0.45 0.47 0.52
LibreOffice 0.43 0.40 0.36 0.45 0.60 0.59 0.54 0.59 0.67 0.68 0.67 0.72 0.77 0.85 0.79 0.80 0.54 0.53 0.49 0.68
Average 0.40 0.37 0.32 0.53 0.68 0.66 0.59 0.68 0.76 0.73 0.70 0.78 0.85 0.83 0.82 0.82 0.58 0.55 0.50 0.63
Tab.4  Performance comparison of different submodules and full model
Project Top-1 Top-3 Top-5 Top-10 MRR
Y+ R+ KeyBERT+ Y+ R+ KeyBERT+ Y+ R+ KeyBERT+ Y+ R+ KeyBERT+ Y+ R+ KeyBERT+
Android 0.32 0.27 0.60 0.61 0.68 0.80 0.70 0.75 0.85 0.79 0.87 0.83 0.48 0.47 0.71
OpenStack 0.28 0.29 0.55 0.54 0.52 0.72 0.68 0.65 0.81 0.84 0.81 0.87 0.53 0.53 0.62
Qt 0.19 0.15 0.51 0.37 0.30 0.61 0.49 0.41 0.73 0.64 0.57 0.77 0.34 0.28 0.52
LibreOffice 0.28 0.29 0.45 0.56 0.58 0.59 0.64 0.66 0.72 0.76 0.76 0.80 0.46 0.45 0.68
Average 0.27 0.25 0.53 0.52 0.52 0.68 0.63 0.62 0.78 0.76 0.75 0.88 0.45 0.43 0.63
Tab.5  Comparison of performance in different feature extraction technologies
  
  
  
  
  
1 X, Xia D, Lo X, Wang X Yang . Who should review this change?: Putting text and file location analyses together for more accurate recommendations. In: Proceedings of the 31st IEEE International Conference on Software Maintenance and Evolution. 2015, 261–270
2 F, Shull C Seaman . Inspecting the history of inspections: an example of evidence-based technology diffusion. IEEE Software, 2008, 25( 1): 88–90
3 A, Chueshev J, Lawall R, Bendraou T Ziadi . Expanding the number of reviewers in open-source projects by recommending appropriate developers. In: Proceedings of the 36th IEEE International Conference on Software Maintenance and Evolution. 2020, 499–510
4 P, Thongtanunam C, Tantithamthavorn R G, Kula N, Yoshida H, Iida K I Matsumoto . Who should review my code? A file location-based code-reviewer recommendation approach for modern code review. In: Proceedings of the 22nd International Conference on Software Analysis, Evolution, and Reengineering. 2015, 141–150
5 Goto I, Tanaka H. Detecting untranslated content for neural machine translation. In: Proceedings of the 1st Workshop on Neural Machine Translation. 2017, 47–55
6 F Stahlberg . Neural machine translation: a review. Journal of Artificial Intelligence Research, 2020, 69: 343–418
7 A, Vaswani N, Shazeer N, Parmar J, Uszkoreit L, Jones A N, Gomez Ł, Kaiser I Polosukhin . Attention is all you need. In: Proceedings of the 31st International Conference on Neural Information Processing Systems. 2017, 6000–6010
8 Yu Y, Wang H, Yin G, Wang T. Reviewer recommendation for pull-requests in GitHub: what can we learn from code review and bug assignment? Information and Software Technology, 2016, 74: 204−218
9 M B, Zanjani H, Kagdi C Bird . Automatically recommending peer reviewers in modern code review. Transactions on Software Engineering, 2016, 42( 6): 530–543
10 A, Bacchelli C Bird . Expectations, outcomes, and challenges of modern code review. In: Proceedings of the 35th International Conference on Software Engineering. 2013, 712–721
11 P C, Rigby M A Storey . Understanding broadcast based peer review on open source software projects. In: Proceedings of the 33rd International Conference on Software Engineering. 2011, 541–550
12 Y, Yu H, Wang V, Filkov P, Devanbu B Vasilescu . Wait for it: determinants of pull request evaluation latency on GitHub. In: Proceedings of the 12th Working Conference on Mining Software Repositories. 2015, 367–371
13 Y, Tymchuk A, Mocci M Lanza . Code review: veni, ViDI, vici. In: Proceedings of the 22nd International Conference on Software Analysis, Evolution, and Reengineering. 2015, 151–160
14 X, Qiu T, Sun Y, Xu Y, Shao N, Dai X Huang . Pre-trained models for natural language processing: a survey. Science China Technological Sciences, 2020, 63( 10): 1872–1897
15 V, Lavrenko W B Croft . Relevance-based language models. ACM SIGIR Forum, 2017, 51( 2): 260–267
16 T, Mikolov I, Sutskever K, Chen G, Corrado J Dean . Distributed representations of words and phrases and their compositionality. In: Proceedings of the 26th International Conference on Neural Information Processing Systems. 2013, 3111–3119
17 J, Pennington R, Socher C Manning . GloVe: global vectors for word representation. In: Proceedings of the 19th Conference on Empirical Methods in Natural Language Processing. 2014, 1532–1543
18 L R, Cherney R C, Kaye J B, Lee Vuuren S van . Impact of personal relevance on acquisition and generalization of script training for aphasia: a preliminary analysis. American Journal of Speech-Language Pathology, 2015, 24( 4): S913–S922
19 N, Lourie Bras R, Le C, Bhagavatula Y Choi . UNICORN on RAINBOW: a universal commonsense reasoning model on a new multitask benchmark. In: Proceedings of the 35th AAAI Conference on Artificial Intelligence. 2021, 13480–13488
20 Z, Yang Z, Dai Y, Yang J, Carbonell R, Salakhutdinov Q V Le . XLNet: generalized autoregressive pretraining for language understanding. In: Proceedings of the 33rd International Conference on Neural Information Processing Systems. 2019, 517
21 Z, Liu W, Lin Y, Shi J Zhao . A robustly optimized BERT pre-training approach with post-training. In: Proceedings of the 20th China National Conference on Chinese Computational Linguistics. 2021, 471–484
22 Chi P H, Chung P H, Wu T H, Hsieh C C, Chen Y H, Li S W, Lee H Y. Audio albert: a lite bert for self-supervised learning of audio representation. In: Proceedings of 2021 IEEE Spoken Language Technology Workshop. 2021, 344−350
23 T B, Brown B, Mann N, Ryder M, Subbiah J, Kaplan P, Dhariwal A, Neelakantan P, Shyam G, Sastry A, Askell S, Agarwal A, Herbert-Voss G, Krueger T, Henighan R, Child A, Ramesh D M, Ziegler J, Wu C, Winter C, Hesse M, Chen E, Sigler M, Litwin S, Gray B, Chess J, Clark C, Berner S, McCandlish A, Radford I, Sutskever D Amodei . Language models are few-shot learners. In: Proceedings of the 34th International Conference on Neural Information Processing Systems. 2020, 159
24 C, Hannebauer M, Patalas S, Stünkel V Gruhn . Automatically recommending code reviewers based on their expertise: an empirical comparison. In: Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering. 2016, 99−110
25 E, Doğan E, Tüzün K A, Tecimer H A Güvenir . Investigating the validity of ground truth in code reviewer recommendation studies. In: Proceedings of the 13th International Symposium on Empirical Software Engineering and Measurement. 2019, 1−6
26 X, Ye Y, Zheng W, Aljedaani M W Mkaouer . Recommending pull request reviewers based on code changes. Soft Computing, 2021, 25( 7): 5619–5632
27 M, Fejzer P, Przymus K Stencel . Profile based recommendation of code reviewers. Journal of Intelligent Information Systems, 2018, 50( 3): 597–619
28 X Ye . Learning to rank reviewers for pull requests. IEEE Access, 2019, 7: 85382–85391
29 N, Firoozeh A, Nazarenko F, Alizon B Daille . Keyword extraction: issues and methods. Natural Language Engineering, 2020, 26( 3): 259–291
30 J, Piskorski N, Stefanovitch G, Jacquet A Podavini . Exploring linguistically-lightweight keyword extraction techniques for indexing news articles in a multilingual set-up. In: Proceedings of the 16th EACL Hackashop on News Media Content Analysis and Automated Report Generation. 2021, 35−44
31 F, Weninger J, Geiger M, Wöllmer B, Schuller G Rigoll . Feature enhancement by deep LSTM networks for ASR in reverberant multisource environments. Computer Speech & Language, 2014, 28( 4): 888–902
32 N, Jiang T, Lutellier L Tan . CURE: code-aware neural machine translation for automatic program repair. In: Proceedings of the 43rd International Conference on Software Engineering. 2021, 1161−1173
33 A, Ouni R G, Kula K Inoue . Search-based peer reviewers recommendation in modern code review. In: Proceedings of the 32nd International Conference on Software Maintenance and Evolution. 2016, 367−377
34 M, Chouchen A, Ouni M W, Mkaouer R G, Kula K Inoue . WhoReview: a multi-objective search-based approach for code reviewers recommendation in modern code review. Applied Soft Computing, 2021, 100: 106908
35 R, Campos V, Mangaravite A, Pasquali A, Jorge C, Nunes A Jatowt . YAKE! Keyword extraction from single documents using multiple local features. Information Sciences, 2020, 509: 257–289
36 N M, Nasrabadi R A King . Image coding using vector quantization: a review. IEEE Transactions on Communications, 1988, 36( 8): 957–971
37 Lima Junior M L, de D M, Soares A, Plastino L Murta . Automatic assignment of integrators to pull requests: the importance of selecting appropriate attributes. Journal of Systems and Software, 2018, 144: 181–196
38 E, Mirsaeedi P C Rigby . Mitigating turnover with code review recommendation: balancing expertise, workload, and knowledge distribution. In: Proceedings of the 42nd International Conference on Software Engineering. 2020, 1183−1195
39 W H A, Al-Zubaidi P, Thongtanunam H K, Dam C, Tantithamthavorn A Ghose . Workload-aware reviewer recommendation using a multi-objective search-based approach. In: Proceedings of the 16th ACM International Conference on Predictive Models and Data Analytics in Software Engineering. 2020, 21−30
40 V Balachandran . Reducing human effort and improving quality in peer code reviews using automatic static analysis and reviewer recommendation. In: Proceedings of the 35th International Conference on Software Engineering. 2013, 931−940
41 S, Asthana R, Kumar R, Bhagwan C, Bird C, Bansal C, Maddila S, Mehta B Ashok . WhoDo: automating reviewer suggestions at scale. In: Proceedings of the 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 2019, 937−945
42 G, Salton C Buckley . Term-weighting approaches in automatic text retrieval. Information Processing & Management, 1988, 24( 5): 513–523
43 R, Mihalcea P Tarau . TextRank: bringing order into text. In: Proceedings of the 2nd Conference on Empirical Methods in Natural Language Processing. 2004, 404−411
44 G, Erkan D R Radev . LexRank: graph-based lexical centrality as salience in text summarization. Journal of Artificial Intelligence Research, 2004, 22: 457–479
45 T D, Nguyen M T Luong . WINGNUS: keyphrase extraction utilizing document logical structure. In: Proceedings of the 5th International Workshop on Semantic Evaluation. 2010, 166−169
46 S, Rebai A, Amich S, Molaei M, Kessentini R Kazman . Multi-objective code reviewer recommendations: balancing expertise, availability and collaborations. Automated Software Engineering, 2020, 27( 3-4): 301–328
47 H A, Çetin E, Doğan E Tüzün . A review of code reviewer recommendation studies: challenges and future directions. Science of Computer Programming, 2021, 208: 102652
48 Kong D, Chen Q, Bao L, Sun C, Xia X, Li S. Recommending code reviewers for proprietary software projects: a large scale study. In: Proceedings of 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). 2022, 630−640
49 R, Li P, Liang P Avgeriou . Code reviewer recommendation for architecture violations: an exploratory study. In: Proceedings of the 27th International Conference on Evaluation and Assessment in Software Engineering. 2023, 42−51
[1] FCS-23256-OF-DY_suppl_1 Download
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed