Please wait a minute...
购物车 申请加入邮件组 English
高级检索 图表检索
Frontiers of Computer Science

ISSN 2095-2228

ISSN 2095-2236(Online)

CN 10-1014/TP

邮发代号 80-970

2019 Impact Factor: 1.275

热点文章  |  下载排行  |  浏览排行
在线投稿 免费RSS订阅
Frontiers of Computer Science  2022, Vol. 16 Issue (4): 164205   https://doi.org/10.1007/s11704-021-0127-0
  本期目录
Handling class imbalance problem in software maintainability prediction: an empirical investigation
Ruchika MALHOTRA1, Kusum LATA1,2()
1. Discipline of Software Engineering, Department of Computer Science & Engineering, Delhi Technological University, Delhi 110042, India
2. University School of Management and Entrepreneurship, East Delhi Campus, Delhi Technological University, Delhi 110095, India
 全文: PDF(8760 KB)   HTML
Abstract：

As the complexity of software systems is increasing; software maintenance is becoming a challenge for software practitioners. The prediction of classes that require high maintainability effort is of utmost necessity to develop cost-effective and high-quality software. In research of software engineering predictive modeling, various software maintainability prediction (SMP) models are evolved to forecast maintainability. To develop a maintainability prediction model, software practitioners may come across situations in which classes or modules requiring high maintainability effort are far less than those requiring low maintainability effort. This condition gives rise to a class imbalance problem (CIP). In this situation, the minority classes’ prediction, i.e., the classes demanding high maintainability effort, is a challenge. Therefore, in this direction, this study investigates three techniques for handling the CIP on ten open-source software to predict software maintainability. This empirical investigation supports the use of resampling with replacement technique (RR) for treating CIP and develop useful models for SMP.
Key words： software maintenance    software maintainability    imbalanced learning
收稿日期: 2020-03-28      出版日期: 2021-12-01
Corresponding Author(s): Kusum LATA   
 引用本文:   
. [J]. Frontiers of Computer Science, 2022, 16(4): 164205.
Ruchika MALHOTRA, Kusum LATA. Handling class imbalance problem in software maintainability prediction: an empirical investigation. Front. Comput. Sci., 2022, 16(4): 164205.
 链接本文:  
https://academic.hep.com.cn/fcs/CN/10.1007/s11704-021-0127-0
https://academic.hep.com.cn/fcs/CN/Y2022/V16/I4/164205
Predicted positive Predicted negative
Actual positive True positive (TPos) False Negative (FNeg)
Actual negative False positive (FPos) True negative (TNeg)
Tab.1  
Software Version #C #CC C%
Betwixt 0.6 – 0.7 290 279 96.20
Ivy 1.4.1 – 2.2.0 619 429 69.30
Bcel 5.0 – 5.1 363 360 99.17
Io 2.0.1 – 2.2 274 272 99.27
Log4j 1.2.14 – 1.2.15 491 437 89.00
Jcs 1.2.6.5 – 1.2.7.9 333 219 65.76
Lang 2.4 – 2.6 434 267 61.52
Ode 1.3.1 – 1.3.2 1060 1004 94.71
Accumulo 1.5.0-1.5.1 1431 828 57.86
Click 2.2-2.3 402 263 65.42
Tab.2  
Dataset No. of outliers
Betwixt 34
Ivy 58
Bcel 26
Io 24
Log4j 32
Jcs 22
Lang 47
Ode 115
Accumulo 90
Click 31
Tab.3  
Dataset #Min. #Maj. IR
Betwixt 27 218 8.07
Ivy 28 343 12.25
Bcel 18 316 17.55
Io 10 238 23.80
Log4j 42 363 8.64
Jcs 27 170 6.29
Lang 27 193 7.14
Ode 57 832 14.59
Accumulo 120 618 5.15
Click 42 190 4.52
Tab.4  
Fig.1  
Dataset Accuracy Sensitivity
NB MLP AB BG LB RF J48 NB MLP AB BG LB RF J48
Bcel 85.03 94.01 94.01 93.41 95.51 94.31 93.41 16.67 22.22 11.11 5.56 22.22 11.11 22.22
Betwixt 83.67 87.35 88.98 87.76 90.72 86.53 85.71 18.52 22.22 0.00 0.00 3.70 7.41 3.70
Io 94.76 96.37 95.97 95.16 98.74 94.38 95.16 40.00 30.00 30.00 0.00 20.00 27.27 0.00
Ivy 90.79 92.72 91.91 91.96 92.72 91.11 91.91 30.77 17.86 3.75 3.57 14.29 10.71 14.29
Jcs 83.76 91.88 87.31 89.34 86.80 88.32 88.32 77.78 62.96 48.15 44.44 40.74 48.15 48.15
Lang 90.00 89.55 90.91 90.91 91.36 90.45 90.91 44.44 51.85 66.67 55.56 55.56 48.15 51.85
Log4j 86.67 90.12 87.16 88.15 90.86 89.63 87.90 47.62 33.33 26.19 9.52 26.19 35.71 23.81
Ode 90.10 93.81 94.15 93.93 94.15 92.46 92.91 36.84 22.81 8.77 10.53 12.28 10.53 22.81
Accumulo 80.89 81.84 83.76 83.06 84.01 81.57 82.93 24.17 20.00 0.00 9.17 7.50 15.83 9.17
Click 77.59 81.03 79.74 81.47 78.88 79.74 79.31 4.76 21.43 0.00 0.00 2.38 19.05 2.38
Dataset Precision Specificity
NB MLP AB BG LB RF J48 NB MLP AB BG LB RF J48
Bcel 07.89 40.00 33.33 16.67 80.00 40.00 33.33 88.92 85.03 98.73 98.42 99.68 99.05 97.47
Betwixt 21.74 37.50 0.00 0.00 14.29 20.00 10.00 91.74 95.41 100 98.62 98.11 96.33 95.87
Io 36.36 60.00 50.00 0.00 40.00 33.33 0.00 97.06 99.16 98.74 99.16 98.74 97.48 99.16
Ivy 33.33 55.56 25.00 25.00 57.14 27.27 40.00 95.34 98.83 99.13 99.13 99.13 97.67 98.25
Jcs 44.68 73.91 54.17 66.67 52.38 59.09 59.09 84.71 96.47 93.53 96.47 94.12 94.71 94.71
Lang 63.16 58.33 62.07 65.22 68.18 65.00 66.67 96.37 94.82 94.30 95.85 96.37 96.37 96.37
Log4j 38.46 53.85 34.38 28.57 64.71 50.00 37.04 91.18 96.69 94.21 97.25 98.35 95.87 95.32
Ode 28.77 54.17 100 66.67 77.78 27.27 40.63 93.75 98.68 100 99.64 99.76 98.08 97.72
Accumulo 36.71 38.71 0.00 40.74 56.25 35.19 39.29 91.91 93.85 100 97.41 98.87 94.34 97.25
Click 14.29 45.00 0.00 0.00 11.11 35.19 0.00 93.68 94.21 97.37 99.47 95.79 93.16 96.32
Dataset AUC G-Mean
NB MLP AB BG LB RF J48 NB MLP AB BG LB RF J48
Bcel 0.57 0.73 0.78 0.75 0.79 0.79 0.62 38.50 46.69 52.37 23.38 47.07 33.17 46.54
Betwixt 0.69 0.65 0.72 0.69 0.68 0.7 0.54 52.11 46.05 0.00 0.00 19.06 26.71 18.84
Io 0.87 0.87 0.87 0.85 0.85 0.79 0.57 62.31 54.54 54.43 0.00 44.44 51.56 0.00
Ivy 0.67 0.66 0.78 0.72 0.74 0.77 0.51 54.16 42.01 18.42 18.82 37.63 32.35 37.46
Jcs 0.86 0.91 0.89 0.87 0.88 0.71 0.71 81.17 77.94 67.11 65.48 61.92 67.53 67.53
Lang 0.83 0.83 0.80 0.86 0.83 0.68 0.83 65.45 70.12 79.29 72.97 73.17 68.12 70.69
Log4j 0.84 0.77 0.78 0.83 0.79 0.77 0.7 65.89 56.77 49.67 30.43 50.75 58.51 47.64
Ode 0.78 0.76 0.77 0.80 0.79 0.75 0.63 58.77 47.44 29.62 32.39 35.00 32.13 47.21
Accumulo 0.62 0.69 0.67 0.69 0.72 0.75 0.54 47.13 43.32 0.00 29.88 27.23 38.65 29.86
Click 0.57 0.64 0.66 0.66 0.67 0.67 0.47 21.12 44.93 0.00 0.00 15.10 42.12 15.14
Tab.5  
Fig.2  
Fig.3  
Fig.4  
Fig.5  
Dataset Classifier RR SM SS Dataset Classifier RR SM SS
NB 61.85 64.84 70.28 NB 52.12 52.81 53.29
MLP 86.00 78.76 68.94 MLP 83.31 82.10 70.37
AB 80.45 78.05 80.51 AB 37.56 77.68 71.62
Bcel BG 93.19 82.59 76.80 Betwixt BG 84.97 84.73 73.98
LB 89.30 75.52 89.24 LB 75.98 84.46 72.20
RF 95.27 75.36 86.78 RF 92.73 76.71 85.15
J48 92.90 88.01 69.92 J48 89.01 85.09 74.07
NB 81.58 87.16 57.45 NB 49.80 62.89 52.49
MLP 92.89 87.74 79.37 MLP 90.19 80.26 67.61
AB 94.10 87.28 79.37 AB 79.97 71.98 63.39
Io BG 93.70 88.26 83.67 Ivy BG 90.41 87.44 64.14
LB 95.71 85.83 73.48 LB 79.09 76.11 63.39
RF 96.11 87.84 73.48 RF 95.82 90.22 41.41
J48 94.91 89.93 77.14 J48 93.97 85.67 69.85
NB 89.57 84.36 85.65 NB 69.50 72.16 74.07
MLP 96.51 91.88 70.27 MLP 91.35 89.15 72.98
AB 87.40 90.91 88.52 AB 88.54 87.50 77.48
Jcs BG 94.02 90.98 78.26 Lang BG 92.33 89.18 67.94
LB 96.10 88.56 86.86 LB 89.77 93.83 82.32
RF 98.02 92.79 83.15 RF 96.21 83.72 87.41
J48 94.48 93.10 83.15 J48 90.98 90.42 73.17
NB 78.35 75.53 72.17 NB 62.61 67.63 68.22
MLP 91.24 85.55 69.01 MLP 86.48 79.37 51.88
AB 73.43 79.36 76.19 AB 72.19 79.47 71.45
Log4j BG 90.50 85.07 72.84 Ode BG 92.33 86.21 75.35
LB 79.51 82.57 80.99 LB 82.71 54.86 35.34
RF 94.08 90.35 76.64 RF 96.05 91.18 70.67
J48 90.35 87.27 78.51 J48 91.66 87.64 70.87
NB 45.16 61.36 50.73 NB 42.78 65.24 31.72
MLP 74.98 73.43 59.41 MLP 81.39 79.22 47.43
AB 45.16 71.55 74.98 AB 68.24 78.90 38.98
Accumulo BG 89.86 85.12 61.18 Click BG 84.61 81.30 34.50
LB 74.63 74.76 67.76 LB 75.00 85.78 46.38
RF 94.65 88.96 64.95 RF 87.35 84.82 46.29
J48 89.37 82.42 59.11 J48 83.99 79.66 44.22
Tab.6  
Dataset Classifier RR SM SS Dataset Classifier RR SM SS
NB 0.78 0.78 0.76 NB 0.77 0.82 0.69
MLP 0.93 0.9 0.77 MLP 0.89 0.89 0.75
AB 0.97 0.91 0.77 AB 0.79 0.87 0.79
Bcel BG 0.97 0.95 0.83 Betwixt BG 0.93 0.93 0.78
LB 0.94 0.93 0.93 LB 0.85 0.92 0.8
RF 0.99 0.96 0.84 RF 0.97 0.95 0.82
J48 0.96 0.9 0.72 J48 0.93 0.88 0.74
NB 0.95 0.97 0.78 NB 0.72 0.75 0.7
MLP 0.94 0.97 0.91 MLP 0.93 0.9 0.74
AB 0.96 0.97 0.88 AB 0.89 0.86 0.8
Io BG 0.98 0.95 0.80 Ivy BG 0.97 0.95 0.78
LB 0.98 0.96 0.89 LB 0.9 0.89 0.81
RF 0.99 0.96 0.89 RF 0.99 0.96 0.78
J48 0.95 0.89 0.76 J48 0.95 0.85 0.77
NB 0.91 0.89 0.89 NB 0.83 0.87 0.83
MLP 0.97 0.95 0.92 MLP 0.95 0.92 0.82
AB 0.95 0.94 0.89 AB 0.95 0.93 0.78
Jcs BG 0.97 0.97 0.86 Lang BG 0.96 0.93 0.83
LB 0.98 0.94 0.91 LB 0.95 0.95 0.82
RF 0.98 0.98 0.89 RF 0.96 0.95 0.82
J48 0.96 0.94 0.89 J48 0.93 0.9 0.69
NB 0.88 0.86 0.82 NB 0.84 0.81 0.83
MLP 0.96 0.93 0.84 MLP 0.91 0.89 0.75
AB 0.86 0.89 0.83 AB 0.87 0.9 0.81
Log4j BG 0.95 0.94 0.82 Ode BG 0.97 0.95 0.81
LB 0.89 0.91 0.82 LB 0.88 0.93 0.81
RF 0.97 0.96 0.84 RF 0.99 0.97 0.9
J48 0.93 0.92 0.79 J48 0.94 0.89 0.74
NB 0.74 0.70 0.713 NB 0.62 0.68 0.51
MLP 0.83 0.80 0.64 MLP 0.87 0.86 0.6
AB 0.83 0.78 0.65 AB 0.73 0.86 0.52
Accumulo BG 0.96 0.92 0.68 Click BG 0.91 0.9 0.54
LB 0.811 0.78 0.70 LB 0.81 0.93 0.59
RF 0.99 0.94 0.70 RF 0.95 0.92 0.95
J48 0.94 0.87 0.64 J48 0.9 0.86 0.57
Tab.7  
Data set (GM) Rank 1 Rank 2 Rank 3 Rank 4 p-value
Apache Bcel RR (3.57) SM (2.57) SS (2.86) No Resample (1.00) 0.002
Apache Betwixt RR (3.29) SM (3.14) SS (2.57) No Resample (1.00) 0.003
Apache IO RR (3.86) SM (3.14) SS (1.86) No Resample (1.14) 0.000
Apache Ivy RR (3.57) SM (3.14) SS (2.00) No Resample (1.29) 0.003
Apache Jcs RR (3.71) SM (3.00) SS (2.14) No Resample (1.14) 0.001
Apache Lang RR (3.57) SM (3.00) SS (2.14) No Resample (1.29) 0.006
Apache Log4j RR (3.43) SM (3.29) SS (2.29) No Resample (1.00) 0.001
Apache Ode RR (3.57) SM (3.14) SS (2.29) No Resample (1.00) 0.001
Apache Accumulo RR (3.14) SM (2.29) SS (2.43) No Resample (1.14) 0.007
Apache Click RR (3.57) SM (3.43) SS (2.00) No Resample (1.00) 0.000
Tab.8  
Data set (AUC) Rank 1 Rank 2 Rank 3 Rank 4 p-value
Apache Bcel RR (3.93) SM (3.00) SS (1.93) No Resample (1.14) 0.000
Apache Betwixt SM (3.57) RR (3.36) SS (2.00) No Resample (1.07) 0.000
Apache IO RR (3.57) SM (3.43) SS (1.71) No Resample (1.29) 0.001
Apache Ivy RR (3.86) SM (3.14) SS (2.00) No Resample (1.00) 0.000
Apache Jcs RR (3.86) SM (3.07) SS (1.86) No Resample (1.21) 0.001
Apache Lang RR (3.64) SM (3.21) No Resample (1.86) SS (1.29) 0.001
Apache Log4j RR (3.71) SM (3.29) SS (1.71) No Resample (1.29) 0.001
Apache Ode RR (3.71) SM (3.14) SS (2.00) No Resample (1.14) 0.001
Apache Accumulo RR (4.00) SM (2.86) No Resample (1.71) SS (1.43) 0.001
Apache Click RR (3.50) SM (3.29) No Resample (1.71) SS (1.50) 0.003
Tab.9  
Pairwise resampling method G-Mean AUC
RR vs. SM Sig+(0.005) Sig+(0.006)
RR vs. SS Sig+(0.00) Sig+(0.00)
Tab.10  
1 A Kaur , K Kaur . Statistical comparison of modelling methods for software maintainability prediction. International Journal of Software Engineering and Knowledge Engineering, 2013, 23( 6): 743– 774
2 L Kumar , S K Rath . Hybrid functional link artificial neural network approach for predicting maintainability of object-oriented software. Journal of Systems and Software, 2016, 121 : 170– 190
3 R Malhotra, K Lata. An exploratory study for predicting maintenance effort using hybridized techniques. In: Proceedings of the 10th Innovations in Software Engineering Conference. 2017, 26– 33
4 L Kumar , D K Naik , S K Rath . Validating the effectiveness of object-oriented metrics for predicting maintainability. Procedia Computer Science, 2015, 57 : 798– 806
5 M O Elish, K O Elish. Application of TreeNet in predicting object-oriented software maintainability: a comparative study. In: Proceeding of the 13th European Conference on Software Maintenance and Reengineering. 2009, 69– 78
6 A Chug , R Malhotra . Benchmarking framework for maintainability prediction of open source software using object oriented metrics. International Journal of Innovative Computing, Information and Control, 2016, 12( 2): 615– 634
7 M M T Thwin , T S Quah . Application of neural networks for software quality prediction using object-oriented metrics. Journal of Systems and Software, 2005, 76( 2): 147– 156
8 C Van Koten , A R Gray . An application of Bayesian network for predicting object-oriented software maintainability. Information and Software Technology, 2006, 48( 1): 59– 67
9 K K Aggarwal , Y Singh , A Kaur , R Malhotra . Application of artificial neural network for predicting maintainability using object-oriented metrics. International Journal of Computer and Information Engineering, 2008, 2( 10): 3552– 3556
10 W Zhang , L Huang , V Ng , J Ge . SMPLearner: learning to predict software maintainability. Automated Software Engineering, 2015, 22( 1): 111– 141
11 L Kumar , S K Rath . Software maintainability prediction using hybrid neural network and fuzzy logic approach with parallel computing concept. International Journal of System Assurance Engineering and Management, 2017, 8( S2): 1487– 1502
12 L Zhou . Performance of corporate bankruptcy prediction models on imbalanced dataset: the effect of sampling methods. Knowledge-Based Systems, 2013, 41 : 16– 25
13 K Napierala , J Stefanowski . Types of minority class examples and their influence on learning classifiers from imbalanced data. Journal of Intelligent Information Systems, 2016, 46( 3): 563– 597
14 O Loyola-González , J F Martínez-Trinidad , J A Carrasco-Ochoa , M García-Borroto . Study of the impact of resampling methods for contrast pattern based classifiers in imbalanced databases. Neurocomputing, 2016, 175 : 935– 947
15 H Yu , C Mu , C Sun , W Yang , X Yang , X Zuo . Support vector machine-based optimized decision threshold adjustment strategy for classifying imbalanced data. Knowledge-Based Systems, 2015, 76 : 67– 78
16 F Cheng , J Zhang , C Wen . Cost-sensitive large margin distribution machine for classification of imbalanced data. Pattern Recognition Letters, 2016, 80 : 107– 112
17 S Wang , X Yao . Using class imbalance learning for software defect prediction. IEEE Transactions on Reliability, 2013, 62( 2): 434– 443
18 T M Khoshgoftaar, K Gao, N Seliya. Attribute selection and imbalanced data: problems in software defect prediction. In: Proceedings of the 22nd IEEE International Conference on Tools with Artificial Intelligence. 2010, 137– 144
19 M Tan, L Tan, S Dara, C Mayeux. Online defect prediction for imbalanced data. In: Proceedings of the 37th IEEE International Conference on Software Engineering. 2015, 99– 108
20 L Pelayo, S Dick. Applying novel resampling strategies to software defect prediction. In: Proceedings of 2007 Annual Meeting of the North American Fuzzy Information Processing Society. 2007, 69– 72
21 Z Sun , Q Song , X Zhu . Using coding-based ensemble learning to improve software defect prediction. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 2012, 42( 6): 1806– 1817
22 M J Siers , M Z Islam . Software defect prediction using a cost sensitive decision forest and voting, and a potential solution to the class imbalance problem. Information Systems, 2015, 51 : 62– 71
23 I H Laradji , M Alshayeb , L Ghouti . Software defect prediction using ensemble learning on selected features. Information and Software Technology, 2015, 58 : 388– 402
24 J Zheng . Cost-sensitive boosting neural networks for software defect prediction. Expert Systems with Applications, 2010, 37( 6): 4537– 4543
25 Choeikiwong T, Vateekul P. Software defect prediction in imbalanced data sets using unbiased support vector machine. In: Kim K J. ed. Information Science and Applications. Berlin, Heidelberg: Springer, 2015, 923−931
26 R Malhotra , M Khanna . An empirical study for software change prediction using imbalanced data. Empirical Software Engineering, 2017, 22( 6): 2806– 2851
27 E Giger, M Pinzger, H C Gall. Can we predict types of code changes? An empirical analysis. In: Proceedings of the 9th IEEE Working Conference on Mining Software Repositories. 2012, 217– 226
28 Y Singh , A Kaur , R Malhotra . Empirical validation of object-oriented metrics for predicting fault proneness models. Software Quality Journal, 2010, 18( 1): 3– 35
29 R C Martin. Agile Software Development: Principles, Patterns, and Practices. Upper Saddle River: Prentice Hall, 2002
30 B Henderson-Sellers. Object-Oriented Metrics: Measures of Complexity. Upper Saddle River: Prentice Hall, 1995
31 H He , E A Garcia . Learning from imbalanced data. IEEE Transactions on Knowledge and Data Engineering, 2009, 21( 9): 1263– 1284
32 K Gao, T M Khoshgoftaar, A Napolitano. Combining feature subset selection and data sampling for coping with highly imbalanced software data. In: Proceedings of the 27th International Conference on Software Engineering and Knowledge Engineering. 2015, 439– 444
33 S Lessmann , B Baesens , C Mues , S Pietsch . Benchmarking classification models for software defect prediction: a proposed framework and novel findings. IEEE Transactions on Software Engineering, 2008, 34( 4): 485– 496
34 R Malhotra, N Pritam, K Nagpal, P Upmanyu. Defect collection and reporting system for Git based open source software. In: Proceedings of 2014 International Conference on Data Mining and Intelligent Computing. 2014, 1– 7
35 N V Chawla , K W Bowyer , L O Hall , W P Kegelmeyer . SMOTE: synthetic minority over-sampling technique. Journal of Artificial Intelligence Research, 2002, 16 : 321– 357
36 J Han, J Pei, M Kamber. Data Mining: Concepts and Techniques. 3rd ed. New York: Elsevier, 2011
37 S Haykin. Neural Networks: A Comprehensive Foundation. Upper Saddle River: Prentice Hall, 1994
38 L Breiman . Random forests. Machine Learning, 2001, 45( 1): 5– 32
39 I H Witten, E Frank. Data Mining: Practical Machine Learning Tools and Techniques. 2nd ed. San Francisco: Morgan Kaufmann, 2005
40 L Breiman . Bagging predictors. Machine Learning, 1996, 24( 2): 123– 140
41 J Friedman , T Hastie , R Tibshirani . Additive logistic regression: a statistical view of boosting (with discussion and a rejoinder by the authors). The Annals of Statistics, 2000, 28( 2): 337– 407
42 Y Ma , G Luo , X Zeng , A Chen . Transfer learning for cross-company software defect prediction. Information and Software Technology, 2012, 54( 3): 248– 256
43 H A Nguyen , T T Nguyen , N H Pham , J Al-Kofahi , T N Nguyen . Clone management for evolving software. IEEE Transactions on Software Engineering, 2012, 38( 5): 1008– 1026
44 E Duala-Ekoko, M P Robillard. Tracking code clones in evolving software. In: Proceedings of the 29th International Conference on Software Engineering. 2007, 158– 167
45 M De Wit, A Zaidman, A Van Deursen. Managing code clones using dynamic change tracking and resolution. In: Proceedings of 2009 IEEE International Conference on Software Maintenance. 2009, 169– 178
[1] Highlights Download
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed