Experience report: investigating bug fixes in machine learning frameworks/libraries

doi:10.1007/s11704-020-9441-1

Front. Comput. Sci.

2021, Vol. 15

Issue (6) : 156212 https://doi.org/10.1007/s11704-020-9441-1

RESEARCH ARTICLE

Experience report: investigating bug fixes in machine learning frameworks/libraries

Xiaobing SUN^1,^2,³, Tianchi ZHOU¹, Rongcun WANG⁴(

), Yucong DUAN⁵, Lili BO^1,³, Jianming CHANG^1,³

¹. School of Information Engineering, Yangzhou University, Yangzhou 225100, China
². State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing 210023, China
³. Jiangsu Engineering Research Center of Knowledge Management and Intelligent Service, Yangzhou University Yangzhou 225127, China
⁴. School of Computer Science and Technology, China University of Mining and Technology, Xuzhou 221116, China
⁵. School of Computer Science and Cyberspace Security, Hainan University, Haikou 570228, China

Download: PDF(919 KB)
Export: BibTeX | EndNote | Reference Manager | ProCite | RefWorks

Abstract

Machine learning (ML) techniques and algorithms have been successfully and widely used in various areas including software engineering tasks. Like other software projects, bugs are also common in ML projects and libraries. In order to more deeply understand the features related to bug fixing in ML projects, we conduct an empirical study with 939 bugs from five ML projects by manually examining the bug categories, fixing patterns, fixing scale, fixing duration, and types of maintenance. The results show that (1) there are commonly seven types of bugs in ML programs; (2) twelve fixing patterns are typically used to fix the bugs in ML programs; (3) 68.80% of the patches belong to micro-scale-fix and small-scale-fix; (4) 66.77% of the bugs in ML programs can be fixed within one month; (5) 45.90% of the bug fixes belong to corrective activity from the perspective of software maintenance. Moreover, we perform a questionnaire survey and send them to developers or users of ML projects to validate the results in our empirical study. The results of our empirical study are basically consistent with the feedback from developers. The findings from the empirical study provide useful guidance and insights for developers and users to effectively detect and fix bugs in MLprojects.

Keywords bug fixing machine learning project empirical study questionnaire survey

Corresponding Author(s): Rongcun WANG

Just Accepted Date: 16 July 2020 Issue Date: 30 August 2021

Cite this article:

Xiaobing SUN,Tianchi ZHOU,Rongcun WANG, et al. Experience report: investigating bug fixes in machine learning frameworks/libraries[J]. Front. Comput. Sci., 2021, 15(6): 156212.

URL:

https://academic.hep.com.cn/fcs/EN/10.1007/s11704-020-9441-1
https://academic.hep.com.cn/fcs/EN/Y2021/V15/I6/156212

1	Z L Cui, J Wang. Distributed intelligent control system of the injection molding machine based on arm controller. In: Proceedings of IEEE International Conference on Computer Science and Automation Engineering. 2011, 339–342
2	E Menasalvas, C Gonzalo-Martin. Challenges of Medical Text and Image Processing: Machine Learning Approaches. Switzerland: Springer, Cham, 2016 https://doi.org/10.1007/978-3-319-50478-0_11
3	N Subrahmanya, P Xu, A El-Bakry, C Reynolds. Advanced Machine Learning Methods for Production Data Pattern Recognition. Texas: Society of Petroleum Engineers, 2014 https://doi.org/10.2118/167839-MS
4	L D Raedt, T Guns, S Nijssen. Constraint programming for data mining and machine learning. In: Proceedings of the 24th AAAI Conference on Artificial Intelligence. 2010, 11–15
5	L Wang, X B Sun, J W Wang, Y C Duan, B Li. Construct bug knowledge graph for bug resolution: poster. In: Proceedings of the 39th IEEE/ACM International Conference on Software Engineering. 2017, 189–191 https://doi.org/10.1109/ICSE-C.2017.102
6	X B Sun, B X Li, H Leung, B Li, Y Li. Msr4sm: using topic models to effectively mining software repositories for software maintenance tasks, Information and Software Technology, 2015, 66: 1–12 https://doi.org/10.1016/j.infsof.2015.05.003
7	X B Sun, X Y Liu, B Li, Y C Duan, H Yang, J J Hu. Exploring topic models in software engineering data analysis: a survey. In: Proceedings of IEEE/ACIS International Conference on Software Engineering. 2016, 357–362 https://doi.org/10.1109/SNPD.2016.7515925
8	H Yang, X B Sun, B Li, Y C Duan. DR_PSF: enhancing developer recommendation by leveraging personalized source-code files. In: Proceedings of the 40th IEEE Annual Conference on Computer, Software and Applications. 2016, 239–244 https://doi.org/10.1109/COMPSAC.2016.101
9	X Xia, D Lo, Y Ding, J M Al-Kofahi , T N Nguyen, X Y Wang. Improving automated bug triaging with specialized topic model. IEEE Transactions on Software Engineering, 2017, 43(3): 272–297 https://doi.org/10.1109/TSE.2016.2576454
10	X B Sun, H Yang, X Xia, B Li. Enhancing developer recommendation with supplementary information via mining historical commits. Journal of Systems and Software, 2017, 134: 355–368 https://doi.org/10.1016/j.jss.2017.09.021
11	Q Huang, X Xia, D Lo. Supervised vs unsupervised models: a holistic look at effort-aware just-in-time defect prediction. In: Proceedings of IEEE International Conference on Software Maintenance and Evolution. 2017, 159–170 https://doi.org/10.1109/ICSME.2017.51
12	Y B Yang, Y M Zhou, J P Liu, Y Y Zhao, H M Lu, L Xu, B W Xu, H Leung. Effort-aware just-in-time defect prediction: simple unsupervised models could be better than supervised models. In: Proceedings of ACM Sigsoft International Symposium on Foundations of Software Engineering. 2016, 157–168 https://doi.org/10.1145/2950290.2950353
13	T C Zhou, X B Sun, X Xia, B Li, X Chen. Improving defect prediction with deep forest. Information and Software Technology, 2019, 114: 204–216 https://doi.org/10.1016/j.infsof.2019.07.003
14	X Y Jing, F Wu, X W Dong, F M Qi, B W Xu. Heterogeneous crosscompany defect prediction by unified metric representation and cca-based transfer learning. In: Proceedings of the 10th Joint Meeting on Foundations of Software Engineering. 2015, 496–507 https://doi.org/10.1145/2786805.2786813
15	F Zhang, Q Zheng, Y Zou, A E Hassan. Cross-project defect prediction using a connectivity-based unsupervised classifier. In: Proceedings of IEEE/ACM International Conference on Software Engineering. 2016, 309–320 https://doi.org/10.1145/2884781.2884839
16	X B Sun, X Peng, B Li, B X Li, W Z Wen. IPSETFUL: an iterative process of selecting test cases for effective fault localization by exploring concept lattice of program spectra. Frontiers of Computer Science, 2016, 10(5): 812–831 https://doi.org/10.1007/s11704-016-5226-y
17	Z G Xu, S Q Ma, X Y Zhang, S F Zhu, B W Xu. Debugging with intelligence via probabilistic inference. In: Proceedings of the 40th International Conference on Software Engineering. 2018, 1171–1181 https://doi.org/10.1145/3180155.3180237
18	T Chappelly, C Cifuentes, P Krishnan, S Gevay. Machine learning for finding bugs: an initial report. In: Proceedings of IEEE Workshop on Machine Learning Techniques for Software Quality Evaluation. 2017, 21–26 https://doi.org/10.1109/MALTESQUE.2017.7882012
19	J Helming, H Arndt, Z Hodaie, M Koegel, N Narayan. Automatic assignment of work items. In: Proceedings of International Conference on Evaluation of Novel Approaches to Software Engineering. 2010, 236–250 https://doi.org/10.1007/978-3-642-23391-3_17
20	C Liu, J Q Yang, L Tan, M Hafiz. R2fix: automatically generating bug fixes from bug reports. In: Proceedings of IEEE International Conference on Software Testing, Verification and Validation. 2013, 282–291 https://doi.org/10.1109/ICST.2013.24
21	F Thung, D Lo, L X Jiang. Automatic recovery of root causes from bugfixing changes. In: Proceedings of Working Conference on Reverse Engineering. 2013, 92–101 https://doi.org/10.1109/WCRE.2013.6671284
22	J Anvik, G C Murphy. Reducing the effort of bug report triage: recommenders for development-oriented decisions. ACM Transactions on Software Engineering and Methodology, 2011, 20(3): 1–35 https://doi.org/10.1145/2000791.2000794
23	V J Hellendoorn, P T Devanbu. Are deep neural networks the best choice for modeling source code? In: Proceedings of the 11th Joint Meeting on Foundations of Software Engineering. 2017, 763–773 https://doi.org/10.1145/3106237.3106290
24	G Yang, T Zhang, B Lee. Utilizing a multi-developer network-based developer recommendation algorithm to fix bugs effectively. In: Proceedings of ACM Symposium on Applied Computing. 2014, 1134–1139 https://doi.org/10.1145/2554850.2555008
25	X D Gu, H Y Zhang, S H Kim. Deep code search. In: Proceedings of the 40th International Conference on Software Engineering. 2018, 933–944 https://doi.org/10.1145/3180155.3180167
26	A Reungsinkonkarn, P Apirukvorapinit. Bug detection using particle swarm optimization with search space reduction. In: Proceedings of International Conference on Intelligent Systems,Modelling and Simulation. 2015, 53–57 https://doi.org/10.1109/ISMS.2015.20
27	H Liu, Z F Xu, Y Z Zou. Deep learning based feature envy detection. In: Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering. 2018, 385–396 https://doi.org/10.1145/3238147.3238166
28	X Hu, G Li, X Xia, D Lo, Z Jin. Deep code comment generation. In: Proceedings of the 26th Conference on Program Comprehension. 2018, 200–210 https://doi.org/10.1145/3196321.3196334
29	Z Ni, B Li, X B Sun, T H Chen, B Tang, X C Shi. Analyzing bug fix for automatic bug cause classification. Journal of Systems and Software, 2020, 163: 110538 https://doi.org/10.1016/j.jss.2020.110538
30	J Guo, J H Cheng, J Cleland-Huang. Semantically enhanced software traceability using deep learning techniques. In: Proceedings of the 39th International Conference on Software Engineering. 2017, 3–14 https://doi.org/10.1109/ICSE.2017.9
31	Z X Liu, X Xia, A E Hassan, D Lo, Z C Xing, X Y Wang. Neural-machinetranslation-based commit message generation: how far are we? In: Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering. 2018, 373–384 https://doi.org/10.1145/3238147.3238190
32	J J Hu, X B Sun, D Lo, B Li. Modeling the evolution of development topics using dynamic topic models. In: Proceedings of IEEE International Conference on Software Analysis, Evolution and Reengineering. 2015, 3–12 https://doi.org/10.1109/SANER.2015.7081810
33	Y C Sun, M Wu, W J Ruan, X W Huang, M Kwiatkowska, D Kroening. Concolic testing for deep neural networks. In: Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering. 2018, 109–119 https://doi.org/10.1145/3238147.3238172
34	L Ma, Y Liu, J J Zhao, Y D Wang, F J Xu, F Y Zhang, J Y Sun, M H Xue, B Li, C Y Chen, T Su, L Li. Deepgauge: multi-granularity testing criteria for deep learning systems. In: Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering. 2018, 120–131 https://doi.org/10.1145/3238147.3238202
35	F Thung, S W Wang, D Lo, L X Jiang. An empirical study of bugs in machine learning systems. In: Proceedings of IEEE International Symposium on Software Reliability Engineering. 2012, 271–280 https://doi.org/10.1109/ISSRE.2012.22
36	Y H Zhang, Y F Chen, S C Cheung, Y F Xiong, L Zhang. An empirical study on tensorflow program bugs. In: Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis. 2018, 129–140 https://doi.org/10.1145/3213846.3213866
37	X B Sun, T C Zhou, G J Li, J J Hu, H Yang, B Li. An empirical study on real bugs for machine learning programs. In: Proceedings of Asia-Pacific Software Engineering Conference. 2017, 348–357 https://doi.org/10.1109/APSEC.2017.41
38	L L Wang, B X Li, H Leung. A new method to encode calling contexts with recursions. Science China Information Sciences, 2016, 59(5): 60–74 https://doi.org/10.1007/s11432-015-5418-3
39	B X Li, L L Wang, H Leung, F Liu. Profiling all paths: a new profiling technique for both cyclic and acyclic paths. Journal of Systems and Software, 2012, 85(7): 1558–1576 https://doi.org/10.1016/j.jss.2012.01.046
40	S Danicic, M R Laurence. Static backward slicing of non-deterministic programs and systems. ACM Transactions on Programming Language and Systems, 2018, 40(3): 11 https://doi.org/10.1145/2886098
41	E Ufuktepe, T Tuglular. A program slicing-based bayesian network model for change impact analysis. In: Proceedings of IEEE International Conference on Software Quality, Reliability and Security. 2018, 490–499 https://doi.org/10.1109/QRS.2018.00062
42	S Roy, A Pandey, B Dolan-Gavitt, Y Hu. Bug synthesis: challenging bugfinding tools with deep faults. In: Proceedings of the 2018 ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 2018, 224–234 https://doi.org/10.1145/3236024.3236084
43	D W Cheng, C Cao, C Xu, X X Ma. Manifesting bugs in machine learning code: an explorative study with mutation testing. In: Proceedings of IEEE International Conference on Software Quality, Reliability and Security. 2018, 313–324 https://doi.org/10.1109/QRS.2018.00044
44	P Bian, B Liang, W C Shi , J J Huang, Y Cai. Nar-miner: discovering negative association rules from code for bug detection. In: Proceedings of ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 2018, 411–422 https://doi.org/10.1145/3236024.3236032
45	C P Wong, J Meinicke, C Kästner. Beyond testing configurable systems: applying variational execution to automatic program repair and higher order mutation testing. In: Proceedings of ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 2018, 749–753 https://doi.org/10.1145/3236024.3264837
46	H Zhong, Z D Su. An empirical study on real bug fixes. In: Proceedings of the 37th IEEE/ACM International Conference on Software Engineering. 2015, 913–923 https://doi.org/10.1109/ICSE.2015.101
47	A Roychoudhury, Y F Xiong. Automated program repair: a step towards software automation. Science China Information Science, 2019, 62(10): 47–49 https://doi.org/10.1007/s11432-019-9947-6
48	C L Goues, M Pradel, A Roychoudhury. Automated program repair. Communications of the ACM, 2019, 62(12): 56–65 https://doi.org/10.1145/3318162
49	Y Yuan, W Banzhaf. Toward better evolutionary program repair: an integrated approach. ACM Transactions Software Engineering and Methodology, 2020, 29(1): 5 https://doi.org/10.1145/3360004
50	J J Jiang, Y F Xiong, X Xia. A manual inspection of defects4J bugs and its implications for automatic program repair. SCIENCE China Informaiton Sciences, 2019, 62(10): 200102 https://doi.org/10.1007/s11432-018-1465-6
51	N Chapin, J E Hale, M D Khan, J F Ramil, W G Tan. Types of software evolution and software maintenance. Journal of Software Maintenance, 2001, 13(1): 3–30 https://doi.org/10.1002/smr.220
52	L Tan, C Liu, ZM Li, X H Wang, Y Y Zhou, C X Zhai. Bug characteristics in open source software. Empirical Software Engineering, 2014, 19(6): 1665–1705 https://doi.org/10.1007/s10664-013-9258-8
53	X L Kong, L M Zhang, E Wong, B X Li. The impacts of techniques, programs and tests on automated program repair: an empirical study. Journal of Systems and Software, 2018, 137: 480–496 https://doi.org/10.1016/j.jss.2017.06.039
54	M Monperrus. Automatic software repair: a bibliography. ACM Computer Survey, 2018, 51(1): 17 https://doi.org/10.1145/3105906
55	J Witschey, O A Zielinska, A K Welk, E R Murphy-Hill, C B Mayhorn, T Zimmermann. Quantifying developers’ adoption of security tools. In: Proceedings of the 10th Joint Meeting on Foundations of Software Engineering. 2015, 260–271 https://doi.org/10.1145/2786805.2786816
56	L Lucia, F Thung, D Lo, L X Jiang. Are faults localizable? In: Proceedings of the 9th IEEE Working Conference on Mining Software Repositories. 2012, 74–77 https://doi.org/10.1109/MSR.2012.6224302
57	M Tufano, C Watson, G Bavota, M Di Penta, M White, D Poshyvanyk. An empirical investigation into learning bug-fixing patches in the wild via neural machine translation. In: Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering. 2018, 832–837 https://doi.org/10.1145/3238147.3240732
58	M M Lehman. On understanding laws, evolution, and conservation in the large-program life cycle. Journal of Systems and Software, 1980, 1: 213–221 https://doi.org/10.1016/0164-1212(79)90022-0
59	M M Lehman, J F Ramil. Software evolution and software evolution processes. Automated Software Engineering, 2002, 14(1–4): 275–309 https://doi.org/10.1023/A:1020557525901
60	J M Zhang, M Harman, L Ma, Y Liu. Machine learning testing: survey, landscapes and horizons. IEEE Transactions on Software Engineering, 2020 https://doi.org/10.1109/TSE.2019.2962027
61	K X Pei, Y Z Cao, J F Yang, S Jana. Deepxplore: automated whitebox testing of deep learning systems. In: Proceedings of the 26th Symposium on Operating Systems Principles. 2017, 1–18 https://doi.org/10.1145/3132747.3132785
62	N Papernot, P D McDaniel, I J Goodfellow, S Jha, Z B Celik, A Swami. Practical black-box attacks against deep learning systems using adversarial examples. 2016, arXiv preprint arXiv:1602.02697 https://doi.org/10.1145/3052973.3053009
63	A Dwarakanath, M Ahuja, S Sikand, R M Rao, R P J C Bose, N Dubash, S Podder. Identifying implementation bugs in machine learning based image classifiers using metamorphic testing. In: Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis. 2018, 118–128 https://doi.org/10.1145/3213846.3213858
64	T Y Zhang, C Y Gao, L Ma, MR Lyu , M Kim . An empirical study of common challenges in developing deep learning applications. In: Proceedings of the 30th IEEE International Symposium on Software Reliability Engineering. 2019, 104–115 https://doi.org/10.1109/ISSRE.2019.00020
65	Q Y Guo, S Chen, X F Xie, L Ma, Q Hu, H T Liu, Y Liu, J J Zhao, X H Li. An empirical study towards characterizing deep learning development and deployment across different frameworks and platforms. In: Proceedings of the 34th IEEE/ACM International Conference on Automated Software Engineering. 2019, 810–822 https://doi.org/10.1109/ASE.2019.00080
66	X F Xie, L Ma, H J Wang, Y K Li, Y Liu, X H Li. Diffchaser: detecting disagreements for deep neural networks. In: Proceedings of the 28th International Joint Conference on Artificial Intelligence. 2019, 5772–5778 https://doi.org/10.24963/ijcai.2019/800
67	M Motwani, S Sankaranarayanan, R Just, Y Brun. Do automated program repair techniques repair hard and important bugs? Empirical Software Engineering, 2018, 23(5): 2901–2947 https://doi.org/10.1007/s10664-017-9550-0
68	S Khatiwada, M Tushev, A Mahmoud. Just enough semantics: an information theoretic approach for ir-based software bug localization. Information & Software Technology, 2018, 93: 45–57 https://doi.org/10.1016/j.infsof.2017.08.012
69	K C Youm, J Ahn, E Lee. Improved bug localization based on code change histories and bug reports. Information and Software Technology, 2017, 82: 177–192 https://doi.org/10.1016/j.infsof.2016.11.002
70	M Wen, J J Chen, R X Wu, D Hao, S C Cheung. Context-aware patch generation for better automated program repair. In: Proceedings of the 40th International Conference on Software Engineering. 2018, 1–11 https://doi.org/10.1145/3180155.3180233
71	M Wen, R X Wu, S C Cheung. Locus: locating bugs from software changes. In: Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering. 2016, 262–273 https://doi.org/10.1145/2970276.2970359
72	S W Wang, D Lo. Amalgam+: composing rich information sources for accurate bug localization. Journal of Software: Evolution and Process, 2016, 28(10): 921–942 https://doi.org/10.1002/smr.1801
73	C Zhou, B Li, X B Sun. Improving software bug-specific named entity recognition with deep neural network. Journal of Systems and Software, 2020, 165: 110572 https://doi.org/10.1016/j.jss.2020.110572
74	C Zhou, B Li, X B Sun, H J Guo. Recognizing software bug-specific named entity in software bug repository. In: Proceedings of the 26th International Conference on Program Comprehension. 2018, 108–119 https://doi.org/10.1145/3196321.3196335
75	J Garcia, Y Feng, J J Shen, S Almanee, Y Xia, Q A Chen. A comprehensive study of autonomous vehicle bugs. In: Proceedings of the 42nd International Conference on Software Engineering. 2020, 385–396 https://doi.org/10.1145/3377811.3380397
76	Y Y Zhao, H Leung, Y B Yang, Y M Zhou, B W Xu. Towards an understanding of change types in bug fixing code. Information and Software Technology, 2017, 86: 37–53 https://doi.org/10.1016/j.infsof.2017.02.003
77	H Zhong, N Meng. Towards reusing hints from past fixes — an exploratory study on thousands of real samples. Empirical Software Engineering, 2018, 23(5): 2521–2549 https://doi.org/10.1007/s10664-017-9584-3
78	E C Campos, M A Maia. Common bug-fix patterns: a large-scale observational study. In: Proceedings of ACM/IEEE International Symposium on Empirical Software Engineering and Measurement. 2017, 404–413 https://doi.org/10.1109/ESEM.2017.55
79	R R Yue, N Meng, Q X Wang. A characterization study of repeated bug fixes. In: Proceedings of IEEE International Conference on Software Maintenance and Evolution. 2017, 422–432 https://doi.org/10.1109/ICSME.2017.16
80	M Soto, F Thung, C P Wong, C L Goues, D Lo. A deeper look into bug fixes: patterns, replacements, deletions, and additions. In: Proceedings of the 13th International Workshop on Mining Software Repositories. 2016, 512–515 https://doi.org/10.1145/2901739.2903495
81	Z Y Wan, D Lo, X Xia, L Cai. Bug characteristics in blockchain systems: a large-scale empirical study. In: Proceedings of the 14th International Conference on Mining Software Repositories. 2017, 413–424 https://doi.org/10.1109/MSR.2017.59
82	X B Sun, X Peng, K Zhang, Y Liu, Y F Cai. How security bugs are fixed and what can be improved: an empirical study with mozilla. Science China Information Sciences, 2019, 62(1): 19102 https://doi.org/10.1007/s11432-017-9459-5
83	H B Braiek, F Khomh. On testing machine learning programs. Journal of Systems and Software, 2020, 164: 110542 https://doi.org/10.1016/j.jss.2020.110542
84	X F Xie, L Ma, F J Xu, M H Xue, H X Chen, Y Liu, J J Zhao, B Li, J X Yin, S See. Deephunter: a coverage-guided fuzz testing framework for deep neural networks. In: Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis. 2019, 146–157 https://doi.org/10.1145/3293882.3330579
85	R Chillarege, I S Bhandari, J K Chaar, M J Halliday. Orthogonal defect classification-a concept for in-process measurements. IEEE Transactions on Software Engineering, 1992, 18(11): 943–956 https://doi.org/10.1109/32.177364
86	Z M Li, L Tan, X H Wang, S Lu, Y Y Zhou, C X Zhai. Have things changed now?: an empirical study of bug characteristics in modern open source software. In: Proceedings of the 1st Workshop on Architectural and System Support for Improving Software Dependability. 2006, 25–33 https://doi.org/10.1145/1181309.1181314
87	X Xia, X Z Zhou, D Lo, X Q Zhao. An empirical study of bugs in software build systems. In: Proceedings of International Conference on Quality Software. 2013, 200–203 https://doi.org/10.1109/QSIC.2013.60
88	M Nayrolles, A Hamou-Lhadj . Towards a classification of bugs to facilitate software maintainability tasks. In: Proceedings of the 1st International Workshop on Software Qualities and Their Dependencies. 2018, 25–32 https://doi.org/10.1145/3194095.3194101
89	J Hernández-González, D Rodríguez, I Inza, R Harrison, J A Lozano. Learning to classify software defects from crowds: a novel approach. Applied Software Computing, 2018, 62: 579–591 https://doi.org/10.1016/j.asoc.2017.10.047
90	M Hamill, K Goseva-Popstojanova. Exploring fault types, detection activities, and failure severity in an evolving safety-critical software system. Software Quality Journal, 2015, 23(2): 229–265 https://doi.org/10.1007/s11219-014-9235-5
91	N Silva, M Vieira. Experience report: orthogonal classification of safety critical issues. In: Proceedings of the 25th IEEE International Symposium on Software Reliability Engineering. 2014, 156–166 https://doi.org/10.1109/ISSRE.2014.25

[1]

Article highlights

Download

[1]	Deheng YANG, Yuhua QI, Xiaoguang MAO, Yan LEI. Evaluating the usage of fault localization in automated program repair: an empirical study[J]. Front. Comput. Sci., 2021, 15(1): 151202-.
[2]	Xiaobing SUN, Hui YANG, Hareton LEUNG, Bin LI, Hanchao (Jerry) LI, Lingzhi LIAO. Effectiveness of exploring historical commits for developer recommendation: an empirical study[J]. Front. Comput. Sci., 2018, 12(3): 528-544.

Viewed

Full text

Abstract

Cited

Shared

Discussed