Please wait a minute...
Shopping Cart Email List Chinese
Advanced Search Fig/Tab
Frontiers of Computer Science

ISSN 2095-2228

ISSN 2095-2236(Online)

CN 10-1014/TP

Postal Subscription Code 80-970

2018 Impact Factor: 1.129

Featured Articles  |  Most Downloaded  |  Most Reading
Online Submission Subscribe to Our RSS Content Feed
Front. Comput. Sci.    2023, Vol. 17 Issue (5) : 175212    https://doi.org/10.1007/s11704-022-1658-8
RESEARCH ARTICLE
Iterative Android automated testing
Yi ZHONG1,2, Mengyu SHI1, Youran XU1, Chunrong FANG1, Zhenyu CHEN1()
1. State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing 210093, China
2. School of Big Data and Computer Science, Chongqing College of Mobile Communication, Chongqing 400065, China
 Download: PDF(9737 KB)   HTML
 Export: BibTeX | EndNote | Reference Manager | ProCite | RefWorks
Abstract

With the benefits of reducing time and workforce, automated testing has been widely used for the quality assurance of mobile applications (APPs). Compared with automated testing, manual testing can achieve higher coverage in complex interactive Activities. And the effectiveness of manual testing is highly dependent on the user operation process (UOP) of experienced testers. Based on the UOP, we propose an iterative Android automated testing (IAAT) method that automatically records, extracts, and integrates UOPs to guide the test logic of the tool across the complex Activity iteratively. The feedback test results can train the UOPs to achieve higher coverage in each iteration. We extracted 50 UOPs and conducted experiments on 10 popular mobile APPs to demonstrate IAAT’s effectiveness compared with Monkey and the initial automated tests. The experimental results show a noticeable improvement in the IAAT compared with the test logic without human knowledge. Under the 60 minutes test time, the average code coverage is improved by 13.98% to 37.83%, higher than the 27.48% of Monkey under the same conditions.
Keywords quality assurance      automated testing      UOP      test coverage     
Corresponding Author(s): Zhenyu CHEN   
Just Accepted Date: 26 July 2022   Issue Date: 15 December 2022
 Cite this article:   
Yi ZHONG,Mengyu SHI,Youran XU, et al. Iterative Android automated testing[J]. Front. Comput. Sci., 2023, 17(5): 175212.
 URL:  
https://academic.hep.com.cn/fcs/EN/10.1007/s11704-022-1658-8
https://academic.hep.com.cn/fcs/EN/Y2023/V17/I5/175212
Fig.1  Verification example. (a) is a validation screen requiring the shortest path to the specified colour area, and (b) is a successful validation screen
Fig.2  IAAT (Iterative Android automated testing) overview. UOP is the user operation process with human knowledge, where the user consists of two types: application-specific scripts written manually across specific screens; playback system LIRAT records. The Activity overlay tree is a fused and formatted Activity tree based on the widget information to store non-duplicated Activity information in UOPs, with nodes in a specific triples format
  
Fig.3  Activity overlay tree information during automated test tool running. Tree 1 is the Activity overlay tree obtained through automated tests; tree 2 is the Activity overlay tree with human knowledge; tree 3 and tree 4 are the Activity overlay tree after iterating through automated tests guided by UOP information
  
Fig.4  Iterative automated testing process
Number APP name Category Version Downloads
A1 Wikipedia Internet v2.7.50305 10 million+
A2 Monkey Development v1.0.0 5 million+
A3 Jamendo Media v1.0.4 500000+
A4 VLC Media v3.2.7 100 million+
A5 Matomo Development v2.4.5 50000+
A6 openHAB Internet v2.10.3 100000+
A7 OsmAnd Map v3.5.5 5 million+
A8 Linphone Phone+SNS v4.2 500000+
A9 AdGuard System v3.3 5 million+
A10 Kixi Internet v3.1.0 500000+
Tab.1  APP information
Subject Total number of pages Monkey Simple traversal Iterative automated test system
AC/% CC/% AC/% CC/% SC/% AC(v.s.highest)/% CC(v.s.highest)/%
A1 32 28.12 42.27 37.5 42.00 31.25 43.75(+6.25) 48.19(+5.92)
A2 5 60.00 42.91 60.00 38.74 80.00 80.00(+0) 62.02(+19.11)
A3 14 35.71 23.59 35.71 35.34 78.57 78.57(+0) 52.23(+16.89)
A4 40 12.50 10.09 2.5 4.82 12.50 12.50(+0) 11.59(+1.5)
A5 4 50.00 / 50.00 / 50.00 50.00(+0) /
A6 11 27.27 30.01 27.27 28.52 36.36 36.36(+0) 36.24(+6.23)
A7 41 17.07 / 2.43 / 7.3 24.39(+7.32) /
A8 23 43.47 14.91 30.43 10.22 34.78 60.86(+17.30) 16.92(+2.01)
A9 53 43.47 / 1.88 / 20.75 24.52(?18.95) /
A10 10 70.00 28.55 30.00 7.32 80.00 90.00(+10) 37.60(+9.05)
Average 38.76 27.48 27.77 23.85 43.15 50.07(+6.92) 37.83(+10.35)
Tab.2  Experimental results by Simple traversal, Monkey and IAAT
Fig.5  Faults detection distribution
Fig.6  Code coverage comparison
Fig.7  Widgets overlay flowchart
Fig.8  Experimental results comparison. (a) is the code coverage comparison between simple traversal, Monkey and IAAT for open source applications; (b) shows the results of simple traversal, Monkey and IAAT for Activity coverage at the control level; (c) shows the average test results under code coverage and Activity coverage for the three tests
  
  
  
  
  
1 F, Pecorelli G, Catolino F, Ferrucci Lucia A, De F Palomba . Software testing and Android applications: a large-scale empirical study. Empirical Software Engineering, 2022, 27( 2): 31
2 C, Peng A, Rajan T Cai . CAT: change-focused android GUI testing. In: Proceedings of 2021 IEEE International Conference on Software Maintenance and Evolution (ICSME). 2021, 460–470
3 N, Salehnamadi A, Alshayban J W, Lin I, Ahmed S, Branham S Malek . Latte: use-case and assistive-service driven automated accessibility testing framework for android. In: Proceedings of 2021 CHI Conference on Human Factors in Computing Systems. 2021, 274
4 W, Ravelo-Méndez C, Escobar-Velásquez M Linares-Vásquez . Kraken: a framework for enabling multi-device interaction-based testing of Android APPs. Science of Computer Programming, 2021, 206: 102627
5 M J, Noh K T Lee . An analysis of the relationship between quality and user acceptance in smartphone APPs. Information Systems and e-Business Management, 2016, 14( 2): 273–291
6 S, Sun X, Fu H, Ruan X, Du B, Luo M Guizani . Real-time behavior analysis and identification for android application. IEEE Access, 2018, 6: 38041–38051
7 D, Amalfitano A R, Fasolino P, Tramontana Carmine S, De A M Memon . Using GUI ripping for automated testing of Android applications. In: Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering. 2012, 258–261
8 R, Huang Q, Zhang D, Towey W, Sun J Chen . Regression test case prioritization by code combinations coverage. Journal of Systems and Software, 2020, 169: 110712
9 G, Cai Q, Su Z Hu . Automated test case generation for path coverage by using grey prediction evolution algorithm with improved scatter search strategy. Engineering Applications of Artificial Intelligence, 2021, 106: 104454
10 Z, Liu C, Chen J, Wang Y, Huang J, Hu Q Wang . Guided bug crush: assist manual GUI testing of android APPs via hint moves. In: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems. 2022, 557
11 H N, Yasin Hamid S H, Ab R J R Yusof . DroidbotX: test case generation tool for android applications using Q-learning. Symmetry, 2021, 13( 2): 310
12 L, Li T F, Bissyandé M, Papadakis S, Rasthofer A, Bartel D, Octeau J, Klein L Traon . Static analysis of android APPs: a systematic literature review. Information and Software Technology, 2017, 88: 67–95
13 P, Kong L, Li J, Gao K, Liu T F, Bissyandé J Klein . Automated testing of android APPs: a systematic literature review. IEEE Transactions on Reliability, 2019, 68( 1): 45–66
14 A, Méndez-Porras C, Quesada-López M Jenkins . Automated testing of mobile applications: a systematic map and review. In: Proceedings of the XVIII IberoAmerican Conference on Software Engineering. 2015, 195
15 A, Pilgun O, Gadyatskaya Y, Zhauniarovich S, Dashevskyi A, Kushniarou S Mauw . Fine-grained code coverage measurement in automated black-box android testing. ACM Transactions on Software Engineering and Methodology, 2020, 29( 4): 23
16 S Liu . Improvement and implementation of android Robotium automated testing framework system. Southeast University, Dissertation, 2017
17 Z Geng . Study and improvement of android automatic testing. Beijing University of Posts and Telecommunications, Dissertation, 2017
18 S R, Choudhary A, Gorla A Orso . Automated test input generation for android: are we there yet? (E). In: Proceedings of the 30th IEEE/ACM International Conference on Automated Software Engineering. 2015, 429–440
19 N, Mirzaei J, Garcia H, Bagheri A, Sadeghi S Malek . Reducing combinatorics in GUI testing of android applications. In: Proceedings of the 38th IEEE/ACM International Conference on Software Engineering. 2016, 559–570
20 Y, Hu I, Neamtiu A Alavi . Automatically verifying and reproducing event-based races in Android APPs. In: Proceedings of the 25th International Symposium on Software Testing and Analysis. 2016, 377–388
21 L, Clapp O, Bastani S, Anand A Aiken . Minimizing GUI event traces. In: Proceedings of the 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering. 2016, 422–434
22 H, Heiskanen M, Maunumaa M Katara . A test process improvement model for automated test generation. In: Proceedings of the 13th International Conference on Product-Focused Software Process Improvement. 2012, 17–31
23 S, Yu C, Fang Y, Feng W, Zhao Z Chen . LIRAT: layout and image recognition driving automated mobile testing of cross-platform. In: Proceedings of the 34th IEEE/ACM International Conference on Automated Software Engineering. 2019, 1066–1069
24 G, Grano A, Ciurumelea S, Panichella F, Palomba H C Gall . Exploring the integration of user feedback in automated testing of Android applications. In: Proceedings of the 25th IEEE International Conference on Software Analysis, Evolution and Reengineering. 2018, 72–83
25 Y, Gu J L Shi . Generality for Technology of Software Testing. Beijing: Tsinghua University Press, 2004
26 R, Mahmood N, Mirzaei S Malek . EvoDroid: segmented evolutionary testing of Android APPs. In: Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering. 2014, 599–609
27 T, Su G, Meng Y, Chen K, Wu W, Yang Y, Yao G, Pu Y, Liu Z Su . Guided, stochastic model-based GUI testing of Android APPs. In: Proceedings of the 11th Joint Meeting on Foundations of Software Engineering. 2017, 245−256
28 K, Mao M, Harman Y Jia . Sapienz: multi-objective automated testing for Android applications. In: Proceedings of the 25th International Symposium on Software Testing and Analysis. 2016, 94−105
29 F, Behrang A Orso . AppTestMigrator: a tool for automated test migration for Android APPs. In: Proceedings of the 42nd IEEE/ACM International Conference on Software Engineering: Companion Proceedings. 2020, 17−20
30 S, Chen L, Fan C, Chen T, Su W, Li Y, Liu L Xu . StoryDroid: automated generation of storyboard for android APPs. In: Proceedings of the 41st IEEE/ACM International Conference on Software Engineering. 2019, 596−607
31 L, Fan T, Su S, Chen G, Meng Y, Liu L, Xu G Pu . Efficiently manifesting asynchronous programming errors in Android APPs. In: Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering. 2018, 486−497
32 M, Pan A, Huang G, Wang T, Zhang X Li . Reinforcement learning based curiosity-driven testing of Android applications. In: Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis. 2020, 153−164
33 Z, Dong M, Böhme L, Cojocaru A Roychoudhury . Time-travel testing of android APPs. In: Proceedings of the 42nd IEEE/ACM International Conference on Software Engineering. 2020, 481−492
34 X, Zhang Z, Chen C, Fang Z Liu . Guiding the crowds for Android testing. In: Proceedings of the 38th International Conference on Software Engineering Companion. 2016, 752−753
35 C Meng . A research on android test automation technology based on dependency injection. Nanjing University, Dissertation, 2017
36 K, Mao M, Harman Y Jia . Crowd intelligence enhances automated mobile testing. In: Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering. 2017, 16−26
[1] FCS-21658-OF-YZ_suppl_1 Download
[1] Pengfei GAO, Yongjie XU, Fu SONG, Taolue CHEN. Model-based automated testing of JavaScript Web applications via longer test sequences[J]. Front. Comput. Sci., 2022, 16(3): 163204-.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed