Model-based automated testing of JavaScript Web applications via longer test sequences

doi:10.1007/s11704-020-0356-7

Front. Comput. Sci.

2022, Vol. 16

Issue (3) : 163204 https://doi.org/10.1007/s11704-020-0356-7

RESEARCH ARTICLE

Model-based automated testing of JavaScript Web applications via longer test sequences

Pengfei GAO¹, Yongjie XU¹, Fu SONG¹(

), Taolue CHEN²

¹. School of Information Science and Technology, ShanghaiTech University, Shanghai 201210, China
². Department of Computer Science, University of Surrey, Guildford GU2 7XH, UK

Download: PDF(6869 KB) HTML
Export: BibTeX | EndNote | Reference Manager | ProCite | RefWorks

Abstract

JavaScript has become one of the most widely used languages for Web development. Its dynamic and event-driven features make it challenging to ensure the correctness of Web applications written in JavaScript. A variety of dynamic analysis techniques have been proposed which are, however, limited in either coverage or scalability. In this paper, we propose a simple, yet effective, model-based automated testing approach to achieve a high code-coverage within the time budget via testing with longer event sequences. We implement our approach as an open-source tool LJS, and perform extensive experiments on 21 publicly available benchmarks. On average, LJS is able to achieve 86.5% line coverage in 10 minutes. Compared with JSDEP, a state-of-the-art breadth-first search based automated testing tool enriched with partial order reduction, the coverage of LJS is 11%–19% higher than that of JSDEP on real-world large Web applications. Our empirical findings support that proper longer test sequences can achieve a higher code coverage in JavaScript Web application testing.

Keywords model-based testing automated testing JavaScript Web applications

Corresponding Author(s): Fu SONG

Just Accepted Date: 29 October 2020 Issue Date: 09 November 2021

Cite this article:

Pengfei GAO,Yongjie XU,Fu SONG, et al. Model-based automated testing of JavaScript Web applications via longer test sequences[J]. Front. Comput. Sci., 2022, 16(3): 163204.

URL:

https://academic.hep.com.cn/fcs/EN/10.1007/s11704-020-0356-7
https://academic.hep.com.cn/fcs/EN/Y2022/V16/I3/163204

Fig.1 Example HTML page and associated JavaScript code

Fig.2 The search tree of our running example, where the missed labels of edges in black color are respectively

A, B, C

, and the missed labels of edges in red color are

S u b m i t

Fig.3 Framework overview of LJS

Fig.4

Fig.5 The FSM models of the running example: the top FSM is constructed using the state abstraction from ARTEMIS and the bottom FSM is constructed using our new state abstraction

Fig.6

Fig.7

Name	$?$ Loc	LJS				JSDEP			p-value	Effect size
Name	$?$ Loc	Coverage/%	sd	$?$ Tests	Max. length	Coverage/%	$?$ Tests	Max. length	p-value	Effect size
case1	59	100.0	0.000	4803	99	100.0	1409	705	NA	NA
case2	72	100.0	0.000	4470	99	100.0	3058	549	NA	NA
case3	165	100.0	0.000	2789	99	100.0	7811	575	NA	NA
case4	196	87.0	0.000	2885	99	77.9	8594	500	$< 1 × 10 ? 5$	1
frog	567	96.6	0.006	10	99	84.6	86	16	$5 × 10 ? 5$	1
cosmos	363	87.9	0.059	268	99	79.5	973	243	$9.6 × 10 ? 4$	0.71
hanoi	246	88.3	0.013	1470	99	82.5	902	225	$1 × 10 ? 5$	0.95
flipflop	525	97.1	0.004	60	99	96.3	284	71	$1 × 10 ? 5$	1
sokoban	3056	88.2	0.027	55	99	77.6	203	51	$5 × 10 ? 5$	1
wormy	570	44.2	0.052	34	99	41.0	323	18	0.70512	0.67
chinabox	338	83.7	0.003	7	99	82.3	92	9	$2 × 10 ? 5$	1
3dmodel	5414	85.0	0.001	9	99	71.5	66	10	$1 × 10 ? 5$	1
cubuild	1014	87.5	0.035	8	99	72.8	153	17	$5 × 10 ? 5$	1
pearlski	960	55.1	0.000	82	99	54.9	214	52	$< 1 × 10 ? 5$	1
speedyeater	784	90.0	0.008	550	99	82.1	1497	374	$6 × 10 ? 5$	1
gallony	300	94.5	0.001	2395	99	94.5	1611	95	$2.4 × 10 ? 4$	0.05
fullhouse	528	92.1	0.012	1168	99	86.3	889	222	$3 × 10 ? 5$	1
ball_pool	1745	93.2	0.005	2	99	74.2	18	3	$6 × 10 ? 5$	1
harehound	468	95.0	0.004	498	99	94.5	1224	116	$1.3 × 10 ? 4$	0.81
match	369	72.5	0.002	1142	99	73.2	4050	845	$1 × 10 ? 5$	0
lady	820	79.1	0.002	0	99	75.7	35	8	$5 × 10 ? 5$	1
Average	883.8	86.5	0.011	1081	99	81.0	1595	224	NA	NA

Tab.1 Coverage of LJS and JSDEP in 600 seconds

Fig.8 Coverage of LJS and JSDEP, as a function of the execution time

Name	State abstraction from [2]				Our state abstraction				p-value of $S$	p-value of $δ$
Name	Coverage/%	$S$	$δ$	Time/s	Coverage/%	$S$	$δ$	Time/s	p-value of $S$	p-value of $δ$
case1	98.6	1.0	2.0	0.3	98.6	1.0	2.0	0.2	NA	NA
case2	98.3	1.0	4.0	0.3	99.4	1.0	4.0	0.3	NA	NA
case3	99.3	1.0	6.0	0.4	98.2	1.0	6.0	0.4	NA	NA
case4	85.6	1.0	8.0	0.4	85.9	1.0	8.0	0.4	NA	NA
frog	95.9	199.0	198.0	96.3	96.0	27.6	102.0	120.7	$< 1 × 10 ? 5$	$< 1 × 10 ? 5$
cosmos	79.9	127.1	196.5	4.8	78.7	68.0	143.7	4.7	$< 1 × 10 ? 5$	$< 1 × 10 ? 5$
hanoi	88.6	105.8	186.7	0.8	88.5	105.9	185.7	0.7	0.96987	0.50301
flipflop	96.6	30.6	113.9	17.5	97.1	28.6	108.3	17.6	0.36409	0.33882
sokoban	86.5	69.6	189.6	18.9	87.7	31.2	126.5	21.0	$< 1 × 10 ? 5$	$< 1 × 10 ? 5$
wormy	42.0	188.3	197.9	32.8	42.0	131.0	186.1	31.8	$< 1 × 10 ? 5$	$< 1 × 10 ? 5$
chinabox	83.6	63.1	154.3	132.1	83.7	68.1	158.5	141.4	0.00052	0.08411
3dmodel	83.7	3.0	25.0	96.7	83.3	1.0	6.0	96.5	$< 1 × 10 ? 5$	$< 1 × 10 ? 5$
cubuild	85.0	86.6	165.5	125.3	84.6	83.4	163.1	121.6	0.14685	0.24113
pearlski	55.1	138.7	196.0	13.5	55.1	72.4	176.4	13.9	$< 1 × 10 ? 5$	$< 1 × 10 ? 5$
speedyeater	85.3	168.0	198.0	1.6	84.4	4.4	60.2	2.1	$< 1 × 10 ? 5$	$< 1 × 10 ? 5$
gallony	94.5	64.8	171.6	0.5	94.5	62.4	173.4	0.5	0.17342	0.39768
fullhouse	92.7	167.4	197.8	0.8	92.2	28.4	119.0	1.0	$< 1 × 10 ? 5$	$< 1 × 10 ? 5$
ball_pool	93.1	42.4	143.0	283.5	93.2	41.9	146.5	237.5	0.57851	0.2788
harehound	81.2	188.6	198.0	0.9	87.2	11.2	93.4	1.7	$< 1 × 10 ? 5$	$< 1 × 10 ? 5$
match	67.9	15.2	176.0	1.0	70.3	4.1	55.3	1.0	$< 1 × 10 ? 5$	$< 1 × 10 ? 5$
lady	79.1	164.2	197.6	896.2	79.2	86.5	174.2	856.1	$< 1 × 10 ? 5$	$< 1 × 10 ? 5$
Average	84.4	87.0	139.3	82.1	84.8	41.0	104.7	79.6	NA	NA

Tab.2 Comparison of state abstraction techniques

Fig.9 Coverage of LJS, as a function of the length bound

Name	Baseline with POR				Long Sequence Generation				p-value
Name	Coverage/%	Max. Length	Time/s	$?$ Tests	Coverage	Max. Length	Time/s	$?$ Tests	p-value
case1	100.0	16	829	65504	100.0	99	600	4796	NA
case2	87.8	9	1125	87040	100.0	99	600	4482	$< 1 × 10 ? 5$
case3	68.6	7	931	54432	100.0	99	600	2800	$< 1 × 10 ? 5$
case4	61.8	6	674	32768	87.0	99	600	2887	$< 1 × 10 ? 5$
frog	88.6	4	664	592	95.1	99	600	6	$5 × 10 ? 5$
cosmos	78.0	5	1072	4415	88.2	99	600	826	$5 × 10 ? 5$
hanoi	86.8	6	889	4276	88.6	99	600	1417	$< 1 × 10 ? 5$
flipflop	97.0	6	1200	419	97.0	99	600	23	$< 1 × 10 ? 5$
sokoban	88.6	5	1200	1697	84.0	99	600	66	$1 × 10 ? 5$
wormy	41.2	13	657	317	49.7	99	600	37	$3.6 × 10 ? 4$
chinabox	78.0	8	729	64	83.7	99	600	10	$3 × 10 ? 5$
3dmodel	72.0	4	1200	93	85.0	99	600	11	$< 1 × 10 ? 5$
cubuild	75.2	6	1019	238	88.3	99	600	10	$5 × 10 ? 5$
pearlski	52.1	7	960	653	51.1	99	600	78	$2.4 × 10 ? 4$
speedyeater	82.3	5	1200	17106	88.0	99	600	548	$< 1 × 10 ? 5$
gallony	94.5	8	1200	9821	92.6	99	600	2039	$1 × 10 ? 5$
fullhouse	79.2	8	1200	12097	75.2	99	600	757	$1 × 10 ? 5$
ball_pool	92.1	6	1200	22	93.4	99	600	3	$5 × 10 ? 5$
harehound	92.2	4	1200	7949	95.5	99	600	487	$4 × 10 ? 5$
match	73.2	5	738	13341	72.1	99	600	1172	$< 1 × 10 ? 5$
lady	73.2	5	696	12	79.2	99	600	2	$5 × 10 ? 5$
Average	79.2	7	980	14898	85.4	99	600	1069	NA

Tab.3 Comparison of event sequence generation algorithms

1	P Saxena, D Akhawe, S Hanna, F Mao, S McCamant, D Song. A symbolic execution framework for JavaScript. In: Proceedings of IEEE Symposium on Security and Privacy. 2010, 513– 528
2	Artzi S, Dolby J, Jensen S H, Møller A, Tip F. A framework for automated testing of JavaScript web applications. In: Proceedings of International Conference on Software Engineering. 2011, 571–580
3	A Mesbah , A Van Deursen , S Lenselink . Crawling ajaxbased web applications through dynamic analysis of user interface state changes. ACM Transactions on the Web, 2012, 6( 1): 1– 30
4	Sen K, Kalasapur S, Brutch T G, Gibbs S. Jalangi: a selective record-replay and dynamic analysis framework for JavaScript. In: Proceedings of Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering. 2013, 488–498
5	Mirshokraie S, Mesbah A, Pattabiraman K. Efficient JavaScript mutation testing. In: Proceedings of IEEE International Conference on Software Testing, Verification and Validation. 2013, 74–83
6	Li G, Andreasen E, Ghosh I. SymJS: automatic symbolic testing of JavaScript Web applications. In: Proceedings of ACM SIGSOFT International Symposium on Foundations of Software Engineering. 2014, 449–459
7	Pradel M, Schuh P, Necula G, Sen K. EventBreak: analyzing the responsiveness of user interfaces through performance-guided test generation. In: Proceedings of ACM International Conference on Object Oriented Programming Systems Languages & Applications. 2014, 33–47
8	S Mirshokraie , A Mesbah , K Pattabiraman . Guided mutation testing for JavaScript Web Applications. IEEE Transactions on Software Engineering, 2015, 41( 5): 429– 444
9	Sen K, Necula G C, Gong L, Choi W. MultiSE: multipath symbolic execution using value summaries. In: Proceedings of Joint Meeting on Foundations of Software Engineering. 2015, 842–853
10	E Andreasen , L Gong , A Møller , M Pradel , M Selakovic , K Sen , C Staicu . A survey of dynamic analysis and test generation for JavaScript. ACM Computing Surveys, 2017, 50( 5): 66:1– 66:36
11	Sung C, Kusano M, Sinha N, Wang C. Static DOM event dependency analysis for testing Web applications. In: Proceedings of ACM SIGSOFT International Symposium on Foundations of Software Engineering. 2016, 447–459
12	Arcuri A. Longer is better: On the role of test sequence length in software testing. In: Proofceedings of International Conference on Software Testing, Verification and Validation. 2010, 469–478
13	Andrews J H, Groce A, Weston M, Xu R G. Random test run length and effectiveness. In: Proofceedings of IEEE/ACM International Conference on Automated Software Engineering. 2008, 19–28
14	Fraser G, Gargantini A. Experiments on the test case length in specification based test case generation. In: Proceedings of International Workshop on Automation of Software Test. 2009, 18–26
15	Carino S, Andrews J H. Evaluating the effect of test case length on GUI test suite performance. In: Proceedings of IEEE/ACM International Workshop on Automation of Software Test. 2015, 13–17
16	Y F Li , P K Das , D L Dowe . Two decades of Web application testing - A survey of recent advances. Information Systems, 2014, 43 : 20– 54
17	Cheng L, Yang Z, Wang C. Systematic reduction of GUI test sequences. In: Proceedings of IEEE/ACM International Conference on Automated Software Engineering. 2017, 849–860
18	Godefroid P, van Leeuwen J, Hartmanis J, Goos G, Wolper P. Partial-Order Methods for the Verification of Concurrent Systems - An Approach to the StateExplosion Problem. Heidelberg: Springer, 1996
19	A Arcuri , L C Briand . A hitchhiker’s guide to statistical tests for assessing randomized algorithms in software engineering. Software Testing, Verification and Reliability, 2014, 24( 3): 219– 250
20	A C Dias-Neto , G H Travassos . A picture from the model-based testing area: Concepts, techniques, and challenges. Advances in Computers, 2010, 80 : 45– 120
21	M Utting , A Pretschner , B Legeard . A taxonomy of model-based testing approaches. Software Testing, Verification & Reliability, 2012, 22( 5): 297– 312
22	Li W, Le Gall F, Spaseski N. A survey on model-based testing tools for test case generation. In: Proceedings of International Conference on Tools and Methods of Program Analysis. 2018, 77–89
23	Jensen C S, Prasad M R, Møller A. Automated testing with targeted event sequence generation. In: Proceedings of International Symposium on Software Testing and Analysis. 2013, 67–77
24	Takala T, Katara M, Harty J. Experiences of systemlevel model-based GUI testing of an android application. In: Proceedings of IEEE International Conference on Software Testing, Verification and Validation. 2011, 377–386
25	Rau A, Hotzkow J, Zeller A. Efficient GUI test generation by learning from tests of other apps. In: Proceedings of International Conference on Software Engineering: Companion Proceedings. 2018, 370–371
26	A A Andrews , J Offutt , R T Alexander . Testing Web applications by modeling with fsms. Software and System Modeling, 2005, 4( 3): 326– 345
27	Ricca F, Tonella P. Analysis and testing of Web applications. In: Proceedings of International Conference on Software Engineering. 2001, 25–34
28	Dallmeier V, Burger M, Orth T, Zeller A. Webmate: a tool for testing Web 2.0 applications. In: Proceedings of Workshop on JavaScript Tools. 2012, 11–15
29	Alshahwan N, Harman M. Automated Web application testing using search based software engineering. In: Proceedings of IEEE/ACM International Conference on Automated Software Engineering. 2011, 3–12
30	Nguyen C, Yoshida H, Prasad M R, Ghosh I, Sen K. Generating succinct test cases using don’t care analysis. In: Proceedings of IEEE International Conference on Software Testing, Verification and Validation. 2015, 1–10
31	Rau A, Hotzkow J, Zeller A. Transferring tests across Web applications. In: Proceedings of International Conference on Web Engineering. 2018, 50–64

[1]

Download

Viewed

Full text

Abstract

Cited

Shared

Discussed