Please wait a minute...
Frontiers of Computer Science

ISSN 2095-2228

ISSN 2095-2236(Online)

CN 10-1014/TP

Postal Subscription Code 80-970

2018 Impact Factor: 1.129

Front. Comput. Sci.    2013, Vol. 7 Issue (5) : 694-709    https://doi.org/10.1007/s11704-013-2320-2
RESEARCH ARTICLE
An evaluation framework for software crowdsourcing
Wenjun WU1(), Wei-Tek TSAI2,3, Wei LI1
1. State Key Laboratory of Software Development Environment, Beihang University, Beijing 100191, China
2. School of Computing, Informatics, and Decision Systems Engineering, Arizona State University, Tempe, AZ85281, USA
3. Department of Computer Science and Technology, INLIST, Tsinghua University, Beijing 100084, China
 Download: PDF(0 KB)  
 Export: BibTeX | EndNote | Reference Manager | ProCite | RefWorks
Abstract

Recently software crowdsourcing has become an emerging area of software engineering. Few papers have presented a systematic analysis on the practices of software crowdsourcing. This paper first presents an evaluation framework to evaluate software crowdsourcing projects with respect to software quality, costs, diversity of solutions, and competition nature in crowdsourcing. Specifically, competitions are evaluated by the min-max relationship from game theory among participants where one party tries to minimize an objective function while the other party tries to maximize the same objective function. The paper then defines a game theory model to analyze the primary factors in these minmax competition rules that affect the nature of participation as well as the software quality. Finally, using the proposed evaluation framework_this paper illustrates two crowdsourcing processes, Harvard-TopCoder and AppStori. The framework demonstrates the sharp contrasts between both crowdsourcing processes as participants will have drastic behaviors in engaging these two projects.

Keywords crowdsourcing      software engineering      competition rules      game theory     
Corresponding Author(s): Wenjun WU   
Issue Date: 01 October 2013
 Cite this article:   
Wenjun WU,Wei-Tek TSAI,Wei LI. An evaluation framework for software crowdsourcing[J]. Front. Comput. Sci., 2013, 7(5): 694-709.
 URL:  
https://academic.hep.com.cn/fcs/EN/10.1007/s11704-013-2320-2
https://academic.hep.com.cn/fcs/EN/Y2013/V7/I5/694
1 A Doan , R Ramakrishnan , A Y Halevy . Crowdsourcing systems on theWorld-WideWeb. Communications of the ACM, 2011, 54(4): 86−96
https://doi.org/10.1145/1924421.1924442
2 K Lakhani , D Garvin , E Lonstein . Topcoder (a): developing software through crowdsourcing. Harvard Business School General Management Unit Case, 2010. Available at SSRN:
3 uTest.
4 J Bosch . From software product lines to software ecosystems. In: Proceedings of the 13th International Software Product Line Conference. 2009, 111−119
5 S Jansen , A Finkelstein , S Brinkkemper . A sense of community: a research agenda for software ecosystems. In: Proceedings of the 31st International Conference on Software Engineering-Companion Volume. 2009, 187−190
6 Apple Store Metrics. , 2012
7 AppStori. , 2012
8 A Kittur . Crowdsourcing, collaboration and creativity. XRDS, 2010, 17(2): 22−26
https://doi.org/10.1145/1869086.1869096
9 R Constantinescu , I M Iacob . Capability maturity model integration. Journal of Applied Quantitative Methods, 2007, 2(1): 187
10 M Atwood . Military standard: defense system software development. Department of Defense, USA, 1988
11 E Schenk , C Guittard . Crowdsourcing: what can be outsourced to the crowd, and why. In: Workshop on Open Source Innovation, Strasbourg, France. 2009
12 R Tong , K Lakhani . Public-private partnerships for organizing and executing prize-based competitions. Berkman Center Research Publication, 2012. Available at SSRN:
13 Algorithm Development Through Crowdsourcing. 2012
14 N Archak , A Sundararajan . Optimal design of crowdsourcing contests. In: Proceedings of the 30th International Conference on Information Systems. 2009, 1−16
15 T W Wu , W Li . Creative software crowdsourcing. Creative Software Crowdsourcing: From Components and Algorithm Development to Project Concept Formations, 2013
16 C Y Baldwin , K B Clark . The architecture of participation: does code architecture mitigate free riding in the open source development model? Management Science, 2006, 52(7): 1116−1127
https://doi.org/10.1287/mnsc.1060.0546
17 D G Rand , A Dreber , T Ellingsen , D Fudenberg , MA Nowak . Positive interactions promote public cooperation. Science, 2009, 325(5945): 1272−1275
https://doi.org/10.1126/science.1177418
18 R Herbrich , T Minka , T Graepel . TrueSkillTM: a bayesian skill rating system. In: Proceedings of the 2006 Annual Conference of Advances in Neural Information Processing Systems. 2007, 19: 569−576
19 TopCoder Inc. , 2013
20 Apple App Store Review Guidelines. 2010
21 N Archak . Money, glory and cheap talk: analyzing strategic behavior of contestants in simultaneous crowdsourcing contests on topcoder. com. In: Proceedings of the 19th International Conference on World Wide Web. 2010, 21−30
https://doi.org/10.1145/1772690.1772694
22 D DiPalantino , M Vojnovic . Crowdsourcing and all-pay auctions. In: Proceedings of the 10th ACM Conference on Electronic Commerce. 2009, 119−128
23 J J Horton , L B Chilton The labor economics of paid crowdsourcing. In: Proceedings of the 11th ACM Conference on Electronic Commerce. 2010, 209−218
24 D F Bacon , Y Chen , D Parkes , M Rao . A market-based approach to software evolution. In: Proceedings of the 24th ACM SIGPLAN Conference Companion on Object Oriented Programming Systems Languages and Applications. 2009, 973−980
https://doi.org/10.1145/1639950.1640066
25 A C Bullinger , K Moeslein . Innovation contests-where are we? In: Proceedings of the 16th Americas Conference on Information Systems. 2010
26 J M Leimeister , M Huber , U Bretschneider , H Krcmar . Leveraging crowdsourcing: activation-supporting components for it-based ideas competition. Journal of Management Information Systems, 2009, 26(1): 197−224
https://doi.org/10.2753/MIS0742-1222260108
27 R Kazman , HM Chen . The metropolis model a new logic for development of crowdsourced systems. Communications of the ACM, 2009, 52(7): 76−84
https://doi.org/10.1145/1538788.1538808
28 D Bratvold , C Armstrong , 2013
[1] Sedigheh KHOSHNEVIS. A search-based identification of variable microservices for enterprise SaaS[J]. Front. Comput. Sci., 2023, 17(3): 173208-.
[2] Peng LI, Junzuo LAI, Yongdong WU. Accountable attribute-based authentication with fine-grained access control and its application to crowdsourcing[J]. Front. Comput. Sci., 2023, 17(1): 171802-.
[3] Tao HAN, Hailong SUN, Yangqiu SONG, Yili FANG, Xudong LIU. Find truth in the hands of the few: acquiring specific knowledge with crowdsourcing[J]. Front. Comput. Sci., 2021, 15(4): 154315-.
[4] Ibrahim ALSEADOON, Aakash AHMAD, Adel ALKHALIL, Khalid SULTAN. Migration of existing software systems to mobile computing platforms: a systematic mapping study[J]. Front. Comput. Sci., 2021, 15(2): 152204-.
[5] Gang WU, Zhiyong CHEN, Jia LIU, Donghong HAN, Baiyou QIAO. Task assignment for social-oriented crowdsourcing[J]. Front. Comput. Sci., 2021, 15(2): 152316-.
[6] Zhenghui HU, Wenjun WU, Jie LUO, Xin WANG, Boshu LI. Quality assessment in competition-based software crowdsourcing[J]. Front. Comput. Sci., 2020, 14(6): 146207-.
[7] Yixuan TANG, Zhilei REN, Weiqiang KONG, He JIANG. Compiler testing: a systematic literature analysis[J]. Front. Comput. Sci., 2020, 14(1): 1-20.
[8] Bo YUAN, Xiaolei ZHOU, Xiaoqiang TENG, Deke GUO. Enabling entity discovery in indoor commercial environments without pre-deployed infrastructure[J]. Front. Comput. Sci., 2019, 13(3): 618-636.
[9] Anil Kumar KARNA, Yuting CHEN, Haibo YU, Hao ZHONG, Jianjun ZHAO. The role of model checking in software engineering[J]. Front. Comput. Sci., 2018, 12(4): 642-668.
[10] Xiaolei ZHOU, Tao CHEN, Deke GUO, Xiaoqiang TENG, Bo YUAN. From one to crowd: a survey on crowdsourcing-based wireless indoor localization[J]. Front. Comput. Sci., 2018, 12(3): 423-450.
[11] Najam NAZAR,He JIANG,Guojun GAO,Tao ZHANG,Xiaochen LI,Zhilei REN. Source code fragment summarization with small-scale crowdsourcing based features[J]. Front. Comput. Sci., 2016, 10(3): 504-517.
[12] Yuan SU,Xi ZHANG,Lixin LIU,Shouyou SONG,Binxing FANG. Understanding information interactions in diffusion: an evolutionary game-theoretic perspective[J]. Front. Comput. Sci., 2016, 10(3): 518-531.
[13] Xiaolan XU,Wenjun WU,Ya WANG,Yuchuan WU. Software crowdsourcing for developing Software-as-a-Service[J]. Front. Comput. Sci., 2015, 9(4): 554-565.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed