|
|
|
An evaluation framework for software crowdsourcing |
Wenjun WU1( ), Wei-Tek TSAI2,3, Wei LI1 |
1. State Key Laboratory of Software Development Environment, Beihang University, Beijing 100191, China
2. School of Computing, Informatics, and Decision Systems Engineering, Arizona State University, Tempe, AZ85281, USA
3. Department of Computer Science and Technology, INLIST, Tsinghua University, Beijing 100084, China |
|
|
|
|
Abstract Recently software crowdsourcing has become an emerging area of software engineering. Few papers have presented a systematic analysis on the practices of software crowdsourcing. This paper first presents an evaluation framework to evaluate software crowdsourcing projects with respect to software quality, costs, diversity of solutions, and competition nature in crowdsourcing. Specifically, competitions are evaluated by the min-max relationship from game theory among participants where one party tries to minimize an objective function while the other party tries to maximize the same objective function. The paper then defines a game theory model to analyze the primary factors in these minmax competition rules that affect the nature of participation as well as the software quality. Finally, using the proposed evaluation framework_this paper illustrates two crowdsourcing processes, Harvard-TopCoder and AppStori. The framework demonstrates the sharp contrasts between both crowdsourcing processes as participants will have drastic behaviors in engaging these two projects.
|
| Keywords
crowdsourcing
software engineering
competition rules
game theory
|
|
Corresponding Author(s):
Wenjun WU
|
|
Issue Date: 01 October 2013
|
|
| 1 |
A Doan , R Ramakrishnan , A Y Halevy . Crowdsourcing systems on theWorld-WideWeb. Communications of the ACM, 2011, 54(4): 86−96
https://doi.org/10.1145/1924421.1924442
|
| 2 |
K Lakhani , D Garvin , E Lonstein . Topcoder (a): developing software through crowdsourcing. Harvard Business School General Management Unit Case, 2010. Available at SSRN:
|
| 3 |
uTest.
|
| 4 |
J Bosch . From software product lines to software ecosystems. In: Proceedings of the 13th International Software Product Line Conference. 2009, 111−119
|
| 5 |
S Jansen , A Finkelstein , S Brinkkemper . A sense of community: a research agenda for software ecosystems. In: Proceedings of the 31st International Conference on Software Engineering-Companion Volume. 2009, 187−190
|
| 6 |
Apple Store Metrics. , 2012
|
| 7 |
AppStori. , 2012
|
| 8 |
A Kittur . Crowdsourcing, collaboration and creativity. XRDS, 2010, 17(2): 22−26
https://doi.org/10.1145/1869086.1869096
|
| 9 |
R Constantinescu , I M Iacob . Capability maturity model integration. Journal of Applied Quantitative Methods, 2007, 2(1): 187
|
| 10 |
M Atwood . Military standard: defense system software development. Department of Defense, USA, 1988
|
| 11 |
E Schenk , C Guittard . Crowdsourcing: what can be outsourced to the crowd, and why. In: Workshop on Open Source Innovation, Strasbourg, France. 2009
|
| 12 |
R Tong , K Lakhani . Public-private partnerships for organizing and executing prize-based competitions. Berkman Center Research Publication, 2012. Available at SSRN:
|
| 13 |
Algorithm Development Through Crowdsourcing. 2012
|
| 14 |
N Archak , A Sundararajan . Optimal design of crowdsourcing contests. In: Proceedings of the 30th International Conference on Information Systems. 2009, 1−16
|
| 15 |
T W Wu , W Li . Creative software crowdsourcing. Creative Software Crowdsourcing: From Components and Algorithm Development to Project Concept Formations, 2013
|
| 16 |
C Y Baldwin , K B Clark . The architecture of participation: does code architecture mitigate free riding in the open source development model? Management Science, 2006, 52(7): 1116−1127
https://doi.org/10.1287/mnsc.1060.0546
|
| 17 |
D G Rand , A Dreber , T Ellingsen , D Fudenberg , MA Nowak . Positive interactions promote public cooperation. Science, 2009, 325(5945): 1272−1275
https://doi.org/10.1126/science.1177418
|
| 18 |
R Herbrich , T Minka , T Graepel . TrueSkillTM: a bayesian skill rating system. In: Proceedings of the 2006 Annual Conference of Advances in Neural Information Processing Systems. 2007, 19: 569−576
|
| 19 |
TopCoder Inc. , 2013
|
| 20 |
Apple App Store Review Guidelines. 2010
|
| 21 |
N Archak . Money, glory and cheap talk: analyzing strategic behavior of contestants in simultaneous crowdsourcing contests on topcoder. com. In: Proceedings of the 19th International Conference on World Wide Web. 2010, 21−30
https://doi.org/10.1145/1772690.1772694
|
| 22 |
D DiPalantino , M Vojnovic . Crowdsourcing and all-pay auctions. In: Proceedings of the 10th ACM Conference on Electronic Commerce. 2009, 119−128
|
| 23 |
J J Horton , L B Chilton The labor economics of paid crowdsourcing. In: Proceedings of the 11th ACM Conference on Electronic Commerce. 2010, 209−218
|
| 24 |
D F Bacon , Y Chen , D Parkes , M Rao . A market-based approach to software evolution. In: Proceedings of the 24th ACM SIGPLAN Conference Companion on Object Oriented Programming Systems Languages and Applications. 2009, 973−980
https://doi.org/10.1145/1639950.1640066
|
| 25 |
A C Bullinger , K Moeslein . Innovation contests-where are we? In: Proceedings of the 16th Americas Conference on Information Systems. 2010
|
| 26 |
J M Leimeister , M Huber , U Bretschneider , H Krcmar . Leveraging crowdsourcing: activation-supporting components for it-based ideas competition. Journal of Management Information Systems, 2009, 26(1): 197−224
https://doi.org/10.2753/MIS0742-1222260108
|
| 27 |
R Kazman , HM Chen . The metropolis model a new logic for development of crowdsourced systems. Communications of the ACM, 2009, 52(7): 76−84
https://doi.org/10.1145/1538788.1538808
|
| 28 |
D Bratvold , C Armstrong , 2013
|
|
Viewed |
|
|
|
Full text
|
|
|
|
|
Abstract
|
|
|
|
|
Cited |
|
|
|
|
| |
Shared |
|
|
|
|
| |
Discussed |
|
|
|
|
