1. State Key Laboratory of Software Development, Beihang University, Beijing 100191, China 2. National Laboratory for Parallel and Distributed Processing, College of Computer, National University of Defense Technology, Changsha 410073, China 3. Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, China 4. College of Computer Science and Technology, Zhejiang University, Hangzhou 310027, China 5. National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing 210023, China
The Internet based cyber-physical world has profoundly changed the information environment for the development of artificial intelligence (AI), bringing a new wave of AI research and promoting it into the new era of AI 2.0. As one of the most prominent characteristics of research in AI 2.0 era, crowd intelligence has attracted much attention from both industry and research communities. Specifically, crowd intelligence provides a novel problem-solving paradigm through gathering the intelligence of crowds to address challenges. In particular, due to the rapid development of the sharing economy, crowd intelligence not only becomes a new approach to solving scientific challenges, but has also been integrated into all kinds of application scenarios in daily life, e.g., online-tooffline (O2O) application, real-time traffic monitoring, and logistics management. In this paper, we survey existing studies of crowd intelligence. First, we describe the concept of crowd intelligence, and explain its relationship to the existing related concepts, e.g., crowdsourcing and human computation. Then, we introduce four categories of representative crowd intelligence platforms. We summarize three core research problems and the state-of-the-art techniques of crowd intelligence. Finally, we discuss promising future research directions of crowd intelligence.
Abraham , I., Alonso , O., Kandylas , V., , 2013. Adaptive crowdsourcing algorithms for the bandit survey problem. Proc. 26th Conf. on Computational Learning Theory, p.882–910.
2
Ballesteros , J., Carbunar , B., Rahman , M., , 2014. Towards safe cities: a mobile and social networking approach.IEEE Trans. Parall. Distr. Syst., 25(9):2451–2462.
3
Basili , V.R., Briand , L.C., Melo , W.L., 1996. A validation of object-oriented design metrics as quality indicators.IEEE Trans. Softw. Eng., 22(10):751–761.
4
Bhattacharya , P., Neamtiu , I., 2010. Fine-grained incremental learning and multi-feature tossing graphs to improve bug triaging. IEEE Int. Conf. on Software Maintenance, p.1–10.
5
Bird , C., Gourley , A., Devanbu , P., , 2006. Mining email social networks. Proc. Int. Workshop on Mining Software Repositories, p.137–143.
6
Bird , C., Pattison , D., de Souza , R., , 2008. Latent social structure in open source projects. Proc. 16th ACM SIGSOFT Int. Symp. on Foundations of Software Engineering, p.24–35.
7
Bird , C., Nagappan , N., Murphy , B., , 2011. Don’t touch my code!: examining the effects of ownership on software quality. Proc. 19th ACM SIGSOFT Symp. and 13th European Conf. on Foundations of Software Engineering, p.4–14.
8
Bollen , J., Mao , H.N., Zeng , X.J., 2011. Twitter mood predicts the stock market. J. Comput. Sci., 2(1):1–8.
9
Bonabeau , E., 2009. Decisions 2.0: the power of collective intelligence.MIT Sloan Manag. Rev., 50(2):45–52.
10
Borne , K.D., Zooniverse Team, 2011. The Zooniverse: a framework for knowledge discovery from citizen science data. American Geophysical Union Fall Meeting.
11
Burke , J.A., Estrin , D., Hansen , M., , 2006. Participatory sensing. Workshop on World-Sensor-Web: Mobile Device Centric Sensor Networks and Applications, p.117–134.
12
Cao , C.C., She , J.Y., Tong , Y.X., , 2012. Whom to ask? Jury selection for decision making tasks on micro-blog services.Proc. VLDB Endow., 5(11):1495–1506.
13
Cao , C.C., Tong , Y.X., Chen , L., , 2013. Wisemarket: a new paradigm for managing wisdom of online social users. Proc. 19th ACM Int. Conf. on Knowledge Discovery and Data Mining, p.455–463.
14
Castaneda , O.F., 2010. Hierarchy in Meritocracy: Community Building and Code Production in the Apache Software Foundation. MS Thesis, Delft University of Technology, Delft, Netherlands.
15
Chen , X., Lin , Q.H., Zhou , D.Y., 2013. Optimistic knowledge gradient policy for optimal budget allocation in crowdsourcing. Proc. 30th Int. Conf. on Machine Learning, p.64–72.
16
Chen , X., Lin , Q.H., Zhou , D.Y., 2015. Statistical decision making for optimal budget allocation in crowd labeling.J. Mach. Learn. Res., 16:1–46.
17
Dantec , C.A.L., Asad , M., Misra , A., , 2015. Planning with crowdsourced data: rhetoric and representation in transportation planning. Proc. 18th ACM Conf. on Computer Supported Cooperative Work, p.1717–1727.
18
Dawid , A.P., Skene , A.M., 1979. Maximum likelihood estimation of observer error-rates using the EM algorithm.Appl. Statist., 28(1):20–28.
19
de Alwis , B., Sillito , J., 2009. Why are software projects moving from centralized to decentralized version control systems? Proc. ICSE Workshop on Cooperative and Human Aspects on Software Engineering, p.36–39.
20
Dekel , O., Shamir , O., 2009. Vox Populi: collecting highquality labels from a crowd. Proc. 22nd Conf. on Learning Theory.
21
Dempster , A.P., Laird , N.M., Rubin , D.B., 1977. Maximum likelihood from incomplete data via the EM algorithm.J. R. Stat. Soc. Ser. B, 39(1):1–38.
22
Difallah , D.E., Demartini , G., Cudré-Mauroux , G.P., 2013. Pick-a-crowd: tell me what you like, and I’ll tell you what to do. Proc. 22nd Int. Conf. on World Wide Web, p.367–374.
23
Difallah , D.E., Demartini , G., Cudré-Mauroux , G.P., 2016. Scheduling human intelligence tasks in multi-tenant crowd-powered systems. Proc. 25th Int. Conf. on World Wide Web, p.855–865.
24
Dong , X.L., Saha , B., Srivastava , D., 2012. Less is more: selecting sources wisely for integration. Proc. VLDB Endow., 6(2):37–48.
25
Erenkrantz , J.R., Taylor , R.N, 2003. Supporting Distributed and Decentralized Projects: Drawing Lessons From the Open Source Community. ISR Technical Report No. UCI-ISR-03-4, Institute for Software Research, University of California, Irvine, USA.
26
Farkas , K., Nagy , A.Z., Tomás , T., , 2014. Participatory sensing based real-time public transport information service. IEEE Int. Conf. on Pervasive Computing and Communications Workshops, p.141–144.
27
Feng , Z.N., Zhu , Y.M., Zhang , Q., , 2014. Trac: truthful auction for location-aware collaborative sensing in mobile crowdsourcing. Proc. IEEE Conf. on Computer Communications, p.1231–1239.
28
Fowler , G., Schectman , J., 2013. Citizen surveillance helps officials put pieces together.The Wall Street Journal, April 17.
29
Gao , C., Zhou , D.Y., 2013. Minimax optimal convergence rates for estimating ground truth from crowdsourced labels. ePrint Archive, arXiv:1310.5764.
30
Gao , D.W., Tong , Y.X., She , J.Y., , 2016. Top-k team recommendation in spatial crowdsourcing.LNCS, 9658:191–204.
Ghosh , R.A., 2005. Understanding Free Software Developers: Findings from the Floss Study. MIT Press, Cambrige, USA.
33
Gousios , G., Pinzger , M., Deursen , A., 2014. An exploratory study of the pull-based software development model. Proc. 36th Int. Conf. on Software Engineering, p.345–355.
34
Gousios , G., Zaidman , A., Storey , M.A., , 2015. Work practices and challenges in pull-based development: the integrator’s perspective. Proc. 37th Int. Conf. on Software Engineering, p.358–368.
35
Han , K., Zhang , C., Luo , J., , 2016. Truthful scheduling mechanisms for powering mobile crowdsensing.IEEE Trans. Comput., 65(1):294–307.
36
Hars , A., Ou , S., 2001. Working for free? Motivations of participating in open source projects. Proc. 34th Annual Hawaii Int. Conf. on System Sciences, p.1–9. h
37
Hassan , A.E., 2009. Predicting faults using the complexity of code changes. Proc. 31st Int. Conf. on Software Engineering, p.78–88.
38
Hertel , G., Niedner , S., Herrmann , S., 2003. Motivation of software developers in open source projects: an Internet-based survey of contributors to the Linux kernel.Res. Polic., 32(7):1159–1177.
39
Ho , C.J., Vaughan , J.W., 2012. Online task assignment in crowdsourcing markets. Proc. 26th AAAI Conf. on Artificial Intelligence, p.45–51.
40
Ho , C.J., Jabbari , S., Vaughan , J.W., 2013. Adaptive task assignment for crowdsourced classification. Proc. 30th Int. Conf. on Machine Learning, p.534–542.
41
Hoffman , M.L., 1981. Is altruism part of human nature?J. Personal. Soc. Psychol., 40(1):121–137.
42
Jaimes , L.G., Vergara-Laurens , I., Labrador , M.A., 2012. A location-based incentive mechanism for participatory sensing systems with budget constraints. Proc. 10th Annual IEEE Int. Conf. on Pervasive Computing and Communications, p.103–108.
Jeong , G., Kim , S., Zimmermann , T., 2009. Improving bug triage with bug tossing graphs. Proc. 7th Joint Meeting of the European Software Engineering Conf. and the ACM SIGSOFT Symp. on the Foundations of Software Engineering, p.111–120.
45
Karger , D.R., Oh , S., Shah , D., 2011. Iterative learning for reliable crowdsourcing systems. Advances in Neural Information Processing Systems, p.1953–1961.
46
Khetan , A., Oh , S., 2016. Achieving budget-optimality with adaptive schemes in crowdsourcing. Advances in Neural Information Processing Systems, p.4844–4852.
47
Kittur , A., Smus , B., Khamkar , S., , 2011. Crowdforge: crowdsourcing complex work. Proc. 24th Annual ACM Symp. on User Interface Software and Technology, p.43–52.
48
Krishna , V., 2009. Auction Theory. Academic Press, New York, USA.
49
Krontiris , I., Albers , A., 2012. Monetary incentives in participatory sensing using multi-attributive auctions.Int. J. Parall. Emerg. Distr. Syst., 27(4):317–336.
50
Law , E., Ahn , L., 2011. Human computation.Synth. Lect. Artif. Intell. Mach. Learn., 5(3):1–121.
51
Lazer , D., Kennedy , R., King , G., , 2014. The parable of Google flu: traps in big data analysis.Science, 343(6167):1203–1205.
52
Lee , J.S., Hoh , B., 2010. Sell your experiences: a market mechanism based incentive for participatory sensing. IEEE Int. Conf. on Pervasive Computing and Communications, p.60–68.
53
Li , G.L., Wang , J.N., Zheng , Y.D., , 2016. Crowdsourced data management: a survey. IEEE Trans. Knowl. Data Eng., 28(9):2296–2319.
54
Li , H.W., Yu , B., 2014. Error rate bounds and iterative weighted majority voting for crowdsourcing. ePrint Archive, arXiv:1411.4086.
55
Li , X., Dong , X.L., Lyons , K., , 2012. Truth finding on the deep Web: is the problem solved?Proc. VLDB Endow., 6(2):97–108.
56
Lintott , C.J., Schawinski , K., Slosar , A., , 2008. Galaxy Zoo: morphologies derived from visual inspection of galaxies from the sloan digital sky survey.Month. Not. R. Astronom. Soc., 389(3):1179–1189.
57
Liu , Q., Peng , J., Ihler , A.T., 2012. Variational inference for crowdsourcing. Advances in Neural Information Processing Systems, p.692–700.
58
Luo , T., Tan , H.P., Xia , L.R., 2014. Profit-maximizing incentive for participatory sensing. Proc. IEEE Conf. on Computer Communications, p.127–135.
59
Malone , T.W., Laubacher , R., Dellarocas , C., 2009. Harnessing Crowds: Mapping the Genome of Collective Intelligence. MIT Sloan Research Paper No. 4732-09, Sloan School of Management, Massachusetts Institute of Technology, MA, USA.
60
Mamykina , L., Manoim , B., Mittal , M., , 2011. Design lessons from the fastest Q&A site in the west. Proc. SIGCHI Conf. on Human Factors in Computing Systems, p.2857–2866.
61
Maslow , A.H., Frager , R., Fadiman , J., , 1970. Motivation and Personality. Harper & Row, New York, USA.
62
Mavridis , P., Gross-Amblard , D., Miklós , Z., 2016. Using hierarchical skills for optimized task assignment in knowledge-intensive crowdsourcing. Proc. 25th Int. Conf. on World Wide Web, p.843–853.
63
Meng , R., Tong , Y.X., Chen , L., , 2015. CrowdTC: crowdsourced taxonomy construction. Proc. IEEE Int. Conf. on Data Mining, p.913–918.
64
Mockus , A., Fielding , R.T., Herbsleb , J.D., 2002. Two case studies of open source software development: Apache and Mozilla.ACM Trans. Softw. Eng. Meth., 11(3): 309–346.
65
Moser , R., Pedrycz , W., Succi , G., 2008. A comparative analysis of the efficiency of change metrics and static code attributes for defect prediction. Proc. 30th Int. Conf. on Software Engineering, p.181–190.
66
Nagappan , N., Ball , T., 2005. Use of relative code churn measures to predict system defect density. Proc. 27th Int. Conf. on Software Engineering, p.284–292.
67
Nakakoji , K., Yamamoto , Y., Nishinaka , Y., , 2002. Evolution patterns of open-source software systems and communities. Proc. Int. Workshop on Principles of Software Evolution, p.76–85.
68
Ok , J., Oh , S., Shin , J., , 2016. Optimality of belief propagation for crowdsourced classification. ePrint Archive, arXiv:1602.03619.
69
Ouyang , W.R., Kaplan , L.M., Martin , P., , 2015. Debiasing crowdsourced quantitative characteristics in local businesses and services. Proc. 14th Int. Conf. on Information Processing in Sensor Networks, p.190–201.
Pierre , L., 1997. Collective intelligence: mankind’s emerging world in cyberspace. Bononno, R., translator. Perseus Books, Cambridge, USA.
72
Quinn , A.J., Bederson , B.B., 2011. Human computation: a survey and taxonomy of a growing field. Proc. SIGCHI Conf. on Human Factors in Computing Systems, p.1403–1412.
Rahman , F., Devanbu , P.T., 2011. Ownership, experience and defects: a fine-grained study of authorship. Proc. 33rd Int. Conf. on Software Engineering, p.491–500.
75
Rahman , F., Devanbu , P.T., 2013. How, and why, process metrics are better. Proc. 35th Int. Conf. on Software Engineering, p.432–441.
76
Rana , R.K., Chou , C.T., Kanhere , S.S., , 2010. Earphone: an end-to-end participatory urban noise mapping system. Proc. 9th ACM/IEEE Int. Conf. on Information Processing in Sensor Networks, p.105–116.
77
Raykar , V.C., Yu , S., 2012. Eliminating spammers and ranking annotators for crowdsourced labeling tasks. J. Mach. Learn. Res., 13(Feb):491–518.
78
Raykar , V.C., Yu , S., Zhao , L.H., , 2009. Supervised learning from multiple experts: whom to trust when everyone lies a bit. Proc. 26th Int. Conf. on Machine Learning, p.889–896.
Raymond , E., 1999. The cathedral and the bazaar.Knowl. Technol. Polic., 12(3):23-49.
81
Rigby , P.C., German , D.M., Cowen , L., , 2014. Peer review on open-source software projects: parameters, statistical models, and theory. ACM Trans. Softw. Eng. Meth., 23(4):No.35.
82
Rogers , E.M., 2010. Diffusion of Innovations. Simon and Schuster, New York, USA.
83
Sakaki , T., Okazaki , M., Matsuo , Y., 2010. Earthquake shakes Twitter users: real-time event detection by social sensors. Proc. 19th Int. Conf. on World Wide Web, p.851–860.
84
Shah , N.B., Zhou , D., 2015. Double or nothing: multiplicative incentive mechanisms for crowdsourcing. Advances in Neural Information Processing Systems, p.1–9.
85
Shah , N.B., Zhou , D., 2016. No oops, you won’t do it again: mechanisms for self-correction in crowdsourcing. Proc. 33rd Int. Conf. on Machine Learning, p.1–10.
86
Shah , N.B., Zhou , D., Peres , Y., 2015. Approval voting and incentives in crowdsourcing. ePrint Archive, arXiv:1502.05696.
87
She , J.Y., Tong , Y.X., Chen , L., 2015a. Utility-aware social event-participant planning. Proc. ACM Int. Conf. on Management of Data, p.1629–1643.
88
She , J.Y., Tong , Y.X., Chen , L., , 2015b. Conflict-aware event-participant arrangement. Proc. 31st IEEE Int. Conf. on Data Engineering, p.735–746.
89
She , J.Y., Tong , Y.X., Chen , L., , 2016. Conflict-aware event-participant arrangement and its variant for online setting.IEEE Trans. Knowl. Data Eng., 28(9):2281–2295.
Smith , J.B., 1994. Collective Intelligence in Computer-Based Collaboration. CRC Press, Boca Raton, USA.
92
Subramanian , A., Kanth , G.S., Vaze , R., 2013. Offline and online incentive mechanism design for smart-phone crowd-sourcing. ePrint Archive, arXiv:1310.1746.
93
Subramanyam , R., Krishnan , M.S., 2003. Empirical analysis of CK metrics for object-oriented design complexity: implications for software defects. ACM Trans. Softw. Eng. Meth., 29(4):297–310.
94
Sullivan , B.L., Wood , C.L., Iliff , M.J., , 2009. eBird: a citizen-based bird observation network in the biological sciences. Biol. Consev., 142(10):2282–2292.
95
Tamrawi , A., Nguyen , T.T., Al-Kofahi , J.M., , 2011. Fuzzy set and cache-based approach for bug triaging. Proc. 19th ACM SIGSOFT Symp. and 13th European Conf. on Foundations of Software Engineering, p.365–375.
96
Tang , J.C., Cebrian , M., Giacobe , N.A., , 2011. Reflecting on the DARPA red balloon challenge.Commun. ACM, 54(4):78–85.
97
Teodoro , R., Ozturk , P., Naaman , M., , 2014. The motivations and experiences of the on-demand mobile workforce. Proc. 17th ACM Conf. on Computer Supported Cooperative Work & Social Computing, p.236–247.
98
Thebault-Spieker , J., Terveen , L.G., Hecht , B., 2015. Avoiding the south side and the suburbs: the geography of mobile crowdsourcing markets. Proc. 18th ACM Conf. on Computer Supported Cooperative Work & Social Computing, p.265–275.
99
Thongtanunam , P., Tantithamthavorn , C., Kula , R.G., , 2015. Who should review my code? A file locationbased code-reviewer recommendation approach for modern code review. IEEE 22nd Int. Conf. on Software Analysis, Evolution, and Reengineering, p.141–150.
100
Tian , T., Zhu , J., 2015. Max-margin majority voting for learning from crowds. Advances in Neural Information Processing Systems, p.1621–1629.
101
Tong , Y.X., Chen , L., Ding , B.L., 2012a. Discovering threshold-based frequent closed itemsets over probabilistic data. Proc. IEEE 28th Int. Conf. on Data Engineering, p.270–281.
Tong , Y.X., Cao , C.C., Zhang , C.J., , 2014a. Crowd-Cleaner: data cleaning for multi-version data on the web via crowdsourcing. Proc. IEEE 28th Int. Conf. on Data Engineering, p.1182–1185.
104
Tong , Y.X., Cao , C.C., Chen , L., 2014b. TCS: efficient topic discovery over crowd-oriented service data. Proc. 20th ACM Int. Conf. on Knowledge Discovery and Data Mining, p.861–870.
Tong , Y.X., Meng , R., She , J.Y., 2015b. On bottleneck-aware arrangement for event-based social networks. Proc. 31st IEEE Int. Conf. on Data Engineering Workshops, p.216–223.
107
Tong , Y.X., She , J.Y., Meng , R., 2016a. Bottleneck-aware arrangement over event-based social networks: the maxmin approach.World Wide Web, 19(6):1151–1177.
108
Tong , Y.X., She , J.Y., Ding , B.L., , 2016b. Online mobile micro-task allocation in spatial crowdsourcing. Proc. 32nd IEEE Int. Conf. on Data Engineering, p.49–60.
109
Tong , Y.X., She , J.Y., Ding , B.L.,, 2016c. Online minimum matching in real-time spatial data: experiments and analysis. Proc. VLDB Endow., 9(12):1053–1064.
Tong , Y.X., Yuan , Y., Cheng , Y.R., , 2017. A survey of spatiotemporal crowdsourced data management techniques.J. Softw., 28(1):35–58 (in Chinese).
112
Tran-Thanh , L., Stein , S., Rogers , A., , 2012. Efficient crowdsourcing of unknown experts using multi-armed bandits. Proc. European Conf. on Artificial Intelligence, p.768–773.
113
Tsay , J., Dabbish , L., Herbsleb , J.D., 2014a. Influence of social and technical factors for evaluating contribution in GitHub. 36th Int. Conf. on Software Engineering, p.356–366.
114
Tsay , J., Dabbish , L., Herbsleb , J.D., 2014b. Let’s talk about it: evaluating contributions through discussion in GitHub. Proc. 22nd ACM Int. Symp. on Foundations of Software Engineering, p.144–154.
115
Vasilescu , B., Yu , Y., Wang , H., , 2015. Quality and productivity outcomes relating to continuous integration in GitHub. Proc. 10th Joint Meeting on Foundations of Software Engineering, p.805–816.
116
Vickrey , W., 1961. Counterspeculation, auctions, and competitive sealed tenders.J. Finan., 16(1):8–37.
117
von Ahn , L., Maurer , B., McMillen , C., , 2008. reCAPTCHA: human-based character recognition via web security measures.Science, 321(5895):1465–1468.
118
Wang , D., Abdelzaher , T.F., Kaplan , L.M., , 2013. Recursive fact-finding: a streaming approach to truth estimation in crowdsourcing applications. IEEE 33rd Int. Conf. on Distributed Computing Systems, p.530–539.
119
Wang , D., Amin , M.T.A., Li , S., , 2014. Using humans as sensors: an estimation-theoretic perspective. Proc. 13th Int. Conf. on Information Processing in Sensor Networks, p.35–46.
120
Wang , H.M., Yin , G., Li , X., , 2015. TRUSTIE: a software development platform for crowdsourcing. In: Li, W., Huhns, M.N., Tsai, W.T., et al. (Eds.), Crowdsourcing. Springer Berlin Heidelberg, Berlin, Germany, p.165–190.
121
Wang , J.N., Li , G.L., Kraska , T., , 2013. Leveraging transitive relations for crowdsourced joins. Proc. ACM Int. Conf. on Management of Data, p.229–240.
122
Wang , L., Zhou , Z.H., 2016. Cost-saving effect of crowdsourcing learning. Proc. 25th Int. Joint Conf. on Artificial Intelligence, p.2111–2117.
123
Wang , W., Zhou , Z.H., 2015. Crowdsourcing label quality: a theoretical study.Sci. China Inform. Sci., 58(11):1–12.
124
Wauthier , F.L., Jordan , M.I., 2011. Bayesian bias mitigation for crowdsourcing. Advances in Neural Information Processing Systems, p.1800–1808.
125
Welinder , P., Branson , S., Perona , P., , 2010. The multidimensional wisdom of crowds. Advances in Neural Information Processing Systems, p.2424–2432.
126
Whitehill , J., Wu , T., Bergsma , J., , 2009. Whose vote should count more: optimal integration of labels from labelers of unknown expertise. Advances in Neural Information Processing Systems, p.2035–2043.
127
Wu , W.J., Tsai , W.T., Li , W., 2013. An evaluation framework for software crowdsourcing.Front. Comput. Sci., 7(5):694–709.
128
Yan , Y., Fung , G.M., Rosales , R., , 2011. Active learning from crowds. Proc. 28th Int. Conf. on Machine Learning, p.1161–1168.
129
Yang , D.J., Fang , X., Xue , G.L., 2013. Truthful incentive mechanisms for k-anonymity location privacy. Proc. IEEE Conf. on Computer Communications, p.2994–3002.
130
Yang , D.J., Xue, G.L., Fang , X., , 2012. Crowdsourcing to smartphones: incentive mechanism design for mobile phone sensing. Proc. 18th Annual Int. Conf. on Mobile Computing and Networking, p.173–184.
131
Ye , Y.W., Kishida , K., 2003. Toward an understanding of the motivation of open source software developers. Proc. 25th Int. Conf. on Software Engineering, p.419–429.
132
Yu , Y., Yin , G., Wang , H., , 2014. Exploring the patterns of social behavior in GitHub. Proc. 1st Int. Workshop on Crowd-Based Software Development Methods and Technologies, p.31-36.
133
Yu , Y., Yin , G., Wang , T., , 2016a. Determinants of pull-based development in the context of continuous integration.Sci. China Inform. Sci., 59(8):080104.
134
Yu , Y., Wang , H.M., Yin , G., , 2016b. Reviewer recommendation for pull-requests in GitHub: what can we learn from code review and bug assignment?Imform. Softw. Technol., 74:204–218.
135
Zhang , C.J., Chen , L., Tong , Y.X., 2014a. MaC: a probabilistic framework for query answering with machinecrowd collaboration. Proc. 23rd ACM Int. Conf. on Information and Knowledge Management, p.11–20.
136
Zhang , C.J., Tong , Y.X., Chen , L., 2014b. Where to: crowdaided path selection.Proc. VLDB Endow., 7(11):2005–2016.
137
Zhang , C.J., Chen , L., Tong , Y., , 2015. Cleaning uncertain data with a noisy crowd. Proc. 31st IEEE Int. Conf. on Data Engineering, p.6–17.
138
Zhang , Y., Chen , X., Zhou , D., , 2014. Spectral methods meet EM: a provably optimal algorithm for crowdsourcing. Advances in Neural Information Processing Systems, p.1260–1268.
139
Zhao , D., Li , X.Y., Ma , H.D., 2014. How to crowdsource tasks truthfully without sacrificing utility: online incentive mechanisms with budget constraint. Proc. IEEE Conf. on Computer Communications, p.1213–1221.
140
Zhong , J.H., Tang , K., Zhou , Z.H., 2015. Active learning from crowds with unsure option. Proc. 24th Int. Joint Conf. on Artificial Intelligence, p.1061–1067.
141
Zhou , D., Basu , S., Mao , Y., , 2012. Learning from the wisdom of crowds by minimax entropy. Advances in Neural Information Processing Systems, p.2195–2203.
142
Zhou , Y., Chen , X., Li , J., 2014. Optimal PAC multiple arm identification with applications to crowdsourcing. Proc. 31st Int. Conf. on Machine Learning, p.217–225.
143
Zhu , Y., Zhang , Q., Zhu , H., , 2014. Towards truthful mechanisms for mobile crowdsourcing with dynamic smartphones. Proc. 34th Int. Conf. on Distributed Computing Systems, p.11–20.
