Classification-oriented dawid skene model for transferring intelligence from crowds to machines

doi:10.1007/s11704-022-2245-8

Front. Comput. Sci.

2023, Vol. 17

Issue (5) : 175332 https://doi.org/10.1007/s11704-022-2245-8

RESEARCH ARTICLE

Classification-oriented dawid skene model for transferring intelligence from crowds to machines

Jiaran LI¹, Richong ZHANG¹(

), Samuel MENSAH², Wenyi QIN¹, Chunming HU¹

¹. Department of Computer Science and Engineering, Beihang University, Beijing 100191, China
². Department of Computer Science, University of Sheffield, Sheffield, S10 2TN, UK

Download: PDF(15012 KB) HTML
Export: BibTeX | EndNote | Reference Manager | ProCite | RefWorks

Abstract

When a crowdsourcing approach is used to assist the classification of a set of items, the main objective is to classify this set of items by aggregating the worker-provided labels. A secondary objective is to assess the workers’ skill levels in this process. A classical model that achieves both objectives is the famous Dawid-Skene model. In this paper, we consider a third objective in this context, namely, to learn a classifier that is capable of labelling future items without further assistance of crowd workers. By extending the Dawid-Skene model to include the item features into consideration, we develop a Classification-Oriented Dawid Skene (CODS) model, which achieves the three objectives simultaneously. The effectiveness of CODS on this three dimensions of the problem space is demonstrated experimentally.

Keywords crowdsourcing information aggregation learning from crowds

Corresponding Author(s): Richong ZHANG

Just Accepted Date: 14 September 2022 Issue Date: 25 December 2022

Cite this article:

Jiaran LI,Richong ZHANG,Samuel MENSAH, et al. Classification-oriented dawid skene model for transferring intelligence from crowds to machines[J]. Front. Comput. Sci., 2023, 17(5): 175332.

URL:

https://academic.hep.com.cn/fcs/EN/10.1007/s11704-022-2245-8
https://academic.hep.com.cn/fcs/EN/Y2023/V17/I5/175332

Fig.1 The Dawid-Skene model

Fig.2 A simplified model for learning classifier from crowds

Tab.1 Dataset summary

	Weather	Movie	Spam $σ g o o d 2$	SLS $σ g o o d 2$	Iono $σ g o o d 2$
MV	0.667	0.886	0.890	0.813	0.895
DS	0.772	0.915	0.913	0.814	0.912
KOS	?	0.819	0.903	0.813	0.897
CPC	?	0.827	0.923	0.812	0.866
LAA	0.778	0.884	0.904	0.815	0.923
CLA	0.789	0.908	0.923	0.816	0.926
CODS	0.801	0.915	0.932	0.818	0.944

Tab.2 Performance of crowd label aggregation

Tab.3 Running time on different datasets compared with other algorithms

	Weather		Movie		Spam $σ g o o d 2$		SLS $σ g o o d 2$		Iono $σ g o o d 2$
	AA	AA	AA	AA	AA	AA	AA	AA	AA	AA
	$60 %$	$40 %$	$60 %$	$40 %$	$60 %$	$40 %$	$60 %$	$40 %$	$60 %$	$40 %$
MV	0.680	0.654	0.890	0.890	0.893	0.898	0.817	0.808	0.905	0.894
DS	0.756	0.726	0.820	0.820	0.906	0.914	0.817	0.811	0.914	0.908
KOS	?	?	0.820	0.824	0.905	0.911	0.816	0.811	0.895	0.890
CPC	?	?	0.846	0.860	0.923	0.924	0.804	0.800	0.886	0.872
LAA	0.761	0.730	0.906	0.904	0.908	0.915	0.813	0.809	0.920	0.908
CLA	0.759	0.728	0.912	0.911	0.919	0.918	0.815	0.811	0.928	0.915
	AA	CA	AA	CA	AA	CA	AA	CA	AA	CA
CODS	0.765	0.706	0.920	0.690	0.924	0.920	0.818	0.766	0.932	0.881

Tab.4 Performance of CODS in crowd-assisted classifier construction against the baselines. For CODS, 60% columns are the aggregation accuracies (AA), and the 40% columns are the classification accuracies (CA)

Fig.3 Crowd-assisted classifier construction: accuracy as a function of the number (

m

) of worker labels per item on

20 %

simulated workers and votes of Spambase, SLS and Ionosphere. Vertical axis: aggregation accuracy. Horizontal axis: number of worker labels per item. (a) Spambase with skilled workers; (b) Spambase with unskilled workers; (c) SLS with skilled workers; (d) SLS with unskilled workers; (e) Ionosphere with skilled workers; (f) Ionosphere with unskilled workers

Fig.4 Crowd worker selection under

?

-greedy and random schemes. Vertical axis: aggregation accuracy. Horizontal axis: batch number (time). (a) Weather accumulated; (b) movie accumulated

Fig.5 Crowd worker selection under

?

-greedy and random schemes. Vertical axis: aggregation accuracy. Horizontal axis: batch number (time). (a) SLS batch; (b) SLS accumulated; (c) spambase batch; (d) spambase accumulated; (e) Ionosphere batch; (f) Ionosphere accumulated

Fig.6 LCFC Algorithm: Classification accuracy as a function of number of iterations on validation set of spambase, SLS and ionosphere. Horizontal-axis represents the number of iterations and vertical-axis represents the classification accuracy. (a) Spambase with 80 skilled, 20 unskilled workers; (b) spambase with 20 skilled, 80 unskilled workers; (c) SLS with 80 skilled, 20 unskilled workers; (d) spambase with 20 skilled, 80 unskilled workers; (e) ionosphere with 80 skilled, 20 unskilled workers. (f) ionosphere with 20 skilled, 80 unskilled workers

1	H, Su J, Deng F F Li . Crowdsourcing annotations for visual object detection. In: Proceedings of 2012 AAAI Workshop on Human Computation. 2012, 40−46
2	P, Welinder S, Branson S, Belongie P Perona . The multidimensional wisdom of crowds. In: Proceedings of the 23rd International Conference on Neural Information Processing Systems. 2010, 2424−2432
3	G, Little L B, Chilton M, Goldman R C Miller . TurKit: human computation algorithms on mechanical Turk. In: Proceedings of the 23rd Annual ACM Symposium on User Interface Software and Technology. 2010, 57−66
4	R, Snow B, O’Connor D, Jurafsky A Y Ng . Cheap and fast—but is it good?: evaluating non-expert annotations for natural language tasks. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing. 2008, 254−263
5	X, Lu T W S Chow . Modeling sequential annotations for sequence labeling with crowds. IEEE Transactions on Cybernetics, 2021, 1–11 https://doi.org/10.1109/TCYB.2021.3117700
6	Lin C H, Mausam, Weld D S. Dynamically switching between synergistic workﬂows for crowdsourcing. In: Proceedings of the 26th AAAI Conference on Artificial Intelligence. 2012, 87−93
7	J, Wang T, Kraska M J, Franklin J Feng . CrowdER: crowdsourcing entity resolution. Proceedings of the VLDB Endowment, 2012, 5( 11): 1483–1494
8	F, Khatib S, Cooper M D, Tyka K, Xu I, Makedon Z, Popović D, Baker F Players . Algorithm discovery by protein folding game players. Proceedings of the National Academy of Sciences of the United States of America, 2011, 108( 47): 18949–18953
9	O F, Zaidan C Callison-Burch . Crowdsourcing translation: professional quality from non-professionals. In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. 2011, 1220−1229
10	M J, Murphy C D, Miller W S, Lasecki J P Bigham . Adaptive time windows for real-time crowd captioning. In: Proceedings of CHI ’13 Extended Abstracts on Human Factors in Computing Systems. 2013, 13−18
11	A P, Dawid A M Skene . Maximum likelihood estimation of observer error-rates using the EM algorithm. Journal of the Royal Statistical Society. Series C (Applied Statistics), 1979, 28( 1): 20–28
12	A, Kurve D J, Miller G Kesidis . Multicategory crowdsourcing accounting for variable task difficulty, worker skill, and worker intention. IEEE Transactions on Knowledge and Data Engineering, 2015, 27( 3): 794–809
13	Zhao Z, Wei F, Zhou M, Chen W, Ng W S H. Crowd-selection query processing in crowdsourcing databases: a task-driven approach. In: Proceedings of the 18th International Conference on Extending Database Technology. 2015, 397−408
14	C, Lof Maarry K, El W T Balke . Skyline queries in crowd-enabled databases. In: Proceedings of the 16th International Conference on Extending Database Technology. 2013, 465−476
15	X, Chen Q, Lin D Zhou . Optimistic knowledge gradient policy for optimal budget allocation in crowdsourcing. In: Proceedings of the 30th International Conference on International Conference on Machine Learning. 2013, III-64−III-72
16	Fan J, Lu M, Ooi B C, Tan W C, Zhang M. A hybrid machine-crowdsourcing system for matching web tables. In: Proceedings of the 30th IEEE International Conference on Data Engineering. 2014, 976−987
17	M, Yousefnezhad S J, Huang D Zhang . WoCE: a framework for clustering ensemble by exploiting the wisdom of crowds theory. IEEE Transactions on Cybernetics, 2018, 48( 2): 486–499
18	J Zhang . Knowledge learning with crowdsourcing: a brief review and systematic perspective. IEEE/CAA Journal of Automatica Sinica, 2022, 9( 5): 749–762
19	L, Jiang H, Zhang F, Tao C Li . Learning from crowds with multiple noisy label distribution propagation. IEEE Transactions on Neural Networks and Learning Systems, 2021, 1–11 https://doi.org/10.1109/TNNLS.2021.3082496
20	F, Tao L, Jiang C Li . Differential evolution-based weighted soft majority voting for crowdsourcing. Engineering Applications of Artificial Intelligence, 2021, 106: 104474
21	A I, Chittilappilly L, Chen S Amer-Yahia . A survey of general-purpose crowdsourcing techniques. IEEE Transactions on Knowledge and Data Engineering, 2016, 28( 9): 2246–2266
22	J, Zhang X, Wu V S Sheng . Learning from crowdsourced labeled data: a survey. Artificial Intelligence Review, 2016, 46( 4): 543–576
23	Y, Yan R, Rosales G, Fung J G Dy . Active learning from crowds. In: Proceedings of the 28th International Conference on Machine Learning. 2011, 1161−1168
24	B, Mozafari P, Sarkar M J, Franklin M I, Jordan S Madden . Active learning for crowd-sourced databases. 2014, arXiv preprint arXiv: 1209.3686
25	V C, Raykar S, Yu L H, Zhao G H, Valadez C, Florin L, Bogoni L Moy . Learning from crowds. Journal of Machine Learning Research, 2010, 11: 1297–1322
26	J, Zhang X, Wu V S Shengs . Active learning with imbalanced multiple noisy labeling. IEEE Transactions on Cybernetics, 2015, 45( 5): 1095–1107
27	Y, Bachrach T, Minka J, Guiver T Graepel . How to grade a test without knowing the answers: a Bayesian graphical model for adaptive crowdsourcing and aptitude testing. In: Proceedings of the 29th International Conference on Machine Learning. 2012, 819−826
28	C J, Ho S, Jabbari J W Vaughan . Adaptive task assignment for crowdsourced classification. In: Proceedings of the 30th International Conference on Machine Learning. 2013, I-534−I-542
29	N, Buchbinder J Naor . Online primal-dual algorithms for covering and packing problems. In: Proceedings of 13th Annual European Symposium on Algorithms. 2005, 689−701
30	Long C, Hua G, Kapoor A. Active visual recognition with expertise estimation in crowdsourcing. In: Proceedings of 2013 IEEE International Conference on Computer Vision. 2013, 3000−3007
31	P, Donmez J G, Carbonell J Schneider . Efficiently learning the accuracy of labeling sources for selective sampling. In: Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 2009, 259−268
32	L, Zhao Y, Zhang G Sukthankar . An active learning approach for jointly estimating worker performance and annotation reliability with crowdsourced data. 2014, arXiv preprint arXiv: 1401.3836
33	D D, Lewis W A Gale . A sequential algorithm for training text classifiers. In: Proceedings of the 17th Annual International ACM-SIGIR Conference on Research and Development in Information Retrieval. 1994, 3−12
34	W G Cochran . Sampling Techniques. 3rd ed. Hoboken: John Wiley & Sons, 1977
35	G, Salton C Buckley . Improving retrieval performance by relevance feedback. In: Jones K S, Willett P, eds. Readings in Information Retrieval. San Francisco: Morgan Kaufmann, 1997, 355−364
36	Dagan I, Engelson S P. Committee-based sampling for training probabilistic classifiers. In: Proceedings of the 12th International Conference on Machine Learning. 1995, 150−157
37	J, Zhu H, Wang E, Hovy M Ma . Confidence-based stopping criteria for active learning for data annotation. ACM Transactions on Speech and Language Processing, 2010, 6( 3): 3
38	D, Dua C Graff . UCI machine learning repository. See Archive.ics.uci.eduml website, 2017
39	D R, Karger S, Oh D Shah . Iterative learning for reliable crowdsourcing systems. In: Proceedings of the 24th International Conference on Neural Information Processing Systems. 2011, 1953−1961
40	Kajino H, Tsuboi Y, Kashima H. Clustering crowds. In: Proceedings of the 27th AAAI Conference on Artificial Intelligence. 2013, 1120−1127
41	L A, Yin J H, Han W N, Zhang Y Yu . Aggregating crowd wisdoms with label-aware autoencoders. In: Proceedings of the 26th International Joint Conference on Artificial Intelligence. 2017, 1325−1331
42	Liu Y F, Zhang W N, Yu Y. Aggregating crowd wisdom with side information via a clustering-based label-aware autoencoder. In: Proceedings of the 29th International Joint Conference on Artificial Intelligence. 2021, 214
43	E, Simpson S, Roberts I, Psorakis A Smith . Dynamic Bayesian combination of multiple imperfect classifiers. In: Guy T V, Karny M, Wolpert D, eds. Decision Making and Imperfection. Berlin: Springer, 2013, 1−35

[1]

FCS-22245-OF-JL_suppl_1

Download

[1]	Xiaochuan LIN, Kaimin WEI, Zhetao LI, Jinpeng CHEN, Tingrui PEI. Aggregation-based dual heterogeneous task allocation in spatial crowdsourcing[J]. Front. Comput. Sci., 2024, 18(6): 186605-.
[2]	Lijuan REN, Liangxiao JIANG, Wenjun ZHANG, Chaoqun LI. Label distribution similarity-based noise correction for crowdsourcing[J]. Front. Comput. Sci., 2024, 18(5): 185323-.
[3]	Yao ZHANG, Liangxiao JIANG, Chaoqun LI. Attribute augmentation-based label integration for crowdsourcing[J]. Front. Comput. Sci., 2023, 17(5): 175331-.
[4]	Peng LI, Junzuo LAI, Yongdong WU. Accountable attribute-based authentication with fine-grained access control and its application to crowdsourcing[J]. Front. Comput. Sci., 2023, 17(1): 171802-.
[5]	Tao HAN, Hailong SUN, Yangqiu SONG, Yili FANG, Xudong LIU. Find truth in the hands of the few: acquiring specific knowledge with crowdsourcing[J]. Front. Comput. Sci., 2021, 15(4): 154315-.
[6]	Gang WU, Zhiyong CHEN, Jia LIU, Donghong HAN, Baiyou QIAO. Task assignment for social-oriented crowdsourcing[J]. Front. Comput. Sci., 2021, 15(2): 152316-.
[7]	Zhenghui HU, Wenjun WU, Jie LUO, Xin WANG, Boshu LI. Quality assessment in competition-based software crowdsourcing[J]. Front. Comput. Sci., 2020, 14(6): 146207-.
[8]	Bo YUAN, Xiaolei ZHOU, Xiaoqiang TENG, Deke GUO. Enabling entity discovery in indoor commercial environments without pre-deployed infrastructure[J]. Front. Comput. Sci., 2019, 13(3): 618-636.
[9]	Xiaolei ZHOU, Tao CHEN, Deke GUO, Xiaoqiang TENG, Bo YUAN. From one to crowd: a survey on crowdsourcing-based wireless indoor localization[J]. Front. Comput. Sci., 2018, 12(3): 423-450.
[10]	Najam NAZAR,He JIANG,Guojun GAO,Tao ZHANG,Xiaochen LI,Zhilei REN. Source code fragment summarization with small-scale crowdsourcing based features[J]. Front. Comput. Sci., 2016, 10(3): 504-517.
[11]	Xiaolan XU,Wenjun WU,Ya WANG,Yuchuan WU. Software crowdsourcing for developing Software-as-a-Service[J]. Front. Comput. Sci., 2015, 9(4): 554-565.
[12]	Wenjun WU, Wei-Tek TSAI, Wei LI. An evaluation framework for software crowdsourcing[J]. Front. Comput. Sci., 2013, 7(5): 694-709.

Viewed

Full text

Abstract

Cited

Shared

Discussed