Attribute augmentation-based label integration for crowdsourcing
Yao ZHANG1, Liangxiao JIANG1(), Chaoqun LI2
1. School of Computer Science, China University of Geosciences, Wuhan 430074, China 2. School of Mathematics and Physics, China University of Geosciences, Wuhan 430074, China
Crowdsourcing provides an effective and low-cost way to collect labels from crowd workers. Due to the lack of professional knowledge, the quality of crowdsourced labels is relatively low. A common approach to addressing this issue is to collect multiple labels for each instance from different crowd workers and then a label integration method is used to infer its true label. However, to our knowledge, almost all existing label integration methods merely make use of the original attribute information and do not pay attention to the quality of the multiple noisy label set of each instance. To solve these issues, this paper proposes a novel three-stage label integration method called attribute augmentation-based label integration (AALI). In the first stage, we design an attribute augmentation method to enrich the original attribute space. In the second stage, we develop a filter to single out reliable instances with high-quality multiple noisy label sets. In the third stage, we use majority voting to initialize integrated labels of reliable instances and then use cross-validation to build multiple component classifiers on reliable instances to predict all instances. Experimental results on simulated and real-world crowdsourced datasets demonstrate that AALI outperforms all the other state-of-the-art competitors.
L, Jiang L, Zhang L, Yu D Wang . Class-specific attribute weighted naive Bayes. Pattern Recognition, 2019, 88: 321–330
2
Y, Dong L, Jiang C Li . Improving data and model quality in crowdsourcing using co-training-based noise correction. Information Sciences, 2022, 583: 174–188
3
Z, Chen L, Jiang C Li . Label distribution-based noise correction for multiclass crowdsourcing. International Journal of Intelligent Systems, 2022, 37( 9): 5752–5767 https://doi.org/10.1002/int.22812
4
N, Zhang J, Xue Y, Ma R, Zhang T, Liang Y A Tan . Hybrid sequence-based android malware detection using natural language processing. International Journal of Intelligent Systems, 2021, 36( 10): 5770–5784
5
Y, Hu Z, Ou X, Xu M Song . A crowdsourcing repeated annotations system for visual object detection. In: Proceedings of the 3rd International Conference on Vision, Image and Signal Processing. 2019, 14
6
E N N, Ocquaye Q, Mao Y, Xue H Song . Cross lingual speech emotion recognition via triple attentive asymmetric convolutional neural network. International Journal of Intelligent Systems, 2021, 36( 1): 53–71
7
V S, Sheng F, Provost P G Ipeirotis . Get another label? Improving data quality and data mining using multiple, noisy labelers. In: Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 2008, 614−622
8
T, Tian J, Zhu B You . Max-margin majority voting for learning from crowds. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019, 41( 10): 2480–2494
9
V S, Sheng J Zhang . Machine learning with crowdsourcing: a brief summary of the past research and future directions. In: Proceedings of the AAAI Conference on Artificial Intelligence. 2019, 9837−9843
10
J Zhang . Knowledge learning with crowdsourcing: a brief review and systematic perspective. IEEE/CAA Journal of Automatica Sinica, 2022, 9( 5): 749–762
11
A P, Dawid A M Skene . Maximum likelihood estimation of observer error-rates using the EM algorithm. Journal of the Royal Statistical Society: Series C (Applied Statistics), 1979, 28( 1): 20–28
V C, Raykar S, Yu L H, Zhao G H, Valadez C, Florin L, Bogoni L Moy . Learning from crowds. The Journal of Machine Learning Research, 2010, 11: 1297–1322
14
M A, Gemalmaz M Yin . Accounting for confirmation bias in crowdsourced label aggregation. In: Proceedings of the 30th International Joint Conference on Artificial Intelligence. 2021, 1729−1735
15
J, Whitehill P, Ruvolo T, Wu J, Bergsma J Movellan . Whose vote should count more: optimal integration of labels from labelers of unknown expertise. In: Proceedings of the 22nd International Conference on Neural Information Processing Systems. 2009, 2035−2043
16
T, Han H, Sun Y, Song Y, Fang X Liu . Find truth in the hands of the few: acquiring specific knowledge with crowdsourcing. Frontiers of Computer Science, 2021, 15( 4): 154315
17
J, Zhang X Wu . Multi-label truth inference for crowdsourcing using mixture models. IEEE Transactions on Knowledge and Data Engineering, 2021, 33( 5): 2083–2095
18
F, Rodrigues F C Pereira . Deep learning from crowds. In: Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence. 2018, 1611−1618
19
M Y, Guan V, Gulshan A M, Dai G E Hinton . Who said what: modeling individual labelers improves classification. In: Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence. 2018, 3109−3118
20
K, Atarashi S, Oyama M Kurihara . Semi-supervised learning from crowds using deep generative models. In: Proceedings of the 32nd AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence. 2018, 1555–1562
21
S Y, Li S J, Huang S Chen . Crowdsourcing aggregation with deep Bayesian learning. Science China Information Sciences, 2021, 64( 3): 130104
22
V S, Sheng J, Zhang B, Gu X Wu . Majority voting and pairing with multiple noisy labeling. IEEE Transactions on Knowledge and Data Engineering, 2019, 31( 7): 1355–1368
23
F, Tao L, Jiang C Li . Label similarity-based weighted soft majority voting and pairing for crowdsourcing. Knowledge and Information Systems, 2020, 62( 7): 2521–2538
24
F, Tao L, Jiang C Li . Differential evolution-based weighted soft majority voting for crowdsourcing. Engineering Applications of Artificial Intelligence, 2021, 106: 104474
25
D R, Karger S, Oh D Shah . Budget-optimal task allocation for reliable crowdsourcing systems. Operations Research, 2014, 62( 1): 1–24
26
H, Li B Yu . Error rate bounds and iterative weighted majority voting for crowdsourcing. 2014, arXiv preprint arXiv: 1411.4086
27
J, Zhang X, Wu V S Sheng . Imbalanced multiple noisy labeling. IEEE Transactions on Knowledge and Data Engineering, 2015, 27( 2): 489–503
28
J, Zhang V S, Sheng J, Wu X Wu . Multi-class ground truth inference in crowdsourcing with clustering. IEEE Transactions on Knowledge and Data Engineering, 2016, 28( 4): 1080–1085
29
J, Zhang M, Wu V S Sheng . Ensemble learning from crowds. IEEE Transactions on Knowledge and Data Engineering, 2019, 31( 8): 1506–1519
30
L, Jiang H, Zhang F, Tao C Li . Learning from crowds with multiple noisy label distribution propagation. IEEE Transactions on Neural Networks and Learning Systems, 2022, 33(11): 6558−6568
31
J, Zhang V S, Sheng B, Nicholson X Wu . CEKA: a tool for mining the wisdom of crowds. The Journal of Machine Learning Research, 2015, 16( 1): 2853–2858
32
I H, Witten E, Frank M A Hall . Data Mining: Practical Machine Learning Tools and Techniques. 3rd ed. Morgan Kaufmann: Elsevier, 2011
33
P, Langley W, Iba K Thompson . An analysis of Bayesian classifiers. In: Proceedings of the Tenth National Conference on Artificial Intelligence. 1992, 223−228
34
J R Quinlan . C4.5: Programs for Machine Learning. San Mateo: Morgan Kaufmann Publishers, 1993
35
Cessie S, le Houwelingen J C van . Ridge estimators in logistic regression. Journal of the Royal Statistical Society: Series C (Applied Statistics), 1992, 41( 1): 191–201
36
J, Alcala-Fdez A, Fernández J, Luengo J, Derrac S, GarcÃa L, Sánchez H Herrera . KEEL data-mining software tool: data set repository, integration of algorithms and experimental analysis framework. Journal of Multiple-Valued Logic and Soft Computing, 2011, 17(2−3): 255−287
37
J Demšar . Statistical comparisons of classifiers over multiple data sets. The Journal of Machine Learning Research, 2006, 7: 1–30
38
L, Jiang L, Zhang C, Li J Wu . A correlation-based feature weighting filter for naive Bayes. IEEE Transactions on Knowledge and Data Engineering, 2019, 31( 2): 201–213
39
A, Oliva A Torralba . Modeling the shape of the scene: a holistic representation of the spatial envelope. International Journal of Computer Vision, 2001, 42( 3): 145–175
