The ClasSi coefficient for the evaluation of ranking quality in the presence of class similarities

doi:10.1007/s11704-012-1175-2

Front Comput Sci

2012, Vol. 6

Issue (5) : 568-580 https://doi.org/10.1007/s11704-012-1175-2

RESEARCH ARTICLE

The ClasSi coefficient for the evaluation of ranking quality in the presence of class similarities

Anca Maria IVANESCU(

), Marc WICHTERICH, Christian BEECKS, Thomas SEIDL

Data Management and Data Exploration Group, RWTH Aachen University, D-52056 Aachen, Germany

Download: PDF(958 KB) HTML
Export: BibTeX | EndNote | Reference Manager | ProCite | RefWorks

Abstract

Evaluationmeasures play an important role in the design of new approaches, and often quality is measured by assessing the relevance of the obtained result set.While many evaluation measures based on precision/recall are based on a binary relevance model, ranking correlation coefficients are better suited for multi-class problems. State-of-the-art ranking correlation coefficients like Kendall’s τ and Spearman’s ρ do not allow the user to specify similarities between differing object classes and thus treat the transposition of objects from similar classes the same way as that of objects from dissimilar classes. We propose ClasSi, a new ranking correlation coefficient which deals with class label rankings and employs a class distance function to model the similarities between the classes. We also introduce a graphical representation of ClasSi which describes how the correlation evolves throughout the ranking.

Keywords ranking quality measure class similarity ClasSi

Corresponding Author(s): IVANESCU Anca Maria,Email:ivanescu@informatik.rwth-aachen.de

Issue Date: 01 October 2012

Cite this article:

Marc WICHTERICH,Christian BEECKS,Thomas SEIDL, et al. The ClasSi coefficient for the evaluation of ranking quality in the presence of class similarities[J]. Front Comput Sci, 2012, 6(5): 568-580.

URL:

https://academic.hep.com.cn/fcs/EN/10.1007/s11704-012-1175-2
https://academic.hep.com.cn/fcs/EN/Y2012/V6/I5/568

1	van Rijsbergen C J. Information Retrieval. 2nd ed. London: Butterworth-Heinemann, 1979
2	Manning C D, Raghavan P, Schütze H. Introduction to Information Retrieval. Cambridge: Cambridge University Press, 2008 doi: 10.1017/CBO9780511809071
3	Flach P A, Blockeel H, Ferri C, Hernández-Orallo J, Struyf J. Decision support for data mining; introduction to ROC analysis and its applications. In: Mladenic D, Lavra?n, Bohanec M, Moyle S, eds. Data Mining and Decision Support: Integration and Collaboration . Boston: Kluwer Academic Publishers, 2003, 81-90
4	Hand D J, Till R J. A simple generalization of the area under the ROC curve for multiple class classification problems. Machine Learning , 2001, 45(2): 171-186 doi: 10.1023/A:1010920819831
5	Ferri C, Hernández-Orallo J, Salido M A. Volume under the ROC surface for multi-class problems. In: Proceedings of the 14th European Conference on Machine Learning . 2003, 108-120
6	Hassan M R, Ramamohanarao K, Karmakar C K, Hossain M M, Bailey J. A novel scalable multi-class ROC for effective visualization and computation. In: Proceedings of the 14th Pacific-Asia Conference on Knowledge Discovery and Data Mining, Part I . 2010, 107-120
7	Kendall M. A new measure of rank correlation. Biometrika , 1938, 30(1-2): 81-89
8	Spearman C. The proof and measurement of association between two things. The American Journal of Psychology , 1987, 100(3/4): 441-471 doi: 10.2307/1422689
9	Kendall M, Gibbons J D. Rank Correlation Methods. London: Edward Arnold, 1990
10	Goodman L A, Kruskal W H. Measures of association for cross classifications. Journal of the American Statistical Association , 1954, 49(268): 732-764
11	Somers R H. A new asymmetric measure of association for ordinal variables. American Sociological Review , 1962, 27(6): 799-811 doi: 10.2307/2090408
12	Ivanescu A, Wichterich M, Seidl T. ClasSi: measuring ranking quality in the presence of object classes with similarity information. In: Proceedings of PAKDD 2011 Quality Issues, Measures of Interestingness and Evaluation of Data Mining Models Workshop . 2011, 185-196
13	Beecks C, Uysal M S, Seidl T. Signature quadratic form distance. In: Proceedings of the 2010 ACM International Conference on Image and Video Retrieval . 2010, 438-445 doi: 10.1145/1816041.1816105
14	Rubner Y, Tomasi C, Guibas L J. The earth mover’s distance as a metric for image retrieval. International Journal of Computer Vision , 2000, 40(2): 99-121 doi: 10.1023/A:1026543900054
15	Wang J Z, Li J, Wiederhold G. Simplicity: semantics-sensitive integrated matching for picture libraries. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2001, 23(9): 947-963 doi: 10.1109/34.955109
16	van de Sande K E A, Gevers T, Snoek C G M. Evaluating color descriptors for object and scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2010, 32(9): 1582-1596 doi: 10.1109/TPAMI.2009.154
17	Hall M, Frank E, Holmes G, Pfahringer B, Reutemann P, Witten I H. The WEKA data mining software: an update. ACM SIGKDD Explorations Newsletter , 2009, 11(1): 10-18 doi: 10.1145/1656274.1656278

[1]	Yunyun WANG, Jiao HAN, Yating SHEN, Hui XUE. Pointwise manifold regularization for semi-supervised learning[J]. Front. Comput. Sci., 2021, 15(1): 151303-.
[2]	Panthadeep BHATTACHARJEE, Pinaki MITRA. A survey of density based clustering algorithms[J]. Front. Comput. Sci., 2021, 15(1): 151308-.
[3]	Qianchen YU, Zhiwen YU, Zhu WANG, Xiaofeng WANG, Yongzhi WANG. Estimating posterior inference quality of the relational infinite latent feature model for overlapping community detection[J]. Front. Comput. Sci., 2020, 14(6): 146323-.
[4]	Parnika PARANJAPE, Meera DHABU, Parag DESHPANDE. A novel classifier for multivariate instance using graph class signatures[J]. Front. Comput. Sci., 2020, 14(4): 144307-.
[5]	Muhammad Aminur RAHAMAN, Mahmood JASIM, Md. Haider ALI, Md. HASANUZZAMAN. Bangla language modeling algorithm for automatic recognition of hand-sign-spelled Bangla sign language[J]. Front. Comput. Sci., 2020, 14(3): 143302-.
[6]	Xibin DONG, Zhiwen YU, Wenming CAO, Yifan SHI, Qianli MA. A survey on ensemble learning[J]. Front. Comput. Sci., 2020, 14(2): 241-258.
[7]	Hui XUE, Haiming XU, Xiaohong CHEN, Yunyun WANG. A primal perspective for indefinite kernel SVM problem[J]. Front. Comput. Sci., 2020, 14(2): 349-363.
[8]	Ying LI, Xiangwei KONG, Haiyan FU, Qi TIAN. Contextual modeling on auxiliary points for robust image reranking[J]. Front. Comput. Sci., 2019, 13(5): 1010-1022.
[9]	Yu HONG, Kai WANG, Weiyi GE, Yingying QIU, Guodong ZHOU. Cursor momentum for fascination measurement[J]. Front. Comput. Sci., 2019, 13(2): 396-412.
[10]	Rizwan Ahmed KHAN, Alexandre MEYER, Hubert KONIK, Saida BOUAKAZ. Saliency-based framework for facial expression recognition[J]. Front. Comput. Sci., 2019, 13(1): 183-198.
[11]	Changlong WANG, Zhiyong FENG, Xiaowang ZHANG, Xin WANG, Guozheng RAO, Daoxun FU. ComR: a combined OWL reasoner for ontology classification[J]. Front. Comput. Sci., 2019, 13(1): 139-156.
[12]	Munish SAINI, Kuljit Kaur CHAHAL. Change profile analysis of open-source software systems to understand their evolutionary behavior[J]. Front. Comput. Sci., 2018, 12(6): 1105-1124.
[13]	Xiaohui TAN, Yachun FAN, Ruiliang GUO. Local features and manifold ranking coupled method for sketch-based 3D model retrieval[J]. Front. Comput. Sci., 2018, 12(5): 1000-1012.
[14]	Shuaiqiang WANG, Yilong YIN. Polygene-based evolutionary algorithms with frequent pattern mining[J]. Front. Comput. Sci., 2018, 12(5): 950-965.
[15]	Chengyu WANG, Guomin ZHOU, Xiaofeng HE, Aoying ZHOU. NERank+: a graph-based approach for entity ranking in document collections[J]. Front. Comput. Sci., 2018, 12(3): 504-517.

Viewed

Full text

Abstract

Cited

Shared

Discussed