Kernel feature extraction methods observed from the viewpoint of generating-kernels

doi:10.1007/s11460-011-0129-z

Front Elect Electr Eng Chin

2011, Vol. 6

Issue (1) : 43-55 https://doi.org/10.1007/s11460-011-0129-z

RESEARCH ARTICLE

Kernel feature extraction methods observed from the viewpoint of generating-kernels

Jian YANG(

)

School of Computer Science and Technology, Nanjing University of Science and Technology, Nanjing 210094, China

Download: PDF(258 KB) HTML
Export: BibTeX | EndNote | Reference Manager | ProCite | RefWorks

Abstract

This paper introduces an idea of generating a kernel from an arbitrary function by embedding the training samples into the function. Based on this idea, we present two nonlinear feature extraction methods: generating kernel principal component analysis (GKPCA) and generating kernel Fisher discriminant (GKFD). These two methods are shown to be equivalent to the function-mapping-space PCA (FMS-PCA) and the function-mapping-space linear discriminant analysis (FMS-LDA) methods, respectively. This equivalence reveals that the generating kernel is actually determined by the corresponding function map. From the generating kernel point of view, we can classify the current kernel Fisher discriminant (KFD) algorithms into two categories: KPCA+ LDA based algorithms and straightforward KFD (SKFD) algorithms. The KPCA+ LDA based algorithms directly work on the given kernel and are not suitable for non-kernel functions, while the SKFD algorithms essentially work on the generating kernel from a given symmetric function and are therefore suitable for non-kernels as well as kernels. Finally, we outline the tensor-based feature extraction methods and discuss ways of extending tensor-based methods to their generating kernel versions.

Keywords kernel methods feature extraction principal component analysis (PCA) Fisher linear discriminant analysis (FLD or LDA) tensor-based methods

Corresponding Author(s): YANG Jian,Email:csjyang@mail.njust.edu.cn

Issue Date: 05 March 2011

Cite this article:

Jian YANG. Kernel feature extraction methods observed from the viewpoint of generating-kernels[J]. Front Elect Electr Eng Chin, 2011, 6(1): 43-55.

URL:

https://academic.hep.com.cn/fee/EN/10.1007/s11460-011-0129-z
https://academic.hep.com.cn/fee/EN/Y2011/V6/I1/43

1	Vapnik V. The Nature of Statistical Learning Theory. New York: Springer, 1995
2	Müller K R, Mika S, R?tsch G, Tsuda K, Sch?lkopf B. An introduction to kernel-based learning algorithms. IEEE Transactions on Neural Networks , 2001, 12(2): 181-201 doi: 10.1109/72.914517
3	Sch?lkopf B, Smola A, Muller K R. Nonlinear component analysis as a kernel eigenvalue problem. Neural Computation , 1998, 10(5): 1299-1319 doi: 10.1162/089976698300017467
4	Mika S, R?tsch G, Weston J, Sch?lkopf B, Müller K R. Fisher discriminant analysis with kernels. In: Proceedings of IEEE International Workshop on Neural Networks for Signal Processing IX . 1999, 41-48
5	Mika S, R?tsch G, Sch?lkopf B, Smola A, Weston J, Müller K R. Invariant feature extraction and classification in kernel spaces. Advances in Neural Information Processing Systems , 1999, 12: 526-532
6	Baudat G, Anouar F. Generalized discriminant analysis using a kernel approach. Neural Computation , 2000, 12(10): 2385-2404 doi: 10.1162/089976600300014980
7	Roth V, Steinhage V. Nonlinear discriminant analysis using kernel functions. In: Solla S A, Leen T K, Mueller K R, eds. Advances in Neural Information Processing Systems . 2000, 12: 568-574
8	Mika S, R?tsch G, Weston J, Sch?lkopf B, Smola A, Müller K R. Constructing descriptive and discriminative non-linear features: Rayleigh coefficients in kernel feature spaces. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2003, 25(5): 623-628 doi: 10.1109/TPAMI.2003.1195996
9	Yang M H. Kernel Eigenfaces vs. kernel Fisherfaces: face recognition using kernel methods. In: Proceedings of the 5th IEEE International Conference on Automatic Face and Gesture Recognition . 2002, 215-220 doi: 10.1109/AFGR.2002.4527207
10	Lu J, Plataniotis K N, Venetsanopoulos A N. Face recognition using kernel direct discriminant analysis algorithms. IEEE Transactions on Neural Networks , 2003, 14(1): 117-126 doi: 10.1109/TNN.2002.806629
11	Sch?lkopf B, Smola A. Learning with Kernels. Cambridge: MIT Press, 2002
12	Shawe-Taylor J, Cristianini N. Kernel Methods for Pattern Analysis. Cambridge: Cambridge University Press, 2004
13	Xu J, Zhang X, Li Y. Kernel MSE algorithm: a unified framework for KFD, LS-SVM, and KRR. In: Proceedings of the International Joint Conference on Neural Networks . 2001, 1486-1491
14	Billings S A, Lee K L. Nonlinear fisher discriminant analysis using a minimum squared error cost function and the orthogonal least squares algorithm. Neural Networks , 2002, 15(2): 263-270
15	Zheng W, Zhao L, Zou C. Foley-Sammon optimal discriminant vectors using kernel approach. IEEE Transactions on Neural Networks , 2005, 16(1): 1-9 doi: 10.1109/TNN.2004.836239
16	Yang J, Jin Z, Yang J Y, Zhang D, Frangi A F. Essence of kernel Fisher discriminant: KPCA plus LDA. Pattern Recognition , 2004, 37(10): 2097-2100 doi: 10.1016/j.patcog.2003.10.015
17	Yang J, Frangi A F, Yang J Y, Zhang D, Jin Z. KPCA plus LDA: a complete kernel Fisher discriminant framework for feature extraction and recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2005, 27(2): 230-244 doi: 10.1109/TPAMI.2005.33
18	Xu Y, Zhang D, Jin Z, Li M, Yang J Y. A fast kernelbased nonlinear discriminant analysis for multi-class problems. Pattern Recognition , 2006, 39(6): 1026-1033 doi: 10.1016/j.patcog.2005.10.029
19	Zhao J, Wang H, Ren H, Kee S C. LBP discriminant analysis for face verification. In: Proceeding of the 2005 IEEE Conference on Computer Vision and Pattern Recognition . 2005, 3: 167
20	Liu C. Capitalize on dimensionality increasing techniques for improving face recognition grand challenge performance. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2006, 28(5): 725-737 doi: 10.1109/TPAMI.2006.90
21	Cevikalp H, Neamtu M, Wilkes M, Barkana A. Discriminative common vector method with kernels. IEEE Transactions on Neural Networks , 2006, 17(6): 1550-1565 doi: 10.1109/TNN.2006.881485
22	Bach F R, Jordan MI. Kernel independent component analysis. Journal of Machine Learning Research , 2002, 3(1): 1-48 doi: 10.1162/153244303768966085
23	Yang J, Gao X, Zhang D, Yang J Y. Kernel ICA: an alternative formulation and its application to face recognition. Pattern Recognition , 2005, 38(10): 1784-1787 doi: 10.1016/j.patcog.2005.01.023
24	Ma J. Function replacement vs. kernel trick. Neurocomputing , 2003, 50: 479-483 doi: 10.1016/S0925-2312(02)00673-2
25	Ma J, Theiler J, Perkins S. Two realizations of a general feature extraction framework. Pattern Recognition , 2004, 37(5): 875-887 doi: 10.1016/j.patcog.2003.10.010
26	Lodhi H, Saunders C, Shawe-Taylor J, Cristianini N, Watkins C. Text classification using string kernels. Journal of Machine Learning Research , 2002, 2(2): 419-444 doi: 10.1162/153244302760200687
27	Chen W S, Yuen P C, Huang J, Lai J. Wavelet kernel construction for kernel discriminant analysis on face recognition. In: Proceeding of the 2006 Conference on Computer Vision and Pattern Recognition Workshop . 2006, 47-52 doi: 10.1109/CVPRW.2006.213
28	Sch?lkopf B. Support vector learning. Dissertation for the Doctoral Degree . Berlin: Berlin Technical University, 1997
29	Camps-Valls G, Martin-Guerrero J, Rojo-Alvarez J, Soria-Olivas E. Fuzzy sigmoid kernel for support vector classifiers. Neurocomputing , 2004, 62: 501-506 doi: 10.1016/j.neucom.2004.07.004
30	Ahonen T, Hadid A, Pietik?inen M. Face description with local binary patterns: application to face recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2006, 28(12): 2037-2041 doi: 10.1109/TPAMI.2006.244
31	Mangasarian L. Generalized support vector machines. In: Smola A J, Bartlett P, Schokopf B, Schuurmans D, eds. Advances in Large Margin Classifiers . 2000, 135-146
32	Golub G H, Van Loan C F. Matrix Computations. 3rd ed. Baltimore: The Johns Hopkins University Press, 1996
33	Swets D L,Weng J. Using discriminant eigenfeatures for image retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence , 1996, 18(8): 831-836 doi: 10.1109/34.531802
34	Belhumeur P N, Hespanha J P, Kriengman D J. Eigenfaces vs. Fisherfaces: Recognition using class specific linear projection. IEEE Transactions on Pattern Analysis and Machine Intelligence , 1997, 19(7): 711-720 doi: 10.1109/34.598228
35	Yang J, Yang J Y. Why can LDA be performed in PCA transformed space? Pattern Recognition , 2003, 36(2): 563-566 doi: 10.1016/S0031-3203(02)00048-1
36	Liu C J, Wechsler H. Robust coding schemes for indexing and retrieval from large face databases. IEEE Transactions on Image Processing , 2000, 9(1): 132-137 doi: 10.1109/83.817604
37	Turk M, Pentland A. Eigenfaces for recognition. Journal of Cognitive Neuroscience , 1991, 3(1): 71-86 doi: 10.1162/jocn.1991.3.1.71
38	Liu K, Cheng Y Q, Yang J Y. Algebraic feature extraction for image recognition based on an optimal discriminant criterion. Pattern Recognition , 1993, 26(6): 903-911 doi: 10.1016/0031-3203(93)90056-3
39	Yang J, Zhang D, Frangi A F, Yang J Y. Two-dimensional PCA: A new approach to appearance-based face representation and recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2004, 26(1): 131-137 doi: 10.1109/TPAMI.2004.1261097
40	Visani M, Garcia C, Laurent C. Comparing robustness of two-dimensional PCA and eigenfaces for face recognition. Lecture Notes in Computer Science , 2004, 3212: 717-724 doi: 10.1007/978-3-540-30126-4_87
41	Zuo W M, Zhang D. K. Wang K. An assembled matrix distance metric for 2DPCA-based image recognition. Pattern Recognition Letters , 2006, 27(3): 210-216
42	Zhang D Q, Zhou Z H. (2D)(2)PCA: Two-directional twodimensional PCA for efficient face representation and recognition. Neurocomputing , 2005, 69(1-3): 224-231 doi: 10.1016/j.neucom.2005.06.004
43	Kong H,Wang L, Teoh E K, Li X, Wang J G, Venkateswarlu R. Generalized 2D principal component analysis for face image representation and recognition. Neural Networks , 2005, 18(5-6): 585-594 doi: 10.1016/j.neunet.2005.06.041
44	Ye J. Generalized low rank approximations of matrices. Machine Learning , 2005, 61(1-3): 167-191 doi: 10.1007/s10994-005-3561-6
45	Hyv?rinen A, Oja E. Independent component analysis: algorithms and applications. Neural Networks , 2000, 13(4-5): 411-430 doi: 10.1016/S0893-6080(00)00026-5
46	He X, Yan S, Hu Y, Niyogi P, Zhang H J. Face recognition using Laplacianfaces. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2005, 27(3): 328-340 doi: 10.1109/TPAMI.2005.55
47	Ye J, Janardan R, Li Q. Two-dimensional linear discriminant analysis. Advances in Neural Information Processing Systems , 2004, 17: 1569-1576
48	Li M, Yuan B. 2D-LDA: a novel statistical linear discriminant analysis for image matrix. Pattern Recognition Letters , 2005, 26(5): 527-532 doi: 10.1016/j.patrec.2004.09.007
49	Xiong H, Swamy M, Ahmad M. Two-dimensional FLD for face recognition. Pattern Recognition , 2005, 38(7): 1121-1124 doi: 10.1016/j.patcog.2004.12.003
50	Yang J, Zhang D, Yong X, Yang J. Two-dimensional linear discriminant transform for face recognition. Pattern Recognition , 2005, 38(7): 1125-1129 doi: 10.1016/j.patcog.2004.11.019
51	Zuo W, Zhang D, Yang J, Wang K. BDPCA plus LDA: A novel fast feature extraction technique for face recognition. IEEE Transactions on Systems, Man, and Cybernetics, Part B , 2006, 36(4): 946-953
52	Zou C, Sun N, Ji Z, Zhao L. 2DCCA: a novel method for small sample size face recognition. In: Proceedings of IEEE Workshop on Applications of Computer Vision . 2007, 43
53	Lee S H, Choi S. Two-dimensional canonical correlation analysis. IEEE Signal Processing Letters , 2007, 14(10): 735-738 doi: 10.1109/LSP.2007.896438
54	Gao Q, Zhang L, Zhang D, Xu H. Independent components extraction from image matrix. Pattern Recognition Letters , 2010, 31(3): 171-178 doi: 10.1016/j.patrec.2009.10.014
55	Chen S B, Zhao H F, Kong M, Luo B. 2d-lpp: a twodimensional extension of locality preserving projections. Neurocomputing , 2007, 70(4-6): 912-921
56	Hu D W, Feng G Y, Zhou Z T. Two-dimensional locality preserving projections (2DLPP) with its application to palmprint recognition. Pattern Recognition , 2007, 40(1): 339-342 doi: 10.1016/j.patcog.2006.06.022
57	Xu D, Yan S, Zhang L, Lin S, Zhang H J, Huang T S. Reconstruction and recognition of tensor-based objects with concurrent subspaces analysis. IEEE Transactions on Circuits and Systems for Video Technology , 2008, 18(1): 36-47 doi: 10.1109/TCSVT.2007.903317
58	Wang H, Ahuja N. A tensor approximation approach to dimensionality reduction. Journal of Computer Vision , 2008, 76(3): 217-229 doi: 10.1007/s11263-007-0053-0
59	Tao D, Li X, Wu X, Maybank S J. General tensor discriminant analysis and gabor features for gait recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2007, 29(10): 1700-1715 doi: 10.1109/TPAMI.2007.1096
60	Lu H, Plataniotis K N, Venetsanopoulos A N. MPCA: multilinear principal component analysis of tensor objects. IEEE Transactions on Neural Networks , 2008, 19(1): 18-39 doi: 10.1109/TNN.2007.901277
61	Zhang L, Gao Q, Zhang D. Directional independent component analysis with tensor representation. In: Proceedings of Computer Vision and Pattern Recognition . 2008, 1-7
62	Sun N, Wang H, Ji Z, Zou C, Zhao L. An efficient algorithm for kernel two-dimensional principal component analysis. Neural Computing & Applications , 2008, 17(1): 59-64 doi: 10.1007/s00521-007-0111-0
63	Liu J, Chen S, Zhou Z H, Tan X. Generalized low-rank approximations of matrices revisited. IEEE Transactions on Neural Networks , 2010, 21(4): 621-632 doi: 10.1109/TNN.2010.2040290

[1]	Wankou YANG, Changyin SUN, Zhenyu WANG. Finger-knuckle-print recognition using Gabor feature and MMDA[J]. Front Elect Electr Eng Chin, 2011, 6(2): 374-380.
[2]	Xudong ZHAO, Peng LIU, Jiafeng LIU, Xianglong TANG. Feature extraction for classification of different weather conditions[J]. Front Elect Electr Eng Chin, 2011, 6(2): 339-346.
[3]	Shikui TU, Lei XU. An investigation of several typical model selection criteria for detecting the number of signals[J]. Front Elect Electr Eng Chin, 2011, 6(2): 245-255.
[4]	Hujun YIN. Advances in adaptive nonlinear manifolds and dimensionality reduction[J]. Front Elect Electr Eng Chin, 2011, 6(1): 72-85.
[5]	Erkki OJA, Zhirong YANG. Orthogonal nonnegative learning for sparse feature extraction and approximate combinatorial optimization[J]. Front Elect Electr Eng Chin, 2010, 5(3): 261-273.
[6]	WANG Tong, SHEN Hongbin, YAO Lixiu, YANG Jie, CHOU Kuochen. PCA for predicting quaternary structure of protein[J]. Front. Electr. Electron. Eng., 2008, 3(4): 376-380.
[7]	WANG Ying, GAO Xinbo. Mass detection algorithm based on support vector machine and relevance feedback[J]. Front. Electr. Electron. Eng., 2008, 3(3): 267-273.
[8]	GUO Qiang, ZHANG Xingzhou, LI Zheng. A feature extraction method for the signal sorting of interleaved radar pulse serial[J]. Front. Electr. Electron. Eng., 2007, 2(3): 330-333.
[9]	. A new feature extraction method using the amplitude fluctuation property of target HRRP for radar automatic target recognition[J]. Front. Electr. Electron. Eng., 2006, 1(2): 171-176.

Viewed

Full text

Abstract

Cited

Shared

Discussed