Dimensionality reduction with latent variable model

doi:10.1007/s11460-012-0179-x

Front Elect Electr Eng

2012, Vol. 7

Issue (1) : 116-126 https://doi.org/10.1007/s11460-012-0179-x

RESEARCH ARTICLE

Dimensionality reduction with latent variable model

Xinbo GAO(

), Xiumei WANG

School of Electronic Engineering, Xidian University, Xi’an 710071, China

Download: PDF(627 KB) HTML
Export: BibTeX | EndNote | Reference Manager | ProCite | RefWorks

Abstract

Over the past few decades, latent variable model (LVM)-based algorithms have attracted considerable attention for the purpose of data dimensionality reduction, which plays an important role in machine learning, pattern recognition, and computer vision. LVM is an effective tool for modeling density of the observed data. It has been used in dimensionality reduction for dealing with the sparse observed samples. In this paper, two LVM-based dimensionality reduction algorithms are presented firstly, i.e., supervised Gaussian process latent variable model and semi-supervised Gaussian process latent variable model. Then, we propose an LVMbased transfer learning model to cope with the case that samples are not independent identically distributed. In the end of each part, experimental results are given to demonstrate the validity of the proposed dimensionality reduction algorithms.

Keywords dimensionality reduction latent variable model pairwise constraints Bregman divergence

Corresponding Author(s): GAO Xinbo,Email:xbgao@mail.xidian.edu.cn

Issue Date: 05 March 2012

Cite this article:

Xinbo GAO,Xiumei WANG. Dimensionality reduction with latent variable model[J]. Front Elect Electr Eng, 2012, 7(1): 116-126.

URL:

https://academic.hep.com.cn/fee/EN/10.1007/s11460-012-0179-x
https://academic.hep.com.cn/fee/EN/Y2012/V7/I1/116

1	Koren Y, Carmel L. Robust linear dimensionality reduction. IEEE Transactions on Visualization and Computer Graphics , 2004, 10(4): 459-470 doi: 10.1109/TVCG.2004.17
2	Duda R O, Hart P E, Stork D G. Pattern Classification. 2nd ed. New York: Wiley, 2000
3	Tao D, Li X, Wu X, HuW, Maybank S J. Supervised tensor learning. Knowledge and Information Systems , 2007, 13(1): 1-42 doi: 10.1007/s10115-006-0050-6
4	Tao D, Li X, Wu X, Maybank S J. General averaged divergences analysis. In: Proceedings of IEEE International Conference on Data Mining . 2007, 302-311
5	He X. Laplacian regularized D-optimal design for active learning and its application to image retrieval. IEEE Transactions on Image Processing , 2010, 19(1): 254-263 doi: 10.1109/TIP.2009.2032342
6	Tao D, Tang X, Li X, Rui Y. Direct kernel biased discriminant analysis: A new content-based image retrieval relevance feedback algorithm. IEEE Transactions on Multimedia , 2006, 8(4): 716-727 doi: 10.1109/TMM.2005.861375
7	Joliffe I. Principal Component Analysis. New York: Springer, 1986
8	Anderson T W. Asymptotic theory for principal component analysis. Annals of Mathematical Statistics , 1963, 34(1): 122-148 doi: 10.1214/aoms/1177704248
9	Sch?lkopf B, Smola A, Müller K R. Nonlinear component analysis as a kernel eigenvalue problem. Neural Computation , 1998, 10(5): 1299-1319 doi: 10.1162/089976698300017467
10	Fiori S. Visualization of Riemannian-manifold-valued elements by multidimensional scaling. Neurocomputing , 2011, 74(6): 983-992 doi: 10.1016/j.neucom.2010.11.015
11	Roweis S T, Saul L K. Nonlinear dimensionality reduction by locally linear embedding. Science , 2000, 290(5500): 2323-2326 doi: 10.1126/science.290.5500.2323
12	Li X, Lin S, Yan S, Xu D. Discriminant locally linear embedding with high-order tensor data. IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics , 2008, 38(2): 342-352 doi: 10.1109/TSMCB.2007.911536
13	Belkin M, Niyogi P. Laplacian eigenmaps for dimensionality reduction and data representation. Neural Computation , 2003, 15(6): 1373-1396 doi: 10.1162/089976603321780317
14	Tenenbaum J B, de Silva V, Langford J C. A global geometric framework for nonlinear dimensionality reduction. Science , 2000, 290(5500): 2319-2323 doi: 10.1126/science.290.5500.2319
15	de Silva V, Tenenbaum J B. Global versus local methods in nonlinear dimensionality reduction. Advances in Neural Information Processing Systems , 2003, 15: 705-712
16	He X, Niyogi P. Locality preserving projections. Advances in Neural Information Processing Systems , 2003, 16: 153-160
17	He X, Yan S, Hu Y, Niyogi P, Zhang H-J. Face recognition using Laplacianfaces. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2005, 27(3): 328-340 doi: 10.1109/TPAMI.2005.55
18	He X, Cai D, Yan S, Zhang H-J. Neighborhood preserving embedding. In: Proceedings of the 10th International Conference on Computer Vision . 2005, 2: 1208-1213
19	Bar-Hillel A, Hertz T, Shental N, Weinshall D. Learning a Mahalanobis metric from equivalence constraints. Journal of Machine Learning Research , 2005, 6(1): 937-965
20	Mika S, R?tsch G,Weston J, Sch?lkopf B, Müller K R. Fisher discriminant analysis with kernels. In: Proceedings of IEEE Signal Processing Society Workshop (Neural Networks for Signal Processing IX) . 1999, 41-48
21	Baudat G, Anouar F. Generalized discriminant analysis using a kernel approach. Neural Computation , 2000, 12(10): 2385-2404 doi: 10.1162/089976600300014980
22	Yan S, Xu D, Zhang B, Zhang H-J, Yang Q, Lin S. Graph embedding and extensions: A general framework for dimensionality reduction. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2007, 29(1): 40-51 doi: 10.1109/TPAMI.2007.250598
23	Sugiyama M. Dimensionality reduction of multimodal labeled data by local Fisher discriminant analysis. Journal of Machine Learning Research , 2007, 8: 1027-1061
24	Tipping M E, Bishop C M. Probabilistic principal component analysis. Journal of the Royal Statistical Society, Series B: Methodological , 1999, 61(3): 611-622 doi: 10.1111/1467-9868.00196
25	Bartholomew D J. Statistical Factor Analysis and Related Methods. New York: Wiley, 2004
26	Bishop C M, Svensen M, Williams C K I. GTM: The generative topographic mapping. Neural Computation , 1998, 10(1): 215-234 doi: 10.1162/089976698300017953
27	Lawrence N D. Gaussian process models for visualization of high dimensional data. Advances in Neural Information Processing Systems , 2004, 16: 329-336
28	Lawrence N D. Probabilistic non-linear principal component analysis with Gaussian process latent variable models. Journal of Machine Learning Research , 2005, 6: 1783-1816
29	Bishop C M. Learning in Graphical Models. Cambridge: MIT Press, 1999
30	Rabiner L R, Juang B H. An introduction to hidden Markov models. IEEE ASSP Magazine , 1986, 3(1): 4-16 doi: 10.1109/MASSP.1986.1165342
31	Blei D M, Ng A Y, Jordan M I. Latent Dirichlet allocation. Journal of Machine Learning Research , 2003, 3: 993-1022
32	Blei D M, Jordan M I. Variational methods for the Dirichlet process. In: Proceedings of the 21st International Conference on Machine Learning . 2004
33	Rabiner L R. A tutorial on hidden Markov models and selected applications in speech recognition. Proceedings of the IEEE , 1989, 77(2): 257-286 doi: 10.1109/5.18626
34	Baker J K. The DRAGON system — An overview. IEEE Transactions on Acoustics, Speech, and Signal Processing , 1975, 23(1): 24-29 doi: 10.1109/TASSP.1975.1162650
35	Zhong J, Gao X, Tian C. Face sketch synthesis using E-HMM and selective ensemble. In: Proceedings of IEEE International Conference on Acoustics, Speech, and Signal Processing . 2007, 1: 485-488
36	Gao X, Zhong J, Li J, Tian C. Face sketch synthesis algorithm based on E-HMM and selective ensemble. IEEE Transactions on Circuits and Systems for Video Technology , 2008, 18(4): 487-496 doi: 10.1109/TCSVT.2008.918770
37	Fritz M, Schiele B. Decomposition, discovery and detection of visual categories using topic models. In: Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition . 2008, 1-8
38	Blei D M, Griffiths T L, Jordan M I. The nested Chinese restaurant process and Bayesian nonparametric inference of topic hierarchies. Journal of the Association for Computing Machinery , 2010, 57(2): 21-30 doi: 10.1145/1667053.1667056
39	Li F, Perona P. A Bayesian hierarchical model for learning natural scene categories. In: Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition . 2005, 2: 524-531
40	Wang C, Blei D, Li F. Simultaneous image classification and annotation. In: Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition . 2009, 1903-1910
41	Niu Z, Hua G, Gao X, Tian Q. Spatial-DiscLDA for visual recognition. In: Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition . 2011, 1769-1776
42	Wang J M, Fleet D J, Hertzmann A. Gaussian process dynamical models for human motion. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2008, 30(2): 283-298 doi: 10.1109/TPAMI.2007.1167
43	Urtasun R, Fleet D J, Fua P. 3D people tracking with Gaussian process dynamical models. In: Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition . 2006, 1: 238-245
44	Navaratnam R, Fitzgibbon A W, Cipolla R. The joint manifold model for semi-supervised multi-valued regression. In: Proceedings of the 11th IEEE International Conference on Computer Vision . 2007, 1-8 doi: 10.1109/ICCV.2007.4408976
45	Gupta A, Chen T, Chen F, Kimber D, Davis L S. Context and observation driven latent variable model for human pose estimation. In: Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition . 2008, 1-8
46	Salzmann M, Urtasun R, Fua P. Local deformation models for monocular 3D shape recovery. In: Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition . 2008, 1-8
47	Gao X, Wang X, Tao D, Li X. Supervised Gaussian process latent variable model for dimensionality reduction. IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics , 2011, 41(2): 425-434 doi: 10.1109/TSMCB.2010.2057422
48	Wang X, Gao X, Yuan Y, Tao D, Li J. Semi-supervised Gaussian process latent variable model with pairwise constraints. Neurocomputing , 2010, 73(10-12): 2186-2195 doi: 10.1016/j.neucom.2010.01.021
49	Gao X, Wang X, Li X, Tao D. Transfer latent variable model based on divergence analysis. Pattern Recognition , 2011, 44(10-11): 2358-2366 doi: 10.1016/j.patcog.2010.06.013
50	Nock R, Nielsen F. On weighting clustering. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2006, 28(8): 1223-1235 doi: 10.1109/TPAMI.2006.168
51	Cayton L. Fast nearest neighbor retrieval for Bregman divergences. In: Proceedings of the 25th International Conference on Machine Learning . 2008, 112-119 doi: 10.1145/1390156.1390171

[1]	Liang LIAO, Yanning ZHANG. MRI image segmentation based on fast kernel clustering analysis[J]. Front Elect Electr Eng Chin, 2011, 6(2): 363-373.
[2]	Lijun ZHANG, Zhengguang CHEN, Miao ZHENG, Xiaofei HE. Robust non-negative matrix factorization[J]. Front Elect Electr Eng Chin, 2011, 6(2): 192-200.
[3]	Hujun YIN. Advances in adaptive nonlinear manifolds and dimensionality reduction[J]. Front Elect Electr Eng Chin, 2011, 6(1): 72-85.

Viewed

Full text

Abstract

Cited

Shared

Discussed