|
|
Pose-robust feature learning for facial expression recognition |
Feifei ZHANG,Yongbin YU,Qirong MAO( ),Jianping GOU,Yongzhao ZHAN |
School of Computer Science and Communication Engineering, Jiangsu University, Zhenjiang 212013, China |
|
|
Abstract Automatic facial expression recognition (FER) from non-frontal views is a challenging research topic which has recently started to attract the attention of the research community. Pose variations are difficult to tackle and many face analysis methods require the use of sophisticated normalization and initialization procedures. Thus head-pose invariant facial expression recognition continues to be an issue to traditional methods. In this paper, we propose a novel approach for pose-invariant FER based on pose-robust features which are learned by deep learning methods — principal component analysis network (PCANet) and convolutional neural networks (CNN) (PRP-CNN). In the first stage, unlabeled frontal face images are used to learn features by PCANet. The features, in the second stage, are used as the target of CNN to learn a feature mapping between frontal faces and non-frontal faces. We then describe the non-frontal face images using the novel descriptions generated by the maps, and get unified descriptors for arbitrary face images. Finally, the pose-robust features are used to train a single classifier for FER instead of training multiple models for each specific pose. Our method, on the whole, does not require pose/ landmark annotation and can recognize facial expression in a wide range of orientations. Extensive experiments on two public databases show that our framework yields dramatic improvements in facial expression analysis.
|
Keywords
facial expression recognition
pose-robust features
principal component analysis network (PCANet)
convolutional neural networks (CNN)
|
Corresponding Author(s):
Qirong MAO
|
Just Accepted Date: 28 October 2015
Online First Date: 06 April 2016
Issue Date: 07 September 2016
|
|
1 |
Zheng W M. Multi-view facial expression recognition based on group sparse reduced-rank regression. IEEE Transactions on Affective Computing, 2014, 5(1): 71–85
https://doi.org/10.1109/TAFFC.2014.2304712
|
2 |
Eleftheriadis S, Rudovic O, Pantic M. Discriminative shared gaussian processes for multiview and view-invariant facial expression recognition. IEEE Transactions on Image Processing, 2015, 24(1): 189–204
https://doi.org/10.1109/TIP.2014.2375634
|
3 |
Liu P, Han S Z, Meng Z B, Tong Y. Facial expression recognition via a boosted deep belief network. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2014, 1805–1812
https://doi.org/10.1109/cvpr.2014.233
|
4 |
Zeng Z, Pantic M, Roisman G I, Huang T S. A survey of affect recognition methods: audio, visual, and spontaneous expressions. IEEE Transactions on Pattern Analysis andMachine Intelligence, 2009, 31(1): 39–58
|
5 |
Moore S, Bowden R. Local binary patterns for multi-view facial expression recognition. Computer Vision and Image Understanding, 2011, 115(4): 541–558
https://doi.org/10.1016/j.cviu.2010.12.001
|
6 |
Hesse N, Gehrig T, Gao H, Ekenel H K. Multi-view facial expression recognition using local appearance features. In: Proceedings of International Conference on Pattern Recognition. 2012, 3533–3536
|
7 |
Rudovic O, Pantic M, Patras I. Coupled Gaussian processes for poseinvariant facial expression recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013, 35(6): 1357–1369
https://doi.org/10.1109/TPAMI.2012.233
|
8 |
Kumano S, Otsuka K, Yamato J, Maeda E, Sato Y. Pose-invariant facial expression recognition using variable-intensity templates. International Journal of Computer Vision, 2009, 83(2): 178–194
https://doi.org/10.1007/s11263-008-0185-x
|
9 |
Biswas A, Ghose M K. Expression invariant face recognition using DWT sift features. International Journal of Computer Applications, 2014, 92(2): 30–32
https://doi.org/10.5120/15983-4901
|
10 |
Jian S, Hu C B, Aggarwal J K. Facial expression recognition with temporal modeling of shapes. In: Proceedings of IEEE International Conference on Computer Vision. 2011, 1642–1649
https://doi.org/10.1109/iccvw.2011.6130446
|
11 |
Girisha H, Sreepathi B, Karibasappa K. Multi-view face recognition using local binary pattern. International Journal of Computer Science and Information Technologies, 2014, 5(3): 2978–2981
|
12 |
Dahmane M, Meunier J. Emotion recognition using dynamic gridbased HoG features. In: Proceedings of IEEE International Conference on Automatic Face & Gesture Recognition andWorkshops. 2011, 884–888
|
13 |
Hu Y X, Zeng Z, Yin L J, Wei X Z, Tu J, Huang T S. A study of non-frontal-view facial expressions recognition. In: Proceedings of International Conference on Pattern Recognition. 2008, 1–4
https://doi.org/10.1109/icpr.2008.4761052
|
14 |
Rudovic O, Patras I, Pantic M. Regression-based multiview facial expression recognition. In: Proceedings of International Conference on Pattern Recognition. 2010, 4121–4124
|
15 |
Hu Y X, Zeng Z H, Yin L J, Wei X Z, Zhou X, Huang T S. Multi-view facial expression recognition. In: Proceedings of the 8th IEEE International Conference on Automatic Face and Gesture Recognition. 2008, 56–61
https://doi.org/10.1109/afgr.2008.4813445
|
16 |
Gupta S K, Agrwal S, Meena Y K, Nain N. A hybrid method of feature extraction for facial expression recognition. In: Proceedings of Signal-Image Technology and Internet-Based Systems. 2011, 422–425
https://doi.org/10.1109/sitis.2011.64
|
17 |
Ding C X, Tao D C. A comprehensive survey on pose-invariant face recognition. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2015
|
18 |
Tong Y, Chen J X, Ji Q. A unified probabilistic framework for spontaneous facial action modeling and unerstanding. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010, 32(2): 258–273
https://doi.org/10.1109/TPAMI.2008.293
|
19 |
Zheng W M, Tang H, Lin Z C, Huang T S. Emotion recognition from arbitrary view facial images. In: Proceedings of European Conference on Computer Vision. 2010, 490–503
https://doi.org/10.1007/978-3-642-15567-3_36
|
20 |
Sung J, Kim D. Real-time facial expression recognition using STAAM and layered GDA classifier. Image and Vision Computing, 2009, 27(9): 1315–1325
https://doi.org/10.1016/j.imavis.2008.11.010
|
21 |
Tang H, Hasegawa-Johnson M, Huang T. Non-frontal view facial expression recognition based on ergodic hidden markov model supervectors. In: Proceedings of IEEE International Conference on Multimedia and Expo. 2010, 1202–1207
https://doi.org/10.1109/icme.2010.5582576
|
22 |
Ranzato M, Susskind J, Mnih V, Hinton G. On deep generative models with applications to recognition. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2011, 2857–2864
https://doi.org/10.1109/cvpr.2011.5995710
|
23 |
Rifai S, Bengio Y, Courville A, Vincent P, Mirza M. Disentangling factors of variation for facial expression recognition. In: Proceedings of European Conference on Computer Vision. 2012, 808–822
https://doi.org/10.1007/978-3-642-33783-3_58
|
24 |
Eleftheriadis S, Rudovic O, Pantic M. Shared Gaussian process latent variable model for multi-view facial expression recognition. In: Proceedings of International Symposium on Visual Computing. 2013, 527–538
https://doi.org/10.1007/978-3-642-41914-0_52
|
25 |
Liu M Y, Shan S G, Wang R P, Chen X L. Learning expressionlets on spatio-temporal manifold for dynamic facial expression recognition. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2014, 1749–1756
https://doi.org/10.1109/cvpr.2014.226
|
26 |
Rifai S, Vincent P, Muller X, Glorot X, Bengio Y. Contracting autoencoders: explicit invariance during feature extraction. In: Proceedings of International Conference on Machine Learning. 2011, 833–840
|
27 |
Saudagare P V, Chaudhari D S. Facial expression recognition using neural network — an overview. International Journal of Soft Computing and Engineering, 2012, 2(1): 224–227
|
28 |
Li J G, Zhang Y M. Learning surf cascade for fast and accurate object detection. In: Proceedings of the 26th IEEE Conference on Computer Vision and Pattern Recognition. 2013, 3468–3475
https://doi.org/10.1109/cvpr.2013.445
|
29 |
Viola P, Jones M. Rapid object detection using a boosted cascade of simple features. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2001, 511–518
https://doi.org/10.1109/cvpr.2001.990517
|
30 |
Yin L J, Wei X Z, Sun Y, Wang J, Rosato M J. A 3D facial expression database for facial behavior research. In: Proceedings of Automatic face and gesture recognition. 2006, 211–216
|
31 |
Dhall A, Goecke R, Lucey S, Gedeon T. Static facial expressions analysis in tough conditions: data, evaluation protocol and benchmark. In: Proceedings of IEEE International Conference on Computer Vision. 2011, 2106–2112
https://doi.org/10.1109/iccvw.2011.6130508
|
32 |
Zheng W M, Tang H, Lin Z C, Huang T S. A novel approach to expression recognition from non-frontal face images. In: Proceedings of the 12th IEEE International Conference on Computer Vision. 2009, 1901–1908
|
33 |
Tariq U, Yang J, Huang T. Maximum margin gmm learning for facial expression recognition. In: Proceedings of Automatic Face and Gesture Recognition. 2013, 1–6
https://doi.org/10.1109/fg.2013.6553794
|
34 |
Tariq U, Yang J C, Huang T S. Supervised super-vector encoding for facial expression recognition. Pattern Recognition, 2014, 89–95
https://doi.org/10.1016/j.patrec.2014.05.011
|
35 |
Jampour M, Mauthner T, Bischof H. Multi-view facial expressions recognition using local linear regression of sparse codes. In: Proceedings of the 20th Computer Vision Winter Workshop Paul Wohlhart, 2015
|
36 |
Tariq U, Yang J C, Huang T S. Multi-view facial expression recognition analysis with generic sparse coding feature. In: Proceedings of European Conference on Computer Vision. 2012, 578–588
https://doi.org/10.1007/978-3-642-33885-4_58
|
37 |
Kan M N, Shan S G, Zhang H H, Lao S H, Chen X L. Multi-view discriminant analysis. In: Proceedings of European Conference on Computer Vision. 2012, 808–821
https://doi.org/10.1007/978-3-642-33718-5_58
|
38 |
Sharma A, Kumar A, Daume H, Jacobs D W. Generalized multiview analysis: a discriminative latent space. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2012, 2160–2167
https://doi.org/10.1109/cvpr.2012.6247923
|
|
Viewed |
|
|
|
Full text
|
|
|
|
|
Abstract
|
|
|
|
|
Cited |
|
|
|
|
|
Shared |
|
|
|
|
|
Discussed |
|
|
|
|