Fusion of visible and thermal images for facial expression recognition

doi:10.1007/s11704-014-2345-1

Front. Comput. Sci.

2014, Vol. 8

Issue (2) : 232-242 https://doi.org/10.1007/s11704-014-2345-1

RESEARCH ARTICLE

Fusion of visible and thermal images for facial expression recognition

Shangfei WANG^1,²(

), Shan HE^1,², Yue WU³, Menghua HE^1,², Qiang JI³

¹. School of Computer Science and Technology, University of Science and Technology of China, Hefei 230027, China
². Key Lab of Computing and Communicating Software of Anhui Province, Hefei 230027, China
³. Department of Electrical, Computer, and Systems Engineering, Rensselaer Polytechnic Institute, Troy NY12180-3590, USA

Download: PDF(510 KB)
Export: BibTeX | EndNote | Reference Manager | ProCite | RefWorks

Abstract

Most present research into facial expression recognition focuses on the visible spectrum, which is sensitive to illumination change. In this paper, we focus on integrating thermal infrared data with visible spectrum images for spontaneous facial expression recognition. First, the active appearance model AAM parameters and three defined head motion features are extracted from visible spectrum images, and several thermal statistical features are extracted from infrared (IR) images. Second, feature selection is performed using the F-test statistic. Third, Bayesian networks BNs and support vector machines SVMs are proposed for both decision-level and feature-level fusion. Experiments on the natural visible and infrared facial expression (NVIE) spontaneous database show the effectiveness of the proposed methods, and demonstrate thermal IR images’ supplementary role for visible facial expression recognition.

Keywords facial expression recognition feature-level fusion decision-level fusion support vector machine Bayesian network thermal infrared images visible spectrum images

Corresponding Author(s): Shangfei WANG

Issue Date: 24 June 2014

Cite this article:

Shangfei WANG,Shan HE,Yue WU, et al. Fusion of visible and thermal images for facial expression recognition[J]. Front. Comput. Sci., 2014, 8(2): 232-242.

URL:

https://academic.hep.com.cn/fcs/EN/10.1007/s11704-014-2345-1
https://academic.hep.com.cn/fcs/EN/Y2014/V8/I2/232

1	Z Zeng, M Pantic, G Roisman, T Huang. A survey of affect recognition methods: audio, visual, and spontaneous expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2009, 31(1): 39−58 https://doi.org/10.1109/TPAMI.2008.52
2	Z Wang, F Liu, L Wang. Survey of facial expression recognition based on computer vision. Computer Engineering, 2006, 32(11): 231−233
3	X m Liu, H c Tan, Y j Zhang. New research advances in facial expression recogntion. Journal of Image and Graphics, 2006, 11(10): 1359−1368
4	Y L Xue, X Mao, Y Guo, S W Lv. The research advance of facial expression recognition in human computer interaction. Journal of Image and Graphics, 2009, 14(5): 764−772
5	V Bettadapura. Face expression recognition and analysis: The state of the art. CoRR, 2012, abs/1203.6722
6	A Wesley, P Buddharaju, R Pienta, I Pavlidis. A comparative analysis of thermal and visual modalities for automated facial expression recognition. Advances in Visual Computing, 2012, 51−60
7	Y Yoshitomi, S Kim, T Kawano, T Kilazoe. Effect of sensor fusion for recognition of emotional states using voice, face image and thermal image of face. In: Proceedings of the 9th IEEE International Workshop on Robot and Human Interactive Communication. 2000, 178−183
8	Z Wang, S Wang. Spontaneous facial expression recognition by using feature-level fusion of visible and thermal infrared images. In: Proceedings of the 2011 IEEE International Workshop on Machine Learning for Signal Processing. 2011, 1−6 https://doi.org/10.1109/MLSP.2011.6064564
9	S Wang, S He. Spontaneous facial expression recognition by fusing thermal infrared and visible images. Intelligent Autonomous Systems, 2013, 194: 263−272 https://doi.org/10.1007/978-3-642-33932-5_25
10	T Cootes, G Edwards, C Taylor. Active appearance models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2001, 23(6): 681−685 https://doi.org/10.1109/34.927467
11	Y Lv, S Wang. A spontaneous facial expression recognition method using head motion and AAM features. In: Proceedings of the 2nd World Congress on Nature and Biologically Inspired Computing. 2010, 334−339
12	S Wang, Z Liu, S Lv, Y Lv, G Wu, P Peng, F Chen, X Wang. A natural visible and infrared facial expression database for expression recognition and emotion inference. IEEE Transactions on Multimedia, 2010, 12(7): 682−691 https://doi.org/10.1109/TMM.2010.2060716
13	G Littlewort, J Whitehill, T Wu, N Butko, P Ruvolo, J Movellan, M Bartlett. The motion in emotion − a cert based approach to the fera emotion challenge. In: Proceedings of the 2011 IEEE International Conference on Automatic Face & Gesture Recognition and Workshops. 2011, 897−902
14	J Cohn, L Reed, Z Ambadar, J Xiao, T Moriyama. Automatic analysis and recognition of brow actions and head motion in spontaneous facial behavior. In: Proceedings of the 2004 IEEE International Conference on Systems, Man and Cybernetics. 2004, 610−616
15	Y Tong, J Chen, Q Ji. A unified probabilistic framework for spontaneous facial action modeling and understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010, 32(2): 258−273 https://doi.org/10.1109/TPAMI.2008.293
16	M Valstar, H Gunes, M Pantic. How to distinguish posed from spontaneous smiles using geometric features. In: Proceedings of the 9th international conference on Multimodal interfaces. 2007, 38−45 https://doi.org/10.1145/1322192.1322202
17	H Gunes, M Pantic. Dimensional emotion prediction from spontaneous head gestures for interaction with sensitive artificial listeners. In: Proceedings of the 10th International Conference on Intelligent Virtual Agents. 2010, 6356: 371−377
18	S Jarlier, D Grandjean, S Delplanque, K N’Diaye, I Cayeux, M Velazco, D Sander, P Vuilleumier, K Scherer. Thermal analysis of facial muscles contractions. IEEE Transactions on Affective Computing, 2011, 2(1): 2−9 https://doi.org/10.1109/T-AFFC.2011.3
19	B Hernández, G Olague, R Hammoud, L Trujillo, E Romero. Visual learning of texture descriptors for facial expression recognition in thermal imagery. Computer Vision and Image Understanding, 2007, 106(2): 258−269 https://doi.org/10.1016/j.cviu.2006.08.012
20	M Khan, R Ward, M Ingleby. Classifying pretended and evoked facial expressions of positive and negative affective states using infrared measurement of skin temperature. ACM Transactions on Applied Perception, 2009, 6(1): Article 6
21	Y Yoshitomi. Facial expression recognition for speaker using thermal image processing and speech recognition system. In: Proceedings of the 10th World Scientific and Engineering Academy and Society International Conference on Applied Computer Science. 2010, 182−186
22	C Puri, L Olson, I Pavlidis, J Levine, J Starren. Stresscam: non-contact measurement of users’ emotional states through thermal imaging. In: Proceedings of the 2005 Conference on Human Factors in Computing Systems. 2005, 1725−1728
23	P Shen, S Wang, Z Liu. Facial expression recognition from infrared thermal videos. Intelligent Autonomous Systems, 2013, 194: 323−333 https://doi.org/10.1007/978-3-642-33932-5_31
24	P Buddharaju, I Pavlidis, C Manohar. Face recognition beyond the visible spectrum. In: Advances in Biometrics, 157−180. Springer, 2008 https://doi.org/10.1007/978-1-84628-921-7_9
25	I Pavlidis, J Levine, P Baukol. Thermal image analysis for anxiety detection. In: Proceedings of the 2001 International Conference on Image Processing. 2001, 315−318
26	T Cootes. am_tools.
27	Y Tong, Y Wang, Z Zhu, Q Ji. Robust facial feature tracking under varying face pose and facial expression. Pattern Recognition, 2007, 40(11): 3195−3208 https://doi.org/10.1016/j.patcog.2007.02.021
28	C Ding. Analysis of gene expression profiles: class discovery and leaf ordering. In: Proceedings of the 6th Annual International Conference on Computational Biology. 2002, 127−136
29	T F Wu, C J Lin, R C Weng. Probability estimates for multi-class classification by pairwise coupling. The Journal of Machine Learning Research, 2004, 5: 975−1005
30	C C Chang, C J Lin. LIBSVM: a library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2011, 2: 27:1−27:27. Software available at
31	P M Long, R A Servedio. Discriminative learning can succeed where generative learning fails. In: Proceedings of the 19th Annual Conference on Learning Theory. 2006, 4005: 319−334
32	A Y Ng, A Jordan. On discriminative vs. generative classifiers: a comparison of logistic regression and naive bayes. Advances in Neural Information Processing Systems, 2002, 14: 841
33	G Hermosilla, J Ruiz-del-Solar, R Verschae, M Correa. A comparative study of thermal face recognition methods in unconstrained environments. Pattern Recognition, 2012, 45(7): 2445−2459 https://doi.org/10.1016/j.patcog.2012.01.001

[1]	Ren QI, Fei GUO, Quan ZOU. String kernels construction and fusion: a survey with bioinformatics application[J]. Front. Comput. Sci., 2022, 16(6): 166904-.
[2]	Zhao-Yue ZHANG, Zi-Jie SUN, Yu-He YANG, Hao LIN. Towards a better prediction of subcellular location of long non-coding RNA[J]. Front. Comput. Sci., 2022, 16(5): 165903-.
[3]	Yali LV, Junzhong MIAO, Jiye LIANG, Ling CHEN, Yuhua QIAN. BIC-based node order learning for improving Bayesian network structure learning[J]. Front. Comput. Sci., 2021, 15(6): 156337-.
[4]	Hui XUE, Haiming XU, Xiaohong CHEN, Yunyun WANG. A primal perspective for indefinite kernel SVM problem[J]. Front. Comput. Sci., 2020, 14(2): 349-363.
[5]	Hui XUE, Sen LI, Xiaohong CHEN, Yunyun WANG. A maximum margin clustering algorithm based on indefinite kernels[J]. Front. Comput. Sci., 2019, 13(4): 813-827.
[6]	Rizwan Ahmed KHAN, Alexandre MEYER, Hubert KONIK, Saida BOUAKAZ. Saliency-based framework for facial expression recognition[J]. Front. Comput. Sci., 2019, 13(1): 183-198.
[7]	Hao ZHENG,Xin GENG. Facial expression recognition via weighted group sparsity[J]. Front. Comput. Sci., 2017, 11(2): 266-275.
[8]	Feifei ZHANG,Yongbin YU,Qirong MAO,Jianping GOU,Yongzhao ZHAN. Pose-robust feature learning for facial expression recognition[J]. Front. Comput. Sci., 2016, 10(5): 832-844.
[9]	Xu YU,Jing YANG,Zhiqiang XIE. Training SVMs on a bound vectors set based on Fisher projection[J]. Front. Comput. Sci., 2014, 8(5): 793-806.
[10]	Jianjun YANG,Yunhai TONG,Zitian WANG,Shaohua TAN. Efficient and effective Bayesian network local structure learning[J]. Front. Comput. Sci., 2014, 8(4): 527-536.
[11]	Shangfei WANG,Menghua HE,Zhen GAO,Shan HE,Qiang JI. Emotion recognition from thermal infrared images using deep Boltzmann machine[J]. Front. Comput. Sci., 2014, 8(4): 609-618.
[12]	Lean YU, Shouyang WANG, Kin Keung LAI. Developing an SVM-based ensemble learning system for customer risk identification collaborating with customer relationship management[J]. Front Comput Sci Chin, 2010, 4(2): 196-203.
[13]	Baoliang LU, Xiaolin WANG, Masao UTIYAMA. Incorporating prior knowledge into learning by dividing training data[J]. Front Comput Sci Chin, 2009, 3(1): 109-122.

Viewed

Full text

Abstract

Cited

Shared

Discussed