Please wait a minute...
Frontiers of Computer Science

ISSN 2095-2228

ISSN 2095-2236(Online)

CN 10-1014/TP

Postal Subscription Code 80-970

2018 Impact Factor: 1.129

Front. Comput. Sci.    2014, Vol. 8 Issue (4) : 609-618    https://doi.org/10.1007/s11704-014-3295-3
RESEARCH ARTICLE
Emotion recognition from thermal infrared images using deep Boltzmann machine
Shangfei WANG1,2,*(),Menghua HE1,2,Zhen GAO1,2,Shan HE1,2,Qiang JI3
1. School of Computer Science and Technology, University of Science and Technology of China, Hefei 230027, China
2. Key Lab of Computing and Communicating Software of Anhui Province, Hefei 230027, China
3. Department of Electrical, Computer, and Systems Engineering, Rensselaer Polytechnic Institute, Troy NY 12180-3590, USA
 Download: PDF(527 KB)  
 Export: BibTeX | EndNote | Reference Manager | ProCite | RefWorks
Abstract

Facial expression and emotion recognition from thermal infrared images has attracted more and more attentions in recent years. However, the features adopted in current work are either temperature statistical parameters extracted from the facial regions of interest or several hand-crafted features that are commonly used in visible spectrum. Till now there are no image features specially designed for thermal infrared images. In this paper, we propose using the deep Boltzmann machine to learn thermal features for emotion recognition from thermal infrared facial images. First, the face is located and normalized from the thermal infrared images. Then, a deep Boltzmann machine model composed of two layers is trained. The parameters of the deep Boltzmann machine model are further fine-tuned for emotion recognition after pre-training of feature learning. Comparative experimental results on the NVIE database demonstrate that our approach outperforms other approaches using temperature statistic features or hand-crafted features borrowed from visible domain. The learned features from the forehead, eye, and mouth are more effective for discriminating valence dimension of emotion than other facial areas. In addition, our study shows that adding unlabeled data from other database during training can also improve feature learning performance.

Keywords emotion recognition      thermal infrared images      deep Boltzmann machine     
Corresponding Author(s): Shangfei WANG   
Issue Date: 11 August 2014
 Cite this article:   
Shangfei WANG,Menghua HE,Zhen GAO, et al. Emotion recognition from thermal infrared images using deep Boltzmann machine[J]. Front. Comput. Sci., 2014, 8(4): 609-618.
 URL:  
https://academic.hep.com.cn/fcs/EN/10.1007/s11704-014-3295-3
https://academic.hep.com.cn/fcs/EN/Y2014/V8/I4/609
1 Zeng Z, Pantic M, Roisman G, Huang T. A survey of affect recognition methods: audio, visual, and spontaneous expressions. IEEE Transactions on Pattern Analysis and Machine Intellignece, 2009, 31(1): 39-58
doi: 10.1109/TPAMI.2008.52
2 Jarlier S, Grandjean D, Delplanque S, N’Diaye K, Cayeux I, Velazco M, Sander D, Vuilleumier P, Scherer K. Thermal analysis of facial muscles contractions. IEEE Transactions on Affective Computing, 2011, 2(1): 2-9
doi: 10.1109/T-AFFC.2011.3
3 Ranzato M, Susskind J, Mnih V, Hinton G. On deep generative models with applications to recognition. In: Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition. 2011, 2857-2864
4 Wang S, Liu Z, Lv S, Lv Y, Wu G, Peng P, Chen F, Wang X. A natural visible and infrared facial expression database for expression recognition and emotion inference. IEEE Transactions on Multimedia, 2010, 682-691
doi: 10.1109/TMM.2010.2060716
5 Khan MM, Ward R D, Ingleby M. Automated classification and recognition of facial expressions using infrared thermal imaging. In: Proceedings of the 2004 IEEE Conference on Cybernetics and Intelligent Systems. 2004, 202-206
6 Oz I, Khan M M. Efficacy of biophysiological measurements at ftfps for facial expression classification: a validation. In: Proceedings of 2012 IEEE-EMBS International Conference on Biomedical and Health Informatics. 2012, 108-111
doi: 10.1109/BHI.2012.6211519
7 Trujillo L, Olague G, Hammoud R, Hernandez B. Automatic feature localization in thermal images for facial expression recognition. In: Proceedings of the 2005 IEEE Conference on Computer Vision and Pattern Recognition. 2005, 14
doi: 10.1109/CVPR.2005.415
8 Jiang G, Song X, Zheng F, Wang P, Omer A M. Facial expression recognition using thermal image. In: Proceedings of the 27th Annual International Conference of the Engineering in Medicine and Biology Society. 2005, 631-633
9 Hern?ndez B, Olague G, Hammoud R, Trujillo L, Romero E. Visual learning of texture descriptors for facial expression recognition in thermal imagery. Computer Vision and Image Understanding. 2007
doi: 10.1016/j.cviu.2006.08.012
10 Yoshitomi Y. Facial expression recognition for speaker using thermal image processing and speech recognition system. In: Proceedings of the 10th WSEAS International Conference on Applied Computer Science. 2010, 182-186
11 Pavlidis I, Levine J. Thermal image analysis for polygraph testing. IEEE Engineering in Medicine and Biology Magazine, 2002, 21(6): 56-64
doi: 10.1109/MEMB.2002.1175139
12 Jain U, Tan B, Li Q. Concealed knowledge identification using facial thermal imaging. In: Proceedings of the 2012 IEEE International Conference on Acoustics, Speech and Signal Processing. 2012, 1677-1680
13 Liu Z, Wang S. Emotion recognition using hidden Markov models from facial temperature sequence. Lecture Notes in Computer Science, 2011, 6975: 240-247
doi: 10.1007/978-3-642-24571-8_26
14 Eom J S, Sohn J H. Emotion recognition using facial thermal images. Journal of the Ergonomics Society of Korea, 2012, 31(3): 427-435
doi: 10.5143/JESK.2012.31.3.427
15 Bernhard Anzengruber A R. “Facelight”: potentials and drawbacks of thermal imaging to infer driver stress. In: Proceedings of the 4th International Conference on Automotive User Interfaces and Interactive Vehicular Applications. 2012, 209-216
16 Krzywicki A T, He G, O’Kane B L. Analysis of facial thermal variations in response to emotion: eliciting film clips. In: Proceedings of SPIE Defense, Security, and Sensing. 2009, 7343: 73412-1-73412-11
17 Merla A, Romani G L. Thermal signatures of emotional arousal: a functional infrared imaging study. In: Proceedings of the 29th Annual International Conference of the Engineering in Medicine and Biology Society. 2007, 247-249
18 Nhan B R, Chau T. Classifying affective states using thermal infrared imaging of the human face. IEEE Transactions on Biomedical Engineering, 2010, 57(4): 979-987
doi: 10.1109/TBME.2009.2035926
19 Ranzato M, Mnih V, Susskind J, Hinton G. Modeling natural images using gated MRFs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013
doi: 10.1109/TPAMI.2013.29
20 Susskind J M, Hinton G E, Movellan J R, Anderson A K. Generating facial expressions with deep belief nets. Affective Computing, Emotion Modelling, Synthesis and Recognition, 2008, 421-440
21 Susskind JM. Interpreting faces with neurally inspired generative models. PhD thesis, Department of Psychology, University of Toronto, 2011
22 Rifai S, Bengio Y, Courville A, Vincent P, irza M. Disentangling factors of variation for facial expression recognition. In: Proceedings of 12th European Conference on Computer Vision, 2012, V6: 808-822
23 Sabzevari M, Toosizadeh S, Quchani S, Abrishami V. A fast and accurate facial expression synthesis system for color face images using face graph and deep belief network. In: Proceedings of the 2010 International Coference on Electronics and Information Engineering. 2010, V2: 354-358
doi: 10.1109/ICEIE.2010.5559797
24 He S, Wang S, Lan W, Fu H, Ji Q. Facial expression recognition using deep Boltzmann machine from thermal infrared images. In: Proceedings of the 2013 Humaine Association Conference on Affective computing and Intelligent Interaction. 2013, 239-244
doi: 10.1109/ACII.2013.46
25 Kim Y, Lee H, Provost E M. Deep learning for robust feature generation in audio-visual emotion recognition. In: Proceedings of the 2013 IEEE International Conference on Acoustics, Speech and Signal Processing. 2013
doi: 10.1109/ICASSP.2013.6638346
26 Stuhlsatz A, Meyer C, Eyben F, ZieIke T, Meier G, Schuller B. Deep neural networks for acoustic emotion recognition: raising the benchmarks. In: Proceedings of the 2011 IEEE International Conference on acoustics, Speech and Signal Processing. 2011, 5688-5691
27 Schmidt E M, Scott J J, Kim Y E. Feature learning in dynamic environments: modeling the acoustic structure of musical emotion. In: Proceedings of the International Society for Music Information Retrieval. 2012, 325-330
28 Martinez H, Bengio Y, Yannakakis G. Learning deep physiological models of affect. IEEE Computational Intelligence Magazine, 2013, 8(2): 20-33
doi: 10.1109/MCI.2013.2247823
29 Salakhutdinov R, Hinton G E. Deep Boltzmann machines. In: Proceedings of the 2009 International Conference on Artificial Intelligence and Statistics. 2009, 448-455
30 Mohamed A, Dahl G E, Hinton G. Acoustic modeling using deep belief networks. IEEE Transactions on Audio, Speech, and Language Processing. 2012
doi: 10.1109/TASL.2011.2109382
31 Hinton G E. Training products of experts by minimizing contrastive divergence. Neural Computation, 2002, 14(8): 1771-1800
doi: 10.1162/089976602760128018
32 Petridis S, Martinez B, Pantic M. The mahnob laughter database. Image and Vision Computing Journal, 2013, 31(2): 186-202
doi: 10.1016/j.imavis.2012.08.014
33 Selinger A, Socolinsky D A. Appearance-based facial recognition using visible and thermal imagery: a comparative study. Technical report, DTIC Document, 2006
34 Otsu N. A threshold selection method from gray-level histograms. IEEE Transactions on Systems, Man and Cybermetics, 1979, 9(1): 62-66
doi: 10.1109/TSMC.1979.4310076
35 He X, Yan S, Hu Y, Niyogi P, Zhang H J. Face recognition using laplacianfaces. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2005, 27(3): 328-340
doi: 10.1109/TPAMI.2005.55
36 Wright J, Yang A, Ganesh A, Sastry S, Ma Y. Robust face recognition via sparse representation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2009, 31(2): 210-227
doi: 10.1109/TPAMI.2008.79
[1] Shangfei WANG,Shan HE,Yue WU,Menghua HE,Qiang JI. Fusion of visible and thermal images for facial expression recognition[J]. Front. Comput. Sci., 2014, 8(2): 232-242.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed