Saliency-based framework for facial expression recognition

doi:10.1007/s11704-017-6114-9

Front. Comput. Sci.

2019, Vol. 13

Issue (1) : 183-198 https://doi.org/10.1007/s11704-017-6114-9

RESEARCH ARTICLE

Saliency-based framework for facial expression recognition

Rizwan Ahmed KHAN^1,²(

), Alexandre MEYER¹(

), Hubert KONIK³, Saida BOUAKAZ¹

¹. Université de Lyon, CNRS, LIRIS, UMR5205, Université Lyon 1, Lyon F-69622, France
². Faculty of Information Technology, Barrett Hodgson University, Karachi 74900, Pakistan
³. Laboratoire Hubert Curien, UMR5516, Université Jean Monnet, Saint-Etienne 42000, France

Download: PDF(912 KB)
Export: BibTeX | EndNote | Reference Manager | ProCite | RefWorks

Abstract

This article proposes a novel framework for the recognition of six universal facial expressions. The framework is based on three sets of features extracted from a face image: entropy, brightness, and local binary pattern. First, saliency maps are obtained using the state-of-the-art saliency detection algorithm “frequency-tuned salient region detection”. The idea is to use saliency maps to determine appropriate weights or values for the extracted features (i.e., brightness and entropy).We have performed a visual experiment to validate the performance of the saliency detection algorithm against the human visual system. Eye movements of 15 subjects were recorded using an eye-tracker in free-viewing conditions while they watched a collection of 54 videos selected from the Cohn-Kanade facial expression database. The results of the visual experiment demonstrated that the obtained saliency maps are consistent with the data on human fixations. Finally, the performance of the proposed framework is demonstrated via satisfactory classification results achieved with the Cohn-Kanade database, FG-NET FEED database, and Dartmouth database of children’s faces.

Keywords facial expression recognition classification salient regions entropy brightness local binary pattern

Corresponding Author(s): Rizwan Ahmed KHAN,Alexandre MEYER

Just Accepted Date: 14 April 2017 Online First Date: 09 May 2018 Issue Date: 31 January 2019

Cite this article:

Rizwan Ahmed KHAN,Alexandre MEYER,Hubert KONIK, et al. Saliency-based framework for facial expression recognition[J]. Front. Comput. Sci., 2019, 13(1): 183-198.

URL:

https://academic.hep.com.cn/fcs/EN/10.1007/s11704-017-6114-9
https://academic.hep.com.cn/fcs/EN/Y2019/V13/I1/183

1	PEkman, W V Friesen, PEllsworth. Emotion in the Human Face: Guidelines for Research and an Integration of Findings. New York: Pergamon Press, 1972
2	PEkman, W V Friesen. Unmasking the Face: A Guide to Recognizing Emotions from Facial Clues. New Jersey: Prentice Hall, 1975
3	PEkman. Telling Lies: Clues to Deceit in the Marketplace, Politics, and Marriage. 3rd ed. New York: W. W. Norton & Company, 2001
4	PEkman. Facial expression of emotion. Psychologist, 1993, 48(4): 384–392 https://doi.org/10.1037/0003-066X.48.4.384
5	PBull. State of the art: nonverbal communication. Psychologist, 2001, 14(12): 644–647
6	V HYngve. On getting a word in edgewise. In: Proceedings of the 6th Regional Meeting of the Chicago Linguistic Society. 1970, 567–578
7	PCarrera-Levillain, J Fernandez-Dols. Neutral faces in context: their emotional meaning and their function. Journal of Nonverbal Behavior, 1994, 18(4): 281–299 https://doi.org/10.1007/BF02172290
8	JMFernandez-Dols, H Wallbott, FSanchez. Emotion category accessibility and the decoding of emotion from facial expression and context. Journal of Nonverbal Behavior, 1991, 15(2): 107–123 https://doi.org/10.1007/BF00998266
9	PEkman. Universals and cultural differences in facial expressions of emotion. In: Proceedings of Nebraska Symposium on Motivation. 1971, 207–283
10	URajashekar, L K Cormack, A CBovik. Visual search: structure from noise. In: Proceedings of Eye Tracking Research & Applications Symposium. 2002, 119–123 https://doi.org/10.1145/507072.507096
11	R AKhan , HKonik, EDinet. Enhanced image saliency model based on blur identification. In: Proceedings of IEEE International Conference on Image and Vision Computing New Zealand (IVCNZ). 2010, 1–7 https://doi.org/10.1109/IVCNZ.2010.6148833
12	RAchanta, SHemami, FEstrada, S Susstrunk. Frequency-tuned salient region detection. In: Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition. 2009, 1597–1604 https://doi.org/10.1109/CVPR.2009.5206596
13	PViola, MJones. Rapid object detection using a boosted cascade of simple features. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2001, 511–518 https://doi.org/10.1109/CVPR.2001.990517
14	GLittlewort, M S Bartlett, IFasel, JSusskind, J Movellan. Dynamics of facial expression extracted automatically from video. Image and Vision Computing, 2006, 24(6): 615–625 https://doi.org/10.1016/j.imavis.2005.09.011
15	YTian. Evaluation of face resolution for expression analysis. In: Proceedings of Computer Vision and Pattern Recognition Workshop, 2004
16	PYang, QLiu, D NMetaxas. Exploring facial expressions with compositional features. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2010, 2638–2644 https://doi.org/10.1109/CVPR.2010.5539978
17	TOjala, M Pietikäinen, DHarwood. A comparative study of texture measures with classification based on featured distribution. Pattern Recognition, 1996, 29(1): 51–59 https://doi.org/10.1016/0031-3203(95)00067-4
18	R AKhan, AMeyer, HKonik, S Bouakaz. Exploring human visual system: study to aid the development of automatic facial expression recognition framework. In: Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). 2012, 49–54 https://doi.org/10.1109/CVPRW.2012.6239186
19	VOjansivu, J Heikkilä. Blur insensitive texture classification using local phase quantization. In: Proceedings of International conference on Image and Signal Processing. 2008, 236–243 https://doi.org/10.1007/978-3-540-69905-7_27
20	R AKhan, AMeyer, HKonik, S Bouakaz. Human vision inspired framework for facial expressions recognition. In: Proceedings of IEEE International Conference on Image Processing (ICIP). 2012, 2593–2596 https://doi.org/10.1109/ICIP.2012.6467429
21	GZhao, M Pietikäinen. Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Transaction on Pattern Analysis and Machine Intelligence, 2007, 29(6): 915–928 https://doi.org/10.1109/TPAMI.2007.1110
22	YZhang, QJi. Active and dynamic information fusion for facial expression understanding from image sequences. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2005, 27(5): 699–714 https://doi.org/10.1109/TPAMI.2005.93
23	M FValstar, IPatras, MPantic. Facial action unit detection using probabilistic actively learned support vector machines on tracked facial point data. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition Workshop. 2005, 76–84 https://doi.org/10.1109/CVPR.2005.457
24	YBai, LGuo, LJin, Q Huang. A novel feature extraction method using pyramid histogram of orientation gradients for smile recognition. In: Proceedings of IEEE International Conference on Image Processing (ICIP). 2009, 3305–3308
25	R AKhan, AMeyer, HKonik, S Bouakaz. Pain detection through shape and appearance features. In: Proceedings of IEEE International Conference on Multimedia and Expo (ICME). 2013, 1–6 https://doi.org/10.1109/ICME.2013.6607608
26	R AKhan, AMeyer , SBouakaz. Automatic affect analysis: from children to adults. In: Proceedings of International Symposium on Visual Computing (ISVC). 2015, 304–313 https://doi.org/10.1007/978-3-319-27863-6_28
27	GZhao, M Pietikäinen. Boosted multi-resolution spatiotemporal descriptors for facial expression recognition. Pattern Recognition Letters, 2009, 30(12): 1117–1127 https://doi.org/10.1016/j.patrec.2009.03.018
28	PLucey, J FCohn, TKanade, J Saragih, ZAmbadar, IMatthews. The extended cohn-kande dataset (CK+): a complete facial expression dataset for action unit and emotion-specified expression. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition Workshops. 2010, 94–101
29	FWallhoff. Facial expressions and emotion database. 2006
30	K ADalrymple, JGomez, BDuchaine. The Dartmouth database of children’s faces: acquisition and validation of a new face stimulus set. PLoS ONE, 2013, 8(11): e79131 https://doi.org/10.1371/journal.pone.0079131
31	YTian, TKanade, J FCohn. Handbook of Face Recognition. Berlin: Springer, 2005
32	TKanade, J FCohn, YTian. Comprehensive database for facial expression analysis. In: Proceedings of IEEE International Conference on Automatic face and Gesture Recognition. 2000, 46–53 https://doi.org/10.1109/AFGR.2000.840611
33	D TLin, D CPan. Integrating a mixed-feature model and multiclass support vector machine for facial expression recognition. Integrated Computer-Aided Engineering, 2009, 16(1): 61–74
34	PEkman, W Friesen. The facial action coding system: a technique for the measurement of facial movements. Consulting Psychologist. 1978
35	MPantic, M F Valstar, RRademaker, LMaat. Web-based database for facial expression analysis. In: Proceedings of IEEE International Conference on Multimedia and Expo. 2005 https://doi.org/10.1109/ICME.2005.1521424
36	IKotsia, S Zafeiriou, s IPita. Texture and shape information fusion for facial expression and facial action unit recognition. Pattern Recognition, 2008, 41(3): 833–851 https://doi.org/10.1016/j.patcog.2007.06.026
37	XYang, DHuang, YWang, L Chen. Automatic 3D facial expression recognition using geometric scattering representation. In: Proceedings of the 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition. 2015, 1–6
38	XZhao, DHuang, EDellandrea, LChen. Automatic 3D facial expression recognition based on a Bayesian belief net and a statistical facial feature model. In: Proceedings of the 20th IEEE International Conference on Pattern Recognition. 2010, 3724–3727 https://doi.org/10.1109/ICPR.2010.907
39	ASavran, BSankur, M TBilge. Comparative evaluation of 3D vs. 2D modality for automatic detection of facial action units. Pattern Recognition, 2012, 45(2): 767–782 https://doi.org/10.1016/j.patcog.2011.07.022
40	LYin, XWei, PLongo, A Bhuvanesh. Analyzing facial expressions using intensity-variant 3D data for human computer interaction. In: Proceedings of the 18th International Conference on Pattern Recognition. 2006, 1248–1251
41	HLi, HDing, DHuang, Y Wang, XZhao, JMorvan, LChen. An efficient multimodal 2D+ 3D feature-based approach to automatic facial expression recognition. Computer Vision and Image Understanding, 2015, 140: 83–92 https://doi.org/10.1016/j.cviu.2015.07.005
42	MRosato, XChen, LYin. Automatic registration of vertex correspondences for 3D facial expression analysis. In: Proceedings of IEEE International Conference on Biometrics: Theory, Applications and Systems. 2008, 1–7 https://doi.org/10.1109/BTAS.2008.4699369
43	VLe, HTang, T SHuang. Expression recognition from 3D dynamic faces using robust spatio-temporal shape features. In: Proceedings of IEEE International Conference and Workshops on Automatic Face and Gesture Recognition. 2011, 414–421 https://doi.org/10.1109/FG.2011.5771435
44	TJost, N Ouerhani, RWartburg, RMüri, H Hügli. Assessing the contribution of color in visual attention. Computer Vision and Image Understanding, 2005, 100(1–2): 107–123 https://doi.org/10.1016/j.cviu.2004.10.009
45	R AKhan, HKonik, EDinet. Visual attention: effects of blur. In: Proceedings of IEEE International Conference on Image Processing. 2011, 3289–3292 https://doi.org/10.1109/ICIP.2011.6116373
46	HCollewijin, M R Steinman, J CErkelens, ZPizlo, JSteen. The Head- Neck Sensory Motor System. New York: Oxford University Press, 1992
47	D WCunningham, M Kleiner, CWallraven, HBülthoff. Manipulating video sequences to determine the components of conversational facial expressions. ACM Transactions on Applied Perception, 2005, 2(3): 251–269 https://doi.org/10.1145/1077399.1077404
48	J DBoucher, PEkman. Facial areas and emotional information. Journal of Communication, 1975, 25(2): 21–29 https://doi.org/10.1111/j.1460-2466.1975.tb00577.x
49	C EShannon, WWeave. The Mathematical Theory of Communication. Urbana, IL: University of Illinois Press, 1963
50	GWyszecki, W SStiles. Color Science: Concepts and Methods, Quantitative Data and Formulae. 2nd ed. New York: John Wiley and Sons, 1982
51	SBezryadin, PBourov. Color coordinate system for accurate color image editing software. In: Proceedings of International Conference on Printing Technology. 2006, 145–148
52	CShan, SGong, P WMcOwan. Facial expression recognition based on local binary patterns: a comprehensive study. Image and Vision Computing, 2009, 27(6): 803–816 https://doi.org/10.1016/j.imavis.2008.08.005
53	LItti, CKoch, ENiebur. A model of saliency-based visual attention for rapid scene analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1998, 20(11): 1254–1259 https://doi.org/10.1109/34.730558
54	CKoch, JHarel, PPerona. Graph-based visual saliency. In: Proceedings of Neural Information Processing Systems. 2006, 545–552
55	XHou, LZhang. Saliency detection: a spectral residual approach. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2007, 1–8 https://doi.org/10.1109/CVPR.2007.383267
56	J BCohen. Visual Color and Color Mixture: The Fundamental Color Space. Urbana: University of Illinois Press, 2000
57	CIEC. Commission internationale de l’Eclairage proceedings. Cambridge: Cambridge University Press, 1931
58	R AKhan, AMeyer, HKonik, S Bouakaz. Framework for reliable, real-time facial expression recognition for low resolution images. Pattern Recognition Letters, 2013, 34(10): 1159–1168 https://doi.org/10.1016/j.patrec.2013.03.022
59	TOjala, M Pietikäinen, TMäenpää. Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Transaction on Pattern Analysis andMachine Intelligence, 2002, 24(7): 971–987
60	CZhan, WLi, POgunbona, F Safaei. A real-time facial expression recognition system for online games. International Journal of Computer Games Technology, 2008 https://doi.org/10.1155/2008/542918
61	EPiatkowska. Facial expression recognition system. Dissertation for the Master’s Degree. Chicago: DePaul University, 2010
62	AKhanum, MMufti, YJaved, Z Shafiq. Fuzzy case-based reasoning for facial expression recognition. Fuzzy Sets and Systems, 2009, 160(2): 231–250 https://doi.org/10.1016/j.fss.2008.05.022
63	SRosdiyana, S Hideyuki. Extraction of the minimum number of gabor wavelet parameters for the recognition of natural facial expressions. Artificial Life and Robotics, 2011, 16(1): 21–31 https://doi.org/10.1007/s10015-011-0871-6
64	H LEgger, D SPine, ENelson, E Leibenluft, MErnst, KTowbin, , AAngold. The NIMH child emotional faces picture set (NIMH-ChEFS): a new set of children’s facial emotion stimuli. International Journal of Methods in Psychiatric Research, 2011, 20(3): 145–156 https://doi.org/10.1002/mpr.343

[1]

Download

[1]	Yunyun WANG, Jiao HAN, Yating SHEN, Hui XUE. Pointwise manifold regularization for semi-supervised learning[J]. Front. Comput. Sci., 2021, 15(1): 151303-.
[2]	Panthadeep BHATTACHARJEE, Pinaki MITRA. A survey of density based clustering algorithms[J]. Front. Comput. Sci., 2021, 15(1): 151308-.
[3]	Qianchen YU, Zhiwen YU, Zhu WANG, Xiaofeng WANG, Yongzhi WANG. Estimating posterior inference quality of the relational infinite latent feature model for overlapping community detection[J]. Front. Comput. Sci., 2020, 14(6): 146323-.
[4]	Je Sen TEH, Weijian TENG, Azman SAMSUDIN, Jiageng CHEN. A post-processing method for true random number generators based on hyperchaos with applications in audio-based generators[J]. Front. Comput. Sci., 2020, 14(6): 146405-.
[5]	Parnika PARANJAPE, Meera DHABU, Parag DESHPANDE. A novel classifier for multivariate instance using graph class signatures[J]. Front. Comput. Sci., 2020, 14(4): 144307-.
[6]	Muhammad Aminur RAHAMAN, Mahmood JASIM, Md. Haider ALI, Md. HASANUZZAMAN. Bangla language modeling algorithm for automatic recognition of hand-sign-spelled Bangla sign language[J]. Front. Comput. Sci., 2020, 14(3): 143302-.
[7]	Xibin DONG, Zhiwen YU, Wenming CAO, Yifan SHI, Qianli MA. A survey on ensemble learning[J]. Front. Comput. Sci., 2020, 14(2): 241-258.
[8]	Hui XUE, Haiming XU, Xiaohong CHEN, Yunyun WANG. A primal perspective for indefinite kernel SVM problem[J]. Front. Comput. Sci., 2020, 14(2): 349-363.
[9]	Changlong WANG, Zhiyong FENG, Xiaowang ZHANG, Xin WANG, Guozheng RAO, Daoxun FU. ComR: a combined OWL reasoner for ontology classification[J]. Front. Comput. Sci., 2019, 13(1): 139-156.
[10]	Munish SAINI, Kuljit Kaur CHAHAL. Change profile analysis of open-source software systems to understand their evolutionary behavior[J]. Front. Comput. Sci., 2018, 12(6): 1105-1124.
[11]	Shuaiqiang WANG, Yilong YIN. Polygene-based evolutionary algorithms with frequent pattern mining[J]. Front. Comput. Sci., 2018, 12(5): 950-965.
[12]	Sudipta ROY, Debnath BHATTACHARYYA, Samir Kumar BANDYOPADHYAY, Tai-Hoon KIM. An improved brain MR image binarization method as a preprocessing for abnormality detection and features extraction[J]. Front. Comput. Sci., 2017, 11(4): 717-727.
[13]	Wei SHAO,Yi DING,Hong-Bin SHEN,Daoqiang ZHANG. Deep model-based feature extraction for predicting protein subcellular localizations from bio-images[J]. Front. Comput. Sci., 2017, 11(2): 243-252.
[14]	Hao ZHENG,Xin GENG. Facial expression recognition via weighted group sparsity[J]. Front. Comput. Sci., 2017, 11(2): 266-275.
[15]	Jian-Hao LUO,Wang ZHOU,Jianxin WU. Image categorization with resource constraints: introduction, challenges and advances[J]. Front. Comput. Sci., 2017, 11(1): 13-26.

Viewed

Full text

Abstract

Cited

Shared

Discussed