Face attribute classification (FAC) is a high-profile problem in biometric verification and face retrieval. Although recent research has been devoted to extracting more delicate image attribute features and exploiting the inter-attribute correlations, significant challenges still remain. Wavelet scattering transform (WST) is a promising non-learned feature extractor. It has been shown to yield more discriminative representations and outperforms the learned representations in certain tasks. Applied to the image classification task, WST can enhance subtle image texture information and create local deformation stability. This paper designs a scattering-based hybrid block, to incorporate frequency-domain (WST) and image-domain features in a channel attention manner (Squeeze-and-Excitation, SE), termed WS-SE block. Compared with CNN, WS-SE achieves a more efficient FAC performance and compensates for the model sensitivity of the small-scale affine transform. In addition, to further exploit the relationships among the attribute labels, we propose a learning strategy from a causal view. The cause attributes defined using the causality-related information can be utilized to infer the effect attributes with a high confidence level. Ablative analysis experiments demonstrate the effectiveness of our model, and our hybrid model obtains state-of-the-art results in two public datasets.
P, Samangouei V M, Patel R Chellappa . Facial attributes for active authentication on mobile devices. Image and Vision Computing, 2017, 58: 181–192
2
Y, Liu Q, Li Z Sun . Attribute-aware face aging with wavelet-based generative adversarial networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019, 11869–11878
3
C, Cao F, Lu C, Li S, Lin X Shen . Makeup removal via bidirectional tunable de-makeup network. IEEE Transactions on Multimedia, 2019, 21( 11): 2750–2761
4
F, Nian X, Chen S, Yang G Lv . Facial attribute recognition with feature decoupling and graph convolutional networks. IEEE Access, 2019, 7: 85500–85512
5
L, Mao Y, Yan J H, Xue H Wang . Deep multi-task multi-label CNN for effective facial attribute classification. IEEE Transactions on Affective Computing, 2022, 13( 2): 818–828
6
Y, Shu Y, Yan S, Chen J H, Xue C, Shen H Wang . Learning spatial-semantic relationship for facial attribute recognition with limited labeled data. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021, 11911–11920
7
C, Szegedy W, Zaremba I, Sutskever J, Bruna D, Erhan I J, Goodfellow R Fergus . Intriguing properties of neural networks. In: Proceedings of the 2nd International Conference on Learning Representations, 2014
8
A, Azulay Y Weiss . Why do deep convolutional networks generalize so poorly to small image transformations?. Journal of Machine Learning Research, 2019, 20( 184): 1–25
9
R Zhang . Making convolutional networks shift-invariant again. In: Proceedings of the 36th International Conference on Machine Learning. 2019, 7324–7334
10
Suchitra S, Chitrakala S, Nithya J. A robust face recognition using automatically detected facial attributes. In: Proceedings of 2014 International Conference on Science Engineering and Management Research (ICSEMR). 2014, 1–5
11
S, Yang Z, Wang J, Liu Z Guo . Controllable sketch-to-image translation for robust face synthesis. IEEE Transactions on Image Processing, 2021, 30: 8797–8810
12
N, Zhang J, Zhao F, Duan Z, Pan Z, Wu M, Zhou X Gu . An end-to-end conditional generative adversarial network based on depth map for 3D craniofacial reconstruction. In: Proceedings of the 30th ACM International Conference on Multimedia. 2022, 759–768
13
Rozsa A, Günther M, Rudd E M, Boult T E. Are facial attributes adversarially robust? In: Proceedings of the 23rd International Conference on Pattern Recognition (ICPR). 2016, 3121–3127
14
A, Rozsa M, Günther E M, Rudd T E Boult . Facial attributes: accuracy and adversarial robustness. Pattern Recognition Letters, 2019, 124: 100–108
15
S Mallat . Wavelets for a vision. Proceedings of the IEEE, 1996, 84( 4): 604–614
16
H, Liang J, Gao N Qiang . A novel framework based on wavelet transform and principal component for face recognition under varying illumination. Applied Intelligence, 2021, 51( 3): 1762–1783
17
N B, Kar D R, Nayak K S, Babu Y D Zhang . A hybrid feature descriptor with Jaya optimised least squares SVM for facial expression recognition. IET Image Processing, 2021, 15( 7): 1471–1483
18
H, Huang R, He Z, Sun T Tan . Wavelet-SRNet: a wavelet-based CNN for multi-scale face super resolution. In: Proceedings of the IEEE International Conference on Computer Vision. 2017, 1698–1706
19
P, Li Y, Hu R, He Z Sun . Global and local consistent wavelet-domain age synthesis. IEEE Transactions on Information Forensics and Security, 2019, 14( 11): 2943–2957
20
S Mallat . Group invariant scattering. Communications on Pure and Applied Mathematics, 2012, 65( 10): 1331–1398
21
J, Bruna S Mallat . Invariant scattering convolution networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013, 35( 8): 1872–1886
22
J, Bruna S Mallat . Classification with scattering operators. In: Proceedings of the CVPR 2011. 2011, 1561–1566
23
A, Singh N Kingsbury . Efficient convolutional network learning using parametric log based dual-tree wavelet ScatterNet. In: Proceedings of the IEEE International Conference on Computer Vision Workshops. 2017, 1140–1147
24
E, Oyallon E, Belilovsky S Zagoruyko . Scaling the scattering transform: Deep hybrid networks. In: Proceedings of the IEEE International Conference on Computer Vision. 2017, 5619–5628
25
Cotter F, Kingsbury N. A learnable scatternet: locally invariant convolutional layers. In: Proceedings of 2019 IEEE International Conference on Image Processing (ICIP). 2019, 350–354
26
Minskiy D, Bober M. Scattering-based hybrid networks: an evaluation and design guide. In: Proceedings of 2021 IEEE International Conference on Image Processing (ICIP). 2021, 2793–2797
27
S, Gauthier B, Thérien L, Alséne-Racicot M, Chaudhary I, Rish E, Belilovsky M, Eickenberg G Wolf . Parametric scattering networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022, 5739–5748
28
E M, Hand R Chellappa . Attributes for improved attributes: a multi-task network utilizing implicit and explicit relationships for facial attribute classification. In: Proceedings of the 31st AAAI Conference on Artificial Intelligence. 2017, 4068–4074
29
J, Cao Y, Li Z Zhang . Partially shared multi-task convolutional neural network with local constraint for face attribute learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2018, 4290–4299
30
Fanhe X, Guo J, Huang Z, Qiu W, Zhang Y. Multi-task learning with knowledge transfer for facial attribute classification. In: Proceedings of 2019 IEEE International Conference on Industrial Technology (ICIT). 2019, 877–882
31
Lai X, Chen S, Wang D H, Zhu S. Multi-task learning with deep dual-path network for facial attribute recognition. In: Proceedings of the 9th International Conference on Computing and Pattern Recognition. 2020, 161–167
32
J, Hu L, Shen G Sun . Squeeze-and-excitation networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2018, 7132–7141
33
J, Lanchantin T, Wang V, Ordonez Y Qi . General multi-label image classification with transformers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021, 16473–16483
34
T, Wiatowski M, Tschannen A, Stanic P, Grohs H Bölcskei . Discrete deep feature extraction: a theory and new architectures. In: Proceedings of the 33rd International Conference on Machine Learning. 2016, 2149–2158
35
T, Wiatowski H Bolcskei . A mathematical theory of deep convolutional neural networks for feature extraction. IEEE Transactions on Information Theory, 2018, 64( 3): 1845–1866
36
J, Andén S Mallat . Deep scattering spectrum. IEEE Transactions on Signal Processing, 2014, 62( 16): 4114–4128
37
T, Angles S Mallat . Generative networks as inverse problems with scattering transforms. In: Proceedings of the 6th International Conference on Learning Representations. 2018
38
J, Wu X, Qiu J, Zhang F, Wu Y, Kong G, Yang L, Senhadji H Shu . Fractional wavelet-based generative scattering networks. Frontiers in Neurorobotics, 2021, 15: 752752
39
D, Minskiy M Bober . Efficient hybrid network: inducting scattering features. In: Proceedings of the 26th International Conference on Pattern Recognition (ICPR). 2022, 2300–2306
40
J Woodward . Causation in biology: stability, specificity, and the choice of levels of explanation. Biology & Philosophy, 2010, 25( 3): 287–318
41
J Pearl . Theoretical impediments to machine learning with seven sparks from the causal revolution. In: Proceedings of the 11th ACM International Conference on Web Search and Data Mining. 2018, 3
42
H, Han A K, Jain F, Wang S, Shan X Chen . Heterogeneous face attribute estimation: A deep multi-task learning approach. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40( 11): 2597–2609
43
K, He X, Zhang S, Ren J Sun . Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016, 770–778
44
E, Tola V, Lepetit P Fua . DAISY: an efficient dense descriptor applied to wide-baseline stereo. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010, 32( 5): 815–830
45
F, Cotter N Kingsbury . Deep learning in the wavelet domain. 2018, arXiv preprint arXiv: 1811.06115
46
M, Andreux T, Angles G, Exarchakis R, Leonarduzzi G, Rochette L, Thiry J, Zarka S, Mallat J, Andén E, Belilovsky J, Bruna V, Lostanlen M, Chaudhary M J, Hirn E, Oyallon S, Zhang C, Cella M Eickenberg . Kymatio: scattering transforms in python. Journal of Machine Learning Research, 2020, 21( 60): 1–6
47
I W, Selesnick R G, Baraniuk N C Kingsbury . The dual-tree complex wavelet transform. IEEE Signal Processing Magazine, 2005, 22( 6): 123–151
48
E, Oyallon S, Zagoruyko G, Huang N, Komodakis S, Lacoste-Julien M, Blaschko E Belilovsky . Scattering networks for hybrid representation learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019, 41( 9): 2208–2221
49
S, Fan X, Wang C, Shi K, Kuang N, Liu B Wang . Debiased graph neural networks with agnostic label selection bias. IEEE Transactions on Neural Networks and Learning Systems, 2022, https://doi.org/10.1109/TNNLS.2022.3141260
50
J, Pearl M, Glymour N P Jewell . Causal Inference in Statistics: A Primer. Chichester: John Wiley & Sons, 2016
51
Y, Sun Y, Chen X, Wang X Tang . Deep learning face representation by joint identification-verification. In: Proceedings of the 27th International Conference on Neural Information Processing Systems. 2014, 1988–1996
52
G B, Huang M, Mattar T, Berg E Learned-Miller . Labeled faces in the wild: A database forstudying face recognition in unconstrained environments. In: Proceedings of the Workshop on Faces in ‘Real-Life’ Images: Detection, Alignment, and Recognition. 2008
53
Boer P T, De D P, Kroese S, Mannor R Y Rubinstein . A tutorial on the cross-entropy method. Annals of Operations Research, 2005, 134( 1): 19–67
54
Z, Liu P, Luo X, Wang X Tang . Deep learning face attributes in the wild. In: Proceedings of the IEEE International Conference on Computer Vision. 2015, 3730–3738
55
E M, Rudd M, Gunther T E Boult . MOON: A mixed objective optimization network for the recognition of facial attributes. In: Proceedings of the 14th European Conference on Computer Vision. 2016, 19–35
56
Günther M, Rozsa A, Boult T E. AFFACT: Alignment-free facial attribute classification technique. In: Proceedings of 2017 IEEE International Joint Conference on Biometrics (IJCB). 2017, 90–99
57
M M, Kalayeh M Shah . On symbiosis of attribute prediction and semantic segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 43( 5): 1620–1635
58
Lingenfelter B, Hand E M. Improving evaluation of facial attribute prediction models. In: Proceedings of the 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021). 2021, 1–7
59
A, Singh N Kingsbury . Multi-resolution dual-tree wavelet scattering network for signal classification. 2017, arXiv preprint arXiv: 1702.03345
60
Singh A, Kingsbury N. Dual-tree wavelet scattering network with parametric log transformation for object classification. In: Proceedings of 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2017, 2622–2626
61
E, Oyallon S Mallat . Deep roto-translation scattering for object classification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015, 2865–2873
62
S, Kang G, Kim C D Yoo . Fair facial attribute classification via causal graph-based attribute translation. Sensors, 2022, 22( 14): 5271
