Representation learning via an integrated autoencoder for unsupervised domain adaptation

doi:10.1007/s11704-022-1349-5

Frontiers of Computer Science

2023, Vol. 17

Issue (5): 175334 https://doi.org/10.1007/s11704-022-1349-5

本期目录

Representation learning via an integrated autoencoder for unsupervised domain adaptation

Yi ZHU^1,^2,³, Xindong WU^2,³, Jipeng QIANG¹, Yunhao YUAN¹, Yun LI¹(

)

¹. School of Information Engineering, Yangzhou University, Yangzhou 225127, China
². Key Laboratory of Knowledge Engineering with Big Data (Ministry of Education of China), Hefei University of Technology, Hefei 230009, China
³. School of Computer Science and Information Engineering, Hefei University of Technology, Hefei 230601, China

全文: PDF(4841 KB) HTML

Abstract：

The purpose of unsupervised domain adaptation is to use the knowledge of the source domain whose data distribution is different from that of the target domain for promoting the learning task in the target domain. The key bottleneck in unsupervised domain adaptation is how to obtain higher-level and more abstract feature representations between source and target domains which can bridge the chasm of domain discrepancy. Recently, deep learning methods based on autoencoder have achieved sound performance in representation learning, and many dual or serial autoencoder-based methods take different characteristics of data into consideration for improving the effectiveness of unsupervised domain adaptation. However, most existing methods of autoencoders just serially connect the features generated by different autoencoders, which pose challenges for the discriminative representation learning and fail to find the real cross-domain features. To address this problem, we propose a novel representation learning method based on an integrated autoencoders for unsupervised domain adaptation, called IAUDA. To capture the inter- and inner-domain features of the raw data, two different autoencoders, which are the marginalized autoencoder with maximum mean discrepancy (mAE $M M D$ ) and convolutional autoencoder (CAE) respectively, are proposed to learn different feature representations. After higher-level features are obtained by these two different autoencoders, a sparse autoencoder is introduced to compact these inter- and inner-domain representations. In addition, a whitening layer is embedded for features processed before the mAE $M M D$ to reduce redundant features inside a local area. Experimental results demonstrate the effectiveness of our proposed method compared with several state-of-the-art baseline methods.

Key words： unsupervised domain adaptation representation learning marginalized autoencoder convolutional autoencoder sparse autoencoder

收稿日期: 2021-06-26 出版日期: 2023-01-03

Corresponding Author(s): Yun LI

引用本文:

. [J]. Frontiers of Computer Science, 2023, 17(5): 175334.
Yi ZHU, Xindong WU, Jipeng QIANG, Yunhao YUAN, Yun LI. Representation learning via an integrated autoencoder for unsupervised domain adaptation. Front. Comput. Sci., 2023, 17(5): 175334.

链接本文:

https://academic.hep.com.cn/fcs/CN/10.1007/s11704-022-1349-5
https://academic.hep.com.cn/fcs/CN/Y2023/V17/I5/175334

Notations	Descriptions
$D s$ , $D t$	Source and target domain
$n s$ , $n t$	The numbers of source and target instances
$x i (s)$ , $x i (t)$	The $i t h$ instance in source and target domain
$y i (s)$	The label of $i t h$ instance in source domain
$x^i (s)$ , $x^i (t)$	The reconstruction of $x i (s)$ , $x i (t)$
$m$	The features number of input data
$c$	The label number of nodes
$W i$	The weight matrix of the level $i$
$K i j$	The adjacent matrix between $x i$ and $x j$
$L$	Laplacian matrices
tr()	Trace operator
$α$ , $β$ , $λ$	Hyper-parameters

Tab.1

Fig.1

Fig.2

Tab.2

Tab.3

Tab.4

Fig.3

Fig.4

Fig.5

Fig.6

Fig.7

Fig.8

1	S J, Pan Q Yang . A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering, 2010, 22( 10): 1345–1359
2	J, Xin Z, Cui P, Zhao T He . Active transfer learning of matching query results across multiple sources. Frontiers of Computer Science, 2015, 9( 4): 595–607
3	K, Weiss T M, Khoshgoftaar D Wang . A survey of transfer learning. Journal of Big Data, 2016, 3( 1): 9
4	Y, Zhang G, Chu P, Li X, Hu X Wu . Three-layer concept drifting detection in text data streams. Neurocomputing, 2017, 260: 393–403
5	B, Du W, Xiong J, Wu L, Zhang L, Zhang D Tao . Stacked convolutional denoising auto-encoders for feature representation. IEEE Transactions on Cybernetics, 2017, 47( 4): 1017–1027
6	Y, Zhu X, Wu P, Li Y, Zhang X Hu . Transfer learning with deep manifold regularized auto-encoders. Neurocomputing, 2019, 369: 145–154
7	M, Caron P, Bojanowski A, Joulin M Douze . Deep clustering for unsupervised learning of visual features. In: Proceedings of the 15th European Conference on Computer Vision (ECCV). 2018, 139–156
8	H, Zhang Y, Zhang X Geng . Practical age estimation using deep label distribution learning. Frontiers of Computer Science, 2021, 15( 3): 153318
9	J, Qiang Z, Qian Y, Li Y, Yuan X Wu . Short text topic modeling techniques, applications, and performance: a survey. IEEE Transactions on Knowledge and Data Engineering, 2022, 34( 3): 1427–1445
10	Y, Zhu X, Hu Y, Zhang P Li . Transfer learning with stacked reconstruction independent component analysis. Knowledge-Based Systems, 2018, 152: 100–106
11	R, Li Q, Jiao W, Cao H S, Wong S Wu . Model adaptation: Unsupervised domain adaptation without source data. In: Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020, 9638–9647
12	M, Chen Z, Xu K Q, Weinberger F Sha . Marginalized denoising autoencoders for domain adaptation. In: Proceedings of the 29th International Conference on Machine Learning. 2012, 1627–1634
13	S, Yang Y, Zhang Y, Zhu P, Li X Hu . Representation learning via serial autoencoders for domain adaptation. Neurocomputing, 2019, 351: 1–9
14	J, Wang W, Feng Y, Chen H, Yu M, Huang P S Yu . Visual domain adaptation with manifold embedded distribution alignment. In: Proceedings of the 26th ACM International Conference on Multimedia. 2018, 402–410
15	N C, Iovanac B M Savoie . Simpler is better: how linear prediction tasks improve transfer learning in chemical autoencoders. The Journal of Physical Chemistry A, 2020, 124( 18): 3679–3685
16	X, Wang Y, Ma Y Cheng . Domain adaptation network based on autoencoder. Chinese Journal of Electronics, 2018, 27( 6): 1258–1264
17	F, Zhuang X, Cheng P, Luo S J, Pan Q He . Supervised representation learning with double encoding-layer autoencoder for transfer learning. ACM Transactions on Intelligent Systems and Technology, 2018, 9( 2): 16
18	C, Sun M, Ma Z, Zhao S, Tian R, Yan X Chen . Deep transfer learning based on sparse autoencoder for remaining useful life prediction of tool in manufacturing. IEEE Transactions on Industrial Informatics, 2019, 15( 4): 2416–2425
19	C, Li S, Zhang Y, Qin E Estupinan . A systematic review of deep transfer learning for machinery fault diagnosis. Neurocomputing, 2020, 407: 121–135
20	R K, Sevakula V, Singh N K, Verma C, Kumar Y Cui . Transfer learning for molecular cancer classification using deep neural networks. IEEE/ACM Transactions on Computational Biology and Bioinformatics, 2019, 16( 6): 2089–2100
21	M, Sun H, Wang P, Liu S, Huang P Fan . A sparse stacked denoising autoencoder with optimized transfer learning applied to the fault diagnosis of rolling bearings. Measurement, 2019, 146: 305–314
22	P, Vincent H, Larochelle I, Lajoie Y, Bengio P A Manzagol . Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. The Journal of Machine Learning Research, 2010, 11: 3371–3408
23	H, Yan Y, Ding P, Li Q, Wang Y, Xu W Zuo . Mind the class weight bias: weighted maximum mean discrepancy for unsupervised domain adaptation. In: Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. 2017, 945–954
24	W W, Lin M W, Mak J T Chien . Multisource I-vectors domain adaptation using maximum mean discrepancy based autoencoders. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2018, 26( 12): 2412–2422
25	S, Yang H, Wang Y, Zhang P, Li Y, Zhu X Hu . Semi-supervised representation learning via dual autoencoders for domain adaptation. Knowledge-Based Systems, 2020, 190: 105161
26	X, Glorot A, Bordes Y Bengio . Domain adaptation for large-scale sentiment classification: a deep learning approach. In: Proceedings of the 28th International Conference on Machine Learning. 2011, 513–520
27	X, Jin F, Zhuang H, Xiong C, Du P, Luo Q He . Multi-task multi-view learning for heterogeneous tasks. In: Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management. 2014, 441–450
28	S, Roy A, Siarohin E, Sangineto S R, Bulò N, Sebe E Ricci . Unsupervised domain adaptation using feature-whitening and consensus loss. In: Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019, 9463–9472
29	S J, Pan I W, Tsang J T, Kwok Q Yang . Domain adaptation via transfer component analysis. IEEE Transactions on Neural Networks, 2011, 22( 2): 199–210
30	B, Sun J, Feng K Saenko . Return of frustratingly easy domain adaptation. In: Proceedings of the 30th AAAI Conference on Artificial Intelligence. 2016, 2058–2065
31	Y, Cao M, Long J Wang . Unsupervised domain adaptation with distribution matching machines. In: Proceedings of the 32nd AAAI Conference on Artificial Intelligence and 30th Innovative Applications of Artificial Intelligence Conference and 8th AAAI Symposium on Educational Advances in Artificial Intelligence. 2018, 341
32	J, Zhang W, Li P Ogunbona . Joint geometrical and statistical alignment for visual domain adaptation. In: Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. 2017, 5150–5158
33	Z, Chen C, Chen X, Jin Y, Liu Z Cheng . Deep joint two-stream wasserstein auto-encoder and selective attention alignment for unsupervised domain adaptation. Neural Computing and Applications, 2020, 32( 11): 7489–7502
34	S, Ben-David J, Blitzer K, Crammer Pereira F. . Analysis of representations for domain adaptation. In: Proceedings of the 19th International Conference on Neural Information Processing Systems. 2007, 137–144
35	S, Yang Y, Zhang H, Wang P, Li X Hu . Representation learning via serial robust autoencoder for domain adaptation. Expert Systems with Applications, 2020, 160: 113635
36	J, Hoffman E, Rodner J, Donahue B, Kulis K Saenko . Asymmetric and category invariant feature transformations for domain adaptation. International Journal of Computer Vision, 2014, 109(1–2): 1–2
37	Y H, Tsai K, Sohn S, Schulter M Chandraker . Domain adaptation for structured output via discriminative patch representations. In: Proceedings of 2019 IEEE/CVF International Conference on Computer Vision. 2019, 1456–1465
38	R, Sharma P, Bhattacharyya S, Dandapat H S Bhatt . Identifying transferable information across domains for cross-domain sentiment classification. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. 2018, 968–978
39	M, Chen S, Zhao H, Liu D Cai . Adversarial-learned loss for domain adaptation. In: Proceedings of the 34th AAAI Conference on Artificial Intelligence. 2020, 3521–3528
40	H, Fan L, Zheng C, Yan Y Yang . Unsupervised person re-identification: clustering and fine-tuning. ACM Transactions on Multimedia Computing, Communications, and Applications, 2018, 14( 4): 83
41	H, Fan P, Liu M, Xu Y Yang . Unsupervised visual representation learning via dual-level progressive similar instance selection. IEEE Transactions on Cybernetics, 2022, 52( 9): 8851–8861
42	J, Qiang X Wu . Unsupervised statistical text simplification. IEEE Transactions on Knowledge and Data Engineering, 2021, 33( 4): 1802–1806
43	J, Qiang P, Chen W, Ding T, Wang F, Xie X Wu . Heterogeneous-length text topic modeling for reader-aware multi-document summarization. ACM Transactions on Knowledge Discovery from Data, 2019, 13( 4): 42
44	J C, Su Y H, Tsai K, Sohn B, Liu S, Maji M Chandraker . Active adversarial domain adaptation. In: Proceedings of 2020 IEEE Winter Conference on Applications of Computer Vision. 2020, 728–737
45	B, Gholami P, Sahu O, Rudovic K, Bousmalis V Pavlovic . Unsupervised multi-target domain adaptation: an information theoretic approach. IEEE Transactions on Image Processing, 2020, 29: 3993–4002
46	F M, Carlucci L, Porzi B, Caputo E, Ricci S R Buló . MultiDIAL: domain alignment layers for (multisource) unsupervised domain adaptation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 43( 12): 4441–4452
47	L, Luo L, Chen S, Hu Y, Lu X Wang . Discriminative and geometry-aware unsupervised domain adaptation. IEEE Transactions on Cybernetics, 2020, 50( 9): 3914–3927
48	P, Vincent H, Larochelle Y, Bengio P A Manzagol . Extracting and composing robust features with denoising autoencoders. In: Proceedings of the 25th International Conference on Machine Learning. 2008, 1096–1103
49	P, Wei Y, Ke C K Goh . Deep nonlinear feature coding for unsupervised domain adaptation. In: Proceedings of the 25th International Joint Conference on Artificial Intelligence. 2016, 2189–2195
50	D, Wang P, Cui W Zhu . Deep asymmetric transfer network for unbalanced domain adaptation. In: Proceedings of the 32nd AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence. 2018, 55

[1]

FCS-21349-of-YZ_suppl_1

Download

Viewed

Full text

Abstract

Cited

Shared

Discussed