1. School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, China 2. Department of Computing, Hong Kong Polytechnic University, Hong Kong 100872, China 3. State Key Lab. for Novel Software Technology, Nanjing University, Nanjing 210094, China 4. 14th Research Institute of China Electronics Technology Group Corporation, Nanjing 210094, China
Traditional image-sentence cross-modal retrieval methods usually aim to learn consistent representations of heterogeneous modalities, thereby to search similar instances in one modality according to the query from another modality in result. The basic assumption behind these methods is that parallel multi-modal data (i.e., different modalities of the same example are aligned) can be obtained in prior. In other words, the image-sentence cross-modal retrieval task is a supervised task with the alignments as ground-truths. However, in many real-world applications, it is difficult to realign a large amount of parallel data for new scenarios due to the substantial labor costs, leading the non-parallel multi-modal data and existing methods cannot be used directly. On the other hand, there actually exists auxiliary parallel multi-modal data with similar semantics, which can assist the non-parallel data to learn the consistent representations. Therefore, in this paper, we aim at “Alignment Efficient Image-Sentence Retrieval” (AEIR), which recurs to the auxiliary parallel image-sentence data as the source domain data, and takes the non-parallel data as the target domain data. Unlike single-modal transfer learning, AEIR learns consistent image-sentence cross-modal representations of target domain by transferring the alignments of existing parallel data. Specifically, AEIR learns the image-sentence consistent representations in source domain with parallel data, while transferring the alignment knowledge across domains by jointly optimizing a novel designed cross-domain cross-modal metric learning based constraint with intra-modal domain adversarial loss. Consequently, we can effectively learn the consistent representations for target domain considering both the structure and semantic transfer. Furthermore, extensive experiments on different transfer scenarios validate that AEIR can achieve better retrieval results comparing with the baselines.
Z, Wang X, Liu J, Lin C, Yang H Li . Multi-attention based cross-domain beauty product image retrieval. Science China Information Sciences, 2020, 63( 2): 120112
2
K, Wang Q, Yin W, Wang S, Wu L Wang . A comprehensive survey on cross-modal retrieval. 2016, arXiv preprint arXiv: 1607.06215
3
Y, Peng J, Qi Z, Ye Y Zhuo . Hierarchical visual-textual knowledge distillation for life-long correlation learning. International Journal of Computer Vision, 2021, 129( 4): 921–941
4
Y, Liu Y Y, Guo J, Fang J L, Fan Y, Hao J M Liu . Survey of research on deep learning image-text cross-modal retrieval. Journal of Frontiers of Computer Science & Technology, 2022, 16( 3): 489–511
5
J, Chi Y Peng . Dual adversarial networks for zero-shot cross-media retrieval. In: Proceedings of the 27th International Joint Conference on Artificial Intelligence. 2018, 663−669
6
Zhen L, Hu P, Wang X, Peng D. Deep supervised cross-modal retrieval. In: Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019, 10386−10395
7
D, Wang X, Gao X, Wang L, He B Yuan . Multimodal discriminative binary embedding for large-scale cross-modal retrieval. IEEE Transactions on Image Processing, 2016, 25( 10): 4540–4554
8
W, Qu D, Wang S, Feng Y, Zhang G Yu . A novel cross-modal hashing algorithm based on multimodal deep learning. Science China Information Sciences, 2017, 60( 9): 092104
9
Wang Z, Liu X, Li H, Sheng L, Yan J, Wang X, Shao J. CAMP: cross-modal adaptive message passing for text-image retrieval. In: Proceedings of 2019 IEEE/CVF International Conference on Computer Vision. 2019, 5763−5772
10
K H, Lee X, Chen G, Hua H, Hu X He . Stacked cross attention for image-text matching. In: Proceedings of the 15th European Conference on Computer Vision. 2018, 212−228
11
Y, Zhang H Lu . Deep cross-modal projection learning for image-text matching. In: Proceedings of the 15th European Conference on Computer Vision. 2018, 707−723
12
Yu F, Tang J, Yin W, Sun Y, Tian H, Wu H, Wang H. ERNIE-ViL: Knowledge enhanced vision-language representations through scene graphs. In: Proceedings of AAAI Conference on Artificial Intelligence. 2021, 3208−3216
13
Y, Peng J, Qi Y Zhuo . MAVA: multi-level adaptive visual-textual alignment by cross-media bi-attention mechanism. IEEE Transactions on Image Processing, 2020, 29: 2728–2741
14
Z, Ji H, Wang J, Han Y Pang . SMAN: stacked multimodal attention network for cross-modal image-text retrieval. IEEE Transactions on Cybernetics, 2022, 52( 2): 1086–1097
15
A, Frome G S, Corrado J, Shlens S, Bengio J, Dean M, Ranzato T Mikolov . DeViSE: a deep visual-semantic embedding model. In: Proceedings of the 26th International Conference on Neural Information Processing Systems. 2013, 2121−2129
16
Song G, Tan X. Sequential learning for cross-modal retrieval. In: Proceedings of 2019 IEEE/CVF International Conference on Computer Vision Workshop. 2019, 4531−4539
17
Feng Y, Ma L, Liu W, Luo J. Unsupervised image captioning. In: Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019, 4120−4129
18
Gu J, Joty S R, Cai J, Zhao H, Yang X, Wang G. Unpaired image captioning via scene graph alignments. In: Proceedings of 2019 IEEE/CVF International Conference on Computer Vision. 2019, 10322−10331
19
P Y, Huang G, Kang W, Liu X, Chang A G Hauptmann . Annotation efficient cross-modal retrieval with adversarial attentive alignment. In: Proceedings of the 27th ACM International Conference on Multimedia. 2019, 1758−1767
20
L, Zhen P, Hu X, Peng R S M, Goh J T Zhou . Deep multimodal transfer learning for cross-modal retrieval. IEEE Transactions on Neural Networks and Learning Systems, 2020, 33( 2): 798–810
21
Chen Q, Liu Y, Albanie S. Mind-the-gap! Unsupervised domain adaptation for text-video retrieval. In: Proceedings of AAAI Conference on Artificial Intelligence. 2021, 1072−1080
22
W, Zhao X, Wu J Luo . Cross-domain image captioning via cross-modal retrieval and model adaptation. IEEE Transactions on Image Processing, 2021, 30: 1180–1192
23
G, Geigle J, Pfeiffer N, Reimers I, Vulić I Gurevych . Retrieve fast, Rerank smart: cooperative and joint approaches for improved cross-modal retrieval. Transactions of the Association for Computational Linguistics, 2022, 10: 503–521
24
Yang Y, Zhang C, Xu Y C, Yu D, Zhan D C, Yang J. Rethinking label-wise cross-modal retrieval from A semantic sharing perspective. In: Proceedings of the 30th International Joint Conference on Artificial Intelligence. 2021, 3300−3306
25
S J, Pan I W, Tsang J T, Kwok Q Yang . Domain adaptation via transfer component analysis. IEEE Transactions on Neural Networks, 2011, 22( 2): 199–210
26
T R, Scott K, Ridgeway M C Mozer . Adapted deep embeddings: A synthesis of methods for k-shot inductive transfer learning. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems. 2018, 76−85
27
Y, Wang C, Wang H, Xue S Chen . Self-corrected unsupervised domain adaptation. Frontiers of Computer Science, 2022, 16( 5): 165323
28
J, Yosinski J, Clune Y, Bengio H Lipson . How transferable are features in deep neural networks? In: Proceedings of the 27th International Conference on Neural Information Processing Systems. 2014, 3320−3328
29
Y, Ganin E, Ustinova H, Ajakan P, Germain H, Larochelle F, Laviolette M, Marchand V Lempitsky . Domain-adversarial training of neural networks. The Journal of Machine Learning Research, 2016, 17( 1): 2096–2030
30
M, Long Z, Cao J, Wang M I Jordan . Conditional adversarial domain adaptation. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems. 2018, 1647−1657
31
Z, Yao Y, Wang M, Long J Wang . Unsupervised transfer learning for spatiotemporal predictive networks. In: Proceedings of the 37th International Conference on Machine Learning. 2020, 999
32
Deng J, Dong W, Socher R, Li L J, Li K, Fei-Fei L. ImageNet: a large-scale hierarchical image database. In: Proceedings of 2009 IEEE Conference on Computer Vision and Pattern Recognition. 2009, 248−255
33
A, Karpathy L Fei-Fei . Deep visual-semantic alignments for generating image descriptions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(4): 664−676
34
R, Kiros R, Salakhutdinov R S Zemel . Unifying visual-semantic embeddings with multimodal neural language models. 2014, arXiv preprint arXiv: 1411.2539
35
R, Socher A, Karpathy Q V, Le C D, Manning A Y Ng . Grounded compositional semantics for finding and describing images with sentences. Transactions of the Association for Computational Linguistics, 2014, 2: 207–218
36
F, Faghri D J, Fleet J R, Kiros S Fidler . VSE++: improving visual-semantic embeddings with hard negatives. In: Proceedings of the British Machine Vision Conference 2018. 2018, 12
37
H, Diao Y, Zhang L, Ma H Lu . Similarity reasoning and filtration for image-text matching. In: Proceedings of the AAAI Conference on Artificial Intelligence. 2021, 1218−1226
38
Tzeng E, Hoffman J, Saenko K, Darrell T. Adversarial discriminative domain adaptation. In: Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. 2017, 2962−2971
39
Z, Luo Y, Zou J, Hoffman L Fei-Fei . Label efficient learning of transferable representations across domains and tasks. In: Proceedings of the 31st International Conference on Neural Information Processing Systems. 2017, 165−177
40
I J, Goodfellow J, Pouget-Abadie M, Mirza B, Xu D, Warde-Farley S, Ozair A C, Courville Y Bengio . Generative adversarial nets. In: Proceedings of the 27th International Conference on Neural Information Processing Systems. 2014, 2672−2680
41
J, Hoffman E, Tzeng T, Darrell K Saenko . Simultaneous deep transfer across domains and tasks. In: Csurka G, ed. Domain Adaptation in Computer Vision Applications. Cham: Springer, 2017, 173−187
42
F, Zhuang Z, Qi K, Duan D, Xi Y, Zhu H, Zhu H, Xiong Q He . A comprehensive survey on transfer learning. Proceedings of the IEEE, 2021, 109( 1): 43–76
43
M J, Huiskes M S Lew . The MIR flickr retrieval evaluation. In: Proceedings of the 1st ACM International Conference on Multimedia Information Retrieval. 2008, 39−43
44
T, Lin M, Maire S, Belongie J, Hays P, Perona D, Ramanan P, Dollár C L Zitnick . Microsoft COCO: common objects in context. In: Proceedings of the 13th European Conference on Computer Vision. 2014, 740−755
45
H Hotelling . Relations between two sets of variates. In: Kotz S, Johnson N L, eds. Breakthroughs in Statistics: Methodology and Distribution. New York: Springer, 1992, 162−190
46
G, Andrew R, Arora J A, Bilmes K Livescu . Deep canonical correlation analysis. In: Proceedings of the 30th International Conference on Machine Learning. 2013, 1247−1255
47
Zhang J, Peng Y, Yuan M. Unsupervised generative adversarial cross-modal hashing. In: Proceedings of AAAI Conference on Artificial Intelligence. 2018, 539−546
48
Chen H, Ding G, Liu X, Lin Z, Liu J, Han J. IMRAM: iterative matching with recurrent attention memory for cross-modal image-text retrieval. In: Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020, 12652−12660
49
S J, Peng Y, He X, Liu Y M, Cheung X, Xu Z Cui . Relation-aggregated cross-graph correlation learning for fine-grained image–text retrieval. IEEE Transactions on Neural Networks and Learning Systems, 2022, doi:
50
Y, Peng Z, Ye J, Qi Y Zhuo . Unsupervised visual-textual correlation learning with fine-grained semantic alignment. IEEE Transactions on Cybernetics, 2022, 52( 5): 3669–3683
51
Saito K, Kim D, Sclaroff S, Darrell T, Saenko K. Semi-supervised domain adaptation via minimax entropy. In: Proceedings of 2019 IEEE/CVF International Conference on Computer Vision. 2019, 8049−8057
52
D P, Kingma J Ba . Adam: A method for stochastic optimization. In: Proceedings of the 3rd International Conference on Learning Representations. 2015
53
J Lin . Divergence measures based on the Shannon entropy. IEEE Transactions on Information Theory, 1991, 37( 1): 145–151
