Contextual modeling on auxiliary points for robust image reranking

doi:10.1007/s11704-018-7403-7

Front. Comput. Sci.

2019, Vol. 13

Issue (5) : 1010-1022 https://doi.org/10.1007/s11704-018-7403-7

RESEARCH ARTICLE

Contextual modeling on auxiliary points for robust image reranking

Ying LI¹, Xiangwei KONG¹(

), Haiyan FU¹, Qi TIAN²

¹. School of Information and Communication Engineering, Dalian University of Technology, Dalian 116024, China
². Department of Computer Science, University of Texas at San Antonio, Texas 78249, USA

Download: PDF(606 KB)
Export: BibTeX | EndNote | Reference Manager | ProCite | RefWorks

Abstract

Image reranking is an effective post-processing step to adjust the similarity order in image retrieval. As key components of initialized ranking lists, top-ranked neighborhoods of a given query usually play important roles in constructing dissimilarity measure. However, the number of pertinent candidates varies with respect to different queries. Thus the images with short lists of ground truth suffer from insufficient contextual information. It consequently introduces noises when using k-nearest neighbor rule to define the context. In order to alleviate this problem, this paper proposes auxiliary points which are added as assistant neighbors in an unsupervised manner. These extra points act on revealing implicit similarity in the metric space and clustering matched image pairs. By isometrically embedding each constructed metric space into the Euclidean space, the image relationships on underlying topological manifolds are locally represented by distance descriptions. Furthermore, by combining Jaccard index with our auxiliary points, we present a contextual modeling on auxiliary points (CMAP) method for image reranking.With richer contextual activations, the Jaccard similarity coefficient defined by local distribution achieves more reliable outputs as well as more stable parameters. Extensive experiments demonstrate the robustness and effectiveness of the proposed method.

Keywords image retrieval unsupervised reranking context construction Jaccard distance query expansion

Corresponding Author(s): Xiangwei KONG

Just Accepted Date: 28 May 2018 Online First Date: 04 September 2018 Issue Date: 25 June 2019

Cite this article:

Ying LI,Xiangwei KONG,Haiyan FU, et al. Contextual modeling on auxiliary points for robust image reranking[J]. Front. Comput. Sci., 2019, 13(5): 1010-1022.

URL:

https://academic.hep.com.cn/fcs/EN/10.1007/s11704-018-7403-7
https://academic.hep.com.cn/fcs/EN/Y2019/V13/I5/1010

1	H Jégou, M Douze, C Schmid. Improving bag-of-features for large scale image search. International Journal of Computer Vision, 2010, 87(3): 316–336 https://doi.org/10.1007/s11263-009-0285-2
2	G Song, X Tan. Hierarchical deep hashing for image retrieval. Frontiers of Computer Science, 2017, 11(2): 253–265 https://doi.org/10.1007/s11704-017-6537-3
3	D G Lowe. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 2004, 60(2): 91–110 https://doi.org/10.1023/B:VISI.0000029664.99615.94
4	Y Li, X Kong, L Zheng, Q Tian. Exploiting hierarchical activations of neural network for image retrieval. In: Proceedings of the 24nd ACM International Conference on Multimedia. 2016, 132–136 https://doi.org/10.1145/2964284.2967197
5	H Jégou, F Perronnin, M Douze, J Sanchez, P Perez, C Schmid. Aggregating local image descriptors into compact codes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2012, 34(9): 1704–1716 https://doi.org/10.1109/TPAMI.2011.235
6	Z Liu, S Wang, Q Tian. Fine-residual VLAD for image retrieval. Neurocomputing, 2016, 173: 1183–1191 https://doi.org/10.1016/j.neucom.2015.08.076
7	L Zheng, S Wang, Z Liu, Q Tian. Packing and padding: coupled multiindex for accurate image retrieval. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2014, 1939–1946 https://doi.org/10.1109/CVPR.2014.250
8	O Chum, A Mikulik, M Perdoch, J Matas. Total recall II: query expansion revisited. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2011, 889–896 https://doi.org/10.1109/CVPR.2011.5995601
9	S Bai, X Bai, Q Tian, L J Latecki. Regularized diffusion process for visual retrieval. In: Proceedings of the AAAI Conference on Artificial Intelligence. 2017, 3967–3973
10	S Bai, Z Zhou, J Wang, X Bai, L J Latecki, Q Tian. Ensemble diffusion for retrieval. In: Proceedings of the IEEE Conference on Computer Vision. 2017, 774–783 https://doi.org/10.1109/ICCV.2017.90
11	S Köknar-Tezel, L J Latecki. Improving SVM classification on imbalanced time series data sets with ghost points. Knowledge and Information Systems, 2011, 28(1): 1–23 https://doi.org/10.1007/s10115-010-0310-3
12	H Jégou, M Douze, C Schmid. On the burstiness of visual elements. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2009, 1169–1176 https://doi.org/10.1109/CVPR.2009.5206609
13	Y Zhu, J Jiang, W Han, Y Ding, Q Tian. Interpretation of users’ feedback via swarmed particles for content-based image retrieval. Information Sciences, 2017, 375: 246–257 https://doi.org/10.1016/j.ins.2016.09.021
14	L Zheng, Y Yang, Q Tian. Sift meets CNN: a decade survey of instance retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 40(5): 1224–1244 https://doi.org/10.1109/TPAMI.2017.2709749
15	K Chen, G Ding, J Han. Attribute-based supervised deep learning model for action recognition. Frontiers of Computer Science, 2017, 11(2): 219–229 https://doi.org/10.1007/s11704-016-6066-5
16	Y Gong, L Wang, R Guo, S Lazebnik. Multi-scale orderless pooling of deep convolutional activation features. In: Proceedings of the European Conference on Computer Vision. 2014, 392–407 https://doi.org/10.1007/978-3-319-10584-0_26
17	A Babenko, A Slesarev, A Chigorin, V Lempitsky. Neural codes for image retrieval. In: Proceedings of the European Conference on Computer Vision. 2014, 584–599 https://doi.org/10.1007/978-3-319-10590-1_38
18	Y Kalantidis, C Mellina, S Osindero. Cross-dimensional weighting for aggregated deep convolutional features. In: Proceedings of the European Conference on Computer Vision. 2016, 685–701 https://doi.org/10.1007/978-3-319-46604-0_48
19	J Y Ng, F Yang, L S Davis. Exploiting local features from deep networks for image retrieval. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 2015, 53–61 https://doi.org/10.1109/CVPRW.2015.7301272
20	A Babenko, V Lempitsky. Aggregating local deep features for image retrieval. In: Proceedings of the IEEE International Conference on Computer Vision. 2015, 1269–1277
21	O Chum, J Philbin, J Sivic, M Isard, A Zisserman. Total recall: automatic query expansion with a generative feature model for object retrieval. In: Proceedings of the IEEE International Conference on Computer Vision. 2007, 1–8 https://doi.org/10.1109/ICCV.2007.4408891
22	D Qin, S Gammeter, L Bossard, T Quack, L V Gool. Hello neighbor: accurate object retrieval with k-reciprocal nearest neighbors. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2011, 777–784 https://doi.org/10.1109/CVPR.2011.5995373
23	H Jégou, C Schmid, H Harzallah, J Verbeek. Accurate image search using the contextual dissimilarity measure. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010, 32(1): 2–11 https://doi.org/10.1109/TPAMI.2008.285
24	S Sun, Y Li, W Zhou, Q Tian, H Li. Local residual similarity for image re-ranking. Information Sciences, 2017, 417: 143–153 https://doi.org/10.1016/j.ins.2017.07.004
25	R Arandjelovi′c, A Zisserman. Three things everyone should know to improve object retrieval. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2012, 2911–2918
26	X Yang, L Prasad, L J Latecki. Affinity learning with diffusion on tensor product graph. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013, 35(1): 28–38 https://doi.org/10.1109/TPAMI.2012.60
27	M Donoser, H Bischof. Diffusion processes for retrieval revisited. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2013, 1320–1327 https://doi.org/10.1109/CVPR.2013.174
28	S Bai, X Bai, Q Tian. Scalable person re-identification on supervised smoothed manifold. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017, 3356–3365 https://doi.org/10.1109/CVPR.2017.358
29	S Bai, X Bai, Q Tian, L J Latecki. Regularized diffusion process on bidirectional context for object retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 99: 1 https://doi.org/10.1109/TPAMI.2018.2828815
30	S Bai, X Bai. Sparse contextual activation for efficient visual reranking. IEEE Transactions on Image Processing, 2016, 25(3): 1056–1069 https://doi.org/10.1109/TIP.2016.2514498
31	D Nister, H Stewenius. Scalable recognition with a vocabulary tree. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2006, 2161–2168 https://doi.org/10.1109/CVPR.2006.264
32	H Jégou, M Douze, C Schmid. Hamming embedding and weak geometric consistency for large scale image search. In: Proceedings of the European Conference on Computer Vision. 2008, 304–317 https://doi.org/10.1007/978-3-540-88682-2_24
33	V Balntas, K Lenc, A Vedaldi, K Mikolajczyk. Hpatches: a benchmark and evaluation of handcrafted and learned local descriptors. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017, 3852–3861 https://doi.org/10.1109/CVPR.2017.410
34	J Philbin, O Chum, M Isard, J Sivic, A Zisserman. Object retrieval with large vocabularies and fast spatial matching. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2007, 1–8 https://doi.org/10.1109/CVPR.2007.383172
35	M J Huiskes, M S Lew. The mir flickr retrieval evaluation. In: Proceedings of the ACM International Conference on Multimedia Information Retrieval. 2008, 39–43 https://doi.org/10.1145/1460096.1460104
36	Y Jia, E Shelhamer, J Donahue, S Karayev, J Long, R Girshick, S Guadarrama, T Darrell. Caffe: convolutional architecture for fast feature embedding. In: Proceedings of the 22nd ACM International Conference on Multimedia. 2014, 675–678 https://doi.org/10.1145/2647868.2654889

[1]

Download

[1]	Kun SU, Gongping YANG, Lu YANG, Peng SU, Yilong YIN. Non-negative locality-constrained vocabulary tree for finger vein image retrieval[J]. Front. Comput. Sci., 2019, 13(2): 318-332.
[2]	Xueming WANG, Zechao LI, Jinhui TANG. Visual understanding by mining social media: recent advances and challenges[J]. Front. Comput. Sci., 2018, 12(3): 406-422.
[3]	Ilyes KHENNAK, Habiba DRIAS. Strength Pareto fitness assignment for pseudo-relevance feedback: application to MEDLINE[J]. Front. Comput. Sci., 2018, 12(1): 163-176.
[4]	Le DONG, Wenpu DONG, Ning FENG, Mengdie MAO, Long CHEN, Gaipeng KONG. Color space quantization-based clustering for image retrieval[J]. Front. Comput. Sci., 2017, 11(6): 1023-1035.
[5]	Ge SONG,Xiaoyang TAN. Hierarchical deep hashing for image retrieval[J]. Front. Comput. Sci., 2017, 11(2): 253-265.
[6]	Weimin TAN,Bo YAN. A survey on high coherence visual media retargeting: recent advances and applications[J]. Front. Comput. Sci., 2016, 10(5): 778-796.

Viewed

Full text

Abstract

Cited

Shared

Discussed