Please wait a minute...
Frontiers of Computer Science

ISSN 2095-2228

ISSN 2095-2236(Online)

CN 10-1014/TP

Postal Subscription Code 80-970

2018 Impact Factor: 1.129

Front. Comput. Sci.    2019, Vol. 13 Issue (4) : 778-788    https://doi.org/10.1007/s11704-017-6613-8
RESEARCH ARTICLE
Joint salient object detection and existence prediction
Huaizu JIANG1, Ming-Ming CHENG2(), Shi-Jie LI1, Ali BORJI3, Jingdong WANG4
1. College of Information and Computer Sciences, University of Massachusetts Amherst, Amherst, MA 01003, USA
2. CCCE, Nankai University Jinnan Campus, Tianjin 300353, China
3. Center for Research in Computer Vision, University of Central Florida, Orlando, FL 32816, USA
4. Microsoft Research, Beijing 100080, China
 Download: PDF(565 KB)  
 Export: BibTeX | EndNote | Reference Manager | ProCite | RefWorks
Abstract

Recent advances in supervised salient object detection modeling has resulted in significant performance improvements on benchmark datasets. However, most of the existing salient object detection models assume that at least one salient object exists in the input image. Such an assumption often leads to less appealing saliencymaps on the background images with no salient object at all. Therefore, handling those cases can reduce the false positive rate of a model. In this paper, we propose a supervised learning approach for jointly addressing the salient object detection and existence prediction problems. Given a set of background-only images and images with salient objects, as well as their salient object annotations, we adopt the structural SVM framework and formulate the two problems jointly in a single integrated objective function: saliency labels of superpixels are involved in a classification term conditioned on the salient object existence variable, which in turn depends on both global image and regional saliency features and saliency labels assignments. The loss function also considers both image-level and regionlevel mis-classifications. Extensive evaluation on benchmark datasets validate the effectiveness of our proposed joint approach compared to the baseline and state-of-the-art models.

Keywords salient object detection      existence prediction      joint inference      saliency detection     
Corresponding Author(s): Ming-Ming CHENG   
Just Accepted Date: 22 August 2017   Online First Date: 25 May 2018    Issue Date: 29 May 2019
 Cite this article:   
Huaizu JIANG,Ming-Ming CHENG,Shi-Jie LI, et al. Joint salient object detection and existence prediction[J]. Front. Comput. Sci., 2019, 13(4): 778-788.
 URL:  
https://academic.hep.com.cn/fcs/EN/10.1007/s11704-017-6613-8
https://academic.hep.com.cn/fcs/EN/Y2019/V13/I4/778
1 LItti, CKoch, ENiebur. A model of saliency-based visual attention for rapid scene analysis. IEEE Transactions on Pattern Snalysis and Machine Intelligence, 1998, 20(11): 1254–1259
https://doi.org/10.1109/34.730558
2 ABorji, LItti. State-of-the-art in visual attention modeling. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013, 35(1): 185–207
https://doi.org/10.1109/TPAMI.2012.89
3 ABorji, D NSihite, LItti. Quantitative analysis of human-model agreement in visual saliency modeling: a comparative study. IEEE Transactions on Image Processing, 2013, 22(1): 55–69
https://doi.org/10.1109/TIP.2012.2210727
4 TLiu, ZYuan, JSun, J Wang, NZheng, XTang, H YShum. Learning to detect a salient object. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2011, 33(2): 353–367
https://doi.org/10.1109/TPAMI.2010.70
5 G XZhang, M MCheng, S MHu, R R Martin. A shape-preserving approach to image resizing. Computer Graphics Forum, 2009, 28(7): 1897–1906
https://doi.org/10.1111/j.1467-8659.2009.01568.x
6 TChen, M MCheng, PTan, A Shamir, S MHu. Sketch2photo: Internet image montage. ACM Transactions on Graphics (TOG), 2009, 28(5): 124
https://doi.org/10.1145/1618452.1618470
7 TChen, PTan, L QMa, M M Cheng, AShamir, S MHu. Poseshop: human image database construction and personalized content synthesis. IEEE Transactions on Visualization and Computer Graphics, 2013, 19(5): 824–837
https://doi.org/10.1109/TVCG.2012.148
8 M MCheng, N JMitra , XHuang, S M Hu. Salientshape: group saliency in image collections. The Visual Computer, 2014, 30(4): 443–453
https://doi.org/10.1007/s00371-013-0867-4
9 JWang, LQuan, JSun, X Tang, H YShum. Picture collage. In: Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 2006, 347–354
10 AAbdulmunem, Y KLai, XSun. Saliency guided local and global descriptors for effective action recognition. Computational Visual Media, 2016, 2(1): 97–106
https://doi.org/10.1007/s41095-016-0033-9
11 JZhang, YHan, JJiang. Tucker decomposition-based tensor learning for human action recognition. Multimedia Systems, 2016, 22(3): 343–353
https://doi.org/10.1007/s00530-015-0464-7
12 S MHu, TChen, KXu, M MCheng, R RMartin. Internet visual media processing: a survey with graphics and vision applications. The Visual Computer, 2013, 29(5): 393–405
https://doi.org/10.1007/s00371-013-0792-6
13 M MCheng, Q BHou, S HZhang, P L Rosin. Intelligent visual media processing: when graphics meets vision. Journal of Computer Science and Technology, 2017, 32(1): 110–121
https://doi.org/10.1007/s11390-017-1681-7
14 HJiang, JWang, ZYuan, Y Wu, NZheng, SLi. Salient object detection: a discriminative regional feature integration approach. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2013, 2083–2090
https://doi.org/10.1109/CVPR.2013.271
15 RZhao , WOuyang, HLi, XWang. Saliency detection by multi-context deep learning. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2015, 1265–1274
https://doi.org/10.1109/CVPR.2015.7298731
16 GLi, YYu. Visual saliency based on multiscale deep features. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2015, 5455–5463
17 FPerazzi, P Krähenbühl, YPritch, AHornung. Saliency filters: contrast based filtering for salient region detection. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2012, 733–740
https://doi.org/10.1109/CVPR.2012.6247743
18 WZhu, SLiang, YWei , J Sun. Saliency optimization from robust background detection. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2014, 2814–2821
https://doi.org/10.1109/CVPR.2014.360
19 XLi, HLu, LZhang, X Ruan, M HYang. Saliency detection via dense and sparse reconstruction. In: Proceedings of IEEE International Conference on Computer Vision. 2013, 2976–2983
https://doi.org/10.1109/ICCV.2013.370
20 YLeCun, LBottou, YBengio, P Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 1998, 86(11): 2278–2324
https://doi.org/10.1109/5.726791
21 AKrizhevsky, I Sutskever, G EHinton. Imagenet classification with deep convolutional neural networks. In: Proceedings of the Neural Information Processing Systems Conference. 2012, 1106–1114
22 ABorji. What is a salient object? a dataset and a baseline model for salient object detection. IEEE Transactions on Image Processing, 2015, 24(2): 742–756
https://doi.org/10.1109/TIP.2014.2383320
23 PWang, JWang, GZeng, J Feng, HZha, SLi. Salient object detection for searched Web images via global saliency. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2012, 3194–3201.
https://doi.org/10.1109/CVPR.2012.6248054
24 YBoykov, V Kolmogorov. An experimental comparison of mincut/ max-flow algorithms for energy minimization in vision. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2004, 26(9): 1124–1137
https://doi.org/10.1109/TPAMI.2004.60
25 ABorji, M MCheng, QHou, H Jiang, JLi. Salient object detection: a survey. 2014, arXiv preprint arXiv:1411.5878
26 ABorji, M MCheng, HJiang, J Li. Salient object detection: a benchmark. IEEE Transactions on Image Processing, 2015, 24(12): 5706–5722
https://doi.org/10.1109/TIP.2015.2487833
27 JHan, NLiu, DZhang. Visual saliency detection and applications: a survey. Frontiers of Computer Science, 2017
28 RAchanta, SHemami, FEstrada, S Süsstrunk. Frequency-tuned salient region detection. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2009, 1597–1604
https://doi.org/10.1109/CVPR.2009.5206596
29 SGoferman, L Zelnik-Manor, ATal. Context-aware saliency detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2012, 34(10): 1915–1926
https://doi.org/10.1109/TPAMI.2011.272
30 YTian, JLi, SYu, THuang. Learning complementary saliency priors for foreground object segmentation in complex scenes. International Journal of Computer Vision, 2015, 111(2): 153–170
https://doi.org/10.1007/s11263-014-0737-1
31 SFang, JLi, YTian, T Huang, XChen. Learning discriminative subspaces on random contrasts for image saliency analysis. IEEE Transactions on Neural Networks and Learning Systems, 2017, 28(5): 1095–1108
https://doi.org/10.1109/TNNLS.2016.2522440
32 RMargolin, ATal, LZelnik-Manor. What makes a patch distinct? In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2013, 1139–1146
https://doi.org/10.1109/CVPR.2013.151
33 M MCheng, N JMitra, XHuang, P H Torr, S MHu. Global contrast based salient region detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 37(3): 569–582
https://doi.org/10.1109/TPAMI.2014.2345401
34 ABorji, LItti. Exploiting local and global patch rarities for saliency detection. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2012, 478–485
https://doi.org/10.1109/CVPR.2012.6247711
35 WQi, M MCheng, ABorji, H Lu, L FBai. SaliencyRank: two-stage manifold ranking for salient object detection. Computational Visual Media, 2015, 1(4): 309–320
https://doi.org/10.1007/s41095-015-0028-y
36 HJiang , JWang, ZYuan, T Liu, NZheng, SLi . Automatic salient object segmentation based on context and shape prior. In: Proceedings of the British Machine Vision Conference (BMVC). 2011
https://doi.org/10.5244/C.25.110
37 P FFelzenszwalb, D P Huttenlocher. Efficient graph-based image segmentation. International Journal of Computer Vision, 2004, 59(2): 16–181
https://doi.org/10.1023/B:VISI.0000022288.19776.77
38 M MCheng, YLiu, QHou, J Bian, PTorr, S MHu, ZTu. HFS: hierarchical feature selection for efficient image segmentation. In: Proceedings of European Conference on Computer Vision. 2016, 867–882
https://doi.org/10.1007/978-3-319-46487-9_53
39 QYan, LXu, JShi, J Jia. Hierarchical saliency detection. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2013, 1155–1162
https://doi.org/10.1109/CVPR.2013.153
40 YWei , FWen, WZhu, J Sun. Geodesic saliency using background priors. In: Proceedings of European Conference on Computer Vision. 2012, 29–42
https://doi.org/10.1007/978-3-642-33712-3_3
41 CYang, LZhang, HLu, XRuan, M HYang. Saliency detection via graph-based manifold ranking. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2013, 3166–3173
https://doi.org/10.1109/CVPR.2013.407
42 BJiang, LZhang, HLu, CYang, M HYang. Saliency detection via absorbing Markov chain. In: Proceedings of IEEE International Conference on Computer Vision. 2013, 1665–1672
https://doi.org/10.1109/ICCV.2013.209
43 JZhang, S Sclaroff. Saliency detection: a boolean map approach. In: Proceedings of IEEE International Conference on Computer Vision. 2013, 153–160
https://doi.org/10.1109/ICCV.2013.26
44 K YChang, T LLiu, H TChen, S H Lai. Fusing generic objectness and visual saliency for salient object detection. In: Proceedings of IEEE International Conference on Computer Vision. 2011, 914–921
45 PJiang, HLing, JYu, JPeng. Salient region detection by UFO: uniqueness, focusness and objectness. In: Proceedings of IEEE International Conference on Computer Vision. 2013, 1976–1983
https://doi.org/10.1109/ICCV.2013.248
46 YJia, MHan. Category-independent object-level saliency detection. In: Proceedings of IEEE International Conference on Computer Vision. 2013, 1761–1768
https://doi.org/10.1109/ICCV.2013.221
47 M MCheng, J Warrell, W YLin, SZheng, VVineet, NCrook. Efficient salient region detection with soft image abstraction. In: Proceedings of IEEE International Conference on Computer Vision. 2013, 1529–1536
https://doi.org/10.1109/ICCV.2013.193
48 LMai, YNiu, FLiu. Saliency aggregation: a data-driven approach. In: Proceedings of IEEE International Conference on Computer Vision. 2013, 1131–1138
https://doi.org/10.1109/CVPR.2013.150
49 SLu, V Mahadevan, NVasconcelos. Learning optimal seeds for diffusion-based salient object detection. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2014, 2790–2797
https://doi.org/10.1109/CVPR.2014.357
50 PMehrani, O Veksler. Saliency segmentation based on learning and graph cut refinement. In: Proceedings of the British Machine Vision Conference (BMVC). 2010, 1–12
https://doi.org/10.5244/C.24.110
51 JKim, DHan, Y WTai, J Kim. Salient region detection via highdimensional color transform. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2014, 883–890
52 PKhuwuthyakorn, A Robles-Kelly, JZhou. Object of interest detection by saliency learning. In: Proceedings of European Conference on Computer Vision. 2010
https://doi.org/10.1007/978-3-642-15552-9_46
53 QHou, M MCheng, XHu, ABorji, ZTu, PTorr. Deeply supervised salient object detection with short connections. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2017, 5300–5309
https://doi.org/10.1109/CVPR.2017.563
54 JZhang, SMa, MSameki, S Sclaroff, MBetke, ZLin, XShen, BPrice, R Mech. Salient object subitizing. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2015, 4045–4054
https://doi.org/10.1109/CVPR.2015.7299031
55 JDeng, WDong, RSocher, L J Li, KLi, F FLi. Imagenet: a large-scale hierarchical image database. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2009, 248–255
https://doi.org/10.1109/CVPR.2009.5206848
56 RGirshick, J Donahue, TDarrell, JMalik. Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2014, 580–587
https://doi.org/10.1109/CVPR.2014.81
57 SYang, PLuo, C CLoy, X Tang. From facial parts responses to face detection: a deep learning approach. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2015, 3676–3684
https://doi.org/10.1109/ICCV.2015.419
58 MCimpoi, SMaji, AVedaldi. Deep filter banks for texture recognition and segmentation. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2015, 3828–3836
https://doi.org/10.1109/CVPR.2015.7299007
59 T YLin, A RoyChowdhury, SMaji. Bilinear CNN models for finegrained visual recognition. In: Proceedings of IEEE International Conference on Computer Vision. 2015, 1449–1457
60 HSu, SMaji, EKalogerakis, ELearned-Miller. Multi-view convolutional neural networks for 3D shape recognition. In: Proceedings of IEEE International Conference on Computer Vision. 2015, 945–953
https://doi.org/10.1109/ICCV.2015.114
61 M DZeiler, RFergus. Visualizing and understanding convolutional networks. In: Proceedings of European Conference on Computer Vision. 2014, 818–833
https://doi.org/10.1007/978-3-319-10590-1_53
62 T M TDo, T Artières. Regularized bundle methods for convex and non-convex risks. Journal of Machine Learning Research, 2012, 13: 3539–3583
63 JXiao, JHays, K AEhinger, A Oliva, ATorralba. SUN database: large-scale scene recognition from abbey to zoo. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2010, 3485–3492
https://doi.org/10.1109/CVPR.2010.5539970
64 MCimpoi, SMaji, IKokkinos, S Mohamed, AVedaldi. Describing textures in the wild. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2014, 3606–3613
https://doi.org/10.1109/CVPR.2014.461
65 XShen, YWu. A unified approach to salient object detection via low rank matrix recovery. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2012, 853–860
66 HHuang, LZhang, H CZhang. Arcimboldo-like collage using internet images. ACM Transactions on Graphics, 2011, 30(6): 155
https://doi.org/10.1145/2070781.2024189
67 HLiu, LZhang, HHuang. Web-image driven best views of 3D shapes. The Visual Computer, 2012, 28(3): 279–287
https://doi.org/10.1007/s00371-011-0638-z
68 YWei, XLiang, YChen, X Shen, M MCheng, JFeng, YZhao, SYan. STC: a simple to complex framework for weakly-supervised semantic segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(11): 2314–2320
https://doi.org/10.1109/TPAMI.2016.2636150
69 A Y SChia, SZhuo, R KGupta, Y W Tai, S YCho, PTan, SLin. Semantic colorization with Internet images. ACM Transactions on Graphics, 2011, 30(6): 156
https://doi.org/10.1145/2070781.2024190
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed