Joint salient object detection and existence prediction
Huaizu JIANG1, Ming-Ming CHENG2(), Shi-Jie LI1, Ali BORJI3, Jingdong WANG4
1. College of Information and Computer Sciences, University of Massachusetts Amherst, Amherst, MA 01003, USA 2. CCCE, Nankai University Jinnan Campus, Tianjin 300353, China 3. Center for Research in Computer Vision, University of Central Florida, Orlando, FL 32816, USA 4. Microsoft Research, Beijing 100080, China
Recent advances in supervised salient object detection modeling has resulted in significant performance improvements on benchmark datasets. However, most of the existing salient object detection models assume that at least one salient object exists in the input image. Such an assumption often leads to less appealing saliencymaps on the background images with no salient object at all. Therefore, handling those cases can reduce the false positive rate of a model. In this paper, we propose a supervised learning approach for jointly addressing the salient object detection and existence prediction problems. Given a set of background-only images and images with salient objects, as well as their salient object annotations, we adopt the structural SVM framework and formulate the two problems jointly in a single integrated objective function: saliency labels of superpixels are involved in a classification term conditioned on the salient object existence variable, which in turn depends on both global image and regional saliency features and saliency labels assignments. The loss function also considers both image-level and regionlevel mis-classifications. Extensive evaluation on benchmark datasets validate the effectiveness of our proposed joint approach compared to the baseline and state-of-the-art models.
LItti, CKoch, ENiebur. A model of saliency-based visual attention for rapid scene analysis. IEEE Transactions on Pattern Snalysis and Machine Intelligence, 1998, 20(11): 1254–1259 https://doi.org/10.1109/34.730558
2
ABorji, LItti. State-of-the-art in visual attention modeling. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013, 35(1): 185–207 https://doi.org/10.1109/TPAMI.2012.89
3
ABorji, D NSihite, LItti. Quantitative analysis of human-model agreement in visual saliency modeling: a comparative study. IEEE Transactions on Image Processing, 2013, 22(1): 55–69 https://doi.org/10.1109/TIP.2012.2210727
4
TLiu, ZYuan, JSun, J Wang, NZheng, XTang, H YShum. Learning to detect a salient object. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2011, 33(2): 353–367 https://doi.org/10.1109/TPAMI.2010.70
TChen, M MCheng, PTan, A Shamir, S MHu. Sketch2photo: Internet image montage. ACM Transactions on Graphics (TOG), 2009, 28(5): 124 https://doi.org/10.1145/1618452.1618470
7
TChen, PTan, L QMa, M M Cheng, AShamir, S MHu. Poseshop: human image database construction and personalized content synthesis. IEEE Transactions on Visualization and Computer Graphics, 2013, 19(5): 824–837 https://doi.org/10.1109/TVCG.2012.148
8
M MCheng, N JMitra , XHuang, S M Hu. Salientshape: group saliency in image collections. The Visual Computer, 2014, 30(4): 443–453 https://doi.org/10.1007/s00371-013-0867-4
9
JWang, LQuan, JSun, X Tang, H YShum. Picture collage. In: Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 2006, 347–354
10
AAbdulmunem, Y KLai, XSun. Saliency guided local and global descriptors for effective action recognition. Computational Visual Media, 2016, 2(1): 97–106 https://doi.org/10.1007/s41095-016-0033-9
11
JZhang, YHan, JJiang. Tucker decomposition-based tensor learning for human action recognition. Multimedia Systems, 2016, 22(3): 343–353 https://doi.org/10.1007/s00530-015-0464-7
12
S MHu, TChen, KXu, M MCheng, R RMartin. Internet visual media processing: a survey with graphics and vision applications. The Visual Computer, 2013, 29(5): 393–405 https://doi.org/10.1007/s00371-013-0792-6
13
M MCheng, Q BHou, S HZhang, P L Rosin. Intelligent visual media processing: when graphics meets vision. Journal of Computer Science and Technology, 2017, 32(1): 110–121 https://doi.org/10.1007/s11390-017-1681-7
14
HJiang, JWang, ZYuan, Y Wu, NZheng, SLi. Salient object detection: a discriminative regional feature integration approach. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2013, 2083–2090 https://doi.org/10.1109/CVPR.2013.271
15
RZhao , WOuyang, HLi, XWang. Saliency detection by multi-context deep learning. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2015, 1265–1274 https://doi.org/10.1109/CVPR.2015.7298731
16
GLi, YYu. Visual saliency based on multiscale deep features. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2015, 5455–5463
17
FPerazzi, P Krähenbühl, YPritch, AHornung. Saliency filters: contrast based filtering for salient region detection. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2012, 733–740 https://doi.org/10.1109/CVPR.2012.6247743
18
WZhu, SLiang, YWei , J Sun. Saliency optimization from robust background detection. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2014, 2814–2821 https://doi.org/10.1109/CVPR.2014.360
19
XLi, HLu, LZhang, X Ruan, M HYang. Saliency detection via dense and sparse reconstruction. In: Proceedings of IEEE International Conference on Computer Vision. 2013, 2976–2983 https://doi.org/10.1109/ICCV.2013.370
20
YLeCun, LBottou, YBengio, P Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 1998, 86(11): 2278–2324 https://doi.org/10.1109/5.726791
21
AKrizhevsky, I Sutskever, G EHinton. Imagenet classification with deep convolutional neural networks. In: Proceedings of the Neural Information Processing Systems Conference. 2012, 1106–1114
22
ABorji. What is a salient object? a dataset and a baseline model for salient object detection. IEEE Transactions on Image Processing, 2015, 24(2): 742–756 https://doi.org/10.1109/TIP.2014.2383320
23
PWang, JWang, GZeng, J Feng, HZha, SLi. Salient object detection for searched Web images via global saliency. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2012, 3194–3201. https://doi.org/10.1109/CVPR.2012.6248054
24
YBoykov, V Kolmogorov. An experimental comparison of mincut/ max-flow algorithms for energy minimization in vision. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2004, 26(9): 1124–1137 https://doi.org/10.1109/TPAMI.2004.60
25
ABorji, M MCheng, QHou, H Jiang, JLi. Salient object detection: a survey. 2014, arXiv preprint arXiv:1411.5878
26
ABorji, M MCheng, HJiang, J Li. Salient object detection: a benchmark. IEEE Transactions on Image Processing, 2015, 24(12): 5706–5722 https://doi.org/10.1109/TIP.2015.2487833
27
JHan, NLiu, DZhang. Visual saliency detection and applications: a survey. Frontiers of Computer Science, 2017
28
RAchanta, SHemami, FEstrada, S Süsstrunk. Frequency-tuned salient region detection. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2009, 1597–1604 https://doi.org/10.1109/CVPR.2009.5206596
29
SGoferman, L Zelnik-Manor, ATal. Context-aware saliency detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2012, 34(10): 1915–1926 https://doi.org/10.1109/TPAMI.2011.272
30
YTian, JLi, SYu, THuang. Learning complementary saliency priors for foreground object segmentation in complex scenes. International Journal of Computer Vision, 2015, 111(2): 153–170 https://doi.org/10.1007/s11263-014-0737-1
31
SFang, JLi, YTian, T Huang, XChen. Learning discriminative subspaces on random contrasts for image saliency analysis. IEEE Transactions on Neural Networks and Learning Systems, 2017, 28(5): 1095–1108 https://doi.org/10.1109/TNNLS.2016.2522440
32
RMargolin, ATal, LZelnik-Manor. What makes a patch distinct? In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2013, 1139–1146 https://doi.org/10.1109/CVPR.2013.151
33
M MCheng, N JMitra, XHuang, P H Torr, S MHu. Global contrast based salient region detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 37(3): 569–582 https://doi.org/10.1109/TPAMI.2014.2345401
34
ABorji, LItti. Exploiting local and global patch rarities for saliency detection. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2012, 478–485 https://doi.org/10.1109/CVPR.2012.6247711
35
WQi, M MCheng, ABorji, H Lu, L FBai. SaliencyRank: two-stage manifold ranking for salient object detection. Computational Visual Media, 2015, 1(4): 309–320 https://doi.org/10.1007/s41095-015-0028-y
36
HJiang , JWang, ZYuan, T Liu, NZheng, SLi . Automatic salient object segmentation based on context and shape prior. In: Proceedings of the British Machine Vision Conference (BMVC). 2011 https://doi.org/10.5244/C.25.110
M MCheng, YLiu, QHou, J Bian, PTorr, S MHu, ZTu. HFS: hierarchical feature selection for efficient image segmentation. In: Proceedings of European Conference on Computer Vision. 2016, 867–882 https://doi.org/10.1007/978-3-319-46487-9_53
39
QYan, LXu, JShi, J Jia. Hierarchical saliency detection. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2013, 1155–1162 https://doi.org/10.1109/CVPR.2013.153
40
YWei , FWen, WZhu, J Sun. Geodesic saliency using background priors. In: Proceedings of European Conference on Computer Vision. 2012, 29–42 https://doi.org/10.1007/978-3-642-33712-3_3
41
CYang, LZhang, HLu, XRuan, M HYang. Saliency detection via graph-based manifold ranking. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2013, 3166–3173 https://doi.org/10.1109/CVPR.2013.407
42
BJiang, LZhang, HLu, CYang, M HYang. Saliency detection via absorbing Markov chain. In: Proceedings of IEEE International Conference on Computer Vision. 2013, 1665–1672 https://doi.org/10.1109/ICCV.2013.209
43
JZhang, S Sclaroff. Saliency detection: a boolean map approach. In: Proceedings of IEEE International Conference on Computer Vision. 2013, 153–160 https://doi.org/10.1109/ICCV.2013.26
44
K YChang, T LLiu, H TChen, S H Lai. Fusing generic objectness and visual saliency for salient object detection. In: Proceedings of IEEE International Conference on Computer Vision. 2011, 914–921
45
PJiang, HLing, JYu, JPeng. Salient region detection by UFO: uniqueness, focusness and objectness. In: Proceedings of IEEE International Conference on Computer Vision. 2013, 1976–1983 https://doi.org/10.1109/ICCV.2013.248
46
YJia, MHan. Category-independent object-level saliency detection. In: Proceedings of IEEE International Conference on Computer Vision. 2013, 1761–1768 https://doi.org/10.1109/ICCV.2013.221
47
M MCheng, J Warrell, W YLin, SZheng, VVineet, NCrook. Efficient salient region detection with soft image abstraction. In: Proceedings of IEEE International Conference on Computer Vision. 2013, 1529–1536 https://doi.org/10.1109/ICCV.2013.193
48
LMai, YNiu, FLiu. Saliency aggregation: a data-driven approach. In: Proceedings of IEEE International Conference on Computer Vision. 2013, 1131–1138 https://doi.org/10.1109/CVPR.2013.150
49
SLu, V Mahadevan, NVasconcelos. Learning optimal seeds for diffusion-based salient object detection. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2014, 2790–2797 https://doi.org/10.1109/CVPR.2014.357
50
PMehrani, O Veksler. Saliency segmentation based on learning and graph cut refinement. In: Proceedings of the British Machine Vision Conference (BMVC). 2010, 1–12 https://doi.org/10.5244/C.24.110
51
JKim, DHan, Y WTai, J Kim. Salient region detection via highdimensional color transform. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2014, 883–890
52
PKhuwuthyakorn, A Robles-Kelly, JZhou. Object of interest detection by saliency learning. In: Proceedings of European Conference on Computer Vision. 2010 https://doi.org/10.1007/978-3-642-15552-9_46
53
QHou, M MCheng, XHu, ABorji, ZTu, PTorr. Deeply supervised salient object detection with short connections. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2017, 5300–5309 https://doi.org/10.1109/CVPR.2017.563
54
JZhang, SMa, MSameki, S Sclaroff, MBetke, ZLin, XShen, BPrice, R Mech. Salient object subitizing. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2015, 4045–4054 https://doi.org/10.1109/CVPR.2015.7299031
55
JDeng, WDong, RSocher, L J Li, KLi, F FLi. Imagenet: a large-scale hierarchical image database. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2009, 248–255 https://doi.org/10.1109/CVPR.2009.5206848
56
RGirshick, J Donahue, TDarrell, JMalik. Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2014, 580–587 https://doi.org/10.1109/CVPR.2014.81
57
SYang, PLuo, C CLoy, X Tang. From facial parts responses to face detection: a deep learning approach. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2015, 3676–3684 https://doi.org/10.1109/ICCV.2015.419
58
MCimpoi, SMaji, AVedaldi. Deep filter banks for texture recognition and segmentation. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2015, 3828–3836 https://doi.org/10.1109/CVPR.2015.7299007
59
T YLin, A RoyChowdhury, SMaji. Bilinear CNN models for finegrained visual recognition. In: Proceedings of IEEE International Conference on Computer Vision. 2015, 1449–1457
60
HSu, SMaji, EKalogerakis, ELearned-Miller. Multi-view convolutional neural networks for 3D shape recognition. In: Proceedings of IEEE International Conference on Computer Vision. 2015, 945–953 https://doi.org/10.1109/ICCV.2015.114
61
M DZeiler, RFergus. Visualizing and understanding convolutional networks. In: Proceedings of European Conference on Computer Vision. 2014, 818–833 https://doi.org/10.1007/978-3-319-10590-1_53
62
T M TDo, T Artières. Regularized bundle methods for convex and non-convex risks. Journal of Machine Learning Research, 2012, 13: 3539–3583
63
JXiao, JHays, K AEhinger, A Oliva, ATorralba. SUN database: large-scale scene recognition from abbey to zoo. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2010, 3485–3492 https://doi.org/10.1109/CVPR.2010.5539970
64
MCimpoi, SMaji, IKokkinos, S Mohamed, AVedaldi. Describing textures in the wild. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2014, 3606–3613 https://doi.org/10.1109/CVPR.2014.461
65
XShen, YWu. A unified approach to salient object detection via low rank matrix recovery. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2012, 853–860
66
HHuang, LZhang, H CZhang. Arcimboldo-like collage using internet images. ACM Transactions on Graphics, 2011, 30(6): 155 https://doi.org/10.1145/2070781.2024189
YWei, XLiang, YChen, X Shen, M MCheng, JFeng, YZhao, SYan. STC: a simple to complex framework for weakly-supervised semantic segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(11): 2314–2320 https://doi.org/10.1109/TPAMI.2016.2636150
69
A Y SChia, SZhuo, R KGupta, Y W Tai, S YCho, PTan, SLin. Semantic colorization with Internet images. ACM Transactions on Graphics, 2011, 30(6): 156 https://doi.org/10.1145/2070781.2024190
