Recent years have witnessed a rapid spread of multi-modality microblogs like Twitter and SinaWeibo composed of image, text and emoticon. Visual sentiment prediction of such microblog based social media has recently attracted ever-increasing research focus with broad application prospect. In this paper, we give a systematic review of the recent advances and cutting-edge techniques for visual sentiment analysis. To this end, in this paper we review the most recent works in this topic, in which detailed comparison as well as experimental evaluation are given over the cuttingedge methods.We further reveal and discuss the future trends and potential directions for visual sentiment prediction.
Picard R W. Affective Computing. MIT Technical Report. 1995
2
Pang B, Lee L, Vaithyanathan S. Thumbs up?: sentiment classification using machine learning techniques. In: Proceedings of the ACL- 02 Conference on Empirical Methods in Natural Language Processing. 2002, 79–86Wilson T, Wiebe J, Hwa R. Just how mad are you? Finding strong and weak opinion clauses. In: Proceedings of National Conference on Artificial Intelligence. 2004, 761–767
3
Zhou L, Li B, Gao W, Wei Z, Wong K F. Unsupervised discovery of discourse relations for eliminating intra-sentence polarity ambiguities. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing. 2011, 162–171
4
Hu M Q, Liu B. Mining and summarizing customer reviews. In: Proceedings of ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 2004, 168–177
https://doi.org/10.1145/1014052.1014073
5
Wilson T, Wiebe J, Hoffmann P. Recognizing contextual polarity in phrase-level sentiment analysis. In: Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing. 2005, 347–354
https://doi.org/10.3115/1220575.1220619
6
Liu J J, Cao Y B, Lin C Y, Huang Y L, Zhou M. Low-quality product review detection in sentiment summarization. In: Proceedings of the Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning. 2007
7
McGlohon M, Glance N S, Reiter Z. Star quality: aggregating reviews to rank products and merchants. In: Proceedings of the International Conference on Weblogs and Social Media. 2010
8
Joshi M, Das D, Gimpel K, Smith N A. Movie reviews and revenues: an experiment in text regression. In: Proceedings of the 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics. 2010, 293–296
9
Feldman R, Rosenfeld B, Bar-Haim R, Fresko M. The stock sonar— sentiment analysis of stocks based on a hybrid approach. In: Proceedings of the 23rd IAAI Conference on Artificial Intelligence. 2011
10
Zhang W B, Skiena S. Trading strategies to exploit blog and news sentiment. In: Proceedings of the International Conference on Weblogs and Social Media. 2010
11
Tumasjan A, Sprenger T O, Sandner P G, Welpe I M. Predicting elections with twitter: what 140 characters reveal about political sentiment. In: Proceedings of the International Conference onWeblogs and Social Media. 2010, 178–185
12
Chen B, Leilei Z, Daniel K, Dongwon L. What is an sentiment about? exploring political standpoints using sentiment scoring model. In: Proceeedings of AAAI Conference on Artificial Intelligence. 2010
13
Yano T, Smith N A. What’s worthy of comment? content and comment volume in political blogs. In: Proceedings of the International AAAI Conference on Weblogs and Social Media. 2010
14
Groh G, Hauffa J. Characterizing social relations via NLP-based sentiment analysis. In: Proceedings of the 5th International AAAI Conference on Weblogs and Social Media. 2011
15
Li B, Feng S, Xiong W, Hu W. Scaring or pleasing: exploit emotional impact of an image. In: Proceedings of the 20th ACM International Conference on Multimedia. 2012, 1365–1366
https://doi.org/10.1145/2393347.2396487
16
Jia J, Wu S, Wang X H, Hu P Y, Cai L H, Tang J. Can we understand van Gogh’s mood?: learning to infer affects from images in social networks. In: Proceedings of the 20th ACM International Conference on Multimedia. 2012, 857–860
https://doi.org/10.1145/2393347.2396330
17
Vonikakis V, Winkler S. Emotion-based sequence of family photos. In: Proceedings of the 20th ACM International Conference onMultimedia. 2012, 1371–1372
https://doi.org/10.1145/2393347.2396490
18
Siersdorfer S, Minack E, Deng F, Hare J. Analyzing and predicting sentiment of images on the social Web. In: Proceedings of the 18th ACM International Conference on Multimedia. 2010, 715–718
https://doi.org/10.1145/1873951.1874060
19
Borth D, Ji R, Chen T, Breuel T, Chang S F. Large-scale visual sentiment ontology and detectors using adjective noun pairs. In: Proceedings of the 21st ACM International Conference on Multimedia. 2013, 223–232
https://doi.org/10.1145/2502081.2502282
20
Yuan J, Mcdonough S, You Q, Luo J. Sentribute: image sentiment analysis from a mid-level perspective. In: Proceedings of the 2nd International Workshop on Issues of Sentiment Discovery and Opinion Mining. 2013
https://doi.org/10.1145/2502069.2502079
21
Cao D L, Ji R R, Lin A Z, Li S Z. Visual sentiment topic model based microblog image sentiment analysis. Multimedia Tools and Applications, 2014: 1–14
https://doi.org/10.1007/s11042-014-2337-z
22
Ji R R, Yao H X, Liu W, Sun X S, Tian Q. Task dependent visual codebook compression, IEEE Transactions on Image Processing, 2012, 21(4): 2282–2293
https://doi.org/10.1109/TIP.2011.2176950
23
Ji R R, Duan L Y, Yao H X, Xie L X, Rui Y, Gao W. Learning to distribute vocabulary indexing for scalable visual search. IEEE Transactions on Multimedia, 2013, 15(1): 153–166
https://doi.org/10.1109/TMM.2012.2225035
24
Ji R R, Gao Y, Liu W, Xie X, Tian Q, Li X L. When location meets social multimedia: a comprehensive survey on location-aware social multimedia. ACM Transactions on Intelligent System and Technology, 2015, 6(1): 1–18
https://doi.org/10.1145/2597181
25
Ji R R, Yao H X, Tian Q, Xu P F, Sun X S, Liu X M, Context-aware semi-local feature detector. ACM Transactions on Intelligent System and Technology, 2012, 3(3): 44–71
https://doi.org/10.1145/2168752.2168758
26
Ji R R, Yu F X, Chang S F. Active query sensing for mobile location search. ACM Transactions on Multimedia Computing, Communications and Applications, 2012, 3S(8): 40–61
27
Ji R R, Gao Y, Zhong B N, Yao H X, Tian Q. Mining flickr landmarks bymodeling reconstruction sparsity. ACM Transactions onMultimedia Computing, Communications and Applications, 2011, 7S(1)
28
Ji R R, Duan L Y, Chen J, Huang T J, Gao W. Mining compact 3D patterns for low bit rate mobile visual search. IEEE Transactions on Image Processing, 2014, 23(7): 3099–3113
https://doi.org/10.1109/TIP.2014.2324291
29
You Q Z, Luo J B, Jin H L, Yang J C. Robust image sentiment analysis using progressively trained and domain transferred deep networks. 2015, arXiv:1509.06041
30
Chen T, Yu F X, Chen J, Cui Y, Chen Y Y, Chang S F. Object-based visual sentiment concept analysis and application. In: Proceedings of the ACM International Conference on Multimedia. 2014, 367–376.
https://doi.org/10.1145/2647868.2654935
31
Li L X, Cao D L, Li S Z, Ji R R. Sentiment analysis of Chinese micro-blog based on multi-modal correlation model. In: Proceedings of the 2015 IEEE International Conference on Image Processing. 2015, 4798–4802
https://doi.org/10.1109/ICIP.2015.7351718
32
Chen F H, Gao Y, CAO D L. Multimodal hypergraph learning for microblog sentiment prediction. In: Proceedings of the 2015 IEEE International Conference on Multimedia and Expo. 2015, 1–6
https://doi.org/10.1109/ICME.2015.7177447
33
Wang M, Cao D L, Li L X, Li S Z, Ji R R. Microblog sentiment analysis based on cross-media bag-of-words model. In: Proceedings of International Conference on Internet Multimedia Computing and Service. 2014
https://doi.org/10.1145/2632856.2632912
34
You Q Z, Luo J B, Jin H L, Yang J C. Cross-modality consistent regression for joint visual-textual sentiment analysis. In: Proceedings of the 9th ACM International Conference on Web Search and Data Mining. 2016, 13–22
https://doi.org/10.1145/2835776.2835779
35
Zhong B N, Yao H X, Chen S, Ji R R, Chin T J, Wang H Z. Visual tracking via weakly supervised learning from multiple imperfect oracles. Pattern Recognition, 2014, 47(3): 1395–1410
https://doi.org/10.1016/j.patcog.2013.10.002
36
Gao Y, Wang F L, Luan H B, Chua T S. Brand data gathering from live social media streams. In: Proceedings of ACM International Conference on Multimedia Retrieval. 2014
https://doi.org/10.1145/2578726.2578748
37
Gao Y, Wang M, Zha Z J, Shen J L, Li X L, Wu X D. Visual-textual joint relevance learning for tag-based social image search. IEEE Transactions on Image Processing, 2013, 22(1): 363–376
https://doi.org/10.1109/TIP.2012.2202676
38
Gao Y, Wang M, Zha Z J, Tian Q, Dai Q H, Zhang N Y. Less is more: efficient 3-D object retrieval with query view selection. IEEE Transactions on Multimedia, 2011, 13(5): 1007–1018
https://doi.org/10.1109/TMM.2011.2160619
39
Gao Y, Tang J H, Hong R C, Yan S C, Dai Q H, Zhang N Y, Chua T S. Camera constraint-free view-based 3-D object retrieval. IEEE Transactions on Image Processing, 2012, 21(4): 2269–2281
https://doi.org/10.1109/TIP.2011.2170081
40
Huang Y C, Liu Q S, Zhang S T, Metaxas D N. Image retrieval via probabilistic hypergraph ranking. In: Proceedings of the 2010 IEEE Conference on Computer Vision and Pattern Recognition. 2010, 3376–3383
https://doi.org/10.1109/CVPR.2010.5540012
41
Gao Y, Wang M, Zha Z J, Shen J L, Li X L,Wu X D. Visual-textual joint relevance learning for tag-based social image search. IEEE Transactions on Image Processing, 2013, 22(1): 363–376
https://doi.org/10.1109/TIP.2012.2202676
42
Cai Z, Cao D L, Ji R R. Video (GIF) sentiment analysis using largescale mid-level ontology. 2015, arXiv:1506.00765
43
Wang Y L, Wang S H, Tang J L, Huan Liu H, Li B X. Unsupervised sentiment analysis for social media images. In: Proceedings of the 24th International Conference on Artificial Intelligence. 2015, 2378–2379
44
Narihira T, Borth D, Yu S X, Ni K, Darrell T. Mapping images to sentiment adjective noun pairs with factorized neural nets. 2015, arXiv preprint arXiv:1511.06838
