Line drawing via saliency map and ETF

doi:10.1007/s11704-021-1027-z

Front. Comput. Sci.

2022, Vol. 16

Issue (5) : 165707 https://doi.org/10.1007/s11704-021-1027-z

RESEARCH ARTICLE

Line drawing via saliency map and ETF

Shiguang LIU(

), Ziqi LIU

College of Intelligence and Computing, Tianjin University, Tianjin 300350, China

Download: PDF(9069 KB) HTML
Export: BibTeX | EndNote | Reference Manager | ProCite | RefWorks

Abstract

Line drawing is a means of superior visual communication which can effectively convey useful information to viewers. Artists usually draw what they see rather than what exists, which means the most attractive object is emphasized while the rest are inhibited. Moreover, artists draw the whole object with coherent lines instead of fractured lines, which also contribute to the outstanding visual effect. From these perspectives, generating line drawings with saliency and coherence remains a great challenge. Existing line drawing generation methods ignore these important properties and cannot generate coherent lines in some cases since they do not take into account how artists draw a picture. To this end, a novel saliency-aware line drawing method was proposed to better grasp the viewer’s attention on an image. First, a saliency enhanced line extraction method combining saliency map and edge tangent flow was proposed to ensure the saliency and coherence of lines, especially in salient but low contrast areas. Then, the salient information was highlighted while irrelevant details were eliminated by inhibiting lines in less salient areas to enhance the saliency of the line drawing. Finally, the transparency of lines was adjusted to further emphasize important information. Various results showed that our method can generate line drawings with better visual saliency and coherence than the state-of-the-art methods.

Keywords line drawing saliency coherence

Corresponding Author(s): Shiguang LIU

Just Accepted Date: 12 July 2021 Issue Date: 24 December 2021

Cite this article:

Shiguang LIU,Ziqi LIU. Line drawing via saliency map and ETF[J]. Front. Comput. Sci., 2022, 16(5): 165707.

URL:

https://academic.hep.com.cn/fcs/EN/10.1007/s11704-021-1027-z
https://academic.hep.com.cn/fcs/EN/Y2022/V16/I5/165707

Fig.1 The famous work “Rain Steam and Speed the Great Western Railway” painted by J.M.W. Turner [5]. Note that the train is clearly drawn while the background is blurred to emphasize the train

Fig.2 Illustration of different types of line drawings. Images (a) through (d) are pencil sketch [7], line drawing with blocks [8], line drawing with hand writing style [1], and line drawing without style [9], respectively

Fig.3 The framework of our proposed saliency enhanced line drawing generation method. It consists of four components: preprocessing, saliency enhanced line extraction, saliency enhancement of the line drawing, and transparency adjustment

Fig.4 Illustration of preprocessing. (a) is the input image, (b) is the smoothed luminance channel of the image translated to the CIE-Lab color space, (c) is the saliency map from [36], (d) is the updated saliency map calculated from Eq. (1), and (e) and (f) show the gradient image and the visualized edge tangent flow, respectively

Fig.5 Illustration of the coherence enhanced line extraction. The pixel

p x

is the one that is processed,

S

and

? S

are pixels along the ETF of the pixel

p x

in different directions,

T

and

? T

are pixels along the gradient direction of the pixel

p x

in different directions

Fig.6 Intermediate results in the saliency enhanced line extraction step. Images (a) and (b) are the results generated from Eq. (3) and Eq. (7)

Fig.7 Illustration of saliency value calculation. In (a), pixels in the circle lie in the same saliency area. In (b), pixels in the circle lie in different saliency area. (a) The first case Saliency map; (b) The second case

Fig.8 Intermediate results in saliency enhancement of line drawing step. Images (a) and (b) are the saliency enhanced line drawing and the final result after transparency adjustment

Fig.9 The results of our method. From top to bottom are original images, the updated saliency map, and our final results, respectively

Fig.10 Our results with different parameters. The second image of (a) is defined as the standard result with

α = 1

β = 1

γ = 0.3

s t h = 0.2

l e v e l = 10

and

s t e p = 0.1

. The other results have the same parameters with it except for the parameter indicated under the image

Fig.11 Comparison of our intermediate result with saliency enhanced line extraction with other methods. Images (a) through (f) are the original image, the saliency map, the gradient image, Kang et al.’s result [12], Wang et al.’s result [13], and our result, respectively

Fig.12 A comparison between our method and the method in [15]. Images (a) through (f) are the original image, the saliency map, the gradient image, the results of [21], our intermediate results, and our final result, respectively

Fig.13 A comparison between our method and the method in [14]. Images (a) through (f) are the original image, the saliency map, the gradient image, the results of [14], our intermediate results, and our final result, respectively

Fig.14 Comparison of our intermediate result with saliency enhancement of the line drawing step with other methods. Images (a) through (f) are original image, saliency map, gradient image, Kang et al.’s result [12], Wang et al.’s result [13], and our result, respectively

Fig.15 The influence of the threshold of our method and Kang et al.’s method [12]. Images (a) through (f) are the original image, the updated saliency map, the result of [12] with a smaller threshold, our result with a smaller threshold, the result of [12] with a greater threshold, and our result with a greater threshold, respectively

Fig.16 Statistical result of the user survey. The

X

-axis represents the questions and the

Y

-axis are the average percentage of positive answer

Fig.17 Failure examples of our method. (a) and (b) are the original images, (c) and (d) are the saliency maps, (e) and (f) are our final results

1	M Son, H Kang, Y Lee, S Lee. Abstract line drawings from 2D images. In: Proceedings of IEEE Pacific Conference on Computer Graphics and Applications. 2007, 333– 342
2	J Zhou, B Li. Automatic generation of pencil-sketch like drawings from personal photos. In: Proceedings of IEEE International Conference on Multimedia and Expo. 2005, 1026−1029
3	J He, S Wang, Y Zhang, J Zhang. A computational fresco sketch generation framework. In: Proceedings of IEEE International Conference on Multimedia and Expo. 2013, 1– 6
4	J Choi , H Cho , J Song , S M Yoon . Sketchhelper: real-time stroke guidance for freehand sketch retrieval. IEEE Transactions on Multimedia, 2019, 21( 8): 2083– 2092
5	M Hata , M Toyoura , X Mao . Automatic generation of accentuated pencil drawing with saliency map and LIC. Visual Computer, 2012, 28( 6−8): 657– 668
6	D Doug , S Anthony . Stylization and abstraction of photographs. ACM Transactions on Graphics, 2002, 21( 3): 769– 776
7	H Winnemoeller, J Xie, A Hertzmann, W Li. Portraitsketch: face sketching assistance for novices. In: Proceedings of Annual ACM Symposium on User Interface Software and Technology. 2014, 407– 417
8	Y T Wu , J S Yeh , F C Wu , Y Y Chuang . Tangent-based binary image abstraction. Journal of Imaging, 2017, 3( 2): 16– 30
9	H Kang , S Lee , C K Chui . Flow-based image abstraction. IEEE Transactions on Visualization and Computer Graphics, 2009, 15( 1): 62– 76
10	A Orzan, A Bousseau, P Barla, J Thollot. Structurepreserving manipulation of photographs. In: Proceeding of International Symposium on Non-Photorealistic Animation and Rendering. 2007, 103– 110
11	P L Rosin , Y K Lai . Artistic minimal rendering with lines and blocks. Graphical Models, 2013, 75( 4): 208– 229
12	H Kang, S Lee, C K Chui. Coherent line drawing. In: Proceedings of International Symposium on NonPhotorealistic Animation and Rendering. 2007, 43– 50
13	S Wang , E Wu , Y Liu , X Liu , Y Chen . Abstract line drawings from photographs using flow-based filters. Computers and Graphics, 2012, 36( 4): 224– 231
14	M Li, Z Lin, R Mech, E Yumer, D Ramanan. Photosketching: inferring contour drawings from images. In: Proceedings of Winter Conference on Applications of Computer Vision. 2019, 1– 10
15	Y Liu, M M Cheng, X Hu, K Wang, X Bai. Richer convolutional features for edge detection. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2017, 3000−3009
16	S Xie , Z Tu . Holistically-nested edge detection. International Journal of Computer Vision, 2017, 125( 1): 3– 18
17	B Sheng , P Li , C Gao , K L Ma . Deep neural representation guided face sketch synthesis. IEEE Transactions on Visualization and Computer Graphics, 2019, 25( 12): 3216– 3230
18	H W Kang , C K Chui , U K Chakraborty . A unified scheme for adaptive stroke-based rendering. Visual Computer, 2006, 22 : 9– 11
19	B Gooch , E Reinhard , A Gooch . Human facial illustrations: creation and psychophysical evaluation. ACM Transactions on Graphics, 2004, 23( 1): 27– 44
20	Y Nie , X Cao , P Li , Q Zhang , Z Zhang , G Li , H Sun . Interactive contour extraction via sketch-alike densevalidation optimization. IEEE Transactions on Circuits and Systems for Video Technology, 2020, 30( 4): 906– 916
21	S Wang , Z Ao , S Yao , Y Chen , G Wang . Adaptive line drawing. Journal of Image and Graphics, 2018, 23( 5): 730– 739
22	J Zhang, R Z Wani, D Xu. Automatic generation of sketch-like pencil drawing from image. In: Proceedings of IEEE International Conference on Multimedia and Expo. 2017, 261– 266
23	R Li , S Sun , X Yang , W Hu . Automatic generation of pencil drawings based on multi-level saliency maps. Journal of Computer-Aided Design and Computer Graphics, 2018, 30( 1): 108– 116
24	B Alexe , T Deselaers , V Ferrari . Measuring the objectness of image windows. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2012, 34( 11): 2189– 2202
25	Q Yan, L Xu, J Shi, J Jia. Hierarchical saliency detection. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2013, 2013−1162
26	K Gu , S Wang , H Yang , W Lin , G Zhai , X Yang , S Member , W Zhang . Saliency-guided quality assessment of screen content images. IEEE Transactions on Multimedia, 2016, 18( 6): 1098– 1100
27	Z Huang, S Liu. Perceptual hashing with visual content understanding for reduced-reference screen content image quality assessment. IEEE Transactions on Circuits and Systems for Video Technology, 2021, 31(7): 2808−2823
28	R Gao, Z Huang, S Liu. Multi-task deep learning for no-reference screen content image quality assessment. In: Proceedings of the 27th International Conference on Multimedia Modeling. 2021, 213– 226
29	K R Jerripothula , J Cai , J Yuan . Image co-segmentation via saliency co-fusion. IEEE Transactions on Multimedia, 2016, 18( 9): 1896– 1909
30	R Quan , J Han , S Member , D Zhang , F Nie , X Qian , X Li . Unsupervised salient object detection via inferring from imperfect saliency models. IEEE Transactions on Multimedia, 2017, 20( 5): 1101– 1112
31	Y Zhao , S Liu , Z Hu , Y Bai , C Shen , X Shi . Separate degree based otsu and signed similarity driven level set for segmenting and counting anthrax spores. Computers and Electronics in Agriculture, 2020, 169 : 1– 15
32	X Qian , H Wang , Y Zhao , X Hou , R Hong , M Wang , Y Y Tang . Image location inference by multisaliency enhancement. IEEE Transactions on Multimedia, 2017, 19( 4): 813– 821
33	Y Fang , W Lin , S Member , B S Lee , C T Lau , Z Chen , C W Lin . Bottom-up saliency detection model based on human visual sensitivity and amplitude spectrum. IEEE Transactions on Multimedia, 2012, 14( 1): 187– 198
34	N Imamoglu , W Lin , Y Fang . A saliency detection model using low-level features based on wavelet transform. IEEE Transactions on Multimedia, 2012, 14( 1): 187– 198
35	C Yang, L Zhang, H Lu, X Ruan, M H Yang. Saliency detection via graph-based manifold ranking. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2013, 3166−3173
36	R Achanta, S Hemami, F Estrada, S Susstrunk. Frequency-tuned salient region detection. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2009, 1597−1604
37	B Cabral, L C Leedom. Imaging vector fields using line integral convolution. In: Proceedings of the 20th Annual Conference on Computer Graphics and Interactive Techniques. 1993, 263– 270
38	H Winnemller , B Gooch , S C Olsen . Real-time video abstraction. ACM Transactions on Graphics, 2006, 25( 3): 1221– 1226
39	Y Jin, P Li, W Wang, S Zhang, D Lin, C Yin. Ganbased pencil drawing learning system for art education on large-scale image datasets with learning analytics. Interactive Learning Environments, 2019
40	Z Chen, Y Jin, B Sheng, P Li, H Sun. Parallel pencil drawing stylization via structure-aware optimization. In: Proceedings of the 31st International Conference on Computer Animation and Social Agents. 2018, 32– 37
41	Y Jin , P Li , B Sheng , Y Nie , J Kim , E Wu . SRNPD: spatial rendering network for pencil drawing stylization. Computer Animation and Virtual Worlds, 2019, 30( 3−4): 1– 13
42	Z Song , S Liu . Sufficient image appearance transfer combining color and texture. IEEE Transactions on Multimedia, 2019, 19( 4): 702– 711
43	S Liu , Z Song , X Zhang , T Zhu . Progressive complex illumination image appearance transfer based on CNN. Journal of Visual Communication and Image Representation, 2019, 64 : 1– 11
44	T Zhu, S Liu. Detail-preserving arbitrary style transfer. In: Proceedings of IEEE International Conference on Multimedia and Expo. 2020, 1– 6
45	S Liu, T Zhu. Structure-guided arbitrary style transfer for artistic image and video. IEEE Transactions on Multimedia, 2021
46	P Li , H Sun , B Sheng , J Shen . Image stylization with enhanced structure on GPU. Science China Information Sciences, 2012, 55( 5): 1093– 1105
47	R Wang, P Li, B Sheng, H Sun, E Wu. Real-time video stylization using spatial-temporal gabor filtering. In: Proceedings of the 15th ACM SIGGRAPH Conference on Virtual-Reality Continuum and Its Applications in Industry. 2016, 299– 307

[1]

Download

[1]	Huaizu JIANG, Ming-Ming CHENG, Shi-Jie LI, Ali BORJI, Jingdong WANG. Joint salient object detection and existence prediction[J]. Front. Comput. Sci., 2019, 13(4): 778-788.
[2]	Xiaohui TAN, Yachun FAN, Ruiliang GUO. Local features and manifold ranking coupled method for sketch-based 3D model retrieval[J]. Front. Comput. Sci., 2018, 12(5): 1000-1012.
[3]	Yongjin LIU, Minjing YU, Qiufang FU, Wenfeng CHEN, Ye LIU, Lexing XIE. Cognitive mechanism related to line drawings and its applications in intelligent process of visual media: a survey[J]. Front. Comput. Sci., 2016, 10(2): 216-232.
[4]	Kezhi LI, Shuang CONG. State of the art and prospects of structured sensing matrices in compressed sensing[J]. Front. Comput. Sci., 2015, 9(5): 665-677.
[5]	Li CUI. SWVFS: a saliency weighted visual feature similarity metric for image quality assessment[J]. Front. Comput. Sci., 2014, 8(1): 145-155.
[6]	CUI Wei, XI Zairong, PAN Yu. Decoherence control for high-temperature reservoirs[J]. Front. Comput. Sci., 2008, 2(2): 129-137.

Viewed

Full text

Abstract

Cited

Shared

Discussed