Nighttime image dehazing using color cast removal and dual path multi-scale fusion strategy

doi:10.1007/s11704-021-0162-x

Front. Comput. Sci.

2022, Vol. 16

Issue (4) : 164706 https://doi.org/10.1007/s11704-021-0162-x

RESEARCH ARTICLE

Nighttime image dehazing using color cast removal and dual path multi-scale fusion strategy

Bo WANG¹(

), Li HU¹, Bowen WEI¹, Zitong KANG¹, Chongyi LI²

¹. School of Physics and Electronic-Electrical Engineering, Ningxia University, Yinchuan 750021, China
². School of Computer Science and Engineering, Nanyang Technological University, Singapore 639798, Singapore

Download: PDF(11009 KB) HTML
Export: BibTeX | EndNote | Reference Manager | ProCite | RefWorks

Abstract

Nighttime image dehazing aims to remove the effect of haze on the images captured in nighttime, which however, raises new challenges such as severe color distortion, more complex lighting conditions, and lower contrast. Instead of estimating the transmission map and atmospheric light that are difficult to be accurately acquired in nighttime, we propose a nighttime image dehazing method composed of a color cast removal and a dual path multi-scale fusion algorithm. We first propose a human visual system (HVS) inspired color correction model, which is effective for removing the color deviation on nighttime hazy images. Then, we propose to use dual path strategy that includes an underexposure and a contrast enhancement path for multi-scale fusion, where the weight maps are achieved by selecting appropriate exposed areas under Gaussian pyramids. Extensive experiments demonstrate that the visual effect of the hazy nighttime images in real-world datasets can be significantly improved by our method regarding contrast, color fidelity, and visibility. In addition, our method outperforms the state-of-the-art methods qualitatively and quantitatively.

Keywords nighttime image dehazing color cast removal dual path multi-scale fusion

Corresponding Author(s): Bo WANG

Just Accepted Date: 11 March 2021 Issue Date: 18 November 2021

Cite this article:

Bo WANG,Li HU,Bowen WEI, et al. Nighttime image dehazing using color cast removal and dual path multi-scale fusion strategy[J]. Front. Comput. Sci., 2022, 16(4): 164706.

URL:

https://academic.hep.com.cn/fcs/EN/10.1007/s11704-021-0162-x
https://academic.hep.com.cn/fcs/EN/Y2022/V16/I4/164706

Fig.1 Flow chart of the proposed framework

Fig.2 The RF structure of different bipolar cells

Fig.3 Overview of our proposed nighttime image dehazing scheme

Fig.4 Results of gamma correction (left:

y < 1

and right:

y > 1

)

Fig.5 Comparison of the result using overexposure (A.

γ = 0.9

) and CLAHE (B.

ClipLimit = 0.01

)

Fig.6

Fig.7 Comparison of the proposed method with widely-used color balance methods. (a) Original image, (b) GE, (c) GW, (d) MRGB, (e) SoG, (f) WGE, (g) DOCC, and (h) our method. (Scene 1: complex artificial light sources)

Fig.8 Comparison of the proposed method with widely-used color balance methods. (a) original image, (b) GE, (c) GW, (d) MRGB, (e) SoG, (f) WGE, (g) DOCC, and (h) our method. (Scene 2: street lamp)

Fig.9 Comparisons on close-up views of the regions enclosed with red rectangle. From the second to the fourth column: original regions, results of DOCC, results of our proposed method

Tab.1 Qualitative comparison in terms of the average values of CCF, Entropy and VCG

Fig.10 Results by various weights combination and close-up views of the regions. From left to right: original hazy images, Results I (

ω G / ω G ω C ω C = 1 / 19 9

), Results II (

ω G / ω G ω C ω C = 3 / 37 7

), Results III (

ω G / ω G ω C ω C = 5 / 55 5

), Results IV (

ω G / ω G ω C ω C = 7 / 73 3

) and Results V (

ω G / ω G ω C ω C = 9 / 91 1

)

Fig.11 Comparisons of proposed method with conventional methods when it comes to severe color cast. The fifth row provides the close-up views of the fourth row, where the region is enclosed with red rectangle. From left to right are original images, the results of Zhang 2014 [25], Li 2015 [26], Zhang 2017, Zhang 2017 (fast) [27], Yang 2018 [28], Liao 2018 [31], Yu 2019 [44] and our method

Fig.12 Comparisons of proposed method with conventional methods when it comes to not severe color cast. The fifth row provides the close-up views of the fourth row, where the region is enclosed with red rectangle. From left to right are original images, the results of Zhang 2014 [25], Li 2015 [26], Zhang 2017, Zhang 2017 (fast) [27], Yang 2018 [28], Liao 2018 [31], Yu 2019 [44] and our method

Fig.13 Comparisons of proposed method with conventional methods when it comes to colorful artificial lights. The fifth row provides the close-up views of the fourth row, where the region is enclosed with red rectangle. From left to right are original images, the results of Zhang 2014 [25], Li 2015 [26], Zhang 2017, Zhang 2017 (fast) [27], Yang 2018 [28], Liao 2018 [31], Yu 2019 [44] and our method

Fig.14 Comparison to different methods with ground truth. The ground truth (a1)?(d1), Zhang 2014 (a3)?(d3), Li 2015 (a4)?(d4), Zhang 2017 (a5)?(d5), Zhang 2017 (fast) (a6)?(d6), Yang 2018 (a7)?(d7), Liao 2018 (a8)?(d8), Yu 2019 (a9)?(d9) and our method (a10)?(d10)

Fig.15 The qualitative comparisons of close-up views of the regions. The original region of ground truth (a2)?(d2), Zhang 2014 (a3)?(d3), Li 2015 (a4)?(d4), Zhang 2017 (a5)?(d5), Yang 2018 (a6)?(d6), Liao 2018 (a7)?(d7), Yu 2019 (a8)?(d8) and our method (a9)?(d9)

Tab.2 Quantitative evaluation of results

Fig.16 Runtime evaluation of our proposed method compared with seven nighttime dehazing methods

Fig.17 Failure cases for nighttime image dehazing

Tab.3 Quantitative comparison in terms of Entropy, VCM and IVM

Fig.18 Comparative results with and without the proposed dual path multi-scale fusion

1	Y Gao , Y Su , Q M Li , H Y Li , J Li . Single image dehazing via self-constructing image fusion. Signal Processing, 2020, 167 : 107284–
2	Li Y, You S D, Brown M S, Tan R T. Haze visibility enhancement: a survey and quantitative benchmarking. Computer Vision and Image Understanding, 2017, 165: 1−16
3	C G Dai , M X Lin , X J Wu , D Zhang . Single hazy image restoration using robust atmospheric scattering model. Signal Processing, 2020, 166 : 107257–
4	Y T Lin , Y Wu , C G Yan , M L Xu , Y Yang . Unsupervised person re-identification via cross-camera similarity exploration. IEEE Transactions on Image Processing, 2020, 29 : 5481– 5490
5	Q Liu , X B Gao , L H He , W Lu . Haze removal for a single visible remote sensing image. Signal Processing, 2017, 137 : 33– 43
6	D Zhao , L Xu , Y H Yan , J Chen , L Y Duan . Multi-scale optimal fusion model for single image dehazing. Signal Processing: Image Communication, 2019, 74 : 253– 265
7	C Y Li , J C Guo , R M Cong , Y W Pang , B Wang . Underwater image enhancement by dehazing with minimum information loss and histogram distribution prior. IEEE Transactions on Image Processing, 2016, 25( 12): 5664– 5677
8	C Y Li , C L Guo , J C Guo , P Han , H Z Fu , R M Cong . PDR-Net: perception-inspired single image dehazing network with refinement. IEEE Transactions on Multimedia, 2020, 22( 3): 704– 716
9	F Yuan , H Huang . Image haze removal via reference retrieval and scene prior. IEEE Transactions on Image Processing, 2018, 27( 9): 4395– 4409
10	P J Liu , S J Horng , J S Lin , T R Li . Contrast in haze removal: configurable contrast enhancement model based on dark channel prior. IEEE Transactions on Image Processing, 2019, 28( 5): 2212– 2227
11	K M He , J Sun , X Tang . Single image haze removal using dark channel prior. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2011, 33( 12): 2341– 2353
12	Q S Zhu , J M Mai , L Shao . A fast single image haze removal algorithm using color attenuation prior. IEEE Transactions on Image Processing, 2015, 24( 11): 3522– 3533
13	Berman D, Treibitz T, Avidan S. Non-Local image dehazing. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2016, 1674−1682
14	S Santra, B Chanda. Day/night unconstrained image dehazing. In: Proceedings of the 23rd International Conference on Pattern Recognition. 2016, 1406−1411
15	G Kim, J Kwon. Robust pixel-wise dehazing algorithm based on advanced haze-relevant features. In: Proceedings of British Machine Vision Conference. 2017, 1−12
16	A P Yang , J Liu , Z Ji , Y W Pan . Detail-preserving single nighttime image dehazing. Journal of Electronic Imaging, 2020, 29( 4): 043010–
17	G D Finlayson, E Trezzi. Shades of gray and colour constancy. In: Proceedings of the 12th Color Imaging Conference: Color Science and Engineering Systems, Technologies, Applications. 2004, 37−41
18	J Van De Weijer , T Gevers , A Gijsenij . Edge-based color constancy. IEEE Transactions on Image Processing, 2007, 16( 9): 2207– 2214
19	S Gao , K Yang , C Li . Color constancy using double-opponency. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 37( 10): 1973– 1985
20	C O Ancuti , C Ancuti , C D Vleeschouwer , P Bekaert . Color balance and fusion for underwater image enhancement. IEEE Transactions on Image Processing, 2018, 27( 1): 379– 393
21	A Galdran . Image dehazing by artificial multiple-exposure image fusion. Signal Processing, 2018, 149 : 135– 147
22	X S Zhang , S B Gao , C Y Li , Y J Li . A retina inspired model for enhancing visibility of hazy images. Frontiers in Computational Neuroscience, 2015, 9 : 1– 13
23	X Zhang , S Gao , R Li , X Du , C Li , Y Li . A retinal mechanism inspired color constancy model. IEEE Transactions on Image Processing, 2016, 25( 3): 1219– 1232
24	Pei S C, Lee T Y. Nighttime haze removal using color transfer pre-processing and dark channel prior. In: Proceedings of the 19th IEEE International Conference on Image Processing. 2012, 957−960
25	J Zhang, Y Cao, Z Wang. Nighttime haze removal based on a new imaging model. In: Proceedings of IEEE International Conference on Image Processing. 2014, 4557−4561
26	Y Li, R T Tan, M S Brown. Nighttime haze removal with glow and multiple light colors. In: Proceedings of IEEE International Conference on Computer Vision. 2015, 226−234
27	J Zhang, Y Cao, S Fang, Y Kang, C W Chen. Fast haze removal for nighttime image using maximum reflectance prior. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2017, 7016−7024
28	M M Yang , J C Liu , Z G Li . Superpixel-based single nighttime image haze removal. IEEE Transactions on Multimedia, 2018, 20( 11): 3008– 3018
29	C Ancuti, C O Ancuti, C D Vleeschouwer, A C Bovik. Night-time dehazing by fusion. In: Proceedings of IEEE International Conference on Image Processing. 2016, 2256−2260
30	C Ancuti , C O Ancuti , C D Vleeschouwer , A C Bovik . Day and night-time dehazing by local airlight estimation. IEEE Transactions on Image Processing, 2020, 29 : 6264– 6275
31	Y Liao, Z Su, X Liang, B Qiu. HDP-Net: haze density prediction network for nighttime dehazing. In: Hong R, Cheng W H, Yamasaki T, Wang M, Ngo C W, eds. Advances in Multimedia Information Processing. Springer, Cham, 2018, 469−480
32	S Kuanar, K R Rao, D Mahapatra, M Bilas. Night time haze and glow removal using deep dilated convolutional network. 2019, arXiv preprint arXiv: 1902.00855
33	C Ledig, L Theis, F Huszar, J Caballero, A Cunningham, A Acosta, A Aitken, A Tejani, J Totz, Z H Wang, W Z Shi. Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2017, 4681−4690
34	D Engin, A Genc, H K Ekenel. Cycle-Dehaze: enhanced CycleGAN for single image dehazing. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 2018, 938―946
35	K H Zhang , W H Luo , Y R Zhong , L Ma , W Liu , H D Li . Adversarial spatio-temporal learning for video deblurring. IEEE Transactions on Image Processing, 2019, 28( 1): 291– 301
36	K H Zhang, W H Luo, Y R Zhong, L Ma, B Stenger, W Liu, H D Li. Deblurring by realistic blurring. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2020, 2737−2746
37	E Reinhard , M Adhikhmin , B Gooch , P Shirley . Color transfer between images. IEEE Computer Graphics and Applications, 2001, 21( 5): 34– 41
38	B B Lee , P R Martin , U Grünert . Retinal connectivity and primate vision. Progress in Retinal and Eye Research, 2010, 29 : 622– 639
39	S B Gao, K F Yang, C Y Li, Y J Li. A color constancy model with double-opponency mechanisms. In: Proceedings of IEEE International Conference on Computer Vision. 2013, 929−936
40	Y N Li , Q G Miao , R Y Liu , J F Song , Y N Quan , Y H Huang . A multi-scale fusion scheme based on haze-relevant features for single image dehazing. Neurocomputing, 2018, 283 : 73– 86
41	Zuiderveld K. Contrast Limited Adaptive Histogram Equalization. Academic Press Professional, Inc., 1994, 474−485
42	A Gijsenij , T Gevers , J Van De Weijer . Computational color constancy: survey and experiments. IEEE Transactions on Image Processing, 2011, 20( 9): 2475– 2489
43	A Gijsenij , T Gevers , J Van De Weijer . Improving color constancy by photometric edge weighting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2012, 34( 5): 918– 929
44	T Yu , K Song , P Miao , G Yang , H Yang , C Chen . Nighttime single image dehazing via pixel-wise alpha blending. IEEE Access, 2019, 7 : 114619– 114630
45	Y Xu , J Wen , L Fei , Z Zhang . Review of video and image defogging algorithms and related studies on image restoration and enhancement. IEEE Access, 2016, 4 : 165–

[1]

Download

Viewed

Full text

Abstract

Cited

Shared

Discussed