Unpaired image to image transformation via informative coupled generative adversarial networks

doi:10.1007/s11704-020-9002-7

Front. Comput. Sci.

2021, Vol. 15

Issue (4) : 154326 https://doi.org/10.1007/s11704-020-9002-7

RESEARCH ARTICLE

Unpaired image to image transformation via informative coupled generative adversarial networks

Hongwei GE, Yuxuan HAN, Wenjing KANG, Liang SUN(

)

College of Computer Science and Technology, Dalian University of Technology, Dalian 116024, China

Download: PDF(887 KB)
Export: BibTeX | EndNote | Reference Manager | ProCite | RefWorks

Abstract

We consider image transformation problems, and the objective is to translate images from a source domain to a target one. The problem is challenging since it is difficult to preserve the key properties of the source images, and to make the details of target being as distinguishable as possible. To solve this problem, we propose an informative coupled generative adversarial networks (ICoGAN). For each domain, an adversarial generator-and-discriminator network is constructed. Basically, we make an approximately-shared latent space assumption by a mutual information mechanism, which enables the algorithm to learn representations of both domains in unsupervised setting, and to transform the key properties of images from source to target.Moreover, to further enhance the performance, a weightsharing constraint between two subnetworks, and different level perceptual losses extracted from the intermediate layers of the networks are combined. With quantitative and visual results presented on the tasks of edge to photo transformation, face attribute transfer, and image inpainting, we demonstrate the ICo- GAN’s effectiveness, as compared with other state-of-the-art algorithms.

Keywords generative adversarial networks image transformation mutual information perceptual loss

Corresponding Author(s): Liang SUN

Just Accepted Date: 28 February 2020 Issue Date: 08 May 2021

Cite this article:

Hongwei GE,Yuxuan HAN,Wenjing KANG, et al. Unpaired image to image transformation via informative coupled generative adversarial networks[J]. Front. Comput. Sci., 2021, 15(4): 154326.

URL:

https://academic.hep.com.cn/fcs/EN/10.1007/s11704-020-9002-7
https://academic.hep.com.cn/fcs/EN/Y2021/V15/I4/154326

1	A Buades, B Coll, J M Morel. A non-local algorithm for image denoising. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2005, 60–65
2	M Elad, M Aharon. Image denoising via sparse and redundant representations over learned dictionaries. IEEE Transactions on Image Processing, 2006, 15(12): 3736–3745 https://doi.org/10.1109/TIP.2006.881969
3	J Pan, W Ren, Z Hu, M H Yang. Learning to deblur images with exemplars. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019, 41(6): 1412–1425 https://doi.org/10.1109/TPAMI.2018.2832125
4	C Cruz, R Mehta, V Katkovnik, K O Egiazarian. Single image superresolution based on wiener filter in similarity domain. IEEE Transactions on Image Processing, 2018, 27(3): 1376–1389 https://doi.org/10.1109/TIP.2017.2779265
5	Y Huang, J Li, X Gao, L He, W Lu. Single image superresolution via multiple mixture prior models. IEEE Transactions on Image Processing, 2018, 27(12): 5904–5917 https://doi.org/10.1109/TIP.2018.2860685
6	D Pathak, P Krahenbuhl, J Donahue, T Darrell, A A Efros. Context encoders: feature learning by inpainting. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016, 2536–2544 https://doi.org/10.1109/CVPR.2016.278
7	D Ding, S Ram, J Rodriguez. Perceptually aware image inpainting. Pattern Recognition, 2018, 83: 174–184 https://doi.org/10.1016/j.patcog.2018.05.025
8	R Zhang, P Isola, A A Efros. Colorful image colorization. In: Proceedings of the European Conference on Computer Vision. 2016, 649–666 https://doi.org/10.1007/978-3-319-46487-9_40
9	C Wang, C Xu, C Wang, D Tao. Perceptual adversarial networks for image-to-image transformation. IEEE Transactions on Image Processing, 2018, 27(8): 4066–4079 https://doi.org/10.1109/TIP.2018.2836316
10	P Isola, J Y Zhu, T Zhou, A A Efros. Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017, 1125–1134 https://doi.org/10.1109/CVPR.2017.632
11	P Sangkloy, J Lu, C Fang, F Yu, J Hays. Scribbler: controlling deep image synthesis with sketch and color. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017, 5400–5409 https://doi.org/10.1109/CVPR.2017.723
12	, J Y Zhu, T Park, P Isola, A A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision. 2017, 2223–2232 https://doi.org/10.1109/ICCV.2017.244
13	M Y Liu, T Breuel, J Kautz. Unsupervised image-to-image translation networks. In: Proceedings of the 31st International Conference on Neural Information Processing Systems. 2017, 700–708
14	T Kim, M Cha, H Kim, J K Lee, J Kim. Learning to discover crossdomain relations with generative adversarial networks. In: Proceedings of the 34th International Conference on Machine Learning. 2017, 1857–1865
15	X Huang, M Y Liu, S Belongie, J Kautz. Multimodal unsupervised imageto-image translation. In: Proceedings of the European Conference on Computer Vision. 2018, 172–189
16	C Dong, C C Loy, K He, X Tang. Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 38(2): 295–307 https://doi.org/10.1109/TPAMI.2015.2439281
17	E Shelhamer, J Long, T Darrell. Fully convolutional networks for semantic segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2017, 39(4): 640–651 https://doi.org/10.1109/TPAMI.2016.2572683
18	A Radford, L Metz, S Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. 2015, arXiv preprint arXiv: 1511.06434
19	I Goodfellow, J Pouget-Abadie, M Mirza, B Xu, D Warde-Farley, S Ozair, Y Bengio. Generative adversarial nets. In: Proceedings of the 27th International Conference on Neural Information Processing Systems. 2014, 2672–2680
20	M Y Liu, O Tuzel. Coupled generative adversarial networks. In: Proceedings of the 30th International Conference on Neural Information Processing Systems. 2016, 469–477
21	W S Lai, J B Huang, N Ahuja, M H Yang. Fast and accurate image superresolution with deep laplacian pyramid networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 41(11): 2599–2613 https://doi.org/10.1109/TPAMI.2018.2865304
22	W Dong, P Wang, W Yin, G Shi. Denoising prior driven deep neural network for image restoration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019, 41(10): 2305–2318 https://doi.org/10.1109/TPAMI.2018.2873610
23	L Ma, Q Sun, S Georgoulis, L V Gool, B Schiele, M Fritz. Disentangled person image generation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018, 99–108 https://doi.org/10.1109/CVPR.2018.00018
24	Z Murez, S Kolouri, D Kriegman, R Ramamoorthi, K Kim. Image to image translation for domain adaptation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018, 4500–4509 https://doi.org/10.1109/CVPR.2018.00473
25	L Tran, X Yin, X Liu. Representation learning by rotating your faces. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019, 41(12): 3007–3021 https://doi.org/10.1109/TPAMI.2018.2868350
26	J Lin, Y Xia, T Qin, Z Chen, T Y Liu. Conditional image-to-image translation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018, 5524–5532 https://doi.org/10.1109/CVPR.2018.00579
27	R Li, J Pan, Z Li, J Tang. Single image dehazing via conditional generative adversarial network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018, 8202–8211 https://doi.org/10.1109/CVPR.2018.00856
28	T C Wang, M Y Liu, J Y Zhu, A Tao, J Kautz, B Catanzaro. Highresolution image synthesis and semantic manipulation with conditional gans. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018, 8798–8807 https://doi.org/10.1109/CVPR.2018.00917
29	K Regmi, A Borji. Cross-view image synthesis using conditional gans. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018, 3501–3510 https://doi.org/10.1109/CVPR.2018.00369
30	B Dolhansky, C C Ferrer. Eye in-painting with exemplar generative adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018, 7902–7911 https://doi.org/10.1109/CVPR.2018.00824
31	X Huang, M Y Liu, S Belongie, J Kautz. Multimodal unsupervised imageto- image translation. In: Proceedings of the European Conference on Computer Vision. 2018, 172–189
32	H Y Lee, H Y Tseng, J B Huang, M Singh, M H Yang. Diverse image-toimage translation via disentangled representations. In: Proceedings of the European Conference on Computer Vision. 2018, 35–51 https://doi.org/10.1007/978-3-030-01246-5_3
33	L Ma, X Jia, S Georgoulis, T Tuytelaars, L Van Gool. Exemplar guided unsupervised image-to-image translation. 2018, arXiv preprint arXiv:1805.11145
34	X Chen, Y Duan, R Houthooft, J Schulman, I Sutskever, P Abbeel. Infogan: interpretable representation learning by information maximizing generative adversarial nets. In: Proceedings of the 30th International Conference on Neural Information Processing Systems. 2016, 2172–2180
35	J Bruna, P Sprechmann, Y LeCun. Super-resolution with deep convolutional sufficient statistics. 2015, arXiv preprint arXiv:1511.05666
36	J Johnson, A Alahi, F F Li. Perceptual losses for real-time style transfer and super-resolution. In: Proceedings of the European Conference on Computer Vision. 2016, 694–711 https://doi.org/10.1007/978-3-319-46475-6_43
37	L Gatys, A S Ecker, M Bethge. Texture synthesis using convolutional neural networks. In: Proceedings of the 28th International Conference on Neural Information Processing Systems. 2015, 262–270
38	J Donahue, P Krähenbühl, T Darrell. Adversarial feature learning. 2016, arXiv preprint arXiv:1605.09782
39	Z Wang, A C Bovik, H R Sheikh, E P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 2004, 13(4): 600–612 https://doi.org/10.1109/TIP.2003.819861
40	A Yu, K Grauman. Fine-grained visual comparisons with local learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2014, 192–199 https://doi.org/10.1109/CVPR.2014.32
41	J Y Zhu, P Krähenbühl, E Shechtman, A A Efros. Generative visual manipulation on the natural image manifold. In: Proceedings of the European Conference on Computer Vision. 2016, 597–613 https://doi.org/10.1007/978-3-319-46454-1_36
42	S Xie, Z Tu. Holistically-nested edge detection. In: Proceedings of the IEEE Conference on Computer Vision. 2015, 1395–1403 https://doi.org/10.1109/ICCV.2015.164
43	R Zhang, P Isola, A A Efros, E Shechtman, O Wang. The unreasonable effectiveness of deep features as a perceptual metric. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018, 586–595 https://doi.org/10.1109/CVPR.2018.00068
44	Z Liu, P Luo, X Wang, X Tang. Deep learning face attributes in the wild. In: Proceedings of the IEEE International Conference on Computer Vision. 2015, 3730–3738 https://doi.org/10.1109/ICCV.2015.425

[1]

Article highlights

Download

[1]	Kaimin WEI, Tianqi LI, Feiran HUANG, Jinpeng CHEN, Zefan HE. Cancer classification with data augmentation based on generative adversarial networks[J]. Front. Comput. Sci., 2022, 16(2): 162601-.
[2]	Farid FEYZI, Saeed PARSA. Inforence: effective fault localization based on information-theoretic analysis and statistical causal inference[J]. Front. Comput. Sci., 2019, 13(4): 735-759.

Viewed

Full text

Abstract

Cited

Shared

Discussed