Intellectual property protection for deep semantic segmentation models

doi:10.1007/s11704-021-1186-y

Front. Comput. Sci.

2023, Vol. 17

Issue (1) : 171306 https://doi.org/10.1007/s11704-021-1186-y

RESEARCH ARTICLE

Intellectual property protection for deep semantic segmentation models

Hongjia RUAN¹, Huihui SONG¹(

), Bo LIU², Yong CHENG¹, Qingshan LIU¹

¹. B-DAT, CICAEET, Nanjing University of Information Science & Technology, Nanjing 211800, China
². JD Finance America Corporation, Mountain View 94089, USA

Download: PDF(13894 KB) HTML
Export: BibTeX | EndNote | Reference Manager | ProCite | RefWorks

Abstract

Deep neural networks have achieved great success in varieties of artificial intelligent fields. Since training a good deep model is often challenging and costly, such deep models are of great value and even the key commercial intellectual properties. Recently, deep model intellectual property protection has drawn great attention from both academia and industry, and numerous works have been proposed. However, most of them focus on the classification task. In this paper, we present the first attempt at protecting deep semantic segmentation models from potential infringements. In details, we design a new hybrid intellectual property protection framework by combining the trigger-set based and passport based watermarking simultaneously. Within it, the trigger-set based watermarking mechanism aims to force the network output copyright watermarks for a pre-defined trigger image set, which enables black-box remote ownership verification. And the passport based watermarking mechanism is to eliminate the ambiguity attack risk of trigger-set based watermarking by adding an extra passport layer into the target model. Through extensive experiments, the proposed framework not only demonstrates its effectiveness upon existing segmentation models, but also shows strong robustness to different attack techniques.

Keywords deep neural networks intellectual property protection trigger-set passport layer

Corresponding Author(s): Huihui SONG

Just Accepted Date: 26 July 2021 Issue Date: 01 March 2022

Cite this article:

Hongjia RUAN,Huihui SONG,Bo LIU, et al. Intellectual property protection for deep semantic segmentation models[J]. Front. Comput. Sci., 2023, 17(1): 171306.

URL:

https://academic.hep.com.cn/fcs/EN/10.1007/s11704-021-1186-y
https://academic.hep.com.cn/fcs/EN/Y2023/V17/I1/171306

Fig.1 The illustration figure to show the trigger set based working principle, which forces the target model to output normal segmentation mask for clean images and predefined watermark pattern for the adversarial sample based trigger images. Here the adversarial perturbation is amplified for the illustration purpose

Fig.2 The overall pipeline of the proposed IP protection framework for one target segmentation model, which consists of two components. The black-box trigger set based watermarking mechanism aims to predict normal masks for clean images and pre-defined watermark patterns for trigger images generated by adversarial samples. And the passport based watermarking mechanism adds an extra passport layer after each convolution layer of the target model and adopts a “fidelity verification” based watermarking mechanism

Tab.1 The performance of different variants of our framework. “OM” denotes the original model without involving any IP protection module, “CleanTrig” means using clean images split as the trigger set, “AdvTrig” means using the proposed adversarial samples as the trigger set, “Pass” means involving the passport based watermarking mechanism. “OM+AdvTrig+Pass” is our default setting, “Ori set” means the original test set

Fig.3 The performance changes by involving different portions of trigger set during training. “sig” and “mul” are the abbreviations of “single-pattern &single-attack-label” and “multi-pattern & multi-attack-label”, while “ori_set” means the original test set

Tab.2 The performance of the watermark model (only use trigger set) by using four different model fine-tuning attack strategies. It shows that our framework can keep both high trigger-set verification accuracy

Tab.3 The performance of our framework (hybrid watermark) by using four different model fine-tuning attack strategies. It shows that our framework can keep both high trigger-set verification accuracy and passport signature accuracy

Fig.4 The performance of the watermark model (only use trigger set) towards the fine-tuning attack that tries to embed a new trigger set. (a) Single-pattern & single-attack-label; (b) multi-pattern & multi-attack-label

Fig.5 The performance of our framework (hybrid watermark) towards the fine-tuning attack that tries to embed a new trigger set. (a) Single-pattern & single-attack-label; (b) multi-pattern & multi-attack-label

Tab.4 The performance of the watermark model embeded a new trigger set by using “RTLL”. It shows that watermark model can keep both high trigger-set verification accuracy and passport signature accuracy. In-brackets is the aprecision after embedding the new watermark, and out-bracket is the precision after “RTLL”

Tab.5 The performance of the watermark model embeded a new trigger set by using “RTAL”. It shows that watermark model can keep both high trigger-set verification accuracy and passport signature accuracy. In-brackets is the aprecision after embedding the new watermark, and out-bracket is the precision after “RTAL”

Fig.6 The performance of trigger set based by applying different model pruning rates. (a) Single-pattern & single-attack-label; (b) multi-pattern & multi-attack-label

Fig.7 The performance of trigger set based and passport signature based verification by applying different model pruning rates. (a) Single-pattern & single-attack-label; (b) multi-pattern & multi-attack-label

Tab.6 The clean test set performance of our framework towards random passport attack under different percentages of flipped signs, where “SP&SA” and “MP&MA” are the abbreviations of “single-pattern & single-attack-label” and “multi-pattern & multi-attack-label” version respectively

Tab.7 The clean test set performance of our framework towards reverse-engineering passport attack under different percentages of flipped signs, where “SP&SA” and “MP&MA” are the abbreviations of “single-pattern & single-attack-label” and “multi-pattern & multi-attack-label” version respectively

1	K He, X Zhang, S Ren, J Sun. Deep residual learning for image recognition. In: Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. 2016, 770–778
2	D Bahdanau, K Cho, Y Bengio. Neural machine translation by jointly learning to align and translate. In: Proceedings of the 3rd International Conference on Learning Representations. 2014
3	A B Nassif , I Shahin , I Attili , M Azzeh , K Shaalan . Speech recognition using deep neural networks: A systematic review. IEEE Access, 2019, 7 : 19143– 19165
4	Y Adi, C Baum, M Cisse, B Pinkas, J Keshet. Turning your weakness into a strength: watermarking deep neural networks by backdooring. In: Proceedings of the 27th USENIX Conference on Security Symposium. 2018, 1615−1631
5	L C Chen, Y Zhu, G Papandreou, F Schroff, H Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In: Proceedings of the 15th European Conference on Computer Vision. 2018, 833–851
6	B D Rouhani , H Chen , F Koushanfar . Deepsigns: a generic watermarking framework for IP protection of deep learning models. IACR Cryptology ePrint Archive, 2018, 2018 : 311–
7	Y Uchida, Y Nagai, S Sakazawa, S Satoh. Embedding watermarks into deep neural networks. In: Proceedings of 2017 ACM on International Conference on Multimedia Retrieval. 2017, 269–277
8	J Zhang, D Chen, J Liao, W Zhang, H Feng, G Hua, N Yu. Deep model intellectual property protection via deep watermarking. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, DOI:
9	J Zhang, Z Gu, J Jang, H Wu, M P Stoecklin, H Huang, I Molloy. Protecting intellectual property of deep neural networks with watermarking. In: Proceedings of the 2018 on Asia Conference on Computer and Communications Security. 2018, 159–159
10	J Zhang, D Chen, J Liao, W Zhang, G Hua, N Yu. Passport-aware normalization for deep model protection. In: Proceedings of the 34th Conference on Neural Information Processing Systems. 2020, 22619–22628
11	L Fan, K W Ng, C S Chan. Rethinking deep neural network ownership verification: embedding passports to defeat ambiguity attacks. In: Proceedings of the 33rd Conference on Neural Information Processing Systems. 2019, 4716−4716
12	L C Chen, G Papandreou, F Schroff, H Adam. Rethinking atrous convolution for semantic image segmentation. 2017, arXiv preprint arXiv: 1706.05587
13	H Chen , B D Rohani , F Koushanfar . Deepmarks: a digital fingerprinting framework for deep neural networks. IACR Cryptology ePrint Archive, 2018, 2018 : 322–
14	J Zhang, D Chen, J Liao, H Fang, W Zhang, W Zhou, H Cui, N Yu. Model watermarking for image processing networks. In: Proceedings of the 34th AAAI Conference on Artificial Intelligence. 2020, 12805–12812
15	J H Lim, C S Chan, K W Ng, L X Fan, Q Yang. Protect, show, attend and tell: empowering image captioning models with ownership protection. 2020, arXiv preprint arXiv: 2008.11009
16	C Szegedy, W Zaremba, I Sutskever, J Bruna, D Erhan, I J Goodfellow, R Fergus. Intriguing properties of neural networks. In: Proceedings of the 2nd International Conference on Learning Representations. 2014
17	N Carlini, D Wagner. Towards evaluating the robustness of neural networks. In: Proceedings of 2017 IEEE Symposium on Security and Privacy. 2017, 39–57
18	X Dong, D Chen, J Bao, C Qin, L Yuan, W Zhang, N H Yu, D Chen. Greedyfool: distortion-aware sparse adversarial attack. In: Proceedings of the 34th Conference on Neural Information Processing Systems. 2020
19	X Dong, J Han, D Chen, J Liu, H Bian, Z Ma, H Li, X Wang, W Zhang, N Yu. Robust superpixel-guided attentional adversarial attack. In: Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020, 12892–12901
20	A Kurakin, I J Goodfellow, S Bengio. Adversarial examples in the physical world. In: Proceedings of the 5th International Conference on Learning Representations. 2017
21	O Poursaeed, I Katsman, B Gao, S Belongie. Generative adversarial perturbations. In: Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2018, 4422−4431
22	J Han, X Dong, R Zhang, D Chen, W Zhang, N Yu, P Luo, X Wang. Once a man: towards multi-target attack via learning multi-target adversarial network once. In: Proceedings of 2019 IEEE/CVF International Conference on Computer Vision. 2019, 5157−5166
23	H Zhou, D Chen, J Liao, K Chen, X Dong, K Liu, W Zhang, G Hua, N Yu. LG-GAN: label guided adversarial network for flexible targeted attack of point cloud based deep networks. In: Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020, 10353−10362
24	C Xie, J Wang, Z Zhang, Y Zhou, L Xie, A Yuille. Adversarial examples for semantic segmentation and object detection. In: Proceedings of 2017 IEEE International Conference on Computer Vision. 2017, 1378−1387
25	A S Razavian, H Azizpour, J Sullivan, S Carlsson. CNN features off-the-shelf: an astounding baseline for recognition. In: Proceedings of 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops. 2014, 512−519
26	K Simonyan, A Zisserman. Very deep convolutional networks for large-scale image recognition. In: Proceedings of the 3rd International Conference on Learning Representations. 2015
27	J Yosinski, J Clune, Y Bengio, H Lipson. How transferable are features in deep neural networks? In: Proceedings of the 27th International Conference on Neural Information Processing Systems. 2014, 3320−3328
28	A See, M T Luong, C D Manning. Compression of neural machine translation models via pruning. In: Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning. 2016, 291−301

[1]

Download

[1]	Jian-Hao LUO,Wang ZHOU,Jianxin WU. Image categorization with resource constraints: introduction, challenges and advances[J]. Front. Comput. Sci., 2017, 11(1): 13-26.

Viewed

Full text

Abstract

Cited

Shared

Discussed