Several biometric privacy-enhancing techniques have been appraised to protect face image privacy. However, a face privacy protection algorithm is usually designed for a specific face recognition algorithm. When the structure or threshold of the face recognition algorithm is fine-tuned, the protection algorithm may be invalid. It will cause the network bloated and make the image distortion target multiple FRAs through the existing technology simultaneously. To address this problem, a fusion technology is developed to cope with the changeable face recognition algorithms via an image perturbation method. The image perturbation is performed by using a GAN-improved algorithm including generator, nozzles and validator, referred to as the Adversarial Fusion algorithm. A nozzle structure is proposed to replace the discriminator. Paralleling multiple face recognition algorithms on the nozzle can improve the compatibility of the generated image. Next, a validator is added to the training network, which takes part in the inverse back coupling of the generator. This component can make the generated graphics have no impact on human vision. Furthermore, the group hunting theory is quoted to make the network stable and up to 4.8 times faster than other models in training. The experimental results show that the Adversarial Fusion algorithm can not only change the image feature distribution by over 42% but also deal with at least 5 commercial face recognition algorithms at the same time.
Citation: Hao Wang, Guangmin Sun, Kun Zheng, Hui Li, Jie Liu, Yu Bai. Privacy protection generalization with adversarial fusion[J]. Mathematical Biosciences and Engineering, 2022, 19(7): 7314-7336. doi: 10.3934/mbe.2022345
Several biometric privacy-enhancing techniques have been appraised to protect face image privacy. However, a face privacy protection algorithm is usually designed for a specific face recognition algorithm. When the structure or threshold of the face recognition algorithm is fine-tuned, the protection algorithm may be invalid. It will cause the network bloated and make the image distortion target multiple FRAs through the existing technology simultaneously. To address this problem, a fusion technology is developed to cope with the changeable face recognition algorithms via an image perturbation method. The image perturbation is performed by using a GAN-improved algorithm including generator, nozzles and validator, referred to as the Adversarial Fusion algorithm. A nozzle structure is proposed to replace the discriminator. Paralleling multiple face recognition algorithms on the nozzle can improve the compatibility of the generated image. Next, a validator is added to the training network, which takes part in the inverse back coupling of the generator. This component can make the generated graphics have no impact on human vision. Furthermore, the group hunting theory is quoted to make the network stable and up to 4.8 times faster than other models in training. The experimental results show that the Adversarial Fusion algorithm can not only change the image feature distribution by over 42% but also deal with at least 5 commercial face recognition algorithms at the same time.
| [1] |
R. M. Mizanur, M. A. Hossain, H. Mouftah, A. EI Saddik, E. Okamoto, Chaos-cryptography based privacy preservation technique for video surveillance, Multimedia Syst., 18 (2012), 145-155. https://doi.org/10.1007/s00530-011-0246-9 doi: 10.1007/s00530-011-0246-9
|
| [2] |
G. Sun, H. Wang, Image encryption and decryption technology based on rubik's cube and dynamic password, J. Beijing Univ. Technol., 47 (2021), 833-841. https://doi.org/10.11936/bjutxb2020120003 doi: 10.11936/bjutxb2020120003
|
| [3] | S. Shan, E. Wenger, J. Zhang, H. Li, H. Zheng, B. Y. Zhao, Fawkes: protecting privacy against unauthorized deep learning models, in 29th USENIX Security Symposium (USENIX Security 20), (2020), 1589-1604. Available from: https://www.usenix.org/conference/usenixsecurity20/presentation/shan. |
| [4] |
J. Yang, J. Liu, R. Han, J. Wu, Transferable face image privacy protection based on federated learning and ensemble models, Complex Intell. Syst., 7 (2021), 2299-2315. https://doi.org/10.1007/s40747-021-00399-6 doi: 10.1007/s40747-021-00399-6
|
| [5] | 2021 White Paper of Innovation in Face recognition industry. Available from: https://www.vzkoo.com/read/11206bd95038173b5831540e5982e1b2.html. |
| [6] |
R. A. Fisher, The use of multiple measurements in taxonomic problems, Ann. Eugen., 7 (1936), 179-188. https://doi.org/10.1111/j.1469-1809.1936.tb02137.x doi: 10.1111/j.1469-1809.1936.tb02137.x
|
| [7] |
G. Cheng, Z. Song, Robust face recognition based on sparse representation in 2D fisherface space, Optik, 125 (2014), 2804-2808. https://doi.org/10.1016/j.ijleo.2013.11.042 doi: 10.1016/j.ijleo.2013.11.042
|
| [8] |
L. Sirovich, M. Kirby, Low-dimensional procedure for the characterization of human faces, J. Opt. Soc. Am. A, 4 (1987), 519-524. https://doi.org/10.1364/JOSAA.4.000519 doi: 10.1364/JOSAA.4.000519
|
| [9] |
M. Turk, A. Pentland, Eigenfaces for recognition, J. Cognit. Neurosci., 3 (1991), 71-86. https://doi.org/10.1162/jocn.1991.3.1.71 doi: 10.1162/jocn.1991.3.1.71
|
| [10] |
T. Ojala, M. Pietikainen, D. Harwood, A comparative study of texture measures with classification based on fea-tured distributions, Pattern Recognit., 29 (1996), 51-59. https://doi.org/10.1016/0031-3203(95)00067-4 doi: 10.1016/0031-3203(95)00067-4
|
| [11] |
Q. Zhang, H. Li, M. Li, L. Ding, Feature extraction of face image based on LBP and 2-D Gabor wavelet transform, Math. Biosci. Eng., 17 (2020), 1578-1592. https://doi.org/10.3934/mbe.2020082 doi: 10.3934/mbe.2020082
|
| [12] |
Z. Peng, L. Tao, G. Xu, H. Zhang, Detecting facial features based on color segmentation and KL transform, J. Tsinghua Univ. (Sci. Technol.), 41 (2001), 218-221. https://doi.org/10.16511/j.cnki.qhdxxb.2001.z1.052 doi: 10.16511/j.cnki.qhdxxb.2001.z1.052
|
| [13] | Y. Taigman, M. Yang, M. A. Ranzato, L. Wolf, Deepface: closing the gap to human-level performance in face verification, in 2014 IEEE Conference on Computer Vision and Pattern Recognition, (2014), 1701-1708. https://doi.org/10.1109/CVPR.2014.220 |
| [14] | Y. Sun, X. Wang, X. Tang, Deep learning face representation from predicting 10,000 classes, in 2014 IEEE Conference on Computer Vision and Pattern Recognition, (2014), 1891-1898. https://doi.org/10.1109/CVPR.2014.244 |
| [15] | Y. Sun, Y. Chen, X. Wang, X. Tang, Deep learning face representation by joint identification-verification, in Advances in Neural Information Processing Systems, 27 (2014). Available from: https://proceedings.neurips.cc/paper/2014/file/e5e63da79fcd2bebbd7cb8bf1c1d0274-Paper.pdf. |
| [16] | Y. Sun, X. Wang, X. Tang, Deeply learned face representations are sparse, selective, and robust, in 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), (2015), 2892-2900. https://doi.org/10.1109/CVPR.2015.7298907 |
| [17] | F. Schroff, D. Kalenichenko, J. Philbin, FaceNet: a unified embedding for face recognition and clustering, in 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), (2015), 815-823. https://doi.org/10.1109/CVPR.2015.7298682 |
| [18] | J. Liu, Y. Deng, T. Bai, Z. Wei, C. Huang, Targeting ultimate accuracy: face recognition via deep embedding, preprint, arXiv: 1506.07310. |
| [19] |
H. Fan, E. Zhou, Approaching human level facial landmark localization by deep learning, Image Vision Comput., 47 (2016), 27-35. https://doi.org/10.1016/j.imavis.2015.11.004 doi: 10.1016/j.imavis.2015.11.004
|
| [20] | W. Liu, Y. Wen, Z. Yu, M. Li, B. Raj, L. Song, Sphereface: deep hypersphere embedding for face recognition, in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), (2017), 6738-6746. https://doi.org/10.1109/CVPR.2017.713 |
| [21] | H. Wang, Y. Wang, Z. Zhou, X. Ji, D. Gong, J. Zhou, et al., Cosface: large margin cosine loss for deep face recognition, in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, (2018), 5265-5274. https://doi.org/10.1109/CVPR.2018.00552 |
| [22] | J. Deng, J. Guo, N. Xue, S. Zafeiriou, ArcFace: additive angular margin loss for deep face recognition, in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), (2019), 4685-4694. https://doi.org/10.1109/CVPR.2019.00482 |
| [23] | X. Tang, D. K. Du, Z. He, J. Liu, PyramidBox: a context-assisted single shot face detector, in Computer Vision - ECCV 2018, (2018), 812-828. https://doi.org/10.1007/978-3-030-01240-3_49 |
| [24] | F. Boutros, N. Damer, F. Kirchbuchner, A. Kuijper, ElasticFace: elastic margin loss for deep face recognition, preprint, arXiv: 2109.09416. |
| [25] | Q. Meng, S. Zhao, Z. Huang, F. Zhou, MagFace: a universal representation for face recognition and quality assessment, in 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), (2021), 14220-14229. https://doi.org/10.1109/CVPR46437.2021.01400 |
| [26] | F. Boutros, N. Damer, M. Fang, F. Kirchbuchner, A. Kuijper, MixFaceNets: extremely efficient face recognition networks, in 2021 IEEE International Joint Conference on Biometrics (IJCB), (2021), 1-8. https://doi.org/10.1109/IJCB52358.2021.9484374 |
| [27] |
F. Boutros, P. Siebke, M. Klemt, N. Damer, F. Kirchbuchner, A. Luijper, PocketNet: extreme lightweight face recognition network using neural architecture search and multistep knowledge distillation, IEEE Access, 10 (2022), 46823-46833. https://doi.org/10.1109/ACCESS.2022.3170561 doi: 10.1109/ACCESS.2022.3170561
|
| [28] |
B. Meden, P. Rot, P. Terhorst, N. Damer, A. Luijper, W. J. Scheirer, Privacy-enhancing face biometrics: a comprehensive survey, IEEE Trans. Inf. Forensics Secur., 16 (2021), 4147-4183. https://doi.org/10.1109/TIFS.2021.3096024 doi: 10.1109/TIFS.2021.3096024
|
| [29] | A. Chattopadhyay, T. E. Boult, PrivacyCam: a privacy preserving camera using uCLinux on the blackfin DSP, in 2007 IEEE Conference on Computer Vision and Pattern Recognition, (2007), 1-8. https://doi.org/10.1109/CVPR.2007.383413 |
| [30] | P. Terhorst, D. Fahrmann, N. Damer, F. Kirchbuchner, A. Luijper, Beyond identity: what information is stored in biometric face templates, in 2020 IEEE International Joint Conference on Biometrics (IJCB), (2020), 1-10, https://doi.org/10.1109/IJCB48548.2020.9304874 |
| [31] |
Z. Zhang, Y. Xu, L. Shao, J. Yang, Discriminative block-diagonal representation learning for image recognition, IEEE Trans. Neural Networks Learn. Syst., 29 (2018), 3111-3125. https://doi.org/10.1109/TNNLS.2017.2712801 doi: 10.1109/TNNLS.2017.2712801
|
| [32] |
P. Terhorst, K. Riehl, N. Damer, P. Rot, B. Bortolato, F. Kirchbuchner, et al., PE-MIU: a training-free privacy-enhancing face recognition approach based on minimum information units, IEEE Access, 8 (2020), 93635-93647. https://doi.org/10.1109/ACCESS.2020.2994960 doi: 10.1109/ACCESS.2020.2994960
|
| [33] |
V. Mirjalili, S. Raschka, A. Ross, FlowSAN: privacy-enhancing semi-adversarial networks to confound arbitrary face-based gender classifiers, IEEE Access, 7 (2019), 99735-99745. https://doi.org/10.1109/ACCESS.2019.2924619 doi: 10.1109/ACCESS.2019.2924619
|
| [34] |
E. M. Newton, L. Sweeney, B. Malin, Preserving privacy by de-identifying face images, IEEE Trans. Knowl. Data Eng., 17 (2005), 232-243. https://doi.org/10.1109/TKDE.2005.32 doi: 10.1109/TKDE.2005.32
|
| [35] |
C. Xiang, C. Tang, Y. Cai, Q. Xu, Privacy-preserving face recognition with outsourced computation, Soft Comput., 20 (2016), 3735-3744. https://doi.org/10.1007/s00500-015-1759-5 doi: 10.1007/s00500-015-1759-5
|
| [36] | F. Tramer, A. Kurakin, N. Papernot, I. Goodfellow, D. Boneh, P. McDaniel, Ensemble adversarial training: attacks and defenses, preprint, arXiv: 1705.07204. |
| [37] |
P. Terhorst, N. Damer, F. Kirchbuchner, A. Luijper, Unsupervised privacy-enhancement of face representations using similarity-sensitive noise transformations, Appl. Intell., 49 (2019), 3043-3060. https://doi.org/10.1007/s10489-019-01432-5 doi: 10.1007/s10489-019-01432-5
|
| [38] |
Y. Li, Y. Wang, D. Li, Privacy-preserving lightweight face recognition, Neurocomputing, 363 (2019), 212-222. https://doi.org/10.1016/j.neucom.2019.07.039 doi: 10.1016/j.neucom.2019.07.039
|
| [39] |
M. A. P. Chamikara, P. Bertok, I. Khalil, D. Liu, S. Camtepe, Privacy preserving face recognition utilizing differential privacy, Comput. Secur., 97 (2020), 101951. https://doi.org/10.1016/j.cose.2020.101951 doi: 10.1016/j.cose.2020.101951
|
| [40] |
Z. Kuang, Z. Guo, J. Fang, J. Yu, N. Babaguchi, J. Fan, Unnoticeable synthetic face replacement for image privacy protection, Neurocomputing, 457 (2021), 322-333. https://doi.org/10.1016/j.neucom.2021.06.061 doi: 10.1016/j.neucom.2021.06.061
|
| [41] | J. C. LIN, P. Fournier-Viger, L. Wu, W. Gan, Y. Djenouri, J. Zhang, PPSF: an open-source privacy-preserving and security mining framework, in 2018 IEEE International Conference on Data Mining Workshops (ICDMW), (2018), 1459-1463. https://doi.org/10.1109/ICDMW.2018.00208 |
| [42] |
K. Zheng, J. Shen, G. Sun, H. Li, Y. Li, Shielding facial physiological information in video, Math. Biosci. Eng., 19 (2022), 5153-5168. https://doi.org/10.3934/mbe.2022241 doi: 10.3934/mbe.2022241
|
| [43] |
J. Lin, G. Srivastava, Y. Zhang, Y. Djenouri, M. Aloqaily, Privacy-preserving multiobjective sanitization model in 6G IoT environments, IEEE Internet Things J., 8 (2021), 5340-5349. https://doi.org/10.1109/JIOT.2020.3032896 doi: 10.1109/JIOT.2020.3032896
|
| [44] |
X. Wang, H. Xue, X. Liu, Q. Pei, A privacy-preserving edge computation-based face verification system for user authentication, IEEE Access, 7 (2019), 14186-14197. https://doi.org/10.1109/ACCESS.2019.2894535 doi: 10.1109/ACCESS.2019.2894535
|
| [45] | W. Shen, Z. Wu, J. Zhang, A face privacy protection algorithm based on block scrambling and deep learning, in Cloud Computing and Security, (2018), 359-369. https://doi.org/10.1007/978-3-030-00012-7_33 |
| [46] | N. Damer, A. Opel, A. Shahverdyan, An overview on multi-biometric score-level fusion, in Proceedings of the 2nd International Conference on Pattern Recognition Applications and Methods (BTSA-2013), (2013), 647-653. https://doi.org/10.5220/0004358306470653 |
| [47] | N. Damer, A. Opel, A. Nouak, Biometric source weighting in multi-biometric fusion: towards a generalized and robust solution, in 2014 22nd European Signal Processing Conference (EUSIPCO), (2014), 1382-1386. Available from: https://ieeexplore.ieee.org/abstract/document/6952496. |
| [48] | N. Damer, F. Maul, C. Busch, Multi-biometric continuous authentication: a trust model for an asynchronous system, in 2016 19th International Conference on Information Fusion (FUSION), (2016), 2192-2199. Available from: https://ieeexplore.ieee.org/abstract/document/7528154. |
| [49] | N. Damer, S. Zienert, Y. Wainakh, A. M. Saladié, F. Kirchbuchner, A. Kuijper, A multi-detector solution towards an accurate and generalized detection of face morphing attacks, in 2019 22th International Conference on Information Fusion (FUSION), (2019), 1-8. Available from: https://ieeexplore.ieee.org/abstract/document/9011378. |
| [50] |
X. Zhang, C. Shi, X. Wang, X. Wu, X. Li, J. Lv, et al., Face inpainting based on GAN by facial prediction and fusion as guidance information, Appl. Soft Comput., 111 (2016), 107626. https://doi.org/10.1016/j.asoc.2021.107626 doi: 10.1016/j.asoc.2021.107626
|
| [51] | K. Simonyan, A. Zisserman, Very deep convolutional networks for large-scale image recognition, preprint, arXiv: 1409.1556. |
| [52] | C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, et al., Intriguing properties of neural networks, preprint, arXiv: 1312.6199. |
| [53] | R. Mutegeki, D. S. Han, Feature-representation transfer learning for human activity recognition, in 2019 International Conference on Information and Communication Technology Convergence (ICTC), (2019), 18-20. https://doi.org/10.1109/ICTC46691.2019.8939979 |
| [54] |
Y. Li, M. Zhu, G. Sun, J. Chen, X. Zhu, J. Yang, Weakly supervised training for eye fundus lesion segmentation in patients with diabetic retinopathy, Math. Biosci. Eng., 19 (2022), 5293-5311. https://doi.org/10.3934/mbe.2022248 doi: 10.3934/mbe.2022248
|
| [55] |
M. Khishe, M. R. Mosavi, Chimp optimization algorithm, Expert Syst. Appl., 149 (2020), 113338. https://doi.org/10.1016/j.eswa.2020.113338 doi: 10.1016/j.eswa.2020.113338
|
| [56] |
C. Boesch, Cooperative hunting roles among taï chimpanzees, Hum. Nat., 13 (2002), 27-46. https://doi.org/10.1007/s12110-002-1013-6 doi: 10.1007/s12110-002-1013-6
|
| [57] |
J. M. Wu, J. C. Lin, P. Fournier-Viger, Y. Djenouri, C. Chen, Z. Li, The density-based clustering method for privacy-preserving data mining, Math. Biosci. Eng., 16 (2019), 1718-1728. https://doi.org/10.3934/mbe.2019082 doi: 10.3934/mbe.2019082
|
| [58] |
I. Aljarah, H. Faris, S. Mirjalili, Optimizing connection weights in neural networks using the whale optimization algorithm, Soft Comput., 22 (2018), 1-15. https://doi.org/10.1007/s00500-016-2442-1 doi: 10.1007/s00500-016-2442-1
|
| [59] | M. Pautov, G. Melnikov, E. Kaziakhmedov, K. Kireev, A. Petiushko, On adversarial patches: real-world attack on ArcFace-100 face recognition system, in 2019 International Multi-Conference on Engineering, Computer and Information Sciences (SIBIRCON), (2019), 0391-0396. https://doi.org/10.1109/SIBIRCON48586.2019.8958134 |
| [60] | A. Radford, L. Metz, S. Chintala, Unsupervised representation learning with deep convolutional generative adversarial networks, preprint, arXiv: 1511.06434. |
| [61] |
C. Sharma, A. Bagga, R. Sobti, M. Shabaz, R. Amin, A robust image encrypted watermarking technique for neurodegenerative disorder diagnosis and its applications, Comput. Math. Methods Med., 2021 (2021), 8081276. https://doi.org/10.1155/2021/8081276 doi: 10.1155/2021/8081276
|
| [62] |
Z. Liu, J. Li, J. Liu, Encrypted face recognition algorithm based on Ridgelet-DCT transform and THM chaos, Math. Biosci. Eng., 19 (2022), 1373-1387. https://doi.org/10.3934/mbe.2022063 doi: 10.3934/mbe.2022063
|
| [63] |
C. Wang, X. Wang, Z. Xia, C. Zhang, Ternary radial harmonic Fourier moments based robust stereo image zero-watermarking algorithm, Inf. Sci., 470 (2019), 109-120. https://doi.org/10.1016/j.ins.2018.08.028 doi: 10.1016/j.ins.2018.08.028
|
| [64] | S. Komkov, A. Petiushko, AdvHat: Real-world adversarial attack on arcface face ID system, in 2020 25th International Conference on Pattern Recognition (ICPR), (2021), 819-826. https://doi.org/10.1109/ICPR48806.2021.9412236 |
| [65] | B. Bortolato, M. Ivanovska, P. Rot, J. Križaj, P. Terhörst, N. Damer, et al., Learning privacy-enhancing face representations through feature disentanglement, in 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020), (2020), 495-502. https://doi.org/10.1109/FG47880.2020.00007 |
| [66] |
S. Li, F. Liu, J. Liang, Z. Cai, Z. Liang, Optimization of face recognition system based on azure IoT edge, Comput. Mater. Continua, 61 (2019), 1377-1389. https://doi.org/10.32604/cmc.2019.06402 doi: 10.32604/cmc.2019.06402
|
| [67] | G. Gamage, I. Sudasingha, I. Perera, D. Meedeniya, Reinstating Dlib correlation human trackers under occlusions in human detection based tracking, in 2018 18th International Conference on Advances in ICT for Emerging Regions (ICTer), (2018), 92-98. https://doi.org/10.1109/ICTER.2018.8615551 |
| [68] |
P. Baldi, K. Hornik, Neural networks and principal component analysis: learning from examples without local minima, Neural Networks, 2 (1989), 53-58. https://doi.org/10.1016/0893-6080(89)90014-2 doi: 10.1016/0893-6080(89)90014-2
|
| [69] | P. Terhörst, M. Huber, N. Damer, P. Rot, F. Kirchbuchner, V. Struc, et al., Privacy evaluation protocols for the evaluation of soft-biometric privacy-enhancing technologies, in BIOSIG 2020 - Proceedings of the 19th International Conference of the Biometrics Special Interest Group, (2020), 215-222. http://dl.gi.de/handle/20.500.12116/34330 |
| [70] |
K. Owusu-Agyemang, Z. Qin, A. Benjamin, H. Xiong, Z. Qin, Guaranteed distributed machine learning: privacy-preserving empirical risk minimization, Math. Biosci. Eng., 18 (2021), 4772-4796. https://doi.org/10.3934/mbe.2021243 doi: 10.3934/mbe.2021243
|
| [71] |
R. N. Abiram, P. Vincent, Identity preserving multi-pose facial expression recognition using fine tuned VGG on the latent space vector of generative adversarial network, Math. Biosci. Eng., 18 (2021), 3699-3717. https://doi.org/10.3934/mbe.2021186 doi: 10.3934/mbe.2021186
|
Figures(12) / Tables(7)
Hao Wang, Guangmin Sun, Kun Zheng, Hui Li, Jie Liu, Yu Bai. Privacy protection generalization with adversarial fusion[J]. Mathematical Biosciences and Engineering, 2022, 19(7): 7314-7336. doi: 10.3934/mbe.2022345
DownLoad: