A novel Bayesian federated learning framework to address multi-dimensional heterogeneity problem

Jianye Yang; Tongjiang Yan; Pengcheng Ren; Jianye Yang; Tongjiang Yan; Pengcheng Ren

doi:10.3934/math.2023769

AIMS Mathematics

2023, Volume 8, Issue 7: 15058-15080. doi: 10.3934/math.2023769

Previous Article Next Article

Research article

A novel Bayesian federated learning framework to address multi-dimensional heterogeneity problem

College of Science, China University of Petroleum, Qingdao 266580, China

Received: 24 February 2023 Revised: 01 April 2023 Accepted: 13 April 2023 Published: 23 April 2023
MSC : 68T09

Federated learning (FL) has attracted a lot of interests as a promising machine learning approach to protect user privacy and data security. It requires the clients to send model parameters to the server rather than private datasets, thus protecting privacy to a significant extent. However, there are several types of heterogeneities (data, model, objective and systems) in FL scenario, posing distinct challenges to the canonical FL algorithm (FedAvg). In this work, we propose a novel FL framework that integrates knowledge distillation and Bayesian inference to address this multi-dimensional heterogeneity problem. On the client side, we approximate the local likelihood function using a scaled multi-dimensional Gaussian probability density function (PDF). Moreover, each client is allowed to design customized model according to the requirement through knowledge distillation. On the server side, a multi-Gaussian product mechanism is employed to construct and maximize the global likelihood function, greatly enhancing the accuracy of the aggregated model in the case of data heterogeneity. Finally, we show in extensive empirical experiments on various datasets and settings that global model and local model can achieve better performance and require fewer communication rounds to converge compared with other FL techniques.
- federated learning,
- heterogeneity problem,
- knowledge distillation,
- Bayesian inference,
- Gaussian probability density function
Citation: Jianye Yang, Tongjiang Yan, Pengcheng Ren. A novel Bayesian federated learning framework to address multi-dimensional heterogeneity problem[J]. AIMS Mathematics, 2023, 8(7): 15058-15080. doi: 10.3934/math.2023769

Related Papers:

Abstract

Federated learning (FL) has attracted a lot of interests as a promising machine learning approach to protect user privacy and data security. It requires the clients to send model parameters to the server rather than private datasets, thus protecting privacy to a significant extent. However, there are several types of heterogeneities (data, model, objective and systems) in FL scenario, posing distinct challenges to the canonical FL algorithm (FedAvg). In this work, we propose a novel FL framework that integrates knowledge distillation and Bayesian inference to address this multi-dimensional heterogeneity problem. On the client side, we approximate the local likelihood function using a scaled multi-dimensional Gaussian probability density function (PDF). Moreover, each client is allowed to design customized model according to the requirement through knowledge distillation. On the server side, a multi-Gaussian product mechanism is employed to construct and maximize the global likelihood function, greatly enhancing the accuracy of the aggregated model in the case of data heterogeneity. Finally, we show in extensive empirical experiments on various datasets and settings that global model and local model can achieve better performance and require fewer communication rounds to converge compared with other FL techniques.

References

[1]	H. B. Mcmahan, E. Moore, D. Ramage, B. A. y Arcas, Federated learning of deep networks using model averaging, arXiv: 1602.05629.
[2]	T. Li, A. Sahu, A. Talwalkar, V. Smith, Federated learning: challenges, methods, and future directions, IEEE Signal Proc. Mag., 37 (2020), 50–60. https://doi.org/10.1109/MSP.2020.2975749 doi: 10.1109/MSP.2020.2975749
[3]	D. Li, J. Wang, FedMD: heterogenous federated learning via model distillation, arXiv: 1910.03581.
[4]	T. Nishio, R. Yonetani, Client selection for federated learning with heterogeneous resources in mobile edge, 2019 IEEE International Conference on Communications (ICC), Shanghai, China, 2019, 1–7. https://doi.org/10.1109/ICC.2019.8761315
[5]	L. Liu, F. Zheng, H. Chen, G. J. Qi, H. Huang, L. Shao, A Bayesian federated learning framework with online Laplace approximation, arXiv: 2102.01936.
[6]	B. Mcmahan, E. Moore, D. Ramage, S. Hampson, B. A. y Arcas, Communication-efficient learning of deep networks from decentralized data, In: Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, New York: PMLR, 2017, 1273–1282.
[7]	B. Wu, X. Dai, P. Zhang, Y. Wang, F. Sun, Y. Wu, FBNet: hardware-aware efficient convnet design via differentiable neural architecture search, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 2019, 10726–10734. https://doi.org/10.1109/CVPR.2019.01099
[8]	C. He, M. Annavaram, S. Avestimehr, Fednas: federated deep learning via neural architecture search, arXiv: 2004.08546.
[9]	T. Shen, J. Zhang, X. Jia, F. Zhang, G. Huang, P. Zhou, et al., Federated mutual learning, arXiv: 2006.16765.
[10]	C. Xie, S. Koyejo, I. Gupta, Asynchronous federated optimization, arXiv: 1903.03934.
[11]	W. Wu, L. He, W. Lin, R. Mao, C. Maple, S. Jarvis, SAFA: a semi-asynchronous protocol for fast federated learning with low overhead, IEEE T. Comput., 70 (2021), 655–668. https://doi.org/10.1109/TC.2020.2994391 doi: 10.1109/TC.2020.2994391
[12]	Y. Zhang, Y. Xu, S. Wei, Y. Wang, Y. Li, X. Shang, Doubly contrastive representation learning for federated image recognition, Pattern Recogn., 139 (2023), 109507. https://doi.org/10.1016/j.patcog.2023.109507 doi: 10.1016/j.patcog.2023.109507
[13]	J. Xiao, C. Du, Z. Duan, W. Guo, A novel server-side aggregation strategy for federated learning in Non-IID situations, 2021 20th International Symposium on Parallel and Distributed Computing (ISPDC), Cluj-Napoca, Romania, 2021, 17–24.
[14]	L. Hu, H. Yan, L. Li, Z. Pan, X. Liu, Z. Zhang, MHAT: an efficient model-heterogenous aggregation training scheme for federated learning, Inform. Sciences, 560 (2021), 493–503. https://doi.org/10.1016/j.ins.2021.01.046 doi: 10.1016/j.ins.2021.01.046
[15]	T. Li, A. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, V. Smith, Federated optimization in heterogeneous networks, Proceedings of Machine Learning and Systems, 2 (2020), 429–450.
[16]	Q. Li, B. He, D. Song, Model-contrastive federated learning, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA, 2021, 10708–10717. https://doi.org/10.1109/CVPR46437.2021.01057
[17]	M. Mendieta, T. Yang, P. Wang, M. Lee, Z. Ding, C. Chen, Local learning matters: rethinking data heterogeneity in federated learning, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, 2022, 8397–8406. https://doi.org/10.1109/cvpr52688.2022.00821
[18]	M. Al-Shedivat, J. Gillenwater, E. Xing, A. Rostamizadeh, Federated learning via posterior averaging: a new perspective and practical algorithms, arXiv: 2010.05273.
[19]	H. Chang, V. Shejwalkar, R. Shokri, A. Houmansadr, Cronus: robust and heterogeneous collaborative learning with black-box knowledge transfer, arXiv: 1912.11279.
[20]	Y. Zhang, T. Xiang, T. Hospedales, H. Lu, Deep mutual learning, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 2018, 4320–4328. https://doi.org/10.1109/CVPR.2018.00454
[21]	C. Blundell, J. Cornebise, K. Kavukcuoglu, D. Wierstra, Weight uncertainty in neural network, The 32nd International Conference on Machine Learning (ICML), Lille, France, 2015, 1613–1622.
[22]	K. Shridhar, F. Laumann, M. Liwicki, A comprehensive guide to bayesian convolutional neural network with variational inference, arXiv: 1901.02731.
[23]	A. Wilson, P. Izmailov, Bayesian deep learning and a probabilistic perspective of generalization, The 34th Conference on Neural Information Processing Systems (NeurIPS), Vancouver, Canada, 2020, 4697–4708. https://doi.org/10.5555/3495724.3496118
[24]	O. Goldreich, S. Micali, A. Wigderson, How to play any mental game, or a completeness theorem for protocols with honest majority, In: Providing sound foundations for cryptography: on the work of shafi goldwasser and silvio micali, New York: Association for Computing Machinery, 2019,307–328. https://doi.org/10.1145/3335741.3335755
[25]	L. T. Phong, Y. Aono, T. Hayashi, L. Wang, S. Moriai, Privacy-preserving deep learning via additively homomorphic encryption, IEEE T. Inf. Foren. Sec., 13 (2018), 1333–1345. https://doi.org/10.1109/TIFS.2017.2787987 doi: 10.1109/TIFS.2017.2787987
[26]	R. Geyer, T. Klein, M. Nabi, Differentially private federated learning: a client level perspective, arXiv: 1712.07557.
[27]	P. Kairouz, H. McMahan, B. Avent, A. Bellet, M. Bennis, A. N. Bhagoji, et al., Advances and open problems in federated learning, Found. Trends Mach. Le., 14 (2021), 1–210. https://doi.org/10.1561/2200000083 doi: 10.1561/2200000083
[28]	Q. Yang, Y. Liu, T. Chen, Y. Tong, Federated machine learning: concept and applications, ACM T. Intel. Syst. Tec., 10 (2019), 12. https://doi.org/10.1145/3298981 doi: 10.1145/3298981
[29]	Y Lecun, L. Bottou, Y. Bengio, P. Haffner, Gradient-based learning applied to document recognition, P. IEEE, 86 (1998), 2278–2324. https://doi.org/10.1109/5.726791 doi: 10.1109/5.726791
[30]	A. Krizhevsky, G. Hinton, Learning multiple layers of features from tiny images, Technical Report TR-2009, University of Toronto, Toronto, 2009.
[31]	Y. Lecun, B. Boser, J. S. Denker, R. E. Howard, W. Habbard, L. D. Jackel, et al., Handwritten digit recognition with a back-propagation network, In: Advances in Neural Information Processing systems 2, San Francisco: Morgan Kaufmann Publishers Inc., 1989,396–404. https://doi.org/10.5555/109230.109279
[32]	A. Ashukha, A. Lyzhov, D. Molchanov, D. Vetrov, Pitfalls of in-domain uncertainty estimation and ensembling in deep learning, arXiv: 2002.06470.
[33]	M. Yurochkin, M. Agarwal, S. Ghosh, K. Greenewald, N. Hoang, Y. Khazaeni, Bayesian nonparametric federated learning of neural networks, The 36th International Conference on Machine Learning, Long Beach, California, USA, 2019, 7252–7261.
[34]	T. H. Hsu, H. Qi, M. Brown, Measuring the effects of non-identical data distribution for federated visual classification, arXiv: 1909.06335.

Reader Comments

Your name:*

Email:*
© 2023 the Author(s), licensee AIMS Press. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0)