Intelligent multicast routing method based on multi-agent deep reinforcement learning in SDWN

Hongwen Hu; Miao Ye; Chenwei Zhao; Qiuxiang Jiang; Xingsi Xue; Hongwen Hu; Miao Ye; Chenwei Zhao; Qiuxiang Jiang; Xingsi Xue

doi:10.3934/mbe.2023765

Mathematical Biosciences and Engineering

2023, Volume 20, Issue 9: 17158-17196. doi: 10.3934/mbe.2023765

Previous Article Next Article

Research article Special Issues

Intelligent multicast routing method based on multi-agent deep reinforcement learning in SDWN

1.
School of Computer Science and Information Security, Guilin University of Electronic Technology, Guilin 541004, China
2.
Guangxi Key Laboratory of Wireless Wideband Communication and Signal Processing, Guilin University of Electronic Technology, Guilin 541004, China
3.
Ministry of Education Key Laboratory of Cognitive Radio and Information Processing, Guilin University of Electronic Technology, Guilin 541004, China

Academic Editor: Yiu-ming Cheung

Received: 17 June 2023 Revised: 11 August 2023 Accepted: 17 August 2023 Published: 04 September 2023

Multicast communication technology is widely applied in wireless environments with a high device density. Traditional wireless network architectures have difficulty flexibly obtaining and maintaining global network state information and cannot quickly respond to network state changes, thus affecting the throughput, delay, and other QoS requirements of existing multicasting solutions. Therefore, this paper proposes a new multicast routing method based on multiagent deep reinforcement learning (MADRL-MR) in a software-defined wireless networking (SDWN) environment. First, SDWN technology is adopted to flexibly configure the network and obtain network state information in the form of traffic matrices representing global network links information, such as link bandwidth, delay, and packet loss rate. Second, the multicast routing problem is divided into multiple subproblems, which are solved through multiagent cooperation. To enable each agent to accurately understand the current network state and the status of multicast tree construction, the state space of each agent is designed based on the traffic and multicast tree status matrices, and the set of AP nodes in the network is used as the action space. A novel single-hop action strategy is designed, along with a reward function based on the four states that may occur during tree construction: progress, invalid, loop, and termination. Finally, a decentralized training approach is combined with transfer learning to enable each agent to quickly adapt to the dynamic changes of network link information and accelerate convergence. Simulation experiments show that MADRL-MR outperforms existing algorithms in terms of throughput, delay, packet loss rate, etc., and can establish more intelligent multicast routes. Code and model are available at https://github.com/GuetYe/MADRL-MR_code.
- software-defined wireless networking,
- multi-agent,
- deep reinforcement learning,
- multicast
Citation: Hongwen Hu, Miao Ye, Chenwei Zhao, Qiuxiang Jiang, Xingsi Xue. Intelligent multicast routing method based on multi-agent deep reinforcement learning in SDWN[J]. Mathematical Biosciences and Engineering, 2023, 20(9): 17158-17196. doi: 10.3934/mbe.2023765

Related Papers:

Abstract

Multicast communication technology is widely applied in wireless environments with a high device density. Traditional wireless network architectures have difficulty flexibly obtaining and maintaining global network state information and cannot quickly respond to network state changes, thus affecting the throughput, delay, and other QoS requirements of existing multicasting solutions. Therefore, this paper proposes a new multicast routing method based on multiagent deep reinforcement learning (MADRL-MR) in a software-defined wireless networking (SDWN) environment. First, SDWN technology is adopted to flexibly configure the network and obtain network state information in the form of traffic matrices representing global network links information, such as link bandwidth, delay, and packet loss rate. Second, the multicast routing problem is divided into multiple subproblems, which are solved through multiagent cooperation. To enable each agent to accurately understand the current network state and the status of multicast tree construction, the state space of each agent is designed based on the traffic and multicast tree status matrices, and the set of AP nodes in the network is used as the action space. A novel single-hop action strategy is designed, along with a reward function based on the four states that may occur during tree construction: progress, invalid, loop, and termination. Finally, a decentralized training approach is combined with transfer learning to enable each agent to quickly adapt to the dynamic changes of network link information and accelerate convergence. Simulation experiments show that MADRL-MR outperforms existing algorithms in terms of throughput, delay, packet loss rate, etc., and can establish more intelligent multicast routes. Code and model are available at https://github.com/GuetYe/MADRL-MR_code.

References

[1]	K. A. Farhan, F. Abdel-Fattah, F. Altarawneh, M. Lafi, Survey paper on multicast routing in mobile ad-hoc networks, in 2019 IEEE Jordan International Joint Conference on Electrical Engineering and Information Technology (JEEIT), IEEE, Conference Proceedings, (2019), 449–452. https://doi.org/10.1109/JEEIT.2019.8717477
[2]	L. Derdouri, C. Pham, E. M. El Amine Zouaoui, N. Zeghib, Performance analysis of self-organised multicast group in multi-radio multi-channel wireless mesh networks, IET Commun., 14 (2020), 693–702. https://doi.org/10.1049/iet-com.2018.6276 doi: 10.1049/iet-com.2018.6276
[3]	P. M. Ruiz, A. F. Gómez-Skarmeta, Approximating optimal multicast trees in wireless multihop networks, in 10th IEEE Symposium on Computers and Communications (ISCC'05), IEEE, Conference Proceedings, (2005), 686–691. https://doi.org/10.1109/ISCC.2005.34
[4]	I. F. Akyildiz, X. Wang, W. Wang, Wireless mesh networks: a survey, Comput. Networks, 47 (2015), 445–487. https://doi.org/10.1109/MCOM.2005.1509968 doi: 10.1109/MCOM.2005.1509968
[5]	S. Costanzo, L. Galluccio, G. Morabito, S. Palazzo, Software defined wireless network (SDWN): An evolvable architecture for W-PANs, in 2015 IEEE 1st International Forum on Research and Technologies for Society and Industry Leveraging a better tomorrow (RTSI), IEEE, Conference Proceedings, (2015), 23–28. https://doi.org/10.1109/RTSI.2015.7325066
[6]	K. Benzekki, A. El Fergougui, A. Elbelrhiti Elalaoui, Software defined networking (SDN): a survey, Secur. Commun. Netw., 9 (2016), 5803–5833. https://doi.org/10.1002/sec.1737 doi: 10.1002/sec.1737
[7]	S. Babu, P. Mithun, B. Manoj, A novel framework for resource discovery and self-configuration in software defined wireless mesh networks, IEEE Trans. Network Serv. Manage., 17 (2019), 132–146. https://doi.org/10.1109/TNSM.2019.2922107 doi: 10.1109/TNSM.2019.2922107
[8]	L. Kou, G. Markowsky, L. Berman, A fast algorithm for Steiner trees, Acta Inf., 15 (1981), 141–145. https://doi.org/10.1007/BF00288961 doi: 10.1007/BF00288961
[9]	H. Takahashi, An approximate solution for steiner problem in graphs, Math. Japonica, 24 (1980), 573–577.
[10]	V. J. Rayward-Smith, The computation of nearly minimal steiner trees in graphs, Int. J. Math. Educ. Sci. Technol., 14 (1983), 15–23. https://doi.org/10.1080/0020739830140103 doi: 10.1080/0020739830140103
[11]	Y. R. Chen, A. Rezapour, W. G. Tzeng, S. C. Tsai, RL-routing: An SDN routing algorithm based on deep reinforcement learning, IEEE Trans. Network Sci. Eng., 7 (2020), 3185–3199. https://doi.org/10.1109/TNSE.2020.3017751 doi: 10.1109/TNSE.2020.3017751
[12]	D. M. Casas-Velasco, O. M. C. Rendon, N. L. da Fonseca, Intelligent routing based on reinforcement learning for software-defined networking, IEEE Trans. Network Serv. Manage., 18 (2020), 870–881. https://doi.org/10.1109/TNSM.2020.3036911 doi: 10.1109/TNSM.2020.3036911
[13]	D. M. Casas-Velasco, O. M. C. Rendon, N. L. da Fonseca, DRSIR: A deep reinforcement learning approach for routing in software-defined networking, IEEE Trans. Network Serv. Manage., 19 (2021), 4807–4820. https://doi.org/10.1109/TNSM.2021.3132491 doi: 10.1109/TNSM.2021.3132491
[14]	J. Zhang, M. Ye, Z. Guo, C. Y. Yen, H. J. Chao, CFR-RL: Traffic engineering with reinforcement learning in SDN, IEEE J. Sel. Areas Commun., 38 (2020), 2249–2259. https://doi.org/10.1109/JSAC.2020.3000371 doi: 10.1109/JSAC.2020.3000371
[15]	Y. Hou, Y. S. Ong, L. Feng, J. M. Zurada, An evolutionary transfer reinforcement learning framework for multiagent systems, IEEE Trans. Evol. Comput., 21 (2017), 601–615. https://doi.org/10.1109/TEVC.2017.2664665 doi: 10.1109/TEVC.2017.2664665
[16]	Y. Yu, P. Qiu, An improved algorithm for Steiner trees, J. Commun., 23 (2002), 35–40.
[17]	L. Zhou, Y. M. Sun, A delay-constrained steiner tree algorithm using MPH, J. Comput. Res. Dev., 45 (2008), 810–816.
[18]	X. Wang, Steiner tree heuristic algorithm based on weighted node, J. Comput. Appl., 34 (2014), 3414–3416.
[19]	L. Farzinvash, Online multicast tree construction with bandwidth and delay constraints in multi-channel multi-radio wireless mesh networks, Telecommun. Syst., 72 (2019), 413–429. https://doi.org/10.1007/s11235-019-00576-6 doi: 10.1007/s11235-019-00576-6
[20]	M. W. Przewozniczek, K. Walkowiak, A. Sen, M. Komarnicki, P. Lechowicz, The transformation of the k-shortest steiner trees search problem into binary dynamic problem for effective evolutionary methods application, Inf. Sci., 479 (2019), 1–19. https://doi.org/10.1016/j.ins.2018.11.015 doi: 10.1016/j.ins.2018.11.015
[21]	K. Walkowiak, A. Kasprzak, M. Wozniak, Algorithms for calculation of candidate trees for efficient multicasting in elastic optical networks, in 2015 17th International Conference on Transparent Optical Networks (ICTON), IEEE, Conference Proceedings, (2015), 1–4. https://doi.org/10.1109/ICTON.2015.7193692
[22]	L. Martins, D. Santos, T. Gomes, R. Girao-Silva, Determining the minimum cost steiner tree for delay constrained problems, IEEE Access, 9 (2021), 144927–144939. https://doi.org/10.1109/ACCESS.2021.3122024 doi: 10.1109/ACCESS.2021.3122024
[23]	X. Zhang, Y. Wang, G. Geng, J. Yu, Delay-optimized multicast tree packing in software-defined networks, IEEE Trans. Serv. Comput., 16 (2021), 261–275. https://doi.org/10.1109/TSC.2021.3106264 doi: 10.1109/TSC.2021.3106264
[24]	M. Hu, J. Li, C. Cai, T. Deng, W. Xu, Y. Dong, Software defined multicast for large-scale multi-layer leo satellite networks, IEEE Trans. Netw. Serv. Manage., 19 (2022), 2119–2130. https://doi.org/10.1109/TNSM.2022.3151552 doi: 10.1109/TNSM.2022.3151552
[25]	V. Annapurna, C. V. Raj, Improving QoS performance of ATM and MPLS using multicast routing and ACO optimization, in 2022 2nd International Conference on Intelligent Technologies (CONIT), IEEE, Conference Proceedings, (2022), 1–6. https://doi.org/10.1109/CONIT55038.2022.9848211
[26]	X. Zhang, X. Shen, Z. Yu, A novel hybrid ant colony optimization for a multicast routing problem, Algorithms, 12 (2019), 18. https://doi.org/10.3390/a12010018 doi: 10.3390/a12010018
[27]	L. Zhang, Y. Huang, W. Chen, W. Guo, G. Liu, X-architecture steiner tree algorithm with limited routing length inside obstacle, in 2021 11th International Conference on Information Technology in Medicine and Education (ITME), (2021), 152–156. https://doi.org/10.1109/ITME53901.2021.00040
[28]	S. Nath, S. Gupta, S. Biswas, R. Banerjee, J. K. Sing, S. K. Sarkar, Gpso hybrid algorithm for rectilinear steiner tree optimization, in 2020 IEEE VLSI DEVICE CIRCUIT AND SYSTEM (VLSI DCS), IEEE, Conference Proceedings, (2020), 365–369. https://doi.org/10.1109/VLSIDCS47293.2020.9179861
[29]	Q. Zhang, S. Yang, M. Liu, J. Liu, L. Jiang, A new crossover mechanism for genetic algorithms for Steiner tree optimization, IEEE Trans. Cybern., 52 (2020), 3147–3158. https://doi.org/10.1109/TCYB.2020.3005047 doi: 10.1109/TCYB.2020.3005047
[30]	H. J. Heo, N. Kim, B. D. Lee, Multicast tree generation technique using reinforcement learning in sdn environments, in 2018 IEEE SmartWorld, Ubiquitous Intelligence & Computing, Advanced & Trusted Computing, Scalable Computing & Communications, Cloud & Big Data Computing, Internet of People and Smart City Innovation (SmartWorld/SCALCOM/UIC/ATC/CBDCom/IOP/SCI), IEEE, Conference Proceedings, (2018), 77–81. https://doi.org/10.1109/SmartWorld.2018.00048
[31]	A. E. Araqi, B. Mahboobi, Joint channel assignment and multicast routing in multi-channel multi-radio wireless mesh networks based on q-learning, in 2019 IEEE Pacific Rim Conference on Communications, Computers and Signal Processing (PACRIM), IEEE, Conference Proceedings, (2019), 1–6. https://doi.org/10.1109/PACRIM47961.2019.8985111
[32]	T. N. Tran, T. V. Nguyen, K. Shim, D. B. Da Costa, B. An, A new deep Q-network design for QoS multicast routing in cognitive radio MANETs, IEEE Access, 9 (2021), 152841–152856. https://doi.org/10.1109/ACCESS.2021.3126844 doi: 10.1109/ACCESS.2021.3126844
[33]	J. Chae, N. Kim, Multicast Tree Generation using Meta Reinforcement Learning in SDN-based Smart Network Platforms, KSII Trans. Internet Inf. Syst., 15 (2021).
[34]	C. Zhao, M. Ye, X. Xue, J. Lv, Q. Jiang, Y. Wang, DRL-M4MR: An intelligent multicast routing approach based on DQN deep reinforcement learning in SDN, Phys. Commun., 55 (2022), 101919. https://doi.org/10.1016/j.phycom.2022.101919 doi: 10.1016/j.phycom.2022.101919
[35]	J. Yang, J. Zhang, H. Wang, Urban traffic control in software defined internet of things via a multi-agent deep reinforcement learning approach, IEEE Trans. Intell. Transp. Syst., 22 (2020), 3742–3754. https://doi.org/10.1109/TITS.2020.3023788 doi: 10.1109/TITS.2020.3023788
[36]	A. Suzuki, R. Kawahara, S. Harada, Cooperative Multi-agent deep reinforcement learning for dynamic virtual network allocation with traffic fluctuations, IEEE Trans. Netw. Serv. Manage., 19 (2022), 1982–2000. https://doi.org/10.1109/TNSM.2022.3149243 doi: 10.1109/TNSM.2022.3149243
[37]	T. Wu, P. Zhou, B. Wang, A. Li, X. Tang, Z. Xu, et al., Joint traffic control and multi-channel reassignment for core backbone network in SDN-IoT: a multi-agent deep reinforcement learning approach, IEEE Trans. Network Sci. Eng., 8 (2020), 231–245. https://doi.org/10.1109/TNSE.2020.3036456 doi: 10.1109/TNSE.2020.3036456
[38]	S. S. Bhavanasi, L. Pappone, F. Esposito, Routing with Graph Convolutional Networks and Multi-Agent Deep Reinforcement Learning, in 2022 IEEE Conference on Network Function Virtualization and Software Defined Networks (NFV-SDN), IEEE, Conference Proceedings, (2022), 72–77. https://doi.org/10.1109/NFV-SDN56302.2022.9974607
[39]	D. K. Dake, J. D. Gadze, G. S. Klogo, H. Nunoo-Mensah, Multi-agent reinforcement learning framework in SND-IoT for transient load detection and prevention, Technologies, 9 (2021), 44. https://doi.org/10.3390/technologies9030044 doi: 10.3390/technologies9030044
[40]	L. Torrey, M. Taylor, Teaching on a budget: Agents advising agents in reinforcement learning, in Proceedings of the 2013 international conference on Autonomous agents and multi-agent systems, Conference Proceedings, (2013), 1053–1060.
[41]	E. Parisotto, J. L. Ba, R. Salakhutdinov, Actor-mimic: Deep multitask and transfer reinforcement learning, preprint, arXiv: 1511.06342.
[42]	F. L. Da Silva, A. H. R. Costa, A survey on transfer learning for multiagent reinforcement learning systems, J. Artif. Intell. Res., 64 (2019), 645–703. https://doi.org/10.1613/jair.1.11396 doi: 10.1613/jair.1.11396
[43]	Y. Li, Z. P. Cai, H. Xu, LLMP: exploiting LLDP for latency measurement in software-defined data center networks, J. Comput. Sci. Technol., 33 (2018), 277–285. https://doi.org/10.1007/s11390-018-1819-2 doi: 10.1007/s11390-018-1819-2
[44]	L. Al Shalabi, Z. Shaaban, Normalization as a preprocessing engine for data mining and the approach of preference matrix, in 2006 International Conference on Dependability of Computer Systems, IEEE, Conference Proceedings, (2006), 207–214. https://doi.org/10.1109/DEPCOS-RELCOMEX.2006.38
[45]	O. Ashour, M. St-Hilaire, T. Kunz, M. Wang, A survey of applying reinforcement learning techniques to multicast routing, in 2019 IEEE 10th Annual Ubiquitous Computing, Electronics & Mobile Communication Conference (UEMCON), IEEE, Conference Proceedings, (2019), 1145–1151. https://doi.org/10.1109/UEMCON47517.2019.8993014
[46]	V. Konda, J. Tsitsiklis, Actor-critic algorithms, in Advances in Neural Information Processing Systems, 12 (1999).
[47]	A. Feriani, E. Hossain, Single and multi-agent deep reinforcement learning for AI-enabled wireless networks: A tutorial, IEEE Commun. Surv. Tutorials, 23 (2021), 1226–1252. https://doi.org/10.1109/COMST.2021.3063822 doi: 10.1109/COMST.2021.3063822
[48]	J. Heydari, V. Ganapathy, M. Shah, Dynamic task offloading in multi-agent mobile edge computing networks, in 2019 IEEE Global Communications Conference (GLOBECOM), IEEE, Conference Proceedings, (2019), 1–6. https://doi.org/10.1109/GLOBECOM38437.2019.9013115
[49]	X. Liu, J. Yu, Z. Feng, Y. Gao, Multi-agent reinforcement learning for resource allocation in IoT networks with edge computing, China Commun., 17 (2020), 220–236. https://doi.org/10.23919/JCC.2020.09.017 doi: 10.23919/JCC.2020.09.017
[50]	J. Cui, Y. Liu, A. Nallanathan, The application of multi-agent reinforcement learning in UAV networks, in 2019 IEEE International Conference on Communications Workshops (ICC Workshops), IEEE, Conference Proceedings, (2019), 1–6. https://doi.org/10.1109/ICCW.2019.8756984
[51]	Z. Zhu, K. Lin, A. K. Jain, J. Zhou, Transfer learning in deep reinforcement learning: A survey, preprint, arXiv: 2009.07888.
[52]	Mininet-WIFI, Access date: March 16, Available from: https://mininet-wifi.github.io/.
[53]	Ryu, Access date: March 16, Available from: https://ryu-sdn.org/.
[54]	Iperf, Access date: March 16, Available from: https://iperf.fr.

Reader Comments

Your name:*

Email:*
© 2023 the Author(s), licensee AIMS Press. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0)