Research and implementation of variable-domain fuzzy PID intelligent control method based on Q-Learning for self-driving in complex scenarios

Yongqiang Yao; Nan Ma; Cheng Wang; Zhixuan Wu; Cheng Xu; Jin Zhang; Yongqiang Yao; Nan Ma; Cheng Wang; Zhixuan Wu; Cheng Xu; Jin Zhang

doi:10.3934/mbe.2023260

Mathematical Biosciences and Engineering

2023, Volume 20, Issue 3: 6016-6029. doi: 10.3934/mbe.2023260

Previous Article Next Article

Research article Special Issues

Research and implementation of variable-domain fuzzy PID intelligent control method based on Q-Learning for self-driving in complex scenarios

1.
Beijing Key Laboratory of Information Service Engineering, Beijing Union University, Beijing 100101, China
2.
Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China

Academic Editor: Yiu-ming Cheung

Received: 24 October 2022 Revised: 20 December 2022 Accepted: 29 December 2022 Published: 18 January 2023

In the control of the self-driving vehicles, PID controllers are widely used due to their simple structure and good stability. However, in complex self-driving scenarios such as curvature curves, car following, overtaking, etc., it is necessary to ensure the stable control accuracy of the vehicles. Some researchers used fuzzy PID to dynamically change the parameters of PID to ensure that the vehicle control remains in a stable state. It is difficult to ensure the control effect of the fuzzy controller when the size of the domain is not selected properly. This paper designs a variable-domain fuzzy PID intelligent control method based on Q-Learning to make the system robust and adaptable, which is dynamically changed the size of the domain to further ensure the control effect of the vehicle. The variable-domain fuzzy PID algorithm based on Q-Learning takes the error and the error rate of change as input and uses the Q-Learning method to learn the scaling factor online so as to achieve online PID parameters adjustment. The proposed method is verified on the Panosim simulation platform.The experiment shows that the accuracy is improved by 15% compared with the traditional fuzzy PID, which reflects the effectiveness of the algorithm.
- self-driving,
- intelligent control,
- Q-Learning,
- variable-domain fuzzy PID
Citation: Yongqiang Yao, Nan Ma, Cheng Wang, Zhixuan Wu, Cheng Xu, Jin Zhang. Research and implementation of variable-domain fuzzy PID intelligent control method based on Q-Learning for self-driving in complex scenarios[J]. Mathematical Biosciences and Engineering, 2023, 20(3): 6016-6029. doi: 10.3934/mbe.2023260

Related Papers:

Abstract

In the control of the self-driving vehicles, PID controllers are widely used due to their simple structure and good stability. However, in complex self-driving scenarios such as curvature curves, car following, overtaking, etc., it is necessary to ensure the stable control accuracy of the vehicles. Some researchers used fuzzy PID to dynamically change the parameters of PID to ensure that the vehicle control remains in a stable state. It is difficult to ensure the control effect of the fuzzy controller when the size of the domain is not selected properly. This paper designs a variable-domain fuzzy PID intelligent control method based on Q-Learning to make the system robust and adaptable, which is dynamically changed the size of the domain to further ensure the control effect of the vehicle. The variable-domain fuzzy PID algorithm based on Q-Learning takes the error and the error rate of change as input and uses the Q-Learning method to learn the scaling factor online so as to achieve online PID parameters adjustment. The proposed method is verified on the Panosim simulation platform.The experiment shows that the accuracy is improved by 15% compared with the traditional fuzzy PID, which reflects the effectiveness of the algorithm.

References

[1]	R. K. Khadanga, A. Kumar, S. Panda, Frequency control in hybrid distributed power systems via type-2 fuzzy pid controller, IET Renewable Power Gener., 15 (2021), 1706–1723. https://doi.org/10.1049/rpg2.12140 doi: 10.1049/rpg2.12140
[2]	M. K. Diab, H. H. Ammar, R. E. Shalaby, Self-driving car lane-keeping assist using pid and pure pursuit control, in 2020 International Conference on Innovation and Intelligence for Informatics, Computing and Technologies (3ICT), IEEE, (2020), 1–6. https://doi.org/10.1109/3ICT51146.2020.9311987
[3]	H. Maghfiroh, M. Ahmad, A. Ramelan, F. Adriyanto, Fuzzy-pid in bldc motor speed control using matlab/simulink, J. Rob. Control (JRC), 3 (2022), 8–13. https://doi.org/10.18196/jrc.v3i1.10964 doi: 10.18196/jrc.v3i1.10964
[4]	J. R. Nayak, B. Shaw, B. K. Sahu, K. A. Naidu, Application of optimized adaptive crow search algorithm based two degree of freedom optimal fuzzy pid controller for agc system, Eng. Sci. Technol. Int. J., 32 (2022), 101061. https://doi.org/10.1016/j.jestch.2021.09.007 doi: 10.1016/j.jestch.2021.09.007
[5]	N. Ma, D. Li, W. He, Y. Deng, J. Li, Y. Gao, et al., Future vehicles: interactive wheeled robots, Sci. China Inf. Sci., 64 (2021), 1–3. https://doi.org/10.1007/s11432-020-3171-4 doi: 10.1007/s11432-020-3171-4
[6]	N. Ma, Y. Gao, J. Li, D. Li, Interactive cognition in self-driving, Chin. Sci.: Inf. Sci., 48 (2018), 1083–1096.
[7]	D. Li, N. Ma, Y. Gao, Future vehicles: learnable wheeled robots, Sci. China Inf. Sci., 63 (2020), 1–8. https://doi.org/10.1007/s11432-019-2787-2 doi: 10.1007/s11432-019-2787-2
[8]	T. Yang, N. Sun, Y. Fang, Adaptive fuzzy control for a class of mimo underactuated systems with plant uncertainties and actuator deadzones: Design and experiments, IEEE Trans. Cybern., 52 (2022), 8213–8226. https://doi.org/10.1109/TCYB.2021.3050475 doi: 10.1109/TCYB.2021.3050475
[9]	S. H. Park, K. W. Kim, W. H. Choi, M. S. Jie, Y. Kim, The autonomous performance improvement of mobile robot using type-2 fuzzy self-tuning PID controller, Adv. Sci. Technol. Lett., 138 (2016), 182–187. https://doi.org/10.14257/astl.2016.138.37 doi: 10.14257/astl.2016.138.37
[10]	P. Parikh, S. Sheth, R. Vasani, J. K. Gohil, Implementing fuzzy logic controller and pid controller to a dc encoder motor–-"a case of an automated guided vehicle", Procedia Manuf., 20 (2018), 219–226. https://doi.org/10.1016/j.promfg.2018.02.032 doi: 10.1016/j.promfg.2018.02.032
[11]	Q. Bu, J. Cai, Y. Liu, M. Cao, L. Dong, R. Ruan, et al., The effect of fuzzy pid temperature control on thermal behavior analysis and kinetics study of biomass microwave pyrolysis, J. Anal. Appl. Pyrolysis, 158 (2021), 105176. https://doi.org/10.1016/j.jaap.2021.105176 doi: 10.1016/j.jaap.2021.105176
[12]	M. S. Jie, W. H. Choi, Type-2 fuzzy pid controller design for mobile robot, Int. J. Control Autom., 9 (2016), 203–214.
[13]	N. Kumar, M. Takács, Z. Vámossy, Robot navigation in unknown environment using fuzzy logic, in 2017 IEEE 15th International Symposium on Applied Machine Intelligence and Informatics (SAMI), IEEE, (2017), 279–284. https://doi.org/10.1109/SAMI.2017.7880317
[14]	T. Muhammad, Y. Guo, Y. Wu, W. Yao, A. Zeeshan, Ccd camera-based ball balancer system with fuzzy pd control in varying light conditions, in 2019 IEEE 16th International Conference on Networking, Sensing and Control (ICNSC), IEEE, (2019), 305–310. https://doi.org/10.1109/ICNSC.2019.8743305
[15]	A. Wong, T. Back, A. V. Kononova, A. Plaat, Deep multiagent reinforcement learning: Challenges and directions, Artif. Intell. Rev., 2022 (2022). https://doi.org/10.1007/s10462-022-10299-x doi: 10.1007/s10462-022-10299-x
[16]	Z. Cao, S. Xu, H. Peng, D. Yang, R. Zidek, Confidence-aware reinforcement learning for self-driving cars, IEEE Trans. Intell. Transp. Syst., 23 (2022), 7419–7430. https://doi.org/10.1109/TITS.2021.3069497 doi: 10.1109/TITS.2021.3069497
[17]	T. Ribeiro, F. Gonçalves, I. Garcia, G. Lopes, A. F. Ribeiro, Q-learning for autonomous mobile robot obstacle avoidance, in 2019 IEEE International Conference on Autonomous Robot Systems and Competitions (ICARSC), IEEE, (2019), 1–7. https://doi.org/10.1109/ICARSC.2019.8733621
[18]	S. Danthala, S. Rao, K. Mannepalli, D. Shilpa, Robotic manipulator control by using machine learning algorithms: A review, Int. J. Mech. Prod. Eng. Res. Dev., 8 (2018), 305–310.
[19]	X. Lei, Z. Zhang, P. Dong, Dynamic path planning of unknown environment based on deep reinforcement learning, J. Rob., 2018 (2018). https://doi.org/10.1155/2018/5781591 doi: 10.1155/2018/5781591
[20]	Y. Shan, B. Zheng, L. Chen, L. Chen, D. Chen, A reinforcement learning-based adaptive path tracking approach for autonomous driving, IEEE Trans. Veh. Technol., 69 (2020), 10581–10595. https://doi.org/10.1109/TVT.2020.3014628 doi: 10.1109/TVT.2020.3014628
[21]	T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, et al., Continuous control with deep reinforcement learning, preprint, arXiv: 1509.02971. https://doi.org/10.48550/arXiv.1509.02971
[22]	P. Ramanathan, K. K. Mangla, S. Satpathy, Smart controller for conical tank system using reinforcement learning algorithm, Measurement, 116 (2018), 422–428. https://doi.org/10.1016/j.measurement.2017.11.007 doi: 10.1016/j.measurement.2017.11.007
[23]	L. Brunke, M. Greeff, A. W. Hall, Z. Yuan, S. Zhou, J. Panerati, et al., Safe learning in robotics: From learning-based control to safe reinforcement learning, Annu. Rev. Control Rob. Auton. Syst., 5 (2022), 411–444. https://doi.org/10.1146/annurev-control-042920-020211 doi: 10.1146/annurev-control-042920-020211
[24]	A. I. Lakhani, M. A. Chowdhury, Q. Lu, Stability-preserving automatic tuning of PID control with reinforcement learning, preprint, arXiv: 2112.15187. https://doi.org/10.20517/ces.2021.15
[25]	O. Dogru, K. Velswamy, F. Ibrahim, Y. Wu, A. S. Sundaramoorthy, B. Huang, et al., Reinforcement learning approach to autonomous pid tuning, Comput. Chem. Eng., 161 (2022), 107760. https://doi.org/10.1016/j.compchemeng.2022.107760 doi: 10.1016/j.compchemeng.2022.107760
[26]	X. Yu, Y. Fan, S. Xu, L. Ou, A self-adaptive sac-pid control approach based on reinforcement learning for mobile robots, Int. J. Robust Nonlinear Control, 32 (2022), 9625–9643. https://doi.org/10.1002/rnc.5662 doi: 10.1002/rnc.5662
[27]	B. Guo, Z. Zhuang, J. S. Pan, S. C. Chu, Optimal design and simulation for pid controller using fractional-order fish migration optimization algorithm, IEEE Access, 9 (2021), 8808–8819. https://doi.org/10.1109/ACCESS.2021.3049421 doi: 10.1109/ACCESS.2021.3049421
[28]	M. Praharaj, D. Sain, B. Mohan, Development, experimental validation, and comparison of interval type-2 mamdani fuzzy pid controllers with different footprints of uncertainty, Inf. Sci., 601 (2022), 374–402.
[29]	Y. Jia, R. Zhang, X. Lv, T. Zhang, Z. Fan, Research on temperature control of fuel-cell cooling system based on variable domain fuzzy pid, Processes, 10 (2022), 534. https://doi.org/10.3390/pr10030534 doi: 10.3390/pr10030534
[30]	J. Wei, L. Gang, W. Tao, G. Kai, Variable universe fuzzy pid control based on adaptive contracting-expanding factors, Eng. Mech., 38 (2021), 23–32. https://doi.org/10.6052/j.issn.1000-4750.2020.11.0786 doi: 10.6052/j.issn.1000-4750.2020.11.0786
[31]	R. S. Sutton, A. G. Barto, Reinforcement Learning: An Introduction, MIT press, 2018.
[32]	P. R. Montague, Reinforcement learning: an introduction, by Sutton, RS and Barto, AG, Trends Cognit. Sci., 3 (1999), 360. https://doi.org/10.1016/S1364-6613(99)01331-5 doi: 10.1016/S1364-6613(99)01331-5
[33]	D. Wang, R. Walters, X. Zhu, R. Platt, Equivariant $ q $ learning in spatial action spaces, in Conference on Robot Learning, PMLR, (2022), 1713–1723.
[34]	E. Anderlini, D. I. Forehand, P. Stansell, Q. Xiao, M. Abusara, Control of a point absorber using reinforcement learning, IEEE Trans. Sustainable Energy, 7 (2016), 1681–1690. https://doi.org/10.1109/TSTE.2016.2568754 doi: 10.1109/TSTE.2016.2568754

Reader Comments

Your name:*

Email:*
© 2023 the Author(s), licensee AIMS Press. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0)