Research on rainy day traffic sign recognition algorithm based on PMRNet

Jing Zhang; Haoliang Zhang; Ding Lang; Yuguang Xu; Hong-an Li; Xuewen Li; Jing Zhang; Haoliang Zhang; Ding Lang; Yuguang Xu; Hong-an Li; Xuewen Li

doi:10.3934/mbe.2023545

Mathematical Biosciences and Engineering

2023, Volume 20, Issue 7: 12240-12262. doi: 10.3934/mbe.2023545

Previous Article Next Article

Research article Special Issues

Research on rainy day traffic sign recognition algorithm based on PMRNet

1.
College of Computer Science and Technology, Xi'an University of Science and Technology, Xi'an 710054, China
2.
College of Energy, Xi'an University of Science and Technology, Xi'an 710054, China
3.
College of Safety Science and Engineering, Xi'an University of Science and Technology, Xi'an 710054, China

Academic Editor: Sai Xu

Received: 06 February 2023 Revised: 10 April 2023 Accepted: 04 May 2023 Published: 18 May 2023

The recognition of traffic signs is of great significance to intelligent driving and traffic systems. Most current traffic sign recognition algorithms do not consider the impact of rainy weather. The rain marks will obscure the recognition target in the image, which will lead to the performance degradation of the algorithm, a problem that has yet to be solved. In order to improve the accuracy of traffic sign recognition in rainy weather, we propose a rainy traffic sign recognition algorithm. The algorithm in this paper includes two modules. First, we propose an image deraining algorithm based on the Progressive multi-scale residual network (PMRNet), which uses a multi-scale residual structure to extract features of different scales, so as to improve the utilization rate of the algorithm for information, combined with the Convolutional long-short term memory (ConvLSTM) network to enhance the algorithm's ability to extract rain mark features. Second, we use the CoT-YOLOv5 algorithm to recognize traffic signs on the recovered images. In this paper, in order to improve the performance of YOLOv5 (You-Only-Look-Once, YOLO), the 3 × 3 convolution in the feature extraction module is replaced by the Contextual Transformer (CoT) module to make up for the lack of global modeling capability of Convolutional Neural Network (CNN), thus improving the recognition accuracy. The experimental results show that the deraining algorithm based on PMRNet can effectively remove rain marks, and the evaluation indicators Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index Measure (SSIM) are better than the other representative algorithms. The mean Average Precision (mAP) of the CoT-YOLOv5 algorithm on the TT100k datasets reaches 92.1%, which is 5% higher than the original YOLOv5.
- image deraining,
- multi-scale residual,
- CoT module,
- traffic sign recognition
Citation: Jing Zhang, Haoliang Zhang, Ding Lang, Yuguang Xu, Hong-an Li, Xuewen Li. Research on rainy day traffic sign recognition algorithm based on PMRNet[J]. Mathematical Biosciences and Engineering, 2023, 20(7): 12240-12262. doi: 10.3934/mbe.2023545

Related Papers:

Abstract

The recognition of traffic signs is of great significance to intelligent driving and traffic systems. Most current traffic sign recognition algorithms do not consider the impact of rainy weather. The rain marks will obscure the recognition target in the image, which will lead to the performance degradation of the algorithm, a problem that has yet to be solved. In order to improve the accuracy of traffic sign recognition in rainy weather, we propose a rainy traffic sign recognition algorithm. The algorithm in this paper includes two modules. First, we propose an image deraining algorithm based on the Progressive multi-scale residual network (PMRNet), which uses a multi-scale residual structure to extract features of different scales, so as to improve the utilization rate of the algorithm for information, combined with the Convolutional long-short term memory (ConvLSTM) network to enhance the algorithm's ability to extract rain mark features. Second, we use the CoT-YOLOv5 algorithm to recognize traffic signs on the recovered images. In this paper, in order to improve the performance of YOLOv5 (You-Only-Look-Once, YOLO), the 3 × 3 convolution in the feature extraction module is replaced by the Contextual Transformer (CoT) module to make up for the lack of global modeling capability of Convolutional Neural Network (CNN), thus improving the recognition accuracy. The experimental results show that the deraining algorithm based on PMRNet can effectively remove rain marks, and the evaluation indicators Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index Measure (SSIM) are better than the other representative algorithms. The mean Average Precision (mAP) of the CoT-YOLOv5 algorithm on the TT100k datasets reaches 92.1%, which is 5% higher than the original YOLOv5.

References

[1]	D. Chattaraj, B.Bera, A. Das, S. Saha, P. Lorenz, Y. Park, Block-CLAP: Blockchain-Assisted certificateless key agreement protocol for internet of vehicles in smart transportation, IEEE Trans. Veh. Technol., 70 (2021), 8092–8107. https://doi.org/10.1109/TVT.2021.3091163 doi: 10.1109/TVT.2021.3091163
[2]	C. Chang, H. Lina, S. Huang, Traffic sign detection and recognition for driving assistance system, Adv. Image Video Process., 6 (2018). https://doi.org/10.14738/aivp.63.4603 doi: 10.14738/aivp.63.4603
[3]	A. Madhu, V. S. Nair, Traffic sign detection and recognition for automated driverless cars based on SSD, Int. J. Trend Sci. Res. Dev., 4 (2020).
[4]	C. Gerhardt, W. Broll, Neural network-based traffic sign recognition in 360° images for semi-automatic road maintenance inventory, in 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), (2020). https://doi.org/10.1109/ITSC45102.2020.9294610
[5]	H. Li, D. Wang, J. Zhang, Z, Li, T. Ma, Image super-resolution reconstruction based on multi-scale dual-attention, Connect. Sci., (2022). https://doi.org/10.1080/09540091.2023.2182487 doi: 10.1080/09540091.2023.2182487
[6]	H. Li, L. Hu, J. Zhang, Irregular mask image inpainting based on progressive generative adversarial networks, Imaging Sci. J., (2023), 1–14. https://doi.org/10.1080/13682199.2023.2180834 doi: 10.1080/13682199.2023.2180834
[7]	J. Zhang, Q. Yan, X. Zhu, K. Yu, Using synthetic data for person tracking under adverse weather conditions, Digital Commun. Networks, 8 (2022), 1–86. https://doi.org/10.1016/j.dcan.2022.08.002 doi: 10.1016/j.dcan.2022.08.002
[8]	A. Kerim, U. Celikcan, E. Erdem, A. Erdem, Using synthetic data for person tracking under adverse weather conditions, Image Vision Comput., 111 (2021), 104187. https://doi.org/10.1016/j.imavis.2021.104187 doi: 10.1016/j.imavis.2021.104187
[9]	S. Huang, Q. Hoang, T. Le, SFA-Net: A selective features absorption network for object detection in rainy weather conditions, IEEE Trans. Neural Networks Learn. Syst., (2022), 2162–2388. https://doi.org/10.1109/TNNLS.2021.3125679 doi: 10.1109/TNNLS.2021.3125679
[10]	S. Di, Q. Feng, C. Li, M. Zhang, H. Zhang, S. Elezovikj, et al., Rainy night scene understanding with near scene semantic adaptation, IEEE Trans. Intell. Trans. Syst., 22 (2021), 1594–1602. https://doi.org/10.1109/TITS.2020.2972912 doi: 10.1109/TITS.2020.2972912
[11]	S. Kim, J. Lee, T. Yoon, Road surface conditions forecasting in rainy weather using artificial neural networks, Safety Sci., 140 (2021), 0925–7535. https://doi.org/10.1016/j.ssci.2021.105302 doi: 10.1016/j.ssci.2021.105302
[12]	R. R. Boukhriss, E. Fendri, M. Hammami, Moving object detection under different weather conditions using full-spectrum light sources, Pattern Recognit. Lett., 129 (2020), 0925–7535. https://doi.org/10.1016/j.ssci.2021.105302 doi: 10.1016/j.ssci.2021.105302
[13]	W. Yang, R. T. Tan, S. Wang, Y. Fang, J. Liu, Single image deraining: From model-based to data-driven and beyond, IEEE Trans. Pattern Anal. Mach. Intell., 43 (2021), 4059–4077. https://doi.org/10.1109/TPAMI.2020.2995190 doi: 10.1109/TPAMI.2020.2995190
[14]	L. J. Deng, T. Z. Huang, X. L. Zhao, T. X. Jiang, A directional global sparse model for single image rain removal, Appl. Math. Model., 59 (2018), 662–679. https://doi.org/10.1016/j.apm.2018.03.001 doi: 10.1016/j.apm.2018.03.001
[15]	H. Wang, Q. Xie, Q. Zhao, D. Meng, A model-driven deep neural network for single image rain removal, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 59 (2020), 3103–3112. https://doi.org/10.1109/CVPR42600.2020.00317
[16]	X. Wang, Z. Li, H. Shan, Z. Tian, W. Zhou, FastDerainNet: A deep learning algorithm for single image deraining, IEEE Access, 8 (2020), 127622–127630. https://doi.org/10.1109/ACCESS.2020.3008324 doi: 10.1109/ACCESS.2020.3008324
[17]	X. Li, J. Wu, Z. Lin, L. Hong, H. Zha, Recurrent squeeze-and-excitation context aggregation net for single image deraining, in Proceedings of the European conference on computer vision (ECCV), 11211 (2020), 262–277. https://doi.org/10.48550/arXiv.1807.05698
[18]	D. Ren, W. Zuo, Q. Hu, P. Zhu, D. Meng, Progressive image deraining networks: A better and simpler baseline, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, (2019), 3937–3946. https://doi.org/10.1109/CVPR.2019.00406
[19]	S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, M. H. Yang, et al., Multi-Stage progressive image restoration, in 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 129 (2021), 14821–14831. https://doi.org/10.1109/CVPR46437.2021.01458
[20]	L. Wang, X. Xu, R. Gui, R. Yang, F. Pu, Learning rotation domain deep mutual information using convolutional LSTM for unsupervised PolSAR image classification, Remote Sens., 12 (2020). https://doi.org/10.3390/rs12244075 doi: 10.3390/rs12244075
[21]	S. Luo, L. Yu, Z. Bi, Y. Li, Traffic sign detection and recognition for intelligent transportation systems: a survey, J. Int. Technol., 21 (2021), 1773–1784. https://doi.org/10.3966/160792642020112106018 doi: 10.3966/160792642020112106018
[22]	X. Li, Z. Xie, X. Deng, Y. Wu, Y. Pi, Traffic sign detection based on improved faster R-CNN for autonomous driving, J. Supercomput., 78 (2022), 7982–8002. https://doi.org/10.1007/s11227-021-04230-4 doi: 10.1007/s11227-021-04230-4
[23]	D. Tabernik, D. Skočaj, Deep learning for large-scale traffic-sign detection and recognition, IEEE Trans. Intell. Trans. Syst., 4 (2020), 1427–1440. https://doi.org/10.1016/j.patrec.2022.06.006 doi: 10.1016/j.patrec.2022.06.006
[24]	J. Du, Understanding of object detection based on CNN family and YOLO, J. Phys. Conf. Ser., 1004 (2018), 012029. https://doi.org/10.1088/1742-6596/1004/1/012029 doi: 10.1088/1742-6596/1004/1/012029
[25]	W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C. Fu, et al., Ssd: Single shot multibox detector, in Computer Vision–ECCV 2016: 14th European Conference, (2016), 21–37. https://doi.org/10.1007/978-3-319-46448-0_2
[26]	J. Wu, S. Liao, Traffic sign detection based on SSD combined with receptive field module and path aggregation network, Comput. Intell. Neurosci., 129 (2022), 1–13. https://doi.org/10.1155/2022/4285436 doi: 10.1155/2022/4285436
[27]	J. Redmon, A. Farhadi, YOLOv3: An incremental improvement, 2018, preprint, arXiv: 0707.0078.
[28]	A. Bochkovskiy, C. Y. Wang, H. Y. M. Liao, Yolov4: Optimal speed and accuracy of object detection, 2020, preprint, arXiv: 2004.10934.
[29]	D. Snegireva, A. Perkova, Traffic sign recognition application using Yolov5 architecture, in 2021 International Russian Automation Conference (RusAutoCon), (2021), 112–126. https://doi.org/10.1109/RusAutoCon52004.2021.9537355
[30]	A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, et al., An image is worth 16x16 words: Transformers for image recognition at scale, 2020, preprint, arXiv: 2010.11929.
[31]	A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, et al., Attention is all you need, Adv. Neural Inform. Process. Syst., 30 (2017). https://doi.org/10.48550/arXiv.1706.03762 doi: 10.48550/arXiv.1706.03762
[32]	Y. Li, T. Yao, Y. Pan, T. Mei, Contextual transformer networks for visual recognition, IEEE Trans. Pattern Anal. Machine Intell., 45 (2022), 1489–1500. https://doi.org/10.1109/TPAMI.2022.3164083 doi: 10.1109/TPAMI.2022.3164083
[33]	K. Huang, C. Tian, J. Su, J. C. Lin, Transformer-based cross reference network for video salient object detection, Pattern Recognit. Lett., 160 (2022), 122–127. https://doi.org/10.1016/j.patrec.2022.06.006 doi: 10.1016/j.patrec.2022.06.006
[34]	J. Zhou, J. Liu, J. Li, M. Huang, S. A. Nawaz, Mixed attention densely residual network for single image super-resolution, Comput. Syst. Sci. Eng., 39 (2021), 133–146. https://doi.org/10.32604/csse.2021.016633 doi: 10.32604/csse.2021.016633
[35]	S. Bande, V. Bhatia, S. Prakash, MSE-based analysis of circular grating self-images for testing beam collimation, Appl. Opt., 59 (2020), 7160–7168. https://doi.org/10.1364/AO.395348 doi: 10.1364/AO.395348
[36]	H. Rezatofighi, N. Tsoi, J. Y. Gwak, A. Sadeghian, S. Savarese, Generalized intersection over union: A metric and a loss for bounding box regression, in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), (2019), 658–666. https://doi.org/10.1109/CVPR.2019.00075
[37]	W. Ma, T. Zhou, J. Qin, Q. Zhou, Z. Cai, Joint-attention feature fusion network and dual-adaptive NMS for object detection, Knowl. Based Syst., 241 (2019). https://doi.org/10.1016/j.knosys.2022.108213 doi: 10.1016/j.knosys.2022.108213
[38]	W. Yang, R. T. Tan, J. Feng, J. Liu, Z. Guo, S. Yan, Deep joint rain detection and removal from a single image, in Proceedings of the IEEE conference on computer vision and pattern recognition, (2017), 1357–1366. https://doi.org/10.1109/CVPR.2017.183
[39]	H. Zhang, V. Sindagi, V. M. Patel, Image de-raining using a conditional generative adversarial network, IEEE Trans. Circuits Syst. Video Technol., 30 (2020), 3943–3956. https://doi.org/10.1109/TCSVT.2019.2920407 doi: 10.1109/TCSVT.2019.2920407
[40]	C. Sun, M. Wen, K. Zhang, P. Meng, R. Cui, Traffic sign detection algorithm based on feature expression enhancement, Multimedia Tools Appl., 80 (2021), 33593–33614. https://doi.org/10.1007/s11042-021-11413-x doi: 10.1007/s11042-021-11413-x
[41]	J. Yan, S. Chen, Y. Zhang, X. Li, Neural architecture search for compressed sensing magnetic resonance image reconstruction, Comput. Med. Imaging Graphics, 85 (2020), 101784. https://doi.org/10.1016/j.compmedimag.2020.101784 doi: 10.1016/j.compmedimag.2020.101784
[42]	M. Malarvel, G. Sethumadhavan, P. C. R. Bhagi, S. Kar, T. Saravanan, A. Krishnan, Anisotropic diffusion based denoising on X-radiography images to detect weld defects, Digital Signal Process., 68 (2017), 112–126. https://doi.org/10.1016/j.dsp.2017.05.014 doi: 10.1016/j.dsp.2017.05.014
[43]	J. H. Shi, H. Y. Lin, A vision system for traffic sign detection and recognition, in 2017 IEEE 26th International Symposium on Industrial Electronics (ISIE), (2017), 1596–1601. https://doi.org/10.1109/ISIE.2017.8001485

Reader Comments

Your name:*

Email:*
© 2023 the Author(s), licensee AIMS Press. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0)