Maritime ship recognition based on convolutional neural network and linear weighted decision fusion for multimodal images

Yongmei Ren; Xiaohu Wang; Jie Yang; Yongmei Ren; Xiaohu Wang; Jie Yang

doi:10.3934/mbe.2023823

Mathematical Biosciences and Engineering

2023, Volume 20, Issue 10: 18545-18565. doi: 10.3934/mbe.2023823

Previous Article Next Article

Research article Special Issues

Maritime ship recognition based on convolutional neural network and linear weighted decision fusion for multimodal images

1.
School of Electrical and Information Engineering, Hunan Institute of Technology, Hengyang 421002, China
2.
College of Intelligent Manufacturing and Mechanical Engineering, Hunan Institute of Technology, Hengyang 421002, China
3.
Hubei Key Laboratory of Broadband Wireless Communication and Sensor Networks, School of Information Engineering, Wuhan University of Technology, Wuhan 430070, China

Academic Editor: Danilo Pelusi

Received: 21 July 2023 Revised: 02 September 2023 Accepted: 11 September 2023 Published: 27 September 2023

Ship images are easily affected by light, weather, sea state, and other factors, making maritime ship recognition a highly challenging task. To address the low accuracy of ship recognition in visible images, we propose a maritime ship recognition method based on the convolutional neural network (CNN) and linear weighted decision fusion for multimodal images. First, a dual CNN is proposed to learn the effective classification features of multimodal images (i.e., visible and infrared images) of the ship target. Then, the probability value of the input multimodal images is obtained using the softmax function at the output layer. Finally, the probability value is processed by linear weighted decision fusion method to perform maritime ship recognition. Experimental results on publicly available visible and infrared spectrum dataset and RGB-NIR dataset show that the recognition accuracy of the proposed method reaches 0.936 and 0.818, respectively, and it achieves a promising recognition effect compared with the single-source sensor image recognition method and other existing recognition methods.
- maritime ship recognition,
- convolutional neural network,
- linear weighted decision fusion,
- visible images,
- infrared images
Citation: Yongmei Ren, Xiaohu Wang, Jie Yang. Maritime ship recognition based on convolutional neural network and linear weighted decision fusion for multimodal images[J]. Mathematical Biosciences and Engineering, 2023, 20(10): 18545-18565. doi: 10.3934/mbe.2023823

Related Papers:

Abstract

Ship images are easily affected by light, weather, sea state, and other factors, making maritime ship recognition a highly challenging task. To address the low accuracy of ship recognition in visible images, we propose a maritime ship recognition method based on the convolutional neural network (CNN) and linear weighted decision fusion for multimodal images. First, a dual CNN is proposed to learn the effective classification features of multimodal images (i.e., visible and infrared images) of the ship target. Then, the probability value of the input multimodal images is obtained using the softmax function at the output layer. Finally, the probability value is processed by linear weighted decision fusion method to perform maritime ship recognition. Experimental results on publicly available visible and infrared spectrum dataset and RGB-NIR dataset show that the recognition accuracy of the proposed method reaches 0.936 and 0.818, respectively, and it achieves a promising recognition effect compared with the single-source sensor image recognition method and other existing recognition methods.

References

[1]	L. Huang, F. X. Wang, Y. L. Zhang, Q. X. Xu, Fine-grained ship classification by combining CNN and Swin transformer, Remote Sens., 14 (2022), 3087. https://doi.org/10.3390/rs14133087 doi: 10.3390/rs14133087
[2]	T. Mustaqim, H. Tsaniya, F. A. Adhiyaksa, N. Suciati, Wavelet transformation and local binary pattern for data augmentation in deep learning-based face recognition, in Proceedings of 10th International Conference on Information and Communication Technology, (2022), 362–367. https://doi.org/10.1109/ICoICT55009.2022.9914875
[3]	Z. M. Zhuang, Z. J. Guo, Y. Yuang, Research on video target tracking technology based on improved SIFT algorithm, in Proceedings of 7th International Conference on Electronics and Information Engineering, (2016), 17–18. https://doi.org/10.1117/12.2265460
[4]	K. Sharma, P. K. Sarangi, L. Rani, G. Singh, A. K. Sahoo, B. P. Rath, Handwritten digit classification using HOG features and SVM classifier, in Proceedings of 2nd International Conference on Advance Computing and Innovative Technologies in Engineering, (2022), 2071–2074. https://doi.org/10.1109/ICACITE53722.2022.9823782
[5]	K. K. Tang, Y. X. Ma, D. R. B. Miao, S. Peng, Z. Q. Gu, Decision fusion networks for image classification, IEEE Trans. Neural Netw. Learn. Syst., (2022), 1–14. https://doi.org/10.1109/TNNLS.2022.3196129 doi: 10.1109/TNNLS.2022.3196129
[6]	Z. Ma, G. D. Huang, Image recognition and analysis: A complex network-based approach, IEEE Access, 10 (2022), 109537–109543. https://doi.org/10.1109/ACCESS.2022.3213675 doi: 10.1109/ACCESS.2022.3213675
[7]	M. Xu, Z. Wang, X. M. Liu, L. H. Ma, A. Shehzad, An efficient pedestrian detection for realtime surveillance systems based on modified YOLOv3, IEEE J. Radio Freq. Identif., 6 (2022), 972–976. https://doi.org/10.1109/JRFID.2022.3212907 doi: 10.1109/JRFID.2022.3212907
[8]	T. W. Zhang, X. L. Zhang, J. Shi, S. J. Wei, A HOG feature fusion method to improve CNN-based SAR ship classification accuracy, in Proceedings of IEEE International Geoscience and Remote Sensing Symposium, (2021), 11–16. https://doi.org/10.1109/IGARSS47720.2021.9553192
[9]	M. Z. Xu, Z. X. Yao, X. P. Kong, Y. C. Xu, Ships classification using deep neural network based on attention mechanism, in Proceedings of 2021 IEEE/OES China Ocean Acoustics, (2021), 1052–1055. https://doi.org/10.1109/COA50123.2021.9519897
[10]	Z. Z. Li, B. J. Zhao, L. B. Tang, Z. Li, F. Feng, Ship classification based on convolutional neural networks, J. Eng., 21 (2019), 7343–7346. https://doi.org/10.1049/joe.2019.0422 doi: 10.1049/joe.2019.0422
[11]	J. W. Li, C. W. Qu, J. Q. Shao, Ship detection in SAR images based on an improved faster R-CNN, in Proceedings of 2017 SAR in Big Data Era: Models, Methods and Applications, (2017), 1–6. https://doi.org/10.1109/BIGSARDATA.2017.8124934
[12]	Y. Y. Wang, C. Wang, H. Zhang, C. Zhang, Q. Y. Fu, Combing single shot multibox detector with transfer learning for ship detection using Chinese Gaofen-3 images, in Proceedings of 2017 Progress in Electromagnetics Research Symposium-fall, (2017), 712–716. https://doi.org/10.1109/PIERS-FALL.2017.8293227
[13]	Y. Y. Wang, C. Wang, H. Zhang, Combining a single shot multibox detector with transfer learning for ship detection using sentinel-1 SAR images, Remote Sens. Lett., 9 (2018), 780–788. https://doi.org/10.1080/2150704X.2018.1475770 doi: 10.1080/2150704X.2018.1475770
[14]	M. Rostami, S. Kolouri, E. Eaton, K. Kim, Deep transfer learning for few-shot SAR image classification. Remote Sens., 11 (2019), 1374. https://doi.org/10.3390/rs11111374 doi: 10.3390/rs11111374
[15]	V. Ganesh, J. Kolluri, A. R. Maada, M. H. Ali, R. Thota, S. Nyalakonda, Real-time video processing for ship detection using transfer learning, in Proceedings of Third International Conference on Image Processing and Capsule Networks, (2022), 685–703. https://doi.org/10.1007/978-3-031-12413-6_54
[16]	Q. Q. Shi, W. Li, R. Tao, X. Sun, L. R. Gao, Ship classification based on multifeature ensemble with convolutional neural network, Remote Sens., 11 (2019), 419. https://doi.org/10.3390/rs11040419 doi: 10.3390/rs11040419
[17]	N. K. Mishra, A. Kumar, K. Choudhury, Deep convolutional neural network based ship images classification, Def. Sci. J., 71 (2021), 200–208. https://doi.org/10.14429/dsj.71.16236 doi: 10.14429/dsj.71.16236
[18]	C. W. Wang, J. F. Pei, S. Y. Luo, W. B. Huo, Y. L. Huang, Y. Zhang, et al., SAR ship target recognition via multiscale feature attention and adaptive-weighed classifier, IEEE Geosci. Remote Sens. Lett., 20 (2023), 4003905. https://doi.org/10.1109/LGRS.2023.3259971 doi: 10.1109/LGRS.2023.3259971
[19]	F. Ucar, D. Korkmaz, A novel ship classification network with cascade deep features for line‑of‑sight sea data, Mach. Vision Appl., 32 (2021), 73. https://doi.org/10.1007/s00138-021-01198-2 doi: 10.1007/s00138-021-01198-2
[20]	K. Aziz, F. Bouchara, Multimodal deep learning for robust recognizing maritime imagery in the visible and infrared spectrums, in Proceedings of the International Conference Image Analysis and Recognition 2018, (2018), 235–244. https://doi.org/10.1007/978-3-319-93000-8_27
[21]	Y. Yang, K. F. Ding, Z. Chen, Ship classification based on convolutional neural networks, Ships Offshore Struct., 17 (2022), 2715–2721. https://doi.org/10.1080/17445302.2021.2016271 doi: 10.1080/17445302.2021.2016271
[22]	X. H. Qiu, M. Li, G. M. Deng, L. T. Wang, Multi-layer convolutional features fusion for dual-band decision-level ship recognition, Opt. Precis. Eng., 29 (2021), 183–190. https://doi.org/10.37188/OPE.20212901.0183 doi: 10.37188/OPE.20212901.0183
[23]	Y. H. Zhang, L. G. Li, Application of improved SqueezeNet in ship classification, Transducer Microsyst. Technol., 41 (2022), 150–152+160. https://doi.org/10.13873/J.1000-9787(2022)01-0150-03 doi: 10.13873/J.1000-9787(2022)01-0150-03
[24]	X. Du, J. Wang, Y. Li, B. Tang, Marine ship identification algorithm based on object detection and fine-grained recognition, in Advanced Intelligent Technologies for Industry. Smart Innovation, Systems and Technologies, (eds. K. Nakamatsu, R. Kountchev, S. Patnaik, J. M. Abe and A. Tyugashev), Academic Press, (2022), 207–215. https://doi.org/10.1007/978-981-16-9735-7_19
[25]	Z. L. Zhang, T. Zhang, Z. Y. Liu, P. J. Zhang, S. S. Tu, Y. J. Li, et al., Fine-grained ship image recognition based on BCNN with inception and AM-softmax, Comput. Mater. Continua., 73 (2022), 1527–1539. https://doi.org/10.32604/cmc.2022.029297 doi: 10.32604/cmc.2022.029297
[26]	L. Huang, F. Wang, Y. Zhang, Q. Xu, Fine-grained ship classification by combining CNN and swin transformer, Remote Sens., 14 (2022), 3087. https://doi.org/10.3390/rs14133087 doi: 10.3390/rs14133087
[27]	W. L. Wang, X. D. Yang, B. Y. Zhang, J. S. Ma, P. Zeng, P. Han, Application of lightweight convolutional neural network in ship classification (in Chinese), Laser Optoelectron. Prog., 60 (2023), 73–80. https://doi.org/10.3788/LOP213033 doi: 10.3788/LOP213033
[28]	W. Sun, J. Yan, A CNN based localization and activity recognition algorithm using multi-receiver CSI measurements and decision fusion, in Proceedings of the 2022 International Conference on Computer, Information and Telecommunication Systems, (2022), 1–7. https://doi.org/10.1109/CITS55221.2022.9832983
[29]	W. N. Zhou, L. H. Sun, Z. J. Xu, A real-time detection method for multi-scale pedestrians in complex environment, J. Electron. Inform. Technol., 43 (2021), 2063–2070. https://doi.org/10.11999/JEIT161032 doi: 10.11999/JEIT161032
[30]	J. L. Guo, Q. Liu, E. Q. Chen, A deep reinforcement learning method for multimodal data fusion in action recognition, IEEE Signal Process. Lett., 29 (2022), 120–124. https://doi.org/10.1109/LSP.2021.3128379 doi: 10.1109/LSP.2021.3128379
[31]	M. M. Zhang, J. Choi, K. Daniilidis, M. T. Wolf, C. Kanan, VAIS: A dataset for recognizing maritime imagery in the visible and infrared spectrums, in Proceedings of the 2015 IEEE Computer Vision and Pattern Recognition Workshops, (2015), 10–16. https://doi.org/10.1109/CVPRW.2015.7301291
[32]	M. Brown, S. Süsstrunk, Multi-spectral SIFT for scene category recognition, in Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition, (2011), 177–184. https://doi.org/10.1109/CVPR.2011.5995637
[33]	N. Saqib, K. F. Haque, V. P. Yanambaka, A. Abdelgawad, Convolutional-neural-network-based handwritten character recognition: an approach with massive multisource data, Algorithms, 15 (2022), 129. https://doi.org/10.3390/a15040129 doi: 10.3390/a15040129
[34]	A. Krizhevsky, I. Sutskever, G. E. Hinton, ImageNet classification with deep convolutional neural networks, in Proceedings of the 25th International Conference on Neural Information Processing Systems, (2012), 1097–1105. http://dx.doi.org/10.1145/3065386
[35]	K. Rainey, J. D. Reeder, A. G. Corelli, Convolution neural networks for ship type recognition, in Proceedings of the SPIE 9844, Automatic Target Recognition XXVI, (2016), 17–21. https://doi.org/10.1117/12.2229366
[36]	Q. S. Zhang, W. Li, L. Li, F. Zhang, H. T. Lang, Infrared and visible image fusion classification based on a codebookless model (in Chinese), J. Beijing Univ. Chem. Technol. (Nat. Sci.), 45 (2018), 71–76.
[37]	M. Wei, HSV fusion of near-infrared image and visible image for scene recognition via sparse recognition using intra-class dictionary, Master's thesis, Nanjing University of Posts and Telecommunications, 2019.

Reader Comments

Your name:*

Email:*
© 2023 the Author(s), licensee AIMS Press. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0)