Few-shot remote sensing scene classification based on multi subband deep feature fusion

Song Yang; Huibin Wang; Hongmin Gao; Lili Zhang; Song Yang; Huibin Wang; Hongmin Gao; Lili Zhang

doi:10.3934/mbe.2023575

Mathematical Biosciences and Engineering

2023, Volume 20, Issue 7: 12889-12907. doi: 10.3934/mbe.2023575

Previous Article Next Article

Research article Special Issues

Few-shot remote sensing scene classification based on multi subband deep feature fusion

1.
College of Computer and Information, Hohai University, Nanjing 211100, China
2.
Faculty of Electronic Information Engineering, Huaiyin Institute of Technology, Huaian 223001, China

Academic Editor: Yiu-ming Cheung

Received: 24 March 2023 Revised: 18 May 2023 Accepted: 25 May 2023 Published: 05 June 2023

Recently, convolutional neural networks (CNNs) have performed well in object classification and object recognition. However, due to the particularity of geographic data, the labeled samples are seriously insufficient, which limits the practical application of CNN methods in remote sensing (RS) image processing. To address the problem of small sample RS image classification, a discrete wavelet-based multi-level deep feature fusion method is proposed. First, the deep features are extracted from the RS images using pre-trained deep CNNs and discrete wavelet transform (DWT) methods. Next, a modified discriminant correlation analysis (DCA) approach is proposed to distinguish easily confused categories effectively, which is based on the distance coefficient of between-class. The proposed approach can effectively integrate the deep feature information of various frequency bands. Thereby, the proposed method obtains the low-dimensional features with good discrimination, which is demonstrated through experiments on four benchmark datasets. Compared with several state-of-the-art methods, the proposed method achieves outstanding performance under limited training samples, especially one or two training samples per class.
- remote sensing scene classification,
- deep feature fusion,
- discriminant correlation analysis,
- discrete wavelet transform
Citation: Song Yang, Huibin Wang, Hongmin Gao, Lili Zhang. Few-shot remote sensing scene classification based on multi subband deep feature fusion[J]. Mathematical Biosciences and Engineering, 2023, 20(7): 12889-12907. doi: 10.3934/mbe.2023575

Related Papers:

Abstract

Recently, convolutional neural networks (CNNs) have performed well in object classification and object recognition. However, due to the particularity of geographic data, the labeled samples are seriously insufficient, which limits the practical application of CNN methods in remote sensing (RS) image processing. To address the problem of small sample RS image classification, a discrete wavelet-based multi-level deep feature fusion method is proposed. First, the deep features are extracted from the RS images using pre-trained deep CNNs and discrete wavelet transform (DWT) methods. Next, a modified discriminant correlation analysis (DCA) approach is proposed to distinguish easily confused categories effectively, which is based on the distance coefficient of between-class. The proposed approach can effectively integrate the deep feature information of various frequency bands. Thereby, the proposed method obtains the low-dimensional features with good discrimination, which is demonstrated through experiments on four benchmark datasets. Compared with several state-of-the-art methods, the proposed method achieves outstanding performance under limited training samples, especially one or two training samples per class.

References

[1]	Y. Yang, S. Newsam, Bag-of-visual-words and spatial extensions for land-use classification, in Proceedings of Sigspatial International Conference on Advances in Geographic Information Systems, (2010), 270–279. https://doi.org/10.1145/1869790.1869829
[2]	Z. Yang, X. Mu, F. Zhao, Scene classification of remote sensing image based on deep network and multi-scale features fusion, Optik, 171 (2018), 287–293. https://doi.org/10.1016/j.ijleo.2018.06.024 doi: 10.1016/j.ijleo.2018.06.024
[3]	T. Hieu, W. Adrian, Remote sensing of coastal hydro-environment with portable unmanned aerial vehicles (pUAVs) a state-of-the-art review, J. Hydro-environ. Res., 37 (2021), 32–45. https://doi.org/10.1016/j.jher.2021.04.003 doi: 10.1016/j.jher.2021.04.003
[4]	G. S. Xia, J. Hu, F. Hu, B. Shi, X. Bai, Y. Zhong, et al., AID: A benchmark data set for performance evaluation of aerial scene classification, IEEE Trans. Geosci. Remote Sens., 55 (2017), 3965–3981, https://doi.org/10.1109/TGRS.2017.2685945 doi: 10.1109/TGRS.2017.2685945
[5]	R. Cinbis, J. Verbeek, C. Schmid, Approximate fisher kernels of non-iid image models for image categorization, IEEE Trans. Pattern Anal. Mach. Intell., 38 (2016), 1084–1098. https://doi.org/10.1109/TPAMI.2015.2484342 doi: 10.1109/TPAMI.2015.2484342
[6]	F. Hu, G. S. Xia, Z. Wang, X. Huang, L. Zhang; H. Sun, Unsupervised feature learning via spectral clustering of multidimensional patches for remotely sensed scene classification, IEEE J. Sel. Topics Appl. Earth Observ. Remote Sens., 8 (2015), 2015–2030. https://doi.org/10.1109/JSTARS.2015.2444405 doi: 10.1109/JSTARS.2015.2444405
[7]	A. Krizhevsky, I. Sutskever, G. E. Hinton, Imagenet classification with deep convolutional neural networks, Commun. ACM, 60 (2017), 84–90. https://doi.org/10.1145/3065386 doi: 10.1145/3065386
[8]	D. Hong, L. Gao, J. Yao, B. Zhang, A. Plaza, J. Chanussot, Graph convolutional networks for hyperspectral image classification, IEEE Trans. Geosci. Remote Sens., 59 (2021). 5966–5978. https://doi.org/10.1109/TGRS.2020.3015157
[9]	K. Simonyan, A. Zisserman, Very deep convolutional networks for large-scale image recognition, preprint, arXiv: 1409.1556. https://doi.org/10.48550/arXiv.1409.1556
[10]	C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, et al., Going deeper with convolutions, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (2015), 1–9.
[11]	K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (2016), 770–778.
[12]	Y. Guo, J. Ji, X. Lu, H. Huo, T. Fang, D. Li, Global-local attention network for aerial scene classification, IEEE Access, 7 (2019), 67200–67212. https://doi.org/10.1109/ACCES2019.2918732 doi: 10.1109/ACCES2019.2918732
[13]	J. Xie, N. He, L. Fang, A. Plaza, Scale-free convolutional neural network for remote sensing scene classification, IEEE Trans. Geosci. Remote Sens., 57 (2019), 6916–6928. https://doi.org/10.1109/TGRS.2019.2909695 doi: 10.1109/TGRS.2019.2909695
[14]	H. Xie, Y. Chen, P. Ghamisi, Remote sensing image scene classification via label augmentation and intra-class constraint, Remote Sens., 13 (2021), 2566–2586. https://doi.org/10.3390/rs13132566 doi: 10.3390/rs13132566
[15]	N. He, L. Fang, S. Li, J. Plaza, A. Plaza, Skip-connected covariance network for remote sensing scene classification, IEEE Trans. Neural Networks Learn. Syst., 31 (2020), 1461–1474. https://doi.org/10.1109/TNNLS.2019.2920374. doi: 10.1109/TNNLS.2019.2920374
[16]	X. Wu, D. Hong, J. Chanussot, Convolutional neural networks for multimodal remote sensing data classification, IEEE Trans. Geosci. Remote Sens., 60 (2022). https://doi.org/10.1109/TGRS.2021.3124913.
[17]	Y. Liu, C. Y. Suen, Y. Liu, L. Ding, Scene classification using hierarchical Wasserstein CNN, IEEE Trans. Geosci. Remote Sens., 57 (2019), 2494–2509. https://doi.org/10.1109/TGRS.2018.2873966 doi: 10.1109/TGRS.2018.2873966
[18]	J. Fang, Y. Yuan, X. Lu, Y. Feng, Robust space-frequency joint representation for remote sensing image scene classification, IEEE Trans. Geosci. Remote Sens., 57 (2019), 7492–7502. https://doi.org/10.1109/TGRS.2019.2913816 doi: 10.1109/TGRS.2019.2913816
[19]	H. Sun, S. Li, X. Zheng, X. Lu, Remote sensing scene classification by gated bidirectional network, IEEE Trans. Geosci. Remote Sens., 58 (2019), 82–96. https://doi.org/10.1109/TGRS.2019.2931801 doi: 10.1109/TGRS.2019.2931801
[20]	N. He, L. Fang, S. Li, A. Plaza, J. Plaza, Remote sensing scene classification using multilayer stacked covariance pooling, IEEE Trans. Geosci. Remote Sens., 56 (2018), 6899–6910. https://doi.org/10.1109/TGRS.2018.2845668 doi: 10.1109/TGRS.2018.2845668
[21]	A. Bahri, S. G. Majelan, S. Mohammadi, M. Noori, K. Mohammadi, Remote sensing image classification via improved crossentropy loss and transfer learning strategy based on deep convolutional neural networks, IEEE Geosci. Remote Sens. Lett., 17 (2020), 1087–1091. https://doi.org/10.1109/LGRS.2019.2937872 doi: 10.1109/LGRS.2019.2937872
[22]	Q. Wang, S. Liu, J. Chanussot, X. Li, Scene classification with recurrent attention of VHR remote sensing images, IEEE Trans. Geosci. Remote Sens., 57 (2019), 1155–1167. https://doi.org/10/1109/TGRS.2018.2864987
[23]	Y. Chen, Y. Li, H. Mao, X. Chai, L. Jiao, A novel deep nearest neighbor neural network for few-shot remote sensing image scene classification, Remote Sens., 15 (2023), 666–684. https://doi.org/10.3390/rs15030666 doi: 10.3390/rs15030666
[24]	N. Jiang, H. Shi, J. Geng, Multi-scale graph-based feature fusion for few-shot remote sensing image scene classification, Remote Sens., 14 (2022), 5550–5568. https://doi.org/10.3390/rs14215550 doi: 10.3390/rs14215550
[25]	W. Huang, Z. Yuan, A. Yang, C. Tang, X. Luo, TAE-Net: Task-adaptive embedding network for few-shot remote sensing scene classification, Remote Sens., 14 (2022), 111–119. https://doi.org/10.3390/rs14010111 doi: 10.3390/rs14010111
[26]	X. Wu, D. Hong, J. Chanussot, UIU-Net: U-Net in U-Net for infrared small object detection, IEEE Trans. Image Process., 32 (2023), 364–376. https://doi.org/10.1109/TIP.2022.3228497 doi: 10.1109/TIP.2022.3228497
[27]	S. Mei, K. Yan, M. Ma, X. Chen, Q. Du, Remote sensing scene classification using sparse representation-based framework with deep feature fusion, IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens., 14 (2021), 5867–5878. https://doi.org/10.1109/JSTARS.2021.3084441 doi: 10.1109/JSTARS.2021.3084441
[28]	Q. Zeng, J. Geng, K. Huang, W. Jiang, J. Guo, Prototype calibration with feature generation for few-shot remote sensing image scene classification, Remote Sens., 13 (2021), 2728–2747. https://doi.org/10.3390/rs13142728 doi: 10.3390/rs13142728
[29]	S. Yang, H. Wang, H. Gao, L. Zhang, Feature fusion method based on discriminant correlation analysis for land use classification with few-shot, in proceedings of the International Conference on Computer Engineering and Artificial Intelligence, (2022), 671–675. https://doi.org/10.1109/ICCEAI55464.2022.00143
[30]	Q. Chen, Z. Chen, W. Luo, Feature transformation for cross-domain few-shot remote sensing scene classification, preprint, arXiv: 2203.02270. https://doi.org/10.48550/arXiv.2203.02270
[31]	H. Y. Tseng, H. Y. Lee, J. B. Huang, M. Yang, Cross-domain few-shot classification via learned feature-wise transformation, preprint, arXiv: 2001.08735. https://doi.org/10.48550/arXiv.2001.08735
[32]	W. Chen, Y. Liu, Z. Kira, Y. Wang, J. Huang, A closer look at few-shot classification, preprint, arXiv: 1904.04232. https://doi.org/10.48550/arXiv.1904.04232
[33]	A. Chowdhury, M. Jiang, C. Jermaine, Few-shot image classification: Just use a library of pre-trained feature extractors and a simple classifier, in Proceedings of the IEEE/CVF International Conference on Computer Vision, (2021), 9425–9434.
[34]	L. Li, J. Han, X. Yao, G. Cheng, L. Guo, DLA-MatchNet for few-shot remote sensing image scene classification, IEEE Trans. Geosci. Remote Sens., 59 (2021), 7844–7853. https://doi.org/10.1109/TGRS.2020.3033336 doi: 10.1109/TGRS.2020.3033336
[35]	W. Li, L. Wang, J. Xu, J. Huo, Y. Gao, J. Luo, Revisiting local descriptor based image-to-class measure for few-shot learning, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, (2019), 7253–7260.
[36]	R. Cao, L. Fang, T. Lu, N. He, Self-attention-based deep feature fusion for remote sensing scene classification, IEEE Geosci. Remote Sens. Lett., 18 (2020), 43–47. https://doi.org/10.1109/LGRS.2020.2968550 doi: 10.1109/LGRS.2020.2968550
[37]	J. Li, K. Zheng, J. Yao, L.Gao, D. Hong, Deep unsupervised blind hyperspectral and multispectral data fusion, IEEE Geosci. Remote Sens. Lett., 19 (2022). https://doi.org/10.1109/LGRS.2022.3151779
[38]	J. Li, D. Hong, L. Gao, J. Yao, K. Zheng, B. Zhang, et al., Deep learning in multimodal remote sensing data fusion: A comprehensive review, Int. J. Appl. Earth Obs. Geoinf., 112 (2022), 102926. https://doi.org/10.1016/j.jag.2022.102926 doi: 10.1016/j.jag.2022.102926
[39]	D. Hong, L. Gao, N. Yokoya, J. Yao, J. Chanussot, Q. Du, et al., More diverse means better: multimodal deep learning meets remote-sensing imagery classification, IEEE Trans. Geosci. Remote Sens., 59 (2021). 4340–4354. https://doi.org/10.1109/TGRS.2020.3016820
[40]	S. Chaib, H. Liu, Y. Gu, H. Yao, Deep feature fusion for VHR remote sensing scene classification, IEEE Trans. Geosci. Remote Sens., 55 (2017), 4775–4784. https://doi.org/10.1109/TGRS.2017.2700322 doi: 10.1109/TGRS.2017.2700322
[41]	H. Wang, X. Wu, Z. Huang, E. P. Xing, High-frequency component helps explain the generalization of convolutional neural networks, in IEEE Conference on Computer Vision and Pattern Recognition, (2020), 8681–8691.
[42]	M. Haghighat, M. Abdel-Mottaleb, W. Alhalabi, Discriminant correlation analysis: real-time feature level fusion for multimodal biometric recognition, IEEE Trans. Inf. Forensics Secur., 11 (2016), 1984–1996. https://doi.org/10.1109/TIFS.2016.2569061 doi: 10.1109/TIFS.2016.2569061
[43]	C. Chang, C. Lin, LIBSVM: A library for support vector machines, ACM Trans. Intell. Syst. Technol., 2 (2011), 1–27. https://doi.org/10.1145/1961189.1961199 doi: 10.1145/1961189.1961199
[44]	G. Xia, W. Yang, J. Delon, Y. Gousseau, H. Sun, H. Maître, Structural high-resolution satellite image indexing, in ISPRS TC VⅡ Symposium-100 Years ISPRS, (2010), 298–303.
[45]	G. Cheng, J. Han, X. Lu, Remote sensing image scene classification: benchmark and state of the art, in Proceedings of the IEEE, 105 (2017), 1865–1883. https://doi.org/10.1109/JPROC.2017.2675998
[46]	H. Li, Z. Cui, Z. Zhu, L. Chen, J. Zhu, H. Huang, et al., RS-MetaNet: Deep metametric learning for few-shot remote sensing scene classification, IEEE Trans. Geosci. Remote Sens., 59 (2020), 6983–6994. https://doi.org/10.1109/TGRS.2020.3027387 doi: 10.1109/TGRS.2020.3027387
[47]	G. Cheng, L. Cai, C. Lang, X, Yao, J, Chen, L. Guo, et al., SPNet: Siamese-prototype network for few-shot remote sensing image scene classification, IEEE Trans. Geosci. Remote Sens., 60 (2022), 5608011. https://doi.org/10.1109/TGRS.2021.3099033 doi: 10.1109/TGRS.2021.3099033

Reader Comments

Your name:*

Email:*
© 2023 the Author(s), licensee AIMS Press. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0)