Single hyperspectral image super-resolution using a progressive upsampling deep prior network

Haijun Wang; Wenli Zheng; Yaowei Wang; Tengfei Yang; Kaibing Zhang; Youlin Shang; Haijun Wang; Wenli Zheng; Yaowei Wang; Tengfei Yang; Kaibing Zhang; Youlin Shang

doi:10.3934/era.2024205

Electronic Research Archive

2024, Volume 32, Issue 7: 4517-4542. doi: 10.3934/era.2024205

Previous Article Next Article

Research article

Single hyperspectral image super-resolution using a progressive upsampling deep prior network

1.
School of Mathematics and Statistics, Henan University of Science and Technology, Luoyang 471000, China
2.
Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100094, China
3.
School of Computer Science, Xi'an Polytechnic University, Xi'an 710048, China

Received: 29 March 2024 Revised: 17 June 2024 Accepted: 25 June 2024 Published: 23 July 2024

Hyperspectral image super-resolution (SR) aims to enhance the spectral and spatial resolution of remote sensing images, enabling more accurate and detailed analysis of ground objects. However, hyperspectral images have high dimensional characteristics and complex spectral patterns. As a result, it is critical to effectively leverage the spatial non-local self-similarity and spectral correlation within hyperspectral images. To address this, we have proposed a novel single hyperspectral image SR method based on a progressive upsampling deep prior network. Specifically, we introduced the spatial-spectral attention fusion unit (S²AF) based on residual connections, in order to extract spatial and spectral information from hyperspectral images. Then we developed the group convolutional upsampling (GCU) to efficiently utilize the spatial and spectral prior information inherent in hyperspectral images. To address the challenges posed by the high dimensionality of hyperspectral images and limited training dataset, we implemented a parameter-sharing grouped convolutional upsampling framework within the GCU to ensure model stability and enhance performance. The experimental results on three benchmark datasets demonstrated that the proposed single hyperspectral image SR using a progressive upsampling deep prior network (PUDPN) method effectively improves the reconstruction quality of hyperspectral images and achieves promising performance.
- deep prior network,
- hyperspectral image super-resolution,
- progressive upsampling,
- deep learning,
- spatial-spectral attention
Citation: Haijun Wang, Wenli Zheng, Yaowei Wang, Tengfei Yang, Kaibing Zhang, Youlin Shang. Single hyperspectral image super-resolution using a progressive upsampling deep prior network[J]. Electronic Research Archive, 2024, 32(7): 4517-4542. doi: 10.3934/era.2024205

Related Papers:

Abstract

Hyperspectral image super-resolution (SR) aims to enhance the spectral and spatial resolution of remote sensing images, enabling more accurate and detailed analysis of ground objects. However, hyperspectral images have high dimensional characteristics and complex spectral patterns. As a result, it is critical to effectively leverage the spatial non-local self-similarity and spectral correlation within hyperspectral images. To address this, we have proposed a novel single hyperspectral image SR method based on a progressive upsampling deep prior network. Specifically, we introduced the spatial-spectral attention fusion unit (S²AF) based on residual connections, in order to extract spatial and spectral information from hyperspectral images. Then we developed the group convolutional upsampling (GCU) to efficiently utilize the spatial and spectral prior information inherent in hyperspectral images. To address the challenges posed by the high dimensionality of hyperspectral images and limited training dataset, we implemented a parameter-sharing grouped convolutional upsampling framework within the GCU to ensure model stability and enhance performance. The experimental results on three benchmark datasets demonstrated that the proposed single hyperspectral image SR using a progressive upsampling deep prior network (PUDPN) method effectively improves the reconstruction quality of hyperspectral images and achieves promising performance.

References

[1]	B. Lu, P. D. Dao, J. Liu, Y. He, J. Shang, Recent advances of hyperspectral imaging technology and applications in agriculture, Remote Sens., 12 (2020), 2659. https://doi.org/10.3390/rs12162659 doi: 10.3390/rs12162659
[2]	B. P. Banerjee, S. Raval, P. J. Cullen, UAV-hyperspectral imaging of spectrally complex environments, Int. J. Remote Sens., 41 (2020), 4136–4159. https://doi.org/10.1080/01431161.2020.1714771 doi: 10.1080/01431161.2020.1714771
[3]	M. Shimoni, R. Haelterman, C. Perneel, Hyperspectral imaging for military and security applications: combining myriad processing and sensing techniques, IEEE Geosci. Remote Sens. Mag., 7 (2019), 101–117. https://doi.org/10.1109/MGRS.2019.2902525 doi: 10.1109/MGRS.2019.2902525
[4]	J. M. Bioucas-Dias, A. Plaza, G. Camps-Valls, P. Scheunders, N. Nasrabadi, J. Chanussot, Hyperspectral remote sensing data analysis and future challenges, IEEE Geosci. Remote Sens. Mag., 1 (2013), 6–36. https://doi.org/10.1109/MGRS.2013.2244672 doi: 10.1109/MGRS.2013.2244672
[5]	W. Xie, X. Jia, Y. Li, J. Lei, Hyperspectral image super-resolution using deep feature matrix factorization, IEEE Trans. Image Process., 57 (2019), 6055–6067. https://doi.org/10.1109/TGRS.2019.2904108 doi: 10.1109/TGRS.2019.2904108
[6]	W. Dong, F. Fu, G. Shi, X. Gao, J. Wu, G. Li, et al., Hyperspectral image super-resolution via non-negative structured sparse representation, IEEE Trans. Image Process., 25 (2016), 2337–2352. https://doi.org/10.1109/TIP.2016.2542360 doi: 10.1109/TIP.2016.2542360
[7]	W. Wan, W. Guo, H. Huang, J. Liu, Nonnegative and nonlocal sparse tensor factorization-based hyperspectral image super-resolution, IEEE Trans. Geosci. Remote Sensing, 58 (2020), 8384–8394. https://doi.org/10.1109/TGRS.2020.2987530 doi: 10.1109/TGRS.2020.2987530
[8]	S. C. Park, M. K. Park, M. G. Kang, Super-resolution image reconstruction: a technical overview, IEEE Signal Process. Mag., 20 (2003), 21–36. https://doi.org/10.1109/MSP.2003.1203207 doi: 10.1109/MSP.2003.1203207
[9]	Q. Wei, N. Dobigeon, J. Y. Tourneret, Bayesian fusion of multiband images, IEEE J. Sel. Top. Signal Process., 9 (2015), 1117–1127. https://doi.org/10.1109/JSTSP.2015.2407855 doi: 10.1109/JSTSP.2015.2407855
[10]	N. Yokoya, T. Yairi, A. Iwasaki, Coupled nonnegative matrix factorization unmixing for hyperspectral and multispectral data fusion, IEEE Trans. Geosci. Remote Sensing, 50 (2011), 528–537. https://doi.org/10.1109/TGRS.2011.2161320 doi: 10.1109/TGRS.2011.2161320
[11]	N. Akhtar, F. Shafait, A. Mian, Sparse spatio-spectral representation for hyperspectral image super-resolution, in Proceedings of the European Conference on Computer Vision (ECCV), Springer, (2014), 63–78. https://doi.org/10.1007/978-3-319-10584-0_5
[12]	Y. Zhou, A. Rangarajan, P. D. Gader, An integrated approach to registration and fusion of hyperspectral and multispectral images, IEEE Trans. Geosci. Remote Sensing, 58 (2020), 3020–3033. https://doi.org/10.1109/TGRS.2019.2946803 doi: 10.1109/TGRS.2019.2946803
[13]	S. He, H. Zhou, Y. Wang, W. Cao, Z. Han, Super-resolution reconstruction of hyperspectral images via low rank tensor modeling and total variation regularization, in 2016 IEEE International Geoscience and Remote Sensing Symposium (IGARSS), IEEE, (2016), 6962–6965. https://doi.org/10.1109/IGARSS.2016.7730816
[14]	R. Dian, L. Fang, S. Li, Hyperspectral image super-resolution via non-local sparse tensor factorization, in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, (2017), 3862–3871. https://doi.org/10.1109/CVPR.2017.411
[15]	H. Huang, J. Yu, W. Sun, Super-resolution mapping via multi-dictionary based sparse representation, in 2014 IEEE International Conference on Acoustics, Speech Signal Processing (ICASSP), IEEE, (2014), 3523–3527. https://doi.org/10.1109/ICASSP.2014.6854256
[16]	Q. Li, Q. Wang, X. Li, Mixed 2D/3D convolutional network for hyperspectral image super-resolution, Remote Sens., 12 (2020), 1660. https://doi.org/10.3390/rs12101660 doi: 10.3390/rs12101660
[17]	J. Hou, Z. Zhu, J. Hou, H. Zeng, J. Wu, J. Zhou, Deep posterior distribution-based embedding for hyperspectral image super-resolution, IEEE Trans. Image Process., 31 (2022), 5720–5732. https://doi.org/10.1109/TIP.2022.3201478 doi: 10.1109/TIP.2022.3201478
[18]	S. Mei, X. Yuan, J. Ji, Y. Zhang, S. Wan, Q. Du, Hyperspectral image spatial super-resolution via 3d full convolutional neural network, Remote Sens., 9 (2017), 1139. https://doi.org/10.3390/rs9111139 doi: 10.3390/rs9111139
[19]	J. Jiang, H. Sun, X. Liu, J. Ma, Learning spatial-spectral prior for super-resolution of hyperspectral imagery, IEEE Trans. Comput. Imaging, 6 (2020), 1082–1096. https://doi.org/10.1109/TCI.2020.2996075 doi: 10.1109/TCI.2020.2996075
[20]	Y. Long, X. Wang, M. Xu, S. Zhang, S. Jiang, S. Jia, Dual self-attention swin transformer for hyperspectral image super-resolution, IEEE Trans. Geosci. Remote Sensing, 61 (2023), 5512012. https://doi.org/10.1109/TGRS.2023.3275146 doi: 10.1109/TGRS.2023.3275146
[21]	M. Zhao, J. Ning, J. Hu, T. Li, Attention-driven dual feature guidance for hyperspectral super-resolution, IEEE Trans. Geosci. Remote Sensing, 61 (2023), 5525116. https://doi.org/10.1109/TGRS.2023.3318013 doi: 10.1109/TGRS.2023.3318013
[22]	Y. Li, L. Zhang, C. Dingl, W. Wei, Y. Zhang, Single hyperspectral Image super-resolution with grouped deep recursive residual network, in 2018 IEEE Fourth International Conference on Multimedia Big Data (BigMM), IEEE, (2018), 1–4. https://doi.org/10.1109/BigMM.2018.8499097
[23]	Q. Wei, J. Bioucas-Dias, N. Dobigeon, J. Y. Tourneret, Hyperspectral and multispectral image fusion based on a sparse representation, IEEE Trans. Geosci. Remote Sensing, 53 (2015), 3658–3668. https://doi.org/10.1109/TGRS.2014.2381272 doi: 10.1109/TGRS.2014.2381272
[24]	Y. Xu, Z. Wu, J. Chanussot, Z. Wei, Nonlocal patch tensor sparse representation for hyperspectral image super-resolution, IEEE Trans. Image Process., 28 (2019), 3034–3047. https://doi.org/10.1109/TIP.2019.2893530 doi: 10.1109/TIP.2019.2893530
[25]	X. H. Han, B. Shi, Y. Zheng, Self-similarity constrained sparse representation for hyperspectral image super-resolution, IEEE Trans. Image Process., 27 (2018), 5625–5637. https://doi.org/10.1109/TIP.2018.2855418 doi: 10.1109/TIP.2018.2855418
[26]	L. Zhang, W. Wei, C. Bai, Y. Gao, Y. Zhang, Exploiting clustering manifold structure for hyperspectral imagery super-resolution, IEEE Trans. Image Process., 27 (2018), 5969–5982. https://doi.org/10.1109/TIP.2018.2862629 doi: 10.1109/TIP.2018.2862629
[27]	M. A. Veganzones, M. Simoes, G. Licciardi, N. Yokoya, J. M. BioucasDias, J. Chanussot, Hyperspectral super-resolution of locally low rank images from complementary multisource data, IEEE Trans. Image Process., 25 (2015), 274–288. https://doi.org/10.1109/TIP.2015.2496263 doi: 10.1109/TIP.2015.2496263
[28]	R. Dian, S. Li, Hyperspectral image super-resolution via subspacebased low tensor multi-rank regularization, IEEE Trans. Image Process., 28 (2019), 5135–5146. https://doi.org/10.1109/TIP.2019.2916734 doi: 10.1109/TIP.2019.2916734
[29]	Q. Xie, M. Zhou, Q. Zhao, D. Meng, W. Zuo, Z. Xu, Multispectral and hyperspectral image fusion by ms/hs fusion net, in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, (2019), 1585–1594. https://doi.org/10.1109/CVPR.2019.00168
[30]	Z. W. Pan, H. L. Shen, Multispectral image super-resolution via RGB image fusion and radiometric calibration, IEEE Trans. Image Process., 28 (2019), 1783–1797. https://doi.org/10.1109/TIP.2018.2881911 doi: 10.1109/TIP.2018.2881911
[31]	C. Dong, C. C. Loy, K. He, X. Tang, Learning a deep convolutional network for image super-resolution, in Computer Vision-ECCV 2014, Springer, (2014), 184–199. https://doi.org/10.1007/978-3-319-10593-2_13
[32]	L. Liebel, M. Körner, Single-image super resolution for multispectral remote sensing data using convolutional neural networks, Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci., 41 (2016), 883–890. https://doi.org/10.5194/isprs-archives-XLI-B3-883-2016 doi: 10.5194/isprs-archives-XLI-B3-883-2016
[33]	Y. Yuan, X. Zheng, X. Lu, Hyperspectral image superresolution by transfer learning, IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens., 10 (2017), 1963–1974. https://doi.org/10.1109/JSTARS.2017.2655112 doi: 10.1109/JSTARS.2017.2655112
[34]	S. Woo, J. Park, J. Y. Lee, I. S. Kweon, CBAM: convolutional block attention module, in Proceedings of the European conference on computer vision (ECCV), IEEE, (2018), 3–19.
[35]	V. Lempitsky, A. Vedaldi, D. Ulyanov, Deep image prior, in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, (2018), 9446–9454. https://doi.org/10.1109/CVPR.2018.00984
[36]	O. Sidorov, J. Y. Hardeberg, Deep hyperspectral prior: single-image denoising, inpainting, super-resolution, in 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), IEEE, (2019), 3844–3851. https://doi.org/10.1109/ICCVW.2019.00477
[37]	C. Dong, C. C. Loy, K. He, X. Tang, Image super-resolution using deep convolutional networks, IEEE Trans. Pattern Anal. Mach. Intell., 38 (2016), 295–307. https://doi.org/10.1109/TPAMI.2015.2439281 doi: 10.1109/TPAMI.2015.2439281
[38]	D. Liu, J. Li, Q. Yuan, A spectral grouping and attention-driven residual dense network for hyperspectral image super-resolution, IEEE Trans. Geosci. Remote Sensing, 59 (2021), 7711–7725. https://doi.org/10.1109/TGRS.2021.3049875 doi: 10.1109/TGRS.2021.3049875
[39]	X. Wang, Q. Hu, J. Jiang, J. Ma, A group-based embedding learning and integration network for hyperspectral image super-resolution, IEEE Trans. Geosci. Remote Sensing, 60 (2022), 5541416. https://doi.org/10.1109/TGRS.2022.3217406 doi: 10.1109/TGRS.2022.3217406
[40]	T. Liu, Y. Liu, C. Zhang, L. Yuan, X. Sui, Q. Chen, Hyperspectral image super-resolution via dual-domain network based on hybrid convolution, IEEE Trans. Geosci. Remote Sensing, 62 (2024), 5512518. https://doi.org/10.1109/TGRS.2024.3370107 doi: 10.1109/TGRS.2024.3370107
[41]	S. Chen, L. Zhang, L. Zhang, Cross-scope spatial-spectral information aggregation for hyperspectral image super-resolution, preprint, arXiv: 2311.17340.
[42]	M. Zhang, C. Zhang, Q. Zhang, J. Guo, X. Gao, J. Zhang, Essaformer: efficient transformer for hyperspectral image super-resolution, in 2023 IEEE/CVF International Conference on Computer Vision (ICCV), IEEE, (2023), 23016–23027. https://doi.org/10.1109/ICCV51070.2023.02109
[43]	X. Huang, L. Zhang, A comparative study of spatial approaches for urban mapping using hyperspectral rosis images over pavia city, Int. J. Remote Sens., 30 (2009), 3205–3221. https://doi.org/10.1080/01431160802559046 doi: 10.1080/01431160802559046
[44]	F. Yasuma, T. Mitsunaga, D. Iso, S. K. Nayar, Generalized assorted pixel camera: postcapture control of resolution, dynamic range, and spectrum, IEEE Trans. Image Process., 19 (2010), 2241–2253. https://doi.org/10.1109/TIP.2010.2046811 doi: 10.1109/TIP.2010.2046811
[45]	N. Yokoya, A. Iwasaki, Airborne hyperspectral data over chikusei, Space Appl. Lab., Univ. Tokyo, Tokyo, 5 (2016), 1–6. https://doi.org/10.1109/TIP.2010.2046811 doi: 10.1109/TIP.2010.2046811
[46]	H. Hou, H. Andrews, Cubic splines for image interpolation and digital filtering, IEEE Trans. Acoust. Speech Signal Process., 26 (1978), 508–517. https://doi.org/10.1109/TASSP.1978.1163154 doi: 10.1109/TASSP.1978.1163154
[47]	Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, et al., Swin transformer: hierarchical vision transformer using shifted windows, in 2021 IEEE/CVF International Conference on Computer Vision (ICCV), IEEE, (2021), 9992–10002. https://doi.org/10.1109/ICCV48922.2021.00986

Reader Comments

Your name:*

Email:*
© 2024 the Author(s), licensee AIMS Press. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0)