Multi-modal brain MRI images enhancement based on framelet and local weights super-resolution

1.   Introduction

Magnetic resonance imaging (MRI) is one of the most important medical imaging technology, it has the following advantages in clinical diagnosis: (1) no ionizing radiation damage; (2) can obtain the body part images that are difficult or inaccessible to other imaging technologies; (3) has better soft tissue resolution than computed tomography (CT); (4) belongs to multi-parameter imaging, which can provide multi-modal diagnostic information. Overall, MRI images show clear anatomical structures and high contrast, and it is also possible to observe pathological tissues with small onset, which is very beneficial for early diagnosis and treatment of brain diseases ^[1,2].

However, when MRI is used to collect images, due to the constraints of some conditions, such as the limited sampling time resulting in the use of a fast imaging method, the collected images are often low-resolution (LR) images. Compared with high-resolution (HR) images, LR images lose many edges and details of biological tissue structures, which is not conducive to subsequent clinical diagnosis. To overcome and compensate for the hardware limitations of MRI scanners, super-resolution reconstruction techniques can be used to recover HR images from the corresponding LR images ^[3,4]. Super-resolution reconstruction technology is an image enhancement method, which improves the spatial resolution of the image through a post-processing algorithm, and optimizes the image edge structure and details.

The super-resolution algorithm of MRI images can be roughly divided into single-image super-resolution and multi-contrast image super-resolution. The super-resolution algorithms of MRI images can be roughly divided into single-modal and multi-modal image super-resolution. Single-modal super-resolution can enhance image quality based on a single image or a sequence of images in the same modal ^[5]. Interpolation is the most traditional super-resolution reconstruction method, such as bicubic interpolation ^[6]. Due to its simple and efficient algorithm, it is widely used to improve the resolution of MRI. However, bicubic assumes that the image has regional smoothness in all regions, it damages the high-frequency component of the image and causes image blurring and edge distortion. One way to improve the interpolation effect is to use prior knowledge of the image. The new edge-directed interpolation (NEDI) ^[7] algorithm estimates the local covariance coefficients by using the statistical features of the LR image's edge structure, and uses the coefficients as the linear regression weights of adjacent pixels to interpolate the image. NEDI improves edge interpolation effects, but is more time-consuming than bicubic. Manjón et al. ^[8] proposed an upsampling model based on the nonlocal similarity of image patches. Wei and Ma ^[9] studied using the edge contrast information of LR images for interpolation, and proposed a contrast-guided interpolation (CGI) algorithm to enhance the sharpness of reconstructed edges through various directional filtering, which is much faster than NEDI. Shi et al. ^[10] and Tourbier et al. ^[11] introduced total variation regularizations to enhance the edge sharpness of the MR image. However, the prior structure information based on a single image is limited, and the enhancement effect of image super-resolution is also limited. Therefore, the image super-resolution method based on dictionary learning is gradually widely used in MR image enhancement. Rueda et al. ^[12] proposed an MR image super-resolution model based on the overcomplete dictionary and sparse representation. Zhang et al. ^[13] combined the sparse representation with nonlocal similarity and sparse derivative prior to improving the edge reconstruction. Jia et al. ^[14] and Jia et al. ^[15] train the overcomplete dictionary from in-plane MR images to reconstruct the isotropic high spatial resolution MR image. Meanwhile, Huang et al. ^[16] use weakly-supervised joint convolutional sparse coding to enhance the resolution of MR images based on a small number of well-registered images of different modalities. Recently, various convolutional networks based on deep learning methods have been widely used for MR image super-resolution ^{[17,18,19,20,21]}, which essentially use deep neural networks to seek better nonlinear mapping from LR images to HR images. However, although these methods can achieve high scores in quantitative evaluation indicators, their common problem is that the models are less explicable and may produce meaningless image contents that do not exist originally ^[22,23].

Multi-modal image super-resolution algorithm is established on the basis of a single-modal image super-resolution algorithm by introducing the structure information of different modal images as the prior knowledge of the reconstruction model. MR images of different modals can be obtained by using different MRI techniques or different software and hardware parameter settings when imaging the same object ^[24,25]. Multi-modal MRI is increasingly being used in complex disease diagnosis and medical research experiments. Different imaging modes can provide specific information about biological tissues, both structural and functional. The synthesis of MR images of various modes is beneficial to more comprehensive observation of the imaging results. Multi-modal image super-resolution uses the structural correlation between different modals to improve the image resolution of one modal by using the information of another modal. Multi-modal images are usually presented as multi-contrast images, among which the most typical ones are T1-weighted and T2-weighted images. The signal generation mechanism of the multi-contrast image can be described by the following signal intensity model ^[26]:

where the signal intensity SI is proportional to the proton density $ \rho $ and further depends on the pulse train parameters repetition time (TR), inversion time (TI) and echo time (TE). In formula (1.1), when TR is long and TE is short, the signal is roughly proportional to $ \rho $. If TR and TE are both short, the signal is dominated by $ \rho(1-e^{-\text{TR/T1}}) $, i.e., T1 weighting image is obtained ^[27]. When TR and TE are both long, the term is dominated by $ \rho e^{{-}\text{TE/T2}} $, i.e., T2 weighting image is obtained. Where "long" and "short" are relative to the typical T1 and T2 constants.

These multi-contrast images have similar anatomical structures due to the same proton density values in corresponding spatial positions, but the difference in signal intensity leads to the difference in contrast. Studies have shown that the structural information shared between multi-contrast images can be used to guide image enhancement and reconstruction tasks such as MRI super-resolution. For example, T2-weighted images can clearly display the edges of many kinds of pathological tissues, but the imaging time is long. T1-weighted images have the advantage of showing normal brain tissue structure, requiring less imaging time, but the lesions are often not as clear as T2-weighted images. In order to solve the contradiction between imaging time and image quality, a compromise method is to collect all the signals of T1-weighted images while only part of the signals of T2-weighted images. In this way, the T2-weighted image obtained by fast imaging has a low resolution, and then the high-resolution T1 image is used as the reference image for super-resolution reconstruction to obtain the high-resolution T2-weighted image.

Rousseau ^[28,29] constructed a super-resolution model for MR images by taking advantage of the similar structure of multi-mode images, and verified the superiority of using different contrast image information for super-resolution compared with single-mode super-resolution. Manjón et al. ^[30] further combined the prior information of the image to be reconstructed and improved the iterative algorithm. Jafari-Khouzani ^[31] proposed a nonlocal means method based on multiple modal features to further improve the edge sharpness of MR images after super-resolution. Lu et al. ^[32] hypothesized that T2-weighted LR image and T1-weighted HR image have similar geometric structures in manifold space and proposed a super-resolution approach via manifold regularized sparse learning. Zheng et al. ^[33] proved that the local regression weights used for interpolation are very similar among T1-weighted and T2-weighted images, and proposed a super-resolution method for T2 images by using the local weights learned from paired T1 images. Then, Zheng et al. ^[34] established an associated model of gradient value between different MRI contrast images, and proposed a new super-resolution method by using gradient information of the HR reference image that has another contrast. In general, the multi-modal image super-resolution reconstruction algorithm makes full use of the local structure similarity of MRI multi-modal image, and introduces the structure information of high-resolution reference image with different modes (different contrast) in the reconstruction process, which makes the reconstruction result more ideal.

This paper proposed a new model for mutli-modal MR image super-resolution based on the work of ^[33]. Considering that in the problem of T2 image super-resolution by using the structural information of T1 image, the method proposed in Zheng et al. ^[33] interpolate T2 using the coefficient matrix calculated from self-defined image edges of T1 image, which can be improved in two aspects: first, it may be inaccurate to define the image edge by gradient threshold; second, the interpolation coefficient matrix can be fine-tuned to better match the internal structural association of T1 and T2 image pairs. Therefore, this paper constructs a modified model of bivariate optimization based on framelet transformation. We use framelet transform to decompose the super-resolution target T2 image into high-frequency and low frequency parts, so that the main energy and edge information of the image can be treated differently during the reconstruction. The advantage of our method lies in that it can more accurately emphasize the weight borrowing of image edges by separating them more finely, and at the same time we carry out global optimization and fine-tuning of the interpolation weights obtained according to T1 image, so as to achieve better super-resolution effects.

2.   Proposed Approach

In this section, we first introduce the framelet transform and regression weights used in the proposed method, then construct our super-resolution model and present the optimization algorithm.

2.1.   Framelet transform and image representation

Framelet transform is a mathematical method that can transform images in space domain and frequency domain ^[35]. To construct framelets, one starts from a compactly supported refinable function (a scaling function) $ \psi\in L^2(\mathbb{R}) $ with a refinement mask (low-pass filter) $ \xi_0 \in L^2(\mathbb{Z}) $ satisfying a refinement equation

Then, for the given compactly supported refinable function, a tight framelet system can be constructed by finding an appropriate set of framelets $ \Phi = \{\phi^1, \cdots, \phi^r\}\subset L^2(\mathbb{R}) $. Let $ \{\xi_1, \cdots, \xi_r\}\subset L^2(\mathbb{Z}) $ be a set of framelet masks (high-pass filters), then the framelets are defined as

Thus, the construction of framelets $ \Phi $ amounts to design a tight framelet filter bank $ \{\xi_0, \xi_1, \cdots, \xi_r\} $ ^[36,37]. The unitary extension principle (UEP) in ^[38] gives the condition for $ \chi(\Phi) $ to form as a tight frame system, i.e., the filter bank $ \{\xi_0, \xi_1, \cdots, \xi_r\} $ satisfies

for almost all $ \omega\in\mathbb{R} $. Here $ \zeta_\xi(\omega) = \sum\limits_l\xi(l)e^{il\omega} $ and $ \delta(\lambda) $ is a delta function. Based on the UEP, a piecewise linear B-spline can be used as the refinable function $ \psi $. The refinement mask is $ \xi_0 = [\frac{1}{4}, \frac{1}{2}, \frac{1}{4}] $, and the two corresponding high-pass filters are

We usually refer to the process of image transformation from spatial domain to frequency domain as framelet decomposition, and vice versa as framelet reconstruction. In the numerical scheme of framelet transform, we use $ \mathcal{F} $ to represent framelet decomposition operator, it decompose the image into low frequency and high frequency coefficients, and can be written as

where $ \mathcal{F}_0 $ denotes the low-pass filter operator that can retain the main information of the image, and $ \mathcal{F}_1 $ consists of remaining band-pass and high-pass filter operators that can separate the edge information of multiple directions of the image. Based on the unitary extension principle (UEP) ^[38], the following equation holds

where $ \mathcal{F}^T $ and $ \mathcal{I} $ are the inverse framelet transform and equivalent transform, respectively.

The framelet decomposition coefficients of an MRI image $ {\bf Y} $ is given by

where $ \mathcal{F}_0{\bf Y} $ is the approximation coefficients, and $ \mathcal{F}_1{\bf Y} $ contains detail coefficients. Accordingly, framelet reconstruction, as the inverse process of decomposition, can be expressed as

Figure 1(c) shows an intuitive example of framelet decomposition on image (b), as we can see that Figure 1(c) contains nine subgraphs after one level decomposition, in which the upper left corner is approximation coefficient and the remaining eight are detail coefficients (we scale the display based on the range of pixel values in each detail subimage for better visual effects).

2.2.   Shared weights in multi-modal MR images

Taking MRI images of T1 and T2 modes as an example, Zheng et al. ^[33] proved that based on NEDI that the local regression weight of one high-resolution image could be used as interpolation coefficient to amplify low-resolution images with different contrast but the same structure in another mode, that is, high-resolution T1 and T2 images shared similar local regression weights. The core idea of ^[33] is to estimate the regression weight from the local edge position of the high-resolution T1 image, and then perform a weighted summation of the four adjacent pixels in the corresponding position in the low-resolution T2 image with the regression weight to obtain a new pixel, which is the interpolation pixel for the target T2 image.

As described in ^[33], the regression weights $ \mathit{\boldsymbol{b}}_i\in\mathbb{R}^{4\times 1} $ estimated from the position of the $ i $-th edge pixel of T1 image is used to interpolate T2 image, and the new pixel $ y_i $ generated in T2 image can be solved by the following formula:

where $ \mathit{\boldsymbol{a}}_i\in\mathbb{R}^{4\times 1} $ contains the four nearest pixels of T2 interpolation image along the diagonal direction at the $ i $-th pixel position. That is, ^[33] proved that in the image edge region, a newly inserted pixel $ y_i $ in T2 image can be represented by a linear combination of its four diagonal neighborhood pixels, while the T1 image with the same structure as T2 can provide the linear regression weights used for interpolation.

2.3.   The proposed model

As mentioned above, the purpose of our model is to recover high-resolution T2-weighted image from low-resolution T2-weighted image $ {\bf L}_\text{T2} $ with the help of high-resolution T1-weighted image. Let's denote $ {\bf X} $ the image magnified by the traditional interpolation method which is the same size as $ {\bf H}_\text{T1} $, and $ {\bf Y} $ as the super-resolution result of our method. We can combine the regression weight vectors of all pixels and write them in matrix form, the initialized regression weighting matrix is denoted as $ {\bf B}\in \mathbb{R}^{P\times P} $, here $ P = M\times N $. When the coordinates of the interpolated pixel is $ (m, n) $, then the $ (m\times n) $-th row of $ {\bf B} $ is the corresponding regression coefficients of $ [b_1, b_2, b_3, b_4] $, and their column coordinates are the pixel numbers of the four diagonal neighborhoods of the current pixel. To allow interpolation weights can be updated and adjusted, we introduce additional variables $ {\bf W}\in \mathbb{R}^{P\times P} $, which is the fine-tuning regression weighting matrix to be solved. Combined with framelet decomposition, our super-resolution model is constructed as the following optimization problem:

where $ \|D{\bf Y}-{\bf L}_\text{T2}\|^2_{\text{F}} $ is the data fidelity term, and $ D $ represents a downsampling operator. Parameters $ \alpha, \beta $ and $ \gamma $ are all non-negative, which are used to balance the constraint terms of super-resolution images and regression weights respectively.

An overview of the proposed approach is summarized in Figure 2. First, the weighting matrix is calculated based on ^[33] for T1-weighted HR image. Second, the T2-weighted LR image is pre-interpolated by the bicubic method. Then, the HR image of interest and its weighted reference image are decomposed by framelet transformation into high-frequency and low-frequency coefficients. Finally, the HR image and the weight matrix are optimized collaboratively based on Eq (2.10).

In particular, the motivation for this model is derived from our experimental observations. By comparing the differences between the ground-truth high-resolution image and the interpolation enhanced image using the method of ^[33], we find that the different values of the high-frequency coefficients is significantly smaller than the low-frequency coefficients. Figure 3(a) displays the average pixel errors of the super-resolution reconstruction result and the HR image on the framelet decomposition coefficients, in which, the first 9 histograms correspond to 9 decomposition subgraphs, and the 10th histograms shows the average error of the 8 detail graphs.

We also illustrate the absolute value of such differences as difference maps in Figure 3(b) and (c), the former shows the reconstruction errors on the low-frequency coefficients between the Figure 1(b) and its interpolated version obtained by ^[33], and the latter shows that on the high-frequency coefficients, darker pixel means a smaller difference. Therefore, this phenomenon prompted us to use wavelet decomposition to separate the super-resolution enhanced image into approximate and detailed components, that is, in the model, through parameter adjustment, more emphasis is placed on the fidelity of high-frequency detail components, while more freedom is given to the low-frequency components of the image.

To sum up, different from ^[33], which only reconstructed the edge pixels of the enlarged LR image, our model considers the global pixel reconstruction and uses framelet decomposition to more finely separate the edge structure of the enlarged LR image in multiple directions. In addition, the regression weights can also be globally fine-tuned in optimization to achieve better results.

2.4.   Numerical algorithm

Our model can be efficiently solved by the alternating direction method of multipliers (ADMM) ^[39]. We introduce an auxiliary variable $ {\bf V} $, and let $ {\bf V} = {\bf WX} $, the optimization problem (2.10) is equivalently written in the following form:

where $ {\bf G} $ is the matrix of the Lagrange multipliers, and $ \rho $ is the penalty parameter.

To solve the super-resolution image $ {\bf Y} $, take the first variation of Eq (2.11) with respect to $ {\bf Y} $ equal to 0:

which actually lead to

where $ K $ is an operator which is defined as

We have the following property for $ K $: assume $ \xi_0 $ is the corresponding low-pass filter of $ \mathcal{F}_0 $, $ \mathcal{T} $ is the fast Fourier transform (FFT), and $ \mathcal{T}(\cdot)^T $ is the complex conjugate of $ \mathcal{T}(\cdot) $, then, $ \mathcal{T}(K) = (\alpha-\beta)\mathcal{T}(\xi_0)^T\mathcal{T}(\xi_0)+\beta $. Thanks to Eq (2.6), we deduce that

Then we have

In numerical scheme, $ \mathcal{T}(\mathcal{F}_0) $ is essentially the $ \mathcal{T}(\xi_0) $. Therefore,

Using FFT, we can obtain the closed form solution of $ {\bf Y} $ from Eq (2.12) directly,

where $ \odot $ is dot product operator, and $ \mathcal{T}^{-1} $ denotes the inverse FFT.

Meanwhile, to solve $ {\bf V} $, similar to $ {\bf Y} $, take the first variation of Eq (2.11) with respect to $ {\bf V} $ equal to 0, we can obtain

where $ S $ is an operator which is defined as

Using FFT, we can obtain the closed form solution of $ {\bf V} $ from Eq (2.19) directly,

To solve the regression weighting matrix $ {\bf W} $, take the first variation of Eq (2.11) with respect to $ {\bf W} $ equal to 0, we can obtain

The last step is to update the Lagrange multiplier $ {\bf G} $ using the fastest descent method, and the update formula is as follows:

Through the iterative update of the above variables, we can get the optimized super-resolution enhanced image $ {\bf Y} $. The stop rule for our algorithm is when the number of iterations $ t $ reaches the maximum $ t_{max} $ or the variable $ {\bf Y} $ reaches the convergence condition $ \|{\bf Y}^{(t)}-{\bf Y}^{(t-1)}\|_\text{F}/\|{\bf Y}^{(t)}\|_\text{F} < \epsilon $. We set $ t_{max} = 300 $ and $ \epsilon = 10^{-4} $ in the following experiments.

Finally, we give some discussions for the computational complexity of the proposed algorithm. The computation of framelet transform $ \mathcal{F} $ is essentially the convolution, thus the complexity of $ \mathcal{F}{\bf Y} $ is $ O(PlogN) $ when using FFT. Besides, since the downsampling operator $ D $ and the T2 reference HR image $ {\bf X} $ are known, the associated calculations can be computed in advance outside of the iterative loop. The computational complexities of the four variables $ \{{\bf{Y}}, {\bf{V}}, {\bf{W}}, {\bf{G}}\} $ that need to be optimized and updated in the iterations are $ O(PlogN+P) $, $ O(PlogN+P) $, $ O(P) $ and $ O(P) $, respectively. Thus, the overall complexity is $ O(PlogN+P) $.

3.   Experiments and analysis

We reiterate that the purpose of this paper is to achieve super-resolution enhancement of low-resolution T2 images by using high-resolution T1 images that can be quickly imaged as prior information, so as to reduce the time required for multi-modal imaging of the same target. In the experiments, we use accurately registered T1 and T2 image pairs as ground-truth, and a simulated data pair and two real data pairs of brain MRI images are used to verify the effectiveness of the proposed method. The synthetic T1 and T2 images shown in Figure 5 are generated from the BrainWeb ^[40] using default parameters and have a spatial resolution of 256 $ \times $ 256 pixels. The first realistic T1 (TR = 2000 ms, TE = 9.7 ms) and T2 (TR = 5000 ms, TE = 97 ms) images that are shown in Figure 6 are acquired by a 3T Siemens Trio Tim MRI scanner (FOV = 230 $ \times $ 187 mm$ ^2 $, slice thickness = 5.0 mm) and have a spatial resolution of 384 $ \times $ 324 pixels. The second realistic T1 (TR = 700 ms, TE = 15 ms) and T2 (TR = 3500 ms, TE = 90 ms) images that are shown in Figure 7 are acquired by a Philips 3T MRI scanner (FOV = 260 $ \times $ 260 mm$ ^2 $, slice thickness = 4.0 mm) and have a spatial resolution of 512 $ \times $ 512 pixels. Besides, in order to simulate low-resolution T2 image, we first smooth the high-resolution T2 image with $ 3\times3 $ Gaussian filter with standard deviation 0.5 and then down-sample it by a factor of two, then the high-resolution T2 image are used as the ground-truth to quantitatively evaluate the performance of our method. These practices are the same as described in ^[33].

As usual, we use Peak Signal-to-Noise Ratio (PSNR) and structural similarity (SSIM) ^[41] to objectively evaluate the image quality of super-resolution results. PSNR and SSIM reflect the global similarity between super-resolution enhanced images and the ground-truth from the perspective of image pixels and image structures respectively. The higher the PSNR value, the closer the reconstructed pixel value is to the pixel value in the HR source image. The higher the SSIM value, the more accurate the reconstructed edge. That is to say, the larger the two values are, the better the super-resolution effect is. In the following experiments, we compare the proposed method with the four methods: bicubic interpolation ^[6], NEDI ^[7], CGI ^[9] and Zheng et al. ^[33], which have similar technical principles, and demonstrate the effectiveness of the proposed approach from both subjective visual effects and objective index evaluations.

For parameter settings, we adopt grid searches to adjust the parameters $ \alpha, \beta $, and $ \gamma $ in Eq (2.10) to make PSNR and SSIM reach the best, while the barrier parameter $ \rho $ is updated iteratively to speed up the convergence (following closely Boyd et al. ^[42]). We found that for the three datasets, selecting the optimal parameters from a fixed parameter set $ \Psi = [0.01, 0.05, 0.1, 0.3, 0.5, 1, 1.5, 3, 5, 10] $ was enough to demonstrate the effectiveness of the proposed method. Taking the Siemens MR image as an example, we observe the influence of model parameters on experimental results according to the numerical changes of quantitative indicators under different parameters. Figure 4 shows the stability of a single parameter's value change on PSNR and SSIM when the other two parameters are fixed as optimal. It can be seen that the experimental results are the least sensitive to $ \gamma $, and desirable results can be achieved as long as $ \alpha $ and $ \beta $ are within the appropriate range, and the influence trends of parameter changes on PSNR and SSIM are basically the same. As for other comparison methods, we also adjusted them to obtain the optimal effects.

The super-resolution visual effects of the three experimental datasets are shown in Figures 5, 6 and 7, respectively. In order to get a better look at the edge enhancement quality of the super-resolution, we give a magnified view of the local detail in the second line of each set of figures.

As we can see from Figure 5, the image enhanced by bicubic interpolation and NEDI algorithm has edge fracture, artifact and blur, while the overall reconstruction quality of CGI is better than bicubic and NEDI, but the recovery effect is not good enough for small structures. The method proposed in ^[33] is an improvement on the basis of CGI, so the edge structure of its reconstructed image is more complete. On the whole, through the visual contrast of local amplification, we can clearly observe that the super-resolution image obtained by our method has the clearest edges and is closest to the HR source image in terms of detail structures. For experiments on real MR images, we can get similar observations in the Philips image comparison in Figure 6, while the visual contrast in Figure 7 is less obvious.

For objective comparison, PSNR and SSIM values obtained by different methods on the three sets of MR images are listed in Table 1. The PSNR values obtained by our method on BrainWeb data and Siemens data are more than 0.6 values higher than the suboptimal method, and the SSIM value is about 0.006 value higher than that of the suboptimal method. Therefore, we can directly observe the differences in the reconstructed edges from the detailed magnification of Figures 5 and 6. For Philips data, however, the proposed method only achieved a PSNR of about 0.1 and SSIM of about 0.001 higher than the suboptimal method, so the visual contrast differences between the results of each method in Figure 7 are not obvious by macroscopic observation.

Besides, all experimental algorithms are implemented with MATLAB on a personal computer with CPU 2.20 GHz and 16 GB memory. The computation time of different methods are shown in Table 2. As can be seen from the table, the proposed method is relatively time-consuming.

To sum up, by comparing two quantitative indicators between different algorithms, we can conclude that subjective visual contrast is basically consistent with objective quantitative evaluation. When the visual contrast difference is not obvious, we rely on quantitative indicators to judge the effectiveness of the algorithm.

4.   Conclusions

In this paper, a super-resolution model for multi-modal MR images is proposed for T1-weighted and T2-weighted image pairs. In order to overcome the inaccuracy of edge localization and fixed interpolation weights of the earlier methods, we use framelet decomposition to finely separate the image edges and detail components, then construct a global interpolation weighting matrix, and uses an iterative algorithm to co-optimize the super-resolution image and interpolation weights. Multiple experimental results show that the proposed method can effectively improve the visual effects and evaluation indexes of the enhanced images. In future work, we will further explore the prior information that can accurately describe the intrinsic correlation of multimodal images, so as to construct more effective image resolution enhancement models. In addition, we will also consider the combination of frequency domain priori and deep learning models to improve the interpretability of the super-resolution results.

Acknowledgments

This work was supported in part by the Zhejiang Provincial Natural Science Foundation of China under Grant No. LQ21F020001, and LQ21A010001, and in part by the National Natural Science Foundation of China under Grant 62101375, and 62006168.

Conflict of interest

The authors declare that there is no conflict of interest.