A novel multi-modal fundus image fusion method for guiding the laser surgery of central serous chorioretinopathy

1.   Introduction

CSCR is a common macular disease characterized by neurosensory retinal detachment (NRD) with or without pigment epithelium detachment (PED) ^[1,2], which often leads to blurred vision, central scotoma and deformed vision, and even results in permanent visual loss in serious cases. At present, laser surgery intervention is an effective measure for the treatment of CSCR, among which the angiography and color fundus images are the key auxiliary tools. As shown in Figure 1, the angiography image can clearly display the location and shape of the leakage areas (hereinafter referred to as lesions), but it has obvious color difference with the actual fundus field of view. On the contrary, the color fundus image cannot provide the location and shape information of the lesions, but it has the advantage of reflecting the actual color of fundus field of vision. In view of the characteristics of multi-modal fundus images, during the laser surgery procedure, ophthalmologists have to read the angiography image carefully to memorize the location and shape of the lesions, and then the corresponding lesion information will be marked on the color fundus image, and the marked positions of which are consistent with the angiography image. With the help of the marked color fundus image, the location and range of lesions in the real field of view are obtained in time by ophthalmologists through memory, which can provide guidance information for the location and area of laser surgery. However, it can be found that the multi-modal fundus images work independently when guiding the laser surgery of CSCR, that is, the ophthalmologists need to process the two kinds of images separately in order to get the final lesion information of real field of view, which not only increases the burden of ophthalmologists, but also causes inaccurate lesion information due to the cross-modal hand-marked deviation. Therefore, by utilizing multi-modal fundus information fusion strategy, we attempt to provide more accurate lesion information for the laser surgery of CSCR and simultaneously reduce the pressure of ophthalmologists. To the best of our knowledge, the multi-modal fundus image fusion for CSCR has not been done yet. Fortunately, the research of natural image fusion and brain image fusion has been widely carried out, which can provide some inspiration for the work of this paper.

Generally, image fusion can be divided into three categories, namely, pixel level, feature level, and decision level, of which the pixel-level image fusion performs directly on the source images and can retain more information. Next, the related research of this fusion scheme is described briefly. The pixel-level image fusion is mainly composed of spatial domain and transform domain ^[3]. In terms of the former, it aims to synthesize an image using techniques such as averaging the source images ^[4], the principal component analysis ^[5], the intensity-hue-saturation transform ^[6] and so on. Although spatial domain methods achieve better results, spectral degradation and contrast reduction ^[7] may be followed. Then the latter, namely transform domain, has emerged as an excellent fusion way to alleviate the above issues. The representation learning-based and multi-scale transform (MST) based methods are typical representatives of this fusion scheme. Yang et al. ^[8] applied the sparse representation (SR) theory to image fusion field. Specifically, the sparse coefficients of source images were firstly gained using an overcomplete dictionary. A choose-max fusion rule was then employed to combine the coefficients. With the help of global dictionary, the final fused image can be reconstructed accordingly. Li et al. ^[9] presented a low-rank representation-based fusion method. The low-rank representation was used instead of sparse representation to extract features from source images, and after that the L1-norm and the same fusion strategy as ^[8] were jointly adopted to reconstruct the fused image. In addition, the MST-based methods have marked predominance in multi-modal image fusion, where the techniques such as discrete wavelet transform (DWT) ^[10], Shearlet ^[11], Contourlet transform (CT) ^[12], NSCT ^[13] and non-subsampled shearlet transform (NSST) ^[14] are frequently applied to decompose the source images into high-frequency and low-frequency components. In particular, the NSCT has achieved great success in multi-modal image fusion by combining with various methods. Yong et al. ^[15] utilized a type-2 fuzzy logic-based rule to fuse the high-frequency components and a local energy algorithm to combine the low-frequency components, and the inverse NSCT was then followed to construct the final brain image. Similarly, a multi-modal image fusion method ^[16] combined with SR and PCNN was raised based on the NSCT decomposition. Especially, they employed the number of ignition times of PCNN to adjust the fusion parameters for the high-frequency components and acquired more detailed information. By applying the maximum entropy of square of the coefficients and the maximum weighted sum-modified laplacian in NSCT domain, respectively, Ganasala et al. ^[17] achieved the fusion of different components successfully. A fuzzy-adaptive reduced PCNN-based fusion way in NSCT domain was presented by Das et al. ^[18], the difference of which was that the fusion of high-frequency components and low-frequency components adopted the same method, namely using PCNN-based strategy with fuzzy-adaptive linking strengths.

NSCT is a shift-invariant, multiscale, and multi-direction expansion with fast implementability ^[19]. However, it cannot be better competent for specific image fusion tasks independently and may cause the degradation of image quality. Therefore, the combination with other fusion techniques like ^{[15,16,17,18]} is a common solution for NSCT and have achieved promising performance. Nevertheless, in order to make the fusion result possess both the content and details of the source images and shield the noise to the greatest extent, manually designing the fusion rules of different components plays an important role in MST-based methods, which makes these methods complicated and tediously. Fortunately, deep learning provides an effective solution for this problem and some research has preliminarily proved its potential. Hou et al. ^[20] put forward an unsupervised image fusion framework for adaptively extracting, fusing and reconstructing the deep features from the source images, which shows the applicability in infrared and visible image fusion task. Li et al. ^[21] also explored the deep learning-based method for image fusion, but only for extracting the multi-layer features of source images. After that the max selection strategy was still carried out on the candidate features generated by the L1-norm and weighted-average rule. Moreover, a fusion scheme ^[22] based on the generative adversarial network (GAN), was intelligently designed. In their method, the idea of generative adversarial training was cleverly integrated into the image fusion process where the fusion parameters of the source images were obtained in an end-to-end way, and achieved gratifying performance.

The aforementioned deep learning methods are beneficial to alleviate the multi-modal image fusion issue in NSCT domain, among which the GAN-based method has given us the most inspiration. In view of the dilemma of information acquisition of CSCR lesions in the laser surgery and the solid research foundation of image fusion mentioned above, this paper attempts to seek an adaptive parameter fusion method without too much manual intervention for multi-modal CSCR fundus image fusion and aims to obtain an assistant tool with the characteristics of both the angiography and color fundus images. The significance of this work is not only to relieve the pressure of ophthalmologists to capture the lesion information of CSCR in traditional laser surgery, but also to further facilitate the safety and efficiency of automatic laser surgery by providing clearer image data. In general, the main contributions of our research are as follows: 1) To the best of our knowledge, the image fusion method is firstly employed in the multi-modal CSCR fundus image scene for promoting to obtain more accurate information of the CSCR lesions, which provides a feasible scheme for better guiding the laser surgery of CSCR. 2) The data set of low-frequency and high-frequency components for the angiography and color fundus images is established though NSCT, which lays the foundation for better boosting the fusion of image content (structure) and detail information in NSCT domain. 3) An improved deep learning-based fusion method is employed to obtain the adaptive fusion parameters of low-frequency components in an end-to-end learning way, which enhances the smoothness of the fused images. Additionally, the PCNN-based fusion scheme is adopted to combine the high-frequency components, which facilitates the integration of detailed information. 4) Qualitative and quantitative experiments are performed on the multi-modal CSCR fundus images to verify the effectiveness of the proposed image fusion method in NSCT domain.

The rest of the paper is organized as follows. Section 2 presents the related work of NSCT, GAN and PCNN. Section 3 describes the details of the proposed image fusion method. Section 4 shows the experiments and discussions. Section 5 concludes this paper.

2.   Related work

2.1.   Non-subsampled contourlet transform

NSCT is a multi-scale transform technique developed on the basis of CT, which can effectively suppress pseudo-Gibbs phenomena ^[23]. Driven by non-subsampled pyramid filter bank (NSPFB) and non-subsampled directional filter bank (NSDFB), NSCT achieves multi-scale and multi-directional successfully ^[13]. Specifically, on the one hand, NSPFB can complete the iterative decomposition task on the lower-frequency component obtained by the previous decomposition, which ensures the multi-scale characteristics of NSCT. And on the other hand, NSDFB performs multi-directional decomposition with l directional levels on the high-frequency components, which can make NSCT obtain 2^l sub-images with more precise directional details on the current scale. Besides, due to the removal of down-sampling and up-sampling operations in the process of multi-scale transform and multi-directional decomposition processes, NSCT has the advantage of shift-invariance property. It is also the above superiorities that make NSCT shine in a variety of image fusion scenes.

2.2.   Generative adversarial network

GAN ^[24] enhances its ability to capture data distribution by skillfully establishing an adversarial game between the generator G and the discriminator D. Since its birth, it has been widely used in various visual tasks ^[25,26]. Yu et al. ^[25] effectively enhanced the capability of cervical cell detection model by introducing images generated from GAN (i.e., data augmentation) when training the convolutional neural network. In the aspect of image fusion, with the help of adversarial training, the integration of infrared and visible images was achieved successfully ^[26]. Whether it is the process of data augmentation or image fusion, the application of GAN are essentially a minimax two-player game between G and D from a mathematical point of view, which can be described as the following formula:

where x is a sample from a real distribution ${{P_{\mathit{data }}}(x)}$ and z denotes a sample from a concrete prior distribution ${{P_z}(z)}$. Because it is difficult to express ${{P_{\mathit{data }}}(x)}$ explicitly, so GAN tries its best to fit the distribution of ${{P_{\mathit{data }}}(x)}$ by fooling D using the generated samples based on z. The whole process can be further described separately as follows:

where ${{D^*}}$ and ${{G^*}}$ represent the optimal G and D, respectively. Alternating optimization drives G to generate more realistic data distribution close to ${{P_{\mathit{data}}}(x)}$, and the capability of D to distinguish true from false is gradually enhanced. Once the adversarial training reaches equilibrium, the final generative model can be applied in the corresponding scene.

2.3.   Pulse-coupled neural network

PCNN is an artificial neural network that simulates the information processing mechanism of mammalian visual cortex, which processes the input signal autonomously through the cooperation of three sub-modules, namely, the receiving field, modulation field and pulse generator module, and has been well implemented in the field of image fusion ^[27,28]. To perform object detection, the PCNN-based image fusion network ^[27] was firstly chosen as an architecture for the fusion of the original and filtered images, and then the target can be highlighted consequently and thus be easily detected. In order to enhance the ability to capture edge information, Tan et al. ^[28] properly introduced multi-scale morphological gradient into the PCNN-based fusion method to improve its performance in remote sensing image fusion task. The same technique is also used in multi-modal brain image fusion task for integrating the information from the fine-structure and coarse-structure layers ^[29]. It can be found that PCNN has performed well in the above fields, especially in the fusion task, which also encourages us to yearn for it in our multi-modal CSCR fundus image fusion scene.

3.   Proposed methodology

The CSCR image fusion scheme proposed in this paper is an innovative application of GAN-based and PCNN-based image fusion methods in NSCT domain. Firstly, NSCT decomposes the CSCR images into high-frequency and low-frequency components, which promotes the smooth fusion of information in the angiography and color fundus images. Secondly, the GAN-based technique is employed for the fusion of low-frequency components, which can determine the adaptive fusion parameters in a learning way and greatly avoid the manual intervention. Simultaneously, PCNN is adopted to fuse the high-frequency components, which can effectively integrate the edge and contour information from the two modal images. Finally, the required fused CSCR image can be obtained by applying a color conversion operation on the fused image Y_AB achieved based on the inverse transform on the above fusion components. The flow chart of the proposed method is shown in Figure 2.

3.1.   Preprocessing

It should be noted that the preprocessing operation needs to be done before the fusion task is formally executed, the parts of which mainly involve the registration of multi-modal fundus images, the clipping of macular region, and the establishment of data set. Generally, registration is used for the alignment of sequence images or multi-modal images to ensure the consistency of spatial information of image content, which is very important for the analysis of medical images ^[30]. Considering the accuracy and efficiency of registration, the scheme in ^[31] is adopted in our CSCR fundus image registration task. After that, in order to speed up the fusion of multi-modal information, the macular region is roughly localized by clipping, which can limit the image fusion range to the occurrence area of CSCR. Following that is the color space conversion and NSCT operation, among which the YIQ color conversion is applied to separate the brightness and color channels and followed by the NSCT operation on the brightness channel to establish the data set based on the high-frequency and low-frequency components. Furthermore, the low-frequency components will be further divided into patches with size of 24 in this paper, which can not only expand the training samples, but also reduce the training cost.

3.2.   Deep learning-based fusion framework

In recent years, deep learning techniques have been involved in various image fusion scenes and achieved good performance ^{[20,21,22,26]}. Among them, the GAN-based fusion schemes ^[22,26] have shown great advantages, which provides a new solution for the determination of adaptive fusion parameters. This section is the work extension of GAN-based image fusion scheme (as shown in Figure 3), which is embodied in three aspects, namely, a new attempt in application domain, a new incorporation in the MST domain and a new adjustment in technique domain. Firstly, we apply the GAN-based fusion scheme to the multi-modal CSCR fundus image information integration task, which broadens its application field and scope. Secondly, the scheme is intriguingly integrated into the MST domain, aiming at capturing the data distribution of the low-frequency components in an end-to-end learning way and driving the generation of adaptive fusion parameters to enhance the smoothness of the fused images. Thirdly, due to the computational cost of NSCT, not all details are filtered into the high-frequency components ^[13], which means that the structure and detailed information need to be processed synchronously in the fusion of low-frequency components. To this issue, the cycle-consistency loss term, inspired by ^[32], is integrated into the deep learning-based fusion scheme to further enhance the learning ability of GAN model for low-frequency component data distribution.

Considering the loss of information, the up-sampling and down-sampling operations in ^[22] are removed in our work and other network structures remain unchanged. Finally, the objective optimization framework corresponding to Eq (2.1) is as follow:

where ${{G_{AB}}}$ s the generator for obtaining the adaptive fusion parameters and generating fusion results, and ${{D_A}}$ and ${{D_B}}$ denote the discriminators that drive ${{G_{AB}}}$ to capture the data distributions of the low- frequency components ${{L_A}}$ and low-frequency component ${{L_B}}$ synchronously. Similarly, corresponding to Eqs (2.2) and (2.3), we define the loss functions for ${{D_A}}$, ${{D_B}}$ and ${{G_{AB}}}$ as follows:

As mentioned above, we adopt the cycle-consistency term to improve the performance of GAN model, that is to say, to enhance the learning ability of the generator ${G_{AB}}$ for low-frequency component data distribution. Specifically, through two generators with the same encoding and decoding structures as ${G_{AB}}$, the fused image is translated back to the corresponding modals respectively to force ${G_{AB}}$ to grasp the data distribution of the source images in detail. The cycle-consistency loss can be written as follows:

where ${{G_A}}$ and ${{G_B}}$ represent the generators of corresponding modals respectively, and ${\left\| \cdot \right\|_1}$ is the L1-norm term that reduces blurs in the fused images. Then, the total loss function can be defined as follows:

where the η, β and λ denote the trade-off parameters adjusting the proportion of the corresponding term, which are set to 0.05, 1.5 and 0.8, respectively. Then, the final optimized framework Eq (3.1) can be rewritten as:

where $G_{AB}^*$, $D_A^*$ and $D_B^*$ represent the optimal ${{G_{AB}}}$, ${{D_A}}$ and ${{D_B}}$ respectively. In practice, the image fusion method based on GAN is not stable in the training process. In order to reduce the model oscillation, this paper follows the training strategy in ^[22], that is, the training quota is allocated according to the performance of the generators and discriminators instead of training in turns.

3.3.   PCNN fusion framework

The existing PCNN model is developed on the basis of biological neuron model ^[33]. Due to simulating the information processing mechanism of mammalian visual cortex, the signal processed by the model is closer to the characteristics of human vision. However, setting a large number of parameters is a defect of the traditional PCNN model. Therefore, a simplified PCNN ^[34] model is introduced into our multi-modal CSCR fundus image fusion research, which is specifically used for the information integration of the high-frequency components in this section. The expressions of the simplified PCNN model can be defined as follows:

where ${S_{ij}^A}$ and ${S_{ij}^B}$ denote the external stimuli determined by the high-frequency components in this paper, and ${F_{ij}^A}$ and ${F_{ij}^B}$ are the feedbacks input of the above inputs respectively. ${L_{ij}^A}$ and ${L_{ij}^B}$ represent the linking parameters of neurons, and the corresponding amplification and attenuation coefficients are $\mathrm{a}_{L}^{A}, V_{L}^{A}, \mathrm{a}_{L}^{B}, V_{L}^{B}$, among which ${\rm{a}}_L^A$ equals ${\rm{a}}_L^B$ and $V_L^A$ equals $V_L^B$ in this paper. ${Y_{kl}^A}$ and ${Y_{kl}^B}$ are the corresponding outputs of neuron (k, l). ${W_{ijkl}^A}$ and ${W_{ijkl}^B}$ are the connection matrixes between neuron (i, j) and neuron (k, l) in the respective low-frequency components, and they can be express as follow:

where $\left(i^{A}, j^{A}\right)$ and $\left(i^{B}, j^{B}\right)$ denote the current neurons in the high-frequency component A and the high-frequency component B respectively, and $\left(k^{A}, l^{A}\right)$ and $\left(k^{A}, l^{A}\right)$ are the corresponding neighbor neurons.

where ${U_{ij}^A}$ and ${U_{ij}^B}$ denote the modulation output results, and ${{\beta ^A}}$ and ${{\beta ^B}}$ are the link strength factors that take the same value.${E_{ij}^A}$ and ${E_{ij}^B}$ are dynamic thresholds used to adjust the firing frequency of neurons, and the corresponding amplification and attenuation coefficients are $\mathrm{a}_{E}^{A}, V_{E}^{A}, \mathrm{a}_{E}^{B}, V_{E}^{B}$, among which ${\rm{a}}_E^A$ equals ${\rm{a}}_E^B$ and $V_E^A$ equals $V_E^B$ in this paper. Then, based on the step function, the pulse value ${Y_{ij}^A}$ and ${Y_{ij}^B}$ of PCNN model can be obtained. Finally, we acquire the fusion map of the high-frequency components by the following formula:

where $F_{A}$ and $F_{B}$ denote the sum of the pulses with N iterations. Based on Eq (3.16), the final fusion result of the high-frequency components can be obtained, that is, $H_{AB}$ shown in Figure 4.

3.4.   The inverse transformation and color space conversion

Through Sections 3.2 and 3.3, the low-frequency components and high-frequency components of multi-modal CSCR fundus images can be integrated by deep learning-based fusion method and PCNN-based fusion method respectively, which greatly reduces the manual intervention of fusion parameters and strengthens the integration of detail information. On this basis, the preliminary fusion results can be obtained by performing the NSCT inverse transform on the low-frequency fusion and high- frequency fusion results, followed by the YIQ-based color space conversion on the above results to obtain the final fusion image based on the RGB color space.

4.   Experimental results and discussions

4.1.   Datasets and experimental settings

In the task of multi-modal CSCR fundus image fusion in this paper, a total of 396 angiography and color fundus image pairs are used to verify the performance of the proposed image fusion method based on NSCT. By relying on the NVIDIA-2080ti GPU with the Tensorflow framework, 356 image pairs are utilized to train the deep learning-based fusion method and the remaining image pairs are used for testing. Moreover, in order to expand the training samples and reduce the training cost, the low-frequency components are further divided into 297,457 patches with the size of 24. Meanwhile, in the training process, the learning rate, batch size and learning decay rate are set to 0.0002, 64 and 0.9 respectively, and the size of all test images is adjusted to 256 to ensure the fusion effect of the whole CSCR area in the test process.

4.2.   Quantitative criteria

Quantitative analysis of image fusion results is one of the important ways to evaluate the fusion methods effectively. In this paper, six commonly used evaluation indexes are applied to analyze the fusion effect of multi-modal CSCR fundus image, namely, average gradient (AG), entropy (EN) ^[35], mutual information (MI) ^[36], Qabf ^[37], Nabf ^[38], structural similarity (SSIM) ^[39]. The details are as follows:

● AG: AG is the average value of the image gradient, which represents the clarity of the fused image and can be expressed as follow:

where F denotes the final fused image, and M and N represent the size of F.

● EN: EN is an objective evaluation to measure the amount of information contained in the fused image. The larger the value is, the more information the fused image carries. The formula is as follow:

where W denotes all gray levels in the final fused image and p(l) represents the proportion of gray value l to all gray levels.

● MI: MI can indicate how much information the fused image carries about the source images, and the greater the MI value is, the better the fusion effect is. The formula of MI can be expressed as follow:

where ${M{I_{AF}}}$ and ${M{I_{BF}}}$ denote the mutual information between the source image A and the fused image F, the source image B and the fused image F, respectively. ${p_{AF}}(l, p)$ and ${p_{BF}}(l, p)$ are the corresponding joint distribution of gray levels.

● Qabf and Nabf: Qabf reflects the quality of the visual information obtained from the input image, a higher value of which means that the quality of the fused image is better. Details of the formula can be found in ^[37]. Nabf represents the noise or artifacts added in fused image due to fusion process, a smaller value of the index indicates a higher image quality. Refer to reference ^[38] for details.

● SSIM: SSIM is a measure to characterize the structural similarity between the fused image and the source images. A larger value means that the fused image is more consistent with the source images in terms of structure. The formula of SSIM is shown as follow:

where ${\mathit{SSI}{\mathit{M}_{AF}}}$ and ${\mathit{SSI}{\mathit{M}_{BF}}}$ denote the structural similarity between the source image A and the fused image F, the source image B and the fused image F, respectively. ${{\mu _A}}$, ${{\mu _B}}$ and ${{\mu _F}}$ are the corresponding mean intensity respectively. ${{\sigma _A}}$ and ${{\sigma _B}}$ are the standard deviations, and ${{\sigma _{AF}}}$ and ${{\sigma _{BF}}}$ denote the covariances between the source images and the fused image F respectively. $C_{1} = \left(K_{1} * 255\right)^{2}$ and $C_{2} = \left(K_{2} * 255\right)^{2}$ are two constants to prevent an unstable results, and K₁ and K₂ are set to 0.01 and 0.03 respectively.

4.3.   Comparison of the proposed method with or without the cycle-consistency module

In this section, we conduct the ablation experiment to verify the effectiveness of the cycle-consistency loss module in the GAN-based fusion method for the low-frequency components. The results of quantitative comparison are shown in Figure 5.

Significantly, in the five evaluation indexes, namely, EN, Qabf, MI, Nabf and SSIM, the deep learning-based fusion scheme equipped with the cycle-consistency loss module has achieved good results in most cases. Especially the MI index, in the absence of the cycle-consistency loss module, the final fusion images obtain less information from the source images, which may lead to the lack of key lesion information and is very unfavourable for the multi-modal CSCR fundus image fusion task. In addition, in terms of the AG index, it can be found that the low-frequency fusion scheme integrated with the cycle-consistency loss module will be slightly inferior to that without the same module, which means a slight difference in image clarity. However, on the basis of other indexes, it can be observed that this is at the cost of image information reduction and noise increase, which is obviously not desirable.

Besides, we show the effect of the cycle-consistency loss module in the fusion scheme based on deep learning through the specific low-frequency fusion experiments. As shown in Figure 6, the first line and the second line are the low-frequency components of the angiography and color fundus images, represented by L_A and L_B respectively. The third line and the fourth line denote the GAN-based fusion method with and without the cycle-consistency loss module respectively, represented by Scheme 1 and Scheme 2. Under the same experimental setup, Scheme 1 can better learn the data distribution of L_A and L_B, and the final low-frequency fusion results not only contain normal structural information, but also better retain the lesion content. But the performance of Scheme 2 is on the contrary. Specifically, the low-frequency fusion images under this scheme are generally not smooth enough, and the fusion quality of the lesion information is poor, which seriously affects the ophthalmologists' judgment of the shape and location of the lesions, and thus cannot guide the CSCR laser surgery well.

As shown in Figure 7, it further conveys the difference of the final fusion results obtained by the proposed fusion method using the above two low-frequency fusion schemes. It can be clearly observed that part of the lesion information is lost in Scheme 2 and the vessels and macular area are not smooth enough, which are consistent with the performance in Figure 6 and confirm the necessity of introducing the cycle-consistency loss module. On the contrary, Scheme 1 not only captures the distribution of lesions in the source images, but also better integrates the normal structural information. More importantly, the transition of each region of the fused image is natural and smooth, which can provide better reference for ophthalmologists to identify the lesion information.

4.4.   Comparison of the fusion strategies for high-frequency components

After the previous section, the importance of the cycle-consistency loss module in the multi-modal CSCR fundus image fusion task is reflected. On this basis, this section further discusses the fusion strategy of high-frequency components, specifically involving the PCNN-based fusion strategy (i.e., strategy 1) and the commonly used choose-max fusion strategy (i.e., strategy 2). On the premise that the GAN-based fusion scheme is the low-frequency component fusion strategy, the results shown in Figure 8 can be obtained by using the above two high-frequency component fusion strategies. Apparently, except for the Nabf index, the PCNN-based fusion strategy can make the proposed fusion method more attractive in the task of this paper. In particular, the three indexes, namely AG, Qabf and MI, show that the strategy can promote the whole fusion method to obtain high-quality and high-definition fusion results which inherit more information from the source images. Admittedly, the choose-max fusion strategy can indeed make the fusion results avoid too much noise, but it cannot make up for the losses of other aspects of the fusion results, such as the image clarity, image quality and the source image information carried by the fused images.

In addition, corresponding to the above analysis, it can be clearly observed from Figure 9 that the fused images obtained by strategy 1 are significantly better than those obtained by strategy 2 in terms of clarity. Moreover, the latter results in a weak contrast of image fusion results, which makes it difficult to distinguish the fundus background area, normal structure and lesion information to a certain extent. On the contrary, the performance of strategy 1 in this aspect is commendable. The above discussions prove the applicability and feasibility of the PCNN-based fusion strategy in the multi-modal CSCR fundus image fusion task in this paper, which also lay the foundation for the combination of the PCNN-based and deep learning-based fusion strategies.

4.5.   Comparison of the proposed and baseline methods

In this part, the proposed multi-modal CSCR fundus image fusion method in NSCT domain is compared with the MST baseline methods, and the fusion techniques of low-frequency components and high-frequency components in the baseline methods adopt the common fusion strategies ^[8,40]. Additionally, both the proposed and baseline methods perform image fusion tasks in the Y channel of YIQ color space and acquire the final fusion images after the same color space conversion operation. The results of quantitative comparison are shown in Figure 10. It can be found that the AG, EN and Qabf values of the proposed method are better than the baseline methods in most cases. Among them, the AG and Qabf indexes have significant advantages, which imply that the fused images are clearer and of good quality. Compared with the Shearlet and NSST fusion methods, it should be pointed out that the proposed fusion scheme is competitive in terms of the MI index. Furthermore, although there are still some gaps in the Nabf and SSIM indexes between our proposed fusion method and some baseline schemes, this slight difference of which also confirms the potential of the proposed method. At the same time, in order to display the fusion performance of multi-modal CSCR fundus images with different fusion methods more conveniently, the average fusion indexes of 40 pairs of CSCR images are specially counted and summarized in Table 1. The quantitative results in the table show that the proposed method is superior to other baseline fusion methods in the average values of AG, EN and Qabf and projects its competitiveness in most cases in other indexes, which is consistent with the above analysis.

After the same color space conversion, the MST baseline methods and the proposed method can obtain the final CSCR fusion images. The specific fusion results are shown in Figure 11. From the perspective of image clarity, the results gained by the proposed fusion method based on NSCT are relatively clear, and the contours of vessels and lesions are easy to identify. From the perspective of contrast, the proposed method can make the final fusion images have better contrast, and make the vessels, macular area and background easy to distinguish. From the perspective of preserving image information, all the methods can achieve the fusion task of the angiography and color fundus images, but on the whole our scheme can better deal with the fusion of structure and detail information of the source images.

Generally speaking, compared with the MST baseline fusion methods, the proposed fusion scheme based on NSCT can be able to better realize the information integration of multi-modal CSCR fundus images, which shows its potential value. The fusion scheme not only innovatively realizes the combination of the deep learning-based and PCNN-based fusion techniques, but also provides a new idea for the research of fusion rules based on MST fusion methods. Moreover, through the information integration of multi-modal fundus images, such an image fusion scheme provides a more convenient auxiliary tool characterized by similar fundus color and clear lesion and structure information for guiding the operation of traditional CSCR laser surgery.

5.   Conclusions

In this paper, a multi-modal CSCR fundus image fusion method in NSCT domain is presented to assist ophthalmologists to identify the CSCR lesion information and normal fundus structure more convenient so as to better guide the laser surgery. This fusion idea is an intriguing combination of the GAN-based and PCNN-based fusion schemes, which is not only the application extension of deep learning-based fusion technique in the low-frequency component fusion task, but also the reappearance of PCNN fusion rule in the information integration field of high-frequency components. On the one hand, in order to alleviate the tedious process of manually designing fusion rules, we introduce the GAN-based method into the fusion scene of low-frequency components and improve its learning ability of the data distribution by integrating the cycle-consistency loss module. On the other hand, the PCNN-based method is employed to promote to fuse the high-frequency components, aiming to enhance the integration of detail information of multi-modal CSCR fundus images. Quantitative and qualitative experiments demonstrate the effectiveness and competitiveness of the proposed method in the fusion scene of CSCR fundus images. Although the proposed fusion method achieves considerable performance and provides a new idea for the research of MST-based fusion methods, we will continue to make efforts from the following two aspects that include: 1) further optimizing the training strategy of the GAN fusion method to improve the training efficiency and 2) further improving the fusion rules of the low-frequency components and high-frequency components to enhance the quality of fused images.

Acknowledgments

This work was financially supported by Fundamental Research Funds for the Central Universities (No.NP2020420) and China Postdoctoral Science Foundation funded project (No.2019M661832). We also appreciate it very much for editors and reviewers with your attention to our paper.

Conflict of interest

The authors declare that we have no conflict of interests regarding this paper.