CTFusion: CNN-transformer-based self-supervised learning for infrared and visible image fusion

1.   Introduction

Due to the hardware limitations of imaging, single sensor types or setups are often unable to fully represent imaging scenes ^[1,2]. For example, visible images contain rich texture details, but they are susceptible to extreme environments and occlusion, leading to target loss in scenes. In contrast, infrared sensors are capable of imaging by capturing the thermal radiation information emitted by the objects, which effectively highlights pedestrians, vehicles, and other significant targets, but lack detail description for the scenes ^[3]. In order to represent the scene accurately and effectively, image fusion has been pushed forward to integrate the complementary features of multiple source views in the same scene, thus generating a high-quality image for the downstream high-level tasks or human perception ^[4]. Specifically, infrared and visible image fusion (IVIF) aims to integrate complementary information from the source images and generates a high-contrast fusion image that can both highlight salient objects and contain rich texture details ^[5]. In the early years, traditional methods usually used related mathematical transformations and manual design of the fusion rules to realize the image fusion ^[3], including wavelet ^[6], pyramid ^[7], and sparse representation ^[8]. However, these manually designed feature representation approaches and fusion rules poorly understand the inherent information of images, which limits the ability to mine statistical characteristics of large samples and the generalizability.

Recently, deep learning has dominated the development of computer vision with its powerful feature extraction and expression capabilities. It has been used in all kinds of fields such as image classification ^[9], object detection ^[10,11,12], semantic segmentation ^[13], and image fusion ^[14]. In order to overcome the shortcomings of traditional algorithms, researchers have explored a large number of image fusion methods based on deep learning, which can be divided into convolutional neural network (CNN)-based and generative adversarial network (GAN)-based IVIF frameworks. Nevertheless, these models exhibit a major weakness: the lack of ground-truth fused images. Some algorithms ^[15,16] choose to generate ground-truth fusion results with existing state-of-the-art (SOTA) fusion methods, whose fusion quality cannot be promised since it highly depends on the quality of the produced "ground-truth". Moreover, most existing deep learning-based IVIF methods utilize CNNs for feature extraction, but CNNs fail to model longrange dependencies owing to their small receptive field, which is an inherent limitation ^[17].

In a word, there are two major problems in most existing IVIF tasks. On the one hand, there is the lack of ground-truth fused images. As mentioned above, though some methods try to generate so-called ground-truth fused images using other SOTA IVIF approaches, the quality of those produced ground-truths cannot be promised, thus largely affecting the consequent fusion results. On the other hand, failure to capture long-range dependencies of CNNs due to their small receptive field becomes a weakness in most deep learning based IVIF methods. To address the aforementioned issues, we propose a CNN-Transformer-based IVIF framework utilizing self-supervised learning, dubbed CTFusion. The pretext task using masked image reconstruction helps better extract features of source images. The CNN-Transformer-based encoder structure can utilize both local and global information. Inspired by ^[18], we adopt a specific image augmentation strategy that will mask some patches of the original images ${\mathit{\boldsymbol{I}}}_{ir}$ and $\mathit{\boldsymbol{I_{vis}}}$ with noise to generate two "source images", ${\mathit{\boldsymbol{\widetilde I}} _{ir}}$ and ${\mathit{\boldsymbol{\widetilde I}}_{vis}}$. Afterwards, they are fed into the CNN-Transformer-based encoders to excavate the intrinsic features ${{\mathit{\boldsymbol{f}}}_{ir}}$ and ${{\mathit{\boldsymbol{f}}}_{vis}}$ in the source images. We then apply two decoders $\mathit{\boldsymbol{D_{ir}}}$ and ${{\mathit{\boldsymbol{D}}}_{vis}}$ to produce the repaired images of ${\mathit{\boldsymbol{\widetilde I}}_{ir}}$ and ${\mathit{\boldsymbol{\widetilde I}}_{vis}}$. In addition, a self-cross perceptual feature fusion (S-CPFF) strategy is elaborated, by which we combine features ${{\mathit{\boldsymbol{f}}}_{ir}}$ and ${{\mathit{\boldsymbol{f}}}_{vis}}$ together to generate the fusion result of ${{\mathit{\boldsymbol{I}}}_{ir}}$ and ${{\mathit{\boldsymbol{I}}}_{vis}}$. The idea of our proposed method can be applied to common image fusion scenarios, since the abovementioned two problems are universal in the image processing field.

The main contributions of this paper are summarized as follows:

● We propose a self-supervised IVIF framework by designing a mask reconstruction task which no longer needs ground-truth to better excavate intrinsic information lying in infrared and visible source images.

● To compensate for the defect in establishing long-range dependencies in CNN-based architectures, we design an encoder that combines a CNN-Block with a Transformer-Block, which enables the network to utilize both local and global information during feature extraction.

● The S-CPFF module is devised to help enhance the extracted modality-specific and modality-common features, obtaining a final fusion result with high quality.

● Extensive experiments conducted on three publicly available datasets demonstrate the effectiveness of our method, as well as show its superior performance when compared with other state-of-the-art (SOTA) models.

2.   Related works

2.1.   Traditional image fusion methods

Traditional fusion frameworks usually realize image fusion in the transform domain and spatial domain through designing appropriate feature extraction details and fusion rules, which generally contain two major categories, multi-scale transform-based methods ^[19,20] and sparse representation-based methods ^{[21,22,23,24,25]}.

Multi-scale transform-based methods first decompose source images into several levels, as the feature extraction, then fuse corresponding layers with particular rule, and reconstruct the target images accordingly, where popular transforms used for decomposition and reconstruction include wavelet ^[26], pyramid ^[27], curvelet ^[28], and their revised versions. However, these methods typically tend to leave out image details in the fused results and lead to halos or undesirable artifacts in the fused result due to the fixed bases used in the multi-scale transform-based methods. The key of sparse representation-based methods is to build over complete dictionaries from a large number of natural images to possibly represent the source images with linear combinations of sparse bases. Although sparse representation-based methods have achieved promising performance, a limited number of dictionaries cannot reflect the full information of input images, obscuring details such as edges and textures in the source images.

2.2.   Deep learning-based image fusion methods

In deep learning-based algorithms, two source images from different modalities are directly input into a fusion network, and then the network outputs the fused image. Specifically, Liu et al. ^[29] proposed a method based on convolutional neural networks, which can deal with activity level measurement and weight assignment in IVIF as a whole to overcome the difficulty of manual design. To get more useful features from source images, Li and Wu ^[15] presented a novel encoding network combined with convolutional layers, a fusion layer, and a dense block in which the output of each layer is connected to every other layer. With the development of generative adversarial networks (GANs), more and more GAN-based IVIF methods appeared. Although CNNs have made great achievements in the field of supervised learning, they still have not progressed much the unsupervised learning. In order to fill the gap between supervised learning and unsupervised learning of CNNs, Ma et al. ^[30] established an adversarial game between a generator and a discriminator, which enabled the final fused image to simultaneously keep the thermal radiation in an infrared image and the textures in a visible image. Meanwhile, generic image fusion frameworks also achieved surprising performance. Li et al. ^[31] proposed a meta learning-based deep framework for the fusion of infrared and visible images which can accept source images of different resolutions and generate a fused image of arbitrary resolution with just a single learned model. Zhang and Ma ^[32] proposed a squeeze-and-decomposition network named SDNet to realize multi-modal and digital photography image fusion in real time. Xu et al. ^[33] used feature extraction and information measurement to automatically estimate the importance of corresponding source images and came up with adaptive information preservation degrees, solving different fusion problems. Recently, Tang et al. ^[34] and Liu et al. ^[35] bridged the gap between image fusion and high-level vision tasks, facilitating the high-level vision tasks with the proposed frameworks.

Though existing approaches devise complicated fusion rules and loss functions to achieve effective fusion, they still fail to effectively learn the characteristics of infrared and visible images for not devising specific task to explore intrinsic features in source images. It is a consensus that feature extraction is a pivotal step in image fusion. If the extracted features cannot comprehensively represent rich characteristics of source images, the quality of fusion results will definitely be worse. In contrast, we design a self-supervised mask reconstruction task to deeply excavate the intrinsic characteristics of infrared and visible images so that our network is able to achieve high-quality IVIF fusion.

2.3.   Vision transformer

Transformer was first proposed by Vaswani et al. ^[36] for machine translation. Its ability to extract features from the global level and effectively depict the correlations between features at different locations has attracted wide attention in the community. Later, researchers made great success in introducing Transformer into computer vision tasks, such as image processing ^[37], object detection ^[38], semantic segmentation ^[39], etc. Dosovitskiy et al. ^[40] introduced Transformer to an image classification task for the first time, proposing Vision Transformer (ViT). Based on ViT, a series of ViT variants were proposed to improve the performance ^[41,42]. Particularly, in the field of image fusion, Vs et al. ^[43] proposed a Transformer based IVIF method. This method used Transformer's encoder to extract image features, obtained the fused features with Spatial-Transformer, and finally reconstructed the fused image through Transformer's decoder.

Since Transformer has a stronger ability to model long-range dependencies, it is suitable to extract global image features. In contrast, CNNs are apt to capture local image features and describe low-level visual features such as structure and texture details, for they extract image features through convolutional kernels, whose receptive fields are limited. To integrate their advantages, we design an encoder that combines a CNN-Block with a Transformer-Block, which enables the network to utilize both local and global information during feature extraction.

3.   Method

3.1.   Overview

As mentioned above, lack of ground-truth fused images remains a tricky problem in IVIF tasks. Therefore, we propose CTFusion to achieve fusion in a self-supervised way, with a pretext reconstruction task aiming at image understanding is elaborated. More specifically, a CNN-Transformer-based encoder is devised to compensate for the defect in establishing long-range dependencies in CNN-based architectures. After the parameters in encoders are optimized, we fix them during the fusion phase and then fuse the extracted feature maps through tailored fusion net S-CPFF, to generate the fused image.

The Pipeline of the encoder training process is shown in Figure 1(a). We use the network to perform the self-supervised image reconstruction task to enable the encoders to extract intrinsic features lying in the source images, i.e., to reconstruct the original image from the masked input image. Concretely, given an original image ${{\mathit{\boldsymbol{I}}}_{in(in \in \left\{ {ir, vis} \right\})}} \in {\mathbb{R}^{H \times W \times 3}}$, masked image ${\mathit{\boldsymbol{\widetilde I}}_{in}}$ is generated by masking several non-overlapping patches with noise. Then, we feed the masked images into respective encoders ${{\mathit{\boldsymbol{E}}}_{ir}}$ and ${{\mathit{\boldsymbol{E}}}_{vis}}$ to obtain corresponding embeddings, which consists of a CNN-Transformer feature extraction (CTFE) module and a feature enhancement (FE) module. The CTFEBlock integrates the advantages of the CNN and the Transformer to model both global and local dependencies. FE aggregates and enhances the features extracted from the CNN-Block and the Transformer-Block. Finally, the image features extracted by the encoders are sent to respective decoders ${{\mathit{\boldsymbol{D}}}_{ir}}$ and ${{\mathit{\boldsymbol{D}}}_{vis}}$ to reconstruct images ${{\mathit{\boldsymbol{I}}}_{recon\_ir}}$, ${{\mathit{\boldsymbol{I}}}_{recon\_vis}} \in {\mathbb{R}^{H \times W \times 3}}$.

After training the encoders, we then use them for image fusion with their parameters fixed, as shown in Figure 1(b). Specifically, two source images ${{\mathit{\boldsymbol{I}}}_{ir}}$ and ${{\mathit{\boldsymbol{I}}}_{vis}}$ are first input to the trained encoders ${{\mathit{\boldsymbol{E}}}_{ir}}$ and ${{\mathit{\boldsymbol{E}}}_{vis}}$ to extract features, and then the fusion result is obtained by fusing the extracted features using the well-designed fusion network S-CPFF.

3.2.   Self-supervised mask reconstruction task

In general, the goal of image fusion is to integrate complementary information from different source images into a synthetic image. Moreover, feature dependency excavation is also the key in image fusion, since the relation understanding is important in feature extraction. Features with rich semantic and structural information are ideal to obtain high-quality fusion results. We divide the input image into non-overlapping patches and then use a random mask $M$ and Gaussian noise $n$ to force encoders to excavate intrinsic information lying in source images.

where $\overline M (\cdot)$ is the logical negation operator of mask $M$.

For each source image pair ${{\mathit{\boldsymbol{I}}}_{ir}}$ and ${{\mathit{\boldsymbol{I}}}_{vis}}$, they share parts of the scene information with each other while retaining some unique information. By randomly masking and filling the remaining with random noise, the encoders are forced to extract more information in source images, better understanding the relations between pixels. After pre-training, the encoders are able to extract more comprehensive features, which can be directly used for the following image fusion task.

3.3.   CNN-Transformer-based encoder-decoder framework

Given source images ${{\mathit{\boldsymbol{I}}}_{ir}}$ and ${{\mathit{\boldsymbol{I}}}_{vis}}$, we first randomly mask subregions and fill the remaining with noise to form ${\mathit{\boldsymbol{\widetilde I}}_{ir}}$ and ${\mathit{\boldsymbol{\widetilde I}}_{vis}}$, which will be sent to the CNN-Transformer-based encoders ${{\mathit{\boldsymbol{E}}}_{ir}}$ and ${{\mathit{\boldsymbol{E}}}_{vis}}$. Each encoder contains a CTFE module and a FE module, whose detailed architectures are shown in Figure 2.

Given that CNNs are adept at modeling local dependencies in images, while the Transformer specializes in modeling global dependencies, we propose CTFE, which combines the CNN and Transformer architectures to model both local and global dependencies in images. Specifically, the CNN-Block contains a residual dense block following the residual dense network ^[44]. As for the Transformer-Block, the masked image ${\mathit{\boldsymbol{\widetilde I}}_{in(in \in \left\{ {ir, vis} \right\})}} \in {\mathbb{R}^{H \times W \times 3}}$ is first divided into a total of $N$ patches with size $\frac{H}{P} \times \frac{W}{P}$, where $N = \frac{{HW}}{{{P^2}}}$ and $P$ is the size of the patches. Passing the patches through a patch embeddings linear projection and $L$ Transformer layers, we can obtain Transformer-embedded feature ${\mathit{\boldsymbol{f}}}_{in(in \in \{ ir, vis\})}^{tf}$. Figure 2 illustrates the architecture of one Transformer layer, which consists of a multi-head attention (MSA) block and a multi-layer perceptron (MLP) block, where layer normalization (LN) is applied before every block, and residual connections are applied after every block. The MLP block consists of two linear layers with a Gaussian Error Linear Unit (GELU) activation function. In order to better integrate the local and global features extracted by CNN and Transformer blocks, we devise the FE module to aggregate and enhance the feature maps ${\mathit{\boldsymbol{f}}}_{in(in \in \{ ir, vis\})}^{cnn}$ and ${\mathit{\boldsymbol{f}}}_{in(in \in \{ ir, vis\})}^{tf}$. Concretely, we concatenate the two feature maps from the CNN-Block and the Transformer-Block in CTFE and send them into four sequentially connected ConvBlock layers to achieve FE, as shown in Figure 2.

where each ConvBlock consists of two convolutional layers with a kernel size of $3 \times 3$, a padding of 1, and two Rectified Linear Unit (ReLU) activation layers, and $[\cdot]$ denotes channel-wise concatenation.

We then feed the obtained feature maps ${\mathit{\boldsymbol{f}}}_{ir}^{en}$ and ${\mathit{\boldsymbol{f}}}_{vis}^{en}$ to decoders ${\mathit{\boldsymbol{D}}}_{ir}$ and ${\mathit{\boldsymbol{D}}}_{vis}$, each composed of two convolutional layers with a kernel size of $3 \times 3$, a padding of 1, and one ReLU activation layer, to reconstruct the corresponding image.

In the mask reconstruction task, we encourage the network to not only learn the pixel-level image reconstruction but also capture the structural and gradient information in the image. The loss of the reconstruction task in each branch can be formalized as follows:

where $\ell _{pixel}$ is the L1 loss function, $\ell _{structure}$ is the structural similarity (SSIM) loss function, and $\ell _{TV}$ is the total variation loss function. $\lambda _1$ and $\lambda _2$ are two hyperparameters empirically set to 20.

$\ell _{pixel}$ ensures pixel-level reconstruction

where ${\mathit{\boldsymbol{I}}}_{recon\_in}$ is the output reconstructed image, and ${\mathit{\boldsymbol{I}}}_{in}$ represents the input unmasked source image.

To better help the model learn structural information from images, we use the structure loss:

Furthermore, $\ell _{TV}$ in VIFNet ^[45] is used to facilitate gradient preservation in the source images and eliminate noise. It is formulated as follows:

where $R(x, y) = {{\mathit{\boldsymbol{I}}}_{recon\_in}}(x, y) - {{\mathit{\boldsymbol{I}}}_{in}}(x, y)$ ($in \in \{ir, vis\}$) denotes the difference between the input image and the reconstructed image, $||\cdot |{|_2}$ is the L2 norm, and x, y represent the horizontal and vertical coordinates of the image's pixels, respectively.

3.4.   Self-cross perceptual feature fusion

How to effectively fuse feature maps to obtain final fusion results remains challenging. Here, we propose S-CPFF, which is able to highlight both self and mutually interested parts in feature maps, naturally improving the qualities of fused images. Concretely, as shown in Figure 3, the S-CPFF has two self-attention (SA) and two cross-attention (CA) modules. The detailed structures of the SA and CA modules are shown in Figure 4. Mathematically, the SA process is denoted as:

where ${{\bf{Q}}_{ir}}$, ${{\bf{Q}}_{vis}} \in {\mathbb{R}^{H \times W \times C}}$, ${{\bf{K}}_{ir}}$, ${{\bf{K}}_{vis}} \in {\mathbb{R}^{H \times W \times C}}$, and ${{\bf{V}}_{ir}}$, ${{\bf{V}}_{vis}} \in {\mathbb{R}^{H \times W \times C}}$ are the results of ${\mathit{\boldsymbol{f}}}_{ir}^{en}$ and ${\mathit{\boldsymbol{f}}}_{vis}^{en}$ passed through $1\times 1$ convolution, respectively. ${\sqrt d }$ is a normalization factor, and $T$ is the transpose operation. The CA process is:

where $vis \to ir$ denotes information flow from visible modal to infrared modal. Then, we concatenate the results after respective SA and CA modules to obtain the self-cross perceptual feature:

Note that in the fusion phase, we fix the parameters of encoders ${{\mathit{\boldsymbol{E}}}_{ir}}$ and ${{\mathit{\boldsymbol{E}}}_{vis}}$ learned in the mask reconstruction task. We then concatenate ${{\mathit{\boldsymbol{f}}}_{ir}}^{\prime}$ and ${{\mathit{\boldsymbol{f}}}_{vis}}^{\prime}$ and feed the concatenated result to a $3 \times 3$ convolution to integrate all the features. Then, the integrated feature is sent to the decoder ${{\mathit{\boldsymbol{D}}}_{fuse}}$ to attain the fused image:

where the detailed structure of ${{\mathit{\boldsymbol{D}}}_{fuse}}$ is shown in Figure 1.

To retain rich edge and texture information in the fused image, we adopt joint gradient loss $\ell _{JGrad}$, which is formulated as

where $\nabla$ is the Laplacian gradient operator. $\max(\cdot)$ denotes taking the maximum value. ${\mathop{\rm O}\nolimits} (\left| x \right|) = x$ denotes finding the original gradient value before taking its absolute value.

We also introduce the intensity loss to preserve the saliency targets in two input images, which can be expressed as

where ${\mathit{\boldsymbol{S}}}_{ I_{ir}}$ and ${\mathit{\boldsymbol{S}}}_{ I_{vis}}$ denote saliency matrices of ${\mathit{\boldsymbol{I}}}_{ir}$ and ${\mathit{\boldsymbol{I}}}_{vis}$, which can be computed according to ^[46]. ${\mathit{\boldsymbol{\omega}}_{ir}}$ and ${ \mathit{\boldsymbol{\omega}}_{vis}}$ are the weight maps for ${\mathit{\boldsymbol{I}}}_{ir}$ and ${\mathit{\boldsymbol{I}}}_{vis}$, respectively. $\odot$ represents the element-wise multiplying operation.

The overall fusion loss is computed by

where ${\lambda _{JG}}$ is the hyper-parameter set to 20.

4.   Experiments and results

In this section, we evaluate the qualitative and quantitative performance of our proposed method by comparing to five SOTA methods, including IFCNN ^[47], PMGI ^[48], CrossFuse ^[49], RFN-Nest ^[50], and FusionGAN ^[30]. We also implement several ablation studies to validate the effectiveness of the proposed modules.

4.1.   Experimental configurations

Dataset: First, 300 multi-modality images from the M3FD ^[35] benchmark are selected and cropped to 360k patches with $256 \times 256$ pixels by random cropping and augmented as the training set in this study. M3FD is a multi-modal dataset with multiple scenarios, where 4,200 aligned image pairs are divided into four typical types, i.e., Day, Cloudy, Night, and Challenge. We perform qualitative and quantitative experiments on three datasets (i.e., Roadscene, TNO, and MSRS). RoadScene is a wildly-used dataset for cross-modality image fusion. The TNO dataset contains multi-spectral nighttime imagery of various military-relevant scenarios in grayscale. The MSRS dataset contains 1444 pairs of aligned infrared and visible images with high quality.

Evaluation metrics: For quantitative evaluation, five statistical metrics are selected to objectively assess the fusion performance, including correlation coefficient (CC) ^[51], cross entropy (CE), ${Q^{CV}}$ ^[52], the sum of correlations of differences (SCD) ^[53], and structural similarity (SSIM) ^[54]. CC evaluates the degree of linear correlation between the fused image and source images. CE reflects the difference of grayscale information between the fusion image and source images. The smaller the CE value is, the smaller the difference between images, which indicates better fusion quality. ${Q^{CV}}$ uses the Sobel operator to extract the edge information of the source images and the fusion result to obtain the edge intensity map G. Smaller ${Q^{CV}}$ is more in line with human visual perception. SCD reflects the correlation level between information transmitted to the fused image and corresponding source images. SSIM approximates image distortion. In addition, a fusion algorithm with larger CC, SCD, and SSIM indicates better fusion performance.

Implementation details: The Adam optimizer ^[55] (${\beta _1}$ = 0.9, and ${\beta _2}$ = 0.999) is responsible for updating the network parameters with initial learning rate of 0.001, which decreases to $10^{-4}$ after 100 epochs. The epochs of the self-supervised mask reconstruction task and training of S-CPFF are both set to 300 with batch size of 4. Our framework is implemented on PyTorch with an NVIDIA 3090 GPU. Note that source images in all the abovementioned datasets are converted to gray to achieve fusion.

4.2.   Comparative experiment

4.2.1.   Experiments on RoadScene dataset

Qualitative results on the RoadScene benchmark are reported in Figure 5, where we highlight two regions in each example. As can be seen, RFN-Nest and FusionGAN suffer blurred edges and background, and IFCNN, PMGI, and CrossFuse all lose texture details to some extent. Instead, our method has the best image contrast and clear structure information.

Quantitative comparisons are shown in Table 1, where we use five metrics, i.e., CC, CE, $Q^{CV}$, SCD, and SSIM, to evaluate all comparison methods. Our method ranks first in CC, CE, SCD, and SSIM, indicating that the generated fused results are of higher similarity to the source images. For $Q^{CV}$, our method also achieves comparable results, which implies that the fusion images of our method are more real. Our model utilizes the Laplacian gradient operator to detect the edges of images, while $Q^{CV}$ uses the Sobel gradient operator, which might be the reason why our method performs suboptimally in $Q^{CV}$.

4.2.2.   Experiments on TNO dataset

We select five pairs of infrared and visible images to visually observe the fusion performance of different algorithms on the TNO dataset. The visualized results are shown in Figure 6. As shown in the first column of Figure 6, our method keeps the best image contrast. It can be seen from the zoomed-in areas that our methods combine complementary as well as modality-common information in source images to the most extent. Meanwhile, the edges of salient targets from infrared images are clear, and texture details from visible images are well kept in our fusion results.

For example, in the second row of Figure 6, though the image contrast of CrossFuse is better than ours, it still loses cloud information. In the first and third rows, people in the foreground are bright in our fused results, while FusionGAN has blurred target edges.

Quantitative performance of our method on the TNO dataset was similar to that of RoadScene, which is shown in Table 2.

4.2.3.   Experiments on MSRS dataset

So as to visually evaluate the fusion performances of different algorithms on the MSRS dataset, three pairs of infrared and visible images are selected, depicted in Figure 7. As illustrated in the red and green boxes in the image, our proposed method favorably maintains textures of source images while keeping the clearest salient edges.

We conduct quantitative comparisons on 40 image pairs from the MSRS dataset to verify the effectiveness of our method, which is presented in Table 3. It can be seen that our method ranks first in four metrics and second in the CC metric. The CE, $Q^{CV}$, SCD, and SSIM metrics demonstrate that our results contain more realistic information. As for CC, it directly matches images by their intensity, without using any analysis of the image structure. Hence, CC is sensitive to intensity changes in the image. In general, image noise, changes in lighting intensity during imaging, and the use of different imaging equipment all cause changes in image intensity, which will further affect CC.

In conclusion, our method is fully capable of excavating inherent important features in source images and integrating them into fused images. Thereby, our method is superior to other SOTA approaches and obtains high-quality fused images.

4.3.   Analysis of generalization ability

To validate the generalization ability of our method, we conduct experiments on datasets for other image fusion tasks, including LLVIP ^[57] for color image fusion and CT-MRI ^[58] for medical image fusion. Fusion results are shown in Figure 8. From the qualitative results we can see that our proposed model perfectly completes other fusion tasks, which strongly proves the generalization ability of our method.

4.4.   Analysis of computational complexity

As shown in Table 4, a complexity evaluation is introduced to evaluate the efficiency of our method from two aspects, i.e., training parameters and runtime. It is worth pointing out that though our method does not perform the best in terms of the model complexity and inference time due to subtle design of various modules, the proposed CTFusion and the best SOTA method are still equal. This indicates the efficiency of our CTFusion, which can serve practical vision tasks well with better visual performance.

5.   Ablation studies

In the ablation study, we demonstrate the effectiveness of the self-supervised mask reconstruction task, CNN-Transformer-based encoder, and the proposed S-CPFF. The experimental results are shown in Figure 9 and Table 5.

First, we remove the mask reconstruction pretext task, simply training a complete encoder-decoder framework. The results show that our proposed self-supervised mask reconstruction task can improve the ability of the framework to excavate intrinsic information. To verify the effectiveness of the CNN-Transformer-based encoder, we conduct an ablation study where the encoders only contain the CNN-Block. From the results we can see that regardless of whether the proposed self-supervised mask reconstruction task is used, adding the Transformer-Block in the encoders always improves the fusion performance. To further prove that our proposed S-CPFF is effective, we replace the fusion net with a simple feature concatenation operation. The ablation study results also show that S-CPFF highlights salient regions in source images and further promises the enhancement of texture details.

6.   Discussion

Source images in this paper are all registered before fusion, which is a common data preprocessing step in an IVIF task. However, in practical scenarios, although the source images can be aligned to a certain extent by carefully adjusting the installation positions of infrared and visible light sensors, it stays impossible to achieve accurate alignment directly by manual installation ^[16,59]. In other words, images captured by different sensors are difficult to strictly align on a pixel level. In the future, we will focus research on misaligned IVIF. Since the source images are misaligned and of different modalities, we need to reduce the modality discrepancy between them, so that the feature alignment can be achieved more easily. Once the features are aligned, the fusion process will not be a problem.

7.   Conclusions

In this paper, we present CTFusion, a CNN-Transformer-based IVIF framework using self-supervised mask reconstruction. The CNN-Transformer-based encoder integrates the advantages of both CNN and transformer so that the network can focus on both local and global information, better understanding dependencies in images. In addition, the designed mask reconstruction task is naturally adaptive to the intrinsic information excavation requirement in IVIF. Extensive experiments on three infrared-visible image datasets demonstrate the effectiveness of the proposed method.

Use of AI tools declaration

The authors declare they have not used artificial intelligence (AI) tools in the creation of this article. All authors reviewed the manuscript.

Acknowledgments

This work was supported by the 2023 Opening Research Fund of Yunnan Key Laboratory of Digital Communications (YNJTKFB-20230686, YNKLDC-KFKT-202301).

Conflict of interest

The authors declare there is no conflict of interest.