Retinal vessel segmentation based on multi-scale feature and style transfer

1.   Introduction

According to data from the World Vision Report ^[1] released by the World Health Organization in 2019, more than 418 million people suffer from eye diseases worldwide ^[2]. Eye diseases have seriously harmed human life and health. Therefore, it is necessary to prevent and screen for eye diseases early.

Information regarding the morphology of retinal vessels is essential for the treatment of many eye diseases and has been widely used in clinical diagnoses ^[3]. Hence, accurately segmenting retinal blood vessels is very important for doctors in diagnosing and treating eye diseases. However, the vessels in the optic cup region are significantly thicker than those in other regions. Additionally, the blood vessels in the fundus image are easily mixed with the background and difficult to distinguish, which leads to retinal vessel segmentation being very challenging.

Researchers have presented many retinal vessel segmentation approaches, which are broadly grouped into two classes: classical segmentation methods and segmentation methods based on deep learning. In the early years, many classical retinal vessel segmentation methods were proposed based on hand-crafted features (e.g., image gradient features and local texture features). For example, Soares et al. ^[4] employed the pixel intensity and two-dimensional Gabor wavelet transform responses taken at multiple scales as the feature vector of pixels to produce pixel-level vessel segmentation results. Orlando et al. ^[5] proposed a fully connected conditional random field method with an improved potential for feature extraction, which can perform fast and accurate blood vessel segmentation in retinal images. Nguyen et al. ^[6] employed a multiscale line detection framework to combine line detectors at varying scales to classify each pixel as either a retinal vessel or background in fundus images. The existing classical methods usually lack sufficiently discriminative information to accurately segment retinal vessels, and their performances are easily affected by the extracted hand-crafted features.

With the wide application of deep learning technology, retinal vessel segmentation approaches based on deep learning have attracted increasing attention. In 2015, Ronneberger et al. ^[7] proposed U-Net for medical image segmentation tasks. U-Net combines the encoder-decoder architecture and skip connections and is a well-known backbone network in the medical image segmentation community. Recently, researchers have presented a series of improved models based on U-Net ^[8]. For example, Wang et al. ^[9] proposed a dual-coded U-Net (DEU-Net), which improves the ability of the network to segment retinal blood vessels pixel-to-pixel. Zhang et al. ^[10] presented an attention guided network (AG-Net), which designs a filter named the attention guided filter to obtain improved segmentation results for fundus images. Wang et al. ^[3] developed a multi-scale integrated context network (MIC-Net) based on U-Net to fully fuse the multi-scale features from the encoder and the decoder to segment retinal vessels in fundus images. Wu et al. ^[11] designed Vesic-Net for vessel segmentation by embedding an initial residual convolution block into the U-type encoder-decoder structure. The residual convolution block is an effective strategy to resolve vanishing/exploding gradient problems and is widely used in various medical image analysis tasks; for example, Shin et al. ^[12] proposed a squeeze-and-excitation super-resolution residual network (SE-SRResNet) for transcranial focused ultrasound simulation.

The retinal vessel segmentation approaches based on deep learning can continuously optimize the model according to the real labels of the training dataset so they can achieve good segmentation performances. However, when deep learning-based models trained well on one public dataset are directly applied to another dataset, the segmentation performance of the models will greatly decrease due to the domain shift problem ^[13]. Domain shifts often exist in different medical image datasets and are very common in practical applications. For example, different hospitals and institutions usually adopt different equipment to take fundus images, which causes a domain shift problem between the images acquired by different equipment. Even when adopting the same equipment, the appearance of the image usually varies with subjects, clinical operators and other factors thus the domain shift problem is still persistant. Figure 1 shows an example of a domain shift. In Figure 1, the style and size of images between two datasets (DRIVE and CHASE_DB1) are different, which means that there is a domain shift between these two databases. Since a domain shift will degrade the model performance, the development of a valid retinal vessel segmentation model that is robust to domain shifts is necessary.

Figure 1.  Comparison of images from different datasets.

DownLoad: Full-Size Img PowerPoint

In this paper, we propose a novel retinal vessel segmentation method based on multi-scale features and a style transfer (MSFST-NET), which can improve the segmentation results and is robust to domain shifts. The contributions of our work are as follows:

1) Small blood vessels in fundus images play a very important role in disease diagnoses. Ordinary segmentation models often use convolutions of the same size to extract features, which leads to poor small blood vessel segmentation. To solve this problem, we introduce the selective kernel (SK) into the segmentation model to learn more multi-scale feature information to improve the segmentation results of small vessels.

2) A domain shift is a common phenomenon in medical images. To alleviate the problem of model performance degradation caused by domain shifts, we propose a style transfer module to reduce the style difference between the source domain dataset and the target domain dataset.

3) To further enhance the robustness of the model, we additionally design a pseudo-label strategy to increase the model's generalization ability and reduce the effect of domain shifts.

4) To verify the effectiveness of our proposed method, we conducted a large number of experiments on the DRIVE and CHASE_DB1 datasets. The experimental results show that our proposed method is effective and superior to other advanced vessel segmentation methods.

2.   Related work

2.1.   Retinal vessel segmentation

Retinal vessel segmentation is used to localize blood vessels in fundus images. Early retinal vessel segmentation approaches are often designed by employing traditional shallow machine learning models, which include matching filtering and blood vessel tracking methods. Recently, with the development of deep learning, a large number of retinal vessel segmentation approaches have been presented by adopting the convolutional neural network (CNN), which achieves a superior performance compared to the approaches based on traditional shallow machine learning and has attracted increasing attention.

The fully convolutional network (FCN), which was proposed by Long et al. ^[14], is a classical semantic segmentation approach. The FCN extends the deep neural network from the classification of the whole image to the classification of each pixel in the image for the first time. To gain more accurate results in the area of medical image segmentation, a large number of improved methods based on FCN have been proposed. For instance, U-Net ^[7] is a deep convolutional network constructed by improving the original skip connection part of an FCN. Subsequently, many more advanced network models have been proposed based on U-Net (e.g., attention U-Net ^[15] and U-Net++) ^[16]. Attention U-Net introduces an attention gate mechanism into U-Net to reduce the redundant feature information brought on by the skip connection. U-Net++ introduces a jump path consisting of dense convolutional blocks and dense jump connections into the structure of U-Net. Meanwhile, U-Net++ adds a deep supervision mechanism to speed up the convergence of network training. In addition to the methods based on U-Net, some retinal vessel segmentation methods have been proposed based on generative adversarial networks ^{[17,18,19,20]}. For example, Yue et al. ^[20] proposed an improved generative adversarial network based on R2U-Net for retinal vessel segmentation. Generative adversarial network-based segmentation methods have difficulty training models; thus, they have not become widespread popular methods.

The abovementioned methods mainly focus on segmentation accuracy while usually ignoring segmentation efficiency. The efficiency of the model is of great significance to practical applications. To improve the segmentation model efficiency, Galdran et al. ^[21] proposed a new model called W-Net, which is a simple extension of the U-Net structure and can obtain an outstanding performance on retinal vessel segmentation. Hence, we employ W-Net as the backbone network of our proposed method and further boost its segmentation performance for small retinal vessels by introducing the multi-scale feature extraction module.

2.2.   Cross-domain segmentation

In recent years, some researchers have found that when a deep learning model trained on a labeled dataset is directly applied to a different dataset, the segmentation performance of the model is greatly reduced ^[13]. This is because different fundus image datasets usually have the issue of domain shift. To alleviate this issue, many cross-domain segmentation methods have been presented to strengthen the domain transferability of models and mitigate cross-domain model performance decline. Cross-domain segmentation methods can be broadly categorized into two main classes: methods based on semi-supervised domain adaptation and methods based on unsupervised domain adaptation.

The methods based on semi-supervised domain adaptation ^{[22,23,24,25,26]} usually adopt fewer labeled images and more unlabeled images in the target domain dataset to mitigate the negative impact of domain shift on model performance. For instance, Xia et al. ^[22] proposed an uncertainty-aware multi-view co-training (UMCT) method to achieve semi-supervised learning and domain adaptation, which can obtain an improved performance for cross-domain medical image segmentation. Chen et al. ^[23] proposed a dual-level domain mixing-based segmentation network that learns domain-invariant region-level and image-level features based on labeled samples from different domains and further enhances the model performance by using pseudo labels of unlabeled data. The performance of semi-supervised domain adaptation methods is limited by the number of labeled images; annotating these medical images is very difficult, which greatly reduces the scope of the practical application of these semi-supervised domain adaptation models.

The methods based on unsupervised domain adaptation aim to enhance model domain transferability and alleviate cross-domain performance degradation by only using unlabeled data in the target domain. In recent years, many unsupervised domain adaptation methods with a relatively good performance have been proposed ^{[27,28,29,30,31,32]}. Wang et al. ^[27] designed a cross-domain segmentation approach based on feature separation. This approach adopts a new unsupervised region adaptive strategy and the disentangled reconstruction neural network (DRNN) to reduce the impact of domain shift on the model's segmentation performance. Zuo et al. ^[28] developed an unsupervised domain adaptation method based on category-level adversarial self-ensembling, which aligns the source and target domain by constraining the descriptions. Xu et al. ^[29] designed self-ensembling attention networks to produce attention-aware features and used them to guide the model to compute the consistency loss in the target domain. Since the methods based on unsupervised domain adaptation do not require any labeled target domain data, they are more convenient for practical applications.

Inspired by these previous studies and to avoid models that are too complex, we design a simple yet effective method to address the issue of domain shift. Specifically, we combine Cycle-GAN ^[33], which is a popular algorithm to transfer images over different domains by cycle consistency losses, and a pseudo-label learning strategy to ensure that our segmentation method can still acquire good segmentation performance when dealing with images from cross-domains.

3.   Proposed method

3.1.   The overall framework of MSFST-NET

The overall structure of our MSFST-NET is shown in Figure 2. First, the labeled source domain images are input into our proposed segmentation model, named MSF-Net, to train and obtain the model with the optimal segmentation performance. Then, based on the style of the source domain images, style transformation is performed on the unlabeled target domain images by the proposed style transfer module. Finally, the target domain images after the style transformation are input into the trained MSF-Net to generate the prediction maps. Prediction maps are used as pseudo-labels that are fused with the labeled source domain images to further increase the segmentation accuracy and strengthen the generalization ability of the network. Overall, MSFST-NET includes the following three parts: the segmentation module MSF-Net, a style transfer module and a pseudo-label learning strategy.

Figure 2.  The overall structure of MSFST-NET.

DownLoad: Full-Size Img PowerPoint

Segmentation module MSF-Net: This part is the basic segmentation model of our proposed MSFST-NET. To make the model lightweight and have a good performance, we use two-cascade three-layer U-Net as the basic architecture of MSF-Net. In addition, considering that retinal vessels have different sizes and scales, it is difficult to simultaneously obtain the important feature information of blood vessels at each scale by only using convolution kernels of the same size. Therefore, we introduce SK into the encoding stage of each U-Net in MSF-Net, that is, we employ convolution kernels of different sizes to learn and adaptively fuse multi-scale features to reduce feature loss and further increase the accuracy of blood vessel segmentation.

Style transfer module: Since the issue of domain shift is common in the community of medical image segmentation, we construct a style transfer module to make the target domain images and the source domain images tend to assimilate in style, thereby reducing the style difference between them and easing the performance degradation when the model is directly applied to another dataset. In this module, the classical Cycle-GAN style transfer algorithm is adopted, and the model trained by the source domain dataset is employed to segment the target domain images after the style transfer to obtain improved segmentation results of the target domain images.

Pseudo-label learning strategy: To further strengthen the model performance when dealing with cross-domain datasets (i.e., there is a domain shift between two datasets), we further design a pseudo-label learning strategy. The pseudo-label of the target domain dataset generated by the style transfer module is utilized as the training data; then, the model is retrained by the target domain dataset and the source domain dataset, which can obtain a model with an improved segmentation performance and a stronger generalization ability.

3.2.   MSF-Net

We adopt a deep learning model containing a two-cascade three-layer U-Net, which is called W-Net ^[21], as the backbone network of MSF-Net; this deep learning model is a lightweight model that has a good segmentation accuracy and few parameters. The original U-Net contains five layers of downsampling and upsampling. To make the model lightweight, W-Net reduces the five-layer network structure to a three-layer network structure that has approximately 0.03 M parameters. To reduce the number of layers and parameters without losing too much model performance, W-Net concatenates two three-layer U-Nets, which has around 0.06 M parameters in total ^[21]. The segmentation results output by the first U-Net in W-Net are fused with the original images as the input of the second U-Net, that is, the blood vessel segmentation prediction map of the first U-Net is used as an attention map to weigh the original image and input it to the second U-Net so that the second U-Net pays more attention to the key parts of the image when segmenting blood vessels.

Although W-Net performs relatively well for retinal vessel segmentation, it cannot capture rich multi-scale image features; hence, the segmentation results of small blood vessels are not satisfactory. To solve this drawback, we embed the SK ^[34] into W-Net to construct a new network named MSF-Net to strengthen the multi-scale feature extraction ability of the model. Figure 3 shows the structure of our MSF-Net.

Figure 3.  The MSF-Net architecture.

DownLoad: Full-Size Img PowerPoint

In MSF-Net, each encoder in the two cascaded U-Net networks contains three stages, and each stage contains a normal convolution operation and an SK module, along with a downsampling operation. Each decoder in U-Net consists of a convolution operation and an upsampling operation. In Figure 3, the blue rectangular block represents the feature map, and the number on each feature map denotes the number of channels. The blue arrow represents a 3 × 3 convolution, batch normalization (BN) and rectified linear unit (ReLU) activation function ^[35] (BN can accelerate network training and increase model accuracy ^[36], and ReLU can enhance nonlinear mapping learning and reduce the computational complexity of the network to prevent gradient disappearance). The red arrow represents the SK module, and BN and ReLU are added after SK. The yellow downward arrows represent the max pooling operation, which is employed to decrease the size of the feature map and extract features. The yellow up arrow represents the upsampling step used to gradually recover the details of the object and the corresponding spatial dimension. The gray arrow represents skip connections, which can provide richer feature information for the decoder.

The SK module ^[34] is a dynamic selective kernel that uses convolution kernels with different receptive field sizes to extract multi-scale feature maps and adaptively selects the appropriate receptive fields for vessels of different sizes, thus obtaining rich multi-scale feature information for retinal vessel segmentation. Figure 4 shows the architecture of the SK module.

Figure 4.  The SK module architecture.

DownLoad: Full-Size Img PowerPoint

The key to the SK module is that it employs convolution kernels with different receptive field sizes to adaptively extract and select multi-scale features, making the feature information learned by the network richer and more discriminative, which is conducive to improving the segmentation results of small blood vessels in the retinal image. SK consists of three operations: split, fuse and select.

The split operation extracts abundant multi-scale feature information and adopts convolution kernels with different receptive field sizes (3 × 3 and 5 × 5) to deal with the input features X to obtain the different scale feature maps $\stackrel{~}{U}$ and $\widehat{U}$, as shown in Figure 4.

The fusion operation first integrates $\stackrel{~}{U}$ and $\widehat{U}$ by elementwise summation to obtain the fusion feature map U. Then, it utilizes the average pooling and fully connected layer to handle U to produce a compact feature map $z$. $z$ will be utilized to learn the adaptive weights ${a}_{c}$ and ${b}_{c}$ for the different scale feature maps. This process can be described by the following formulas:

where ${F}_{fc}$ represents the fully connected layer, $BN$ is the batch normalization, ${\delta }_{relu}$ represents the ReLU activation function, W is a learnable parameter and $W\in {R}^{d*c}$, $d$ represents the dimensions of the fully connected layer, ${F}_{gp}$ denotes the global average pooling operation utilized to obtain the channel-wise statistics $s$, specifically, ${s}_{c}$ represents the c-th element of $\mathrm{s}$, U_c denotes the c-th channel of U, and h and w are the height and width of the feature map, respectively.

The selection operation adaptively selects the multi-scale features by the different attention weights. First, it employs the softmax operation to calculate the weights ${a}_{c}$ and ${b}_{c}$ of the feature maps with different scales by Eq (3). Then, the c-th channel ${V}_{c}$ of the final feature map V is obtained by Eq (4):

where ${A}_{c}$ represents the c-th row of $A$ , ${B}_{c}$ is the c-th row of $B$, $A$ and $B$ are learnable parameters, and $A, B\in {R}^{C*d}$.

3.3.   Style transfer module

Due to the differences in the shooting devices and the shooting angles, there are often stylistic differences between images from two different retinal vessel datasets, which is called a domain shift. Therefore, when a segmentation model trained on one fundus image dataset is directly applied to another fundus image dataset, the performance of the model will be greatly reduced. To address this problem, we develop a style transfer module to reduce the differences between the two datasets as much as possible to alleviate the domain shift problem. In the style transfer module, we adopt a classical model, termed Cycle-GAN ^[33], to transfer the style of the target domain images to that of source domain images. In medical image segmentation, interpretability is very important. Cycle-GAN can transform target domain images into source-like domain images, which is a more intuitive and highly interpretable approach. Furthermore, the transformed images can be directly utilized in our segmentation model without retraining. In contrast, other advanced domain generalization methods usually require aligning features from different domains in a high-dimensional feature space, which is less interpretable and creates greater training challenges. Hence, we selected the relatively simple yet effective Cycle-GAN as our style transfer module.

One advantage of Cycle-Gan is that it can be trained without paired datasets, and it can retain the details of the original image content as much as possible during style transfer. The Cycle-GAN structure is shown in Figure 5. I_t represents the target domain dataset and I_s represents the source domain dataset. There are two generators and two discriminators in the Cycle-GAN model: F is an image generator that converts the source domain style to the target domain style, and D_x is a discriminator that decides whether the generated image style is consistent with the target domain image style. G is an image generator that converts the target domain style to the source domain style and D_y is a discriminator that determines whether the style of the image generated by G is consistent with that of the source domain image.

Figure 5.  The architecture of Cycle-GAN.

DownLoad: Full-Size Img PowerPoint

Cycle-GAN mainly uses cycle consistency constraints to ensure that the content is consistent between the output image of the generator and the original input image. As shown in Figure 5, in the process of cycle consistency, given two images, I_t and I_s, from different style domains, image I_t is first used to generate image I_t2s with image I_s style through the generator G, and then I_t2s is utilized to generate the image I_t' with the same style as I_t through the generator F. In this process, if the content and style of the image I_t and the image I_t' are consistent, it signifies that the image has gone through a cycle and the content and style of the image remain consistent before and after the cycle, which is the basic principle of cycle consistency.

Cycle consistency can ensure that the content of the original input image is preserved when the generator produces the style-transferred image so that the generated image can be returned to the input image by another generator during the second transformation. In our work, when performing a style transfer on fundus images, cycle consistency can preserve as much of the key information of blood vessels as possible.

3.4.   Pseudo-label learning strategy

In the previous section, the style transfer module was proposed to ease the domain transfer problem between the different datasets to some extent. To further solve this problem and strengthen the generalization ability of our model, our work adopts a pseudo-label learning strategy. The pseudo-label learning strategy is a method based on the idea of semi-supervised learning, and its main purpose is to enhance the generalization ability of the model by employing unlabeled data.

The semi-supervised training process based on pseudo-label learning mainly contains the following four steps. First, the labeled source domain dataset is employed to train the segmentation model and save the optimal model parameters. Second, the trained model is utilized to estimate the labels of unlabeled target domain samples, and the estimated results are used as the pseudo-labels of these unlabeled target domain samples. Third, the labeled source domain samples and the target domain samples with pseudo-labels are employed as new training data to retrain the segmentation model, and the first to third steps are repeated until the model converges. Finally, the images in the target domain test dataset are predicted to obtain the final blood vessel segmentation results. The details of the pseudo-label learning strategy are shown in Algorithm 1.

3.5.   Loss function

Our proposed method adopts the binary cross-entropy (BCE) loss function ^[37]. Specifically, the formula of our final loss function ${f}_{\text{loss}}$ is as follows:

where $x$ is the input image, $y$ is the true label in the ground truth, ${P}_{1}$ is the prediction result of the first U-Net, ${P}_{2}$ is the prediction result of the second U-Net, and ${L}_{BCE}$ represents the BCE loss, which is calculated based on the cross-entropy between the prediction result generated by the model and the true label in the ground truth.

4.   Experiment results

Our proposed MSFST-NET is composed of MSF-Net, which is a style transfer module and a pseudo-label learning strategy. In this section, we first verify the effectiveness of MSF-Net for the retinal blood vessel segmentation task. Second, based on MSF-Net, the style transfer module and pseudo-label strategy are tested and evaluated to verify their effectiveness in solving the domain shift problem. Finally, the performance of our proposed MSFST-NET is evaluated by comparing it to other advanced methods and constructing an additional ablation experiment.

4.1.   Datasets and evaluation criteria

The experimental datasets used in this paper are DRIVE ^[38] and CHASE_DB1 ^[39], which are frequently adopted in the fundus image segmentation community. Figure 6 shows the example images from these two datasets. In Figure 6, the images in the first row are from DRIVE, and the images in the second row are from CHASE_DB1.

Figure 6.  The images from the DRIVE and CHASE_DB1 datasets.

DownLoad: Full-Size Img PowerPoint

The DRIVE dataset ^[38] contains a total of 40 color fundus images, where seven images contain diabetic lesions, and the remaining 33 images are normal. The resolution of each image is 584 × 565 pixels. The first 20 images in DRIVE are employed as the training data, and the last 20 images are adopted as the test data. Moreover, DRIVE provides a nearly circular mask with a diameter of approximately 540 pixels for each image, which is used to indicate the region for the model training.

The CHASE_DB1 dataset ^[39] consists of 28 color retinal images with a resolution of 999 × 960 pixels. The images in this dataset are taken from the left and right eyes of 14 school-aged children. CHASE_DB1 was evenly separated into two parts, each containing 14 images, and used as a training set and a test set. Additionally, this dataset provides a circular mask for each image.

To evaluate the performance of the proposed method, we adopt five evaluation criteria commonly used in retinal vascular segmentation, which include sensitivity (Se), accuracy (Acc), comprehensive evaluation index (F1-score), AUC (area under the curve) and Matthews correlation coefficient (MCC). The larger the value of these indicators, the better the prediction result. The AUC is calculated by using the implementation provided in the scikit-learn Python library. The specific formulas of the other evaluation criteria are as follows:

where TP, TN, FP and FN are the number of true positive, true negative, false positive and false negative pixels, respectively.

4.2.   Implementation details

In the experiment, our method is trained by the PyTorch 11.4 framework and implemented on a desktop PC with an NvidiaGeForceGTX·3060TI and 8 GB RAM.

1) Data enhancement

In the medical image processing domain, labeling data is difficult, so the medical image dataset is relatively small. To fully train the segmentation model, we adopt a variety of online data augmentation methods to preprocess the image data in the experiment, including horizontal flip, vertical flip, random 45-degree rotation transformation, and horizontal and vertical offset transformation.

2) Parameter setting

In the experiments, we utilize DRIVE as the target domain dataset and utilize CHASE_DB1 as the source domain dataset. In model training, the Adam optimization technology is used, and the number of iterations is set to 50 in each cycle, for a total of 20 rounds of training. The batch size is set to four. The learning rate is initially set to 0.01, and then cyclically decreased using the cosine law until the learning rate reaches $1\times 1{0}^{-8}$.

4.3.   Verifying the performance of the MSF-Net module

We compare our proposed MSF-Net to some methods with superior performances in recent years. Tables 1 and 2 show the segmentation results of MSF-Net and other methods on the DRIVE and CHASEDB1 datasets. It should be mentioned that in this experiment, there is no domain shift problem to consider, that is, the training and testing samples are from the same database. Here, we only evaluate the segmentation performance of MSF-Net, and we test the effectiveness of our proposed style transfer module, pseudo-label learning strategy and the proposed overall method MSFST-NET when dealing with the issue of domain shift in Sections 3.4 and 3.5.

From Table 1, we find that the AUC, Acc, Se and F1-score values of MSF-Net are 98.50%, 96.05%, 83.67% and 84.46%, respectively. In general, they are better than those of other methods. The Acc of MSF-Net is slightly lower than those of other methods; for example, the Acc of MSF-Net is slightly lower than that of Wu et al. ^[40], but the AUC, Se and F1-score of MSF-Net are higher than those of Wu et al. ^[40] by 0.08%, 3.24% and 2.67%, respectively. Hence, the overall performance of MSF-Net is superior to that of Wu et al. ^[40] on the DRIVE dataset. As shown in Table 2, the AUC, Acc, Se and F1-score of MSF-Net are 98.46%, 96.21%, 83.26% and 81.44%, respectively. MSF-Net gains an optimal performance in the two indicators of AUC and F1-score. The Acc of MSF-Net is not optimal; for example, the Acc is slightly lower than that of Wu et al. ^[40], but the AUC, Se and F1-score of MSF-Net are higher than those of Wu et al. ^[40] by 1.00%, 0.26% and 1.39, respectively. Moreover, this demonstrates that the overall performance of MSF-Net is superior to that of Wu et al. ^[40] on the CHASE_DB1 dataset. The Se of MSF-Net is slightly lower than that of SegR-Net ^[41] by 0.03% on the CHASE_DB1 dataset, but the F1-score of MSF-Net is higher than that of SegR-Net ^[41] by 1.14% on the CHASE_DB1 dataset, and both the Se and F1-score are higher than those of SegR-Net ^[41] by 1.61% and 3.49% on the DRIVE dataset, respectively. Therefore, by comprehensively comparing all the indicators, our proposed MSF-Net has the optimal segmentation precision for retinal blood vessels compared to other previously used methods.

Table 1.  Comparison of MSF-Net and different segmentation methods on DRIVE.

Method	Year	AUC	Acc	Se	F1-score
U-Net [7]	2018	97.55%	95.31%	75.37%	81.42%
Yan et al. [42]	2018	97.50%	95.40%	76.30%	-
Kai Hu [43]	2018	97.59%	95.33%	77.72%	80.98%
BTS-UNet [44]	2019	98.06%	95.61%	78.91%	82.49%
CENe [45]	2019	97.79%	95.45%	83.09%	-
S-Unet [46]	2019	98.21%	95.67%	83.12%	-
IterNet [47]	2020	98.16%	95.73%	77.35%	82.05%
RVSeg-Net [48]	2020	98.17%	96.81%	81.07%	-
Cheng et al. [49]	2020	97.93%	95.59%	76.72%	-
Du et al. [50]	2021	97.80%	95.56%	78.14%
Wu et al [51]	2021	98.16%	95.65%	78.69%	82.21%
Wu et al. [40]	2022	98.42%	96.86%	80.43%	81.79%
SegR-Net [41]	2023	-	-	82.06%	80.97%
MSF-Net	-	98.50%	96.05%	83.67%	84.46%

---

 | Show Table

DownLoad: CSV

Table 2.  Comparison of MSF-Net and different segmentation methods on CHASE_DB1.

Method	Year	AUC	Acc	Se	F1-score
U-Net [7]	2018	97.72%	95.78%	82.88%	77.83%
LadderNet [52]	2018	98.39%	95.33%	79.78%	80.31%
DEU-Net [9]	2019	98.12%	96.61%	80.37%	80.37%
AG-Net [10]	2019	97.79%	97.43%	81.86%	-
Lü et al. [53]	2020	97.82%	96.17%	81.35%	-
MSCNN-AM [54]	2020	98.38%	96.44%	81.32%	-
RVSeg-Net [48]	2020	98.33%	97.26%	80.69%	-
Cheng et al. [49]	2020	97.85%	94.88%	89.67%	-
Du et al. [50]	2021	97.84%	95.90%	81.95%
Wu et al [51]	2021	98.46%	97.02%	79.42%	80.57%
Wu et al. [40]	2022	97.46%	97.46%	83.00%	80.05%
SegR-Net [41]	2023	-	-	83.29%	80.30%
MSF-Net	-	98.46%	96.21%	83.26%	81.44%

---

 | Show Table

DownLoad: CSV

To further demonstrate the performance of MSF-Net, we visualize and compare the segmentation results of MSF-Net and the classical U-Net. Figures 7 and 8 show the segmentation results of different approaches on DRIVE and CHASEDB1.

Figure 7.  Segmentation results of the images from DRIVE.

DownLoad: Full-Size Img PowerPoint

Figure 8.  Segmentation results of the images from CHASE_DB1.

DownLoad: Full-Size Img PowerPoint

From Figures 7 and 8, we find that compared to U-Net, our MSF-Net can obtain a result that is closer to the ground truth. For example, MSF-Net can segment very small blood vessels well, as shown in the region marked by the red box in Figure 7. Moreover, U-Net misclassifies a large number of background pixels as blood vessels, while our MSF-Net can distinguish the background and blood vessels well, as shown in the region marked by the red box in Figure 8.

Based on the above results and analysis, we can conclude that compared to other segmentation approaches, the segmentation results of our MSF-Net are more consistent with the ground truth, that is, MSF-net achieves a better segmentation performance. This is because MSF-Net can extract more blood vessel feature information of different scales by improving the convolution operation based on SK, so it can effectively distinguish the background and blood vessels, especially for small blood vessels.

4.4.   Verifying the performance of the style transfer module and pseudo-label learning strategy

In this experiment, we employ DRIVE as the target domain dataset and adopt CHASE_DB1 as the source domain dataset. After the style transfer, the DRIVE dataset is named DRIVE_style. The images obtained by the style transfer module are shown in Figure 9. In Figure 9, the images in the first row are from the source domain (CHASE_DB1), the images in the second row are from the target domain (DRIVE), and the images in the third row are the images obtained by transferring the style of the image in the target domain to the style of the source domain. From Figure 9, we can see that after the style transfer, the style of the images in the DRIVE dataset has generally been transformed into the style of the images in the CHASE_DB1 dataset.

Figure 9.  The style transfer results.

DownLoad: Full-Size Img PowerPoint

To further verify the positive effect of style transfer on the retinal vessel segmentation task, the style-transferred images obtained by our style transfer module are input into the segmentation model MSF-Net to test the segmentation performance, and the experimental results are shown in Figure 10. In Figure 10, (a) is the original image in DRIVE, (b) is the segmentation result of the original image by employing the MSF-Net model trained on CHASE_DB1, (c) is the style-transferred image of the original image in DRIVE obtained by the style transfer module, (d) is the segmentation result of the image in (c) when using the MSF-Net model trained on CHASE_DB1, and (e) is the corresponding ground truth. Comparing the regions marked by the red rectangular box in Figure 10, it can be found that the segmentation results of the style-transferred images are significantly better than those of the original image. Therefore, this experiment verifies that our style transfer module can alleviate the degradation of model performance caused by the domain transfer.

Figure 10.  The style-transferred image segmentation results.

DownLoad: Full-Size Img PowerPoint

To further improve the generalization ability of our proposed approach, the pseudo-label learning method is adopted after the style transfer. To illustrate the validity of the pseudo-label learning strategy, we compare the segmentation results obtained by the MSF-Net model with and without the pseudo-label learning strategy, as shown in Figure 11. In Figure 11, (a) is the original image in DRIVE, (b) is the segmentation result of the original image, (c) is the segmentation result of the style-transferred image obtained by the style transfer module, (d) is the segmentation result obtained after the style transfer and the pseudo-label learning strategies, and (e) is the ground truth. The abovementioned results are obtained by using MSF-Net trained on the training set of CHASE_DB1. Observing the red rectangular box in Figure 11, it can be found that with the gradual addition of the style transfer module and a pseudo-label learning strategy, the details of the blood vessels in the segmentation results become increasingly clear, which indicates that the use of the pseudo-label learning strategy based on style transfer can further mitigate the adverse effects of the domain shift on the segmentation performance of the model.

Figure 11.  The segmentation results of the model without and with pseudo-label learning.

DownLoad: Full-Size Img PowerPoint

4.5.   Comparison and analysis of the performance of MSFST-Net

In this section, we first test the performance of our proposed overall segmentation method MSFST-Net. Then, we perform an ablation experiment to quantitatively analyze the validity of each component in MSFST-Net. Finally, we test the effect of different parameters in the style transfer module on the segmentation performance.

4.5.1.   Comparison of segmentation results

In this experiment, to verify the ability of the method to deal with cross-domain segmentation problems CHASE_DB1 is utilized as the source domain dataset, DRIVE is used as the target domain dataset, and we compare MSFST-Net with some cross-domain retinal vessel segmentation methods with superior performances in recent years. The adopted evaluation metrics include AUC, F1-Score, and MCC. The comparison results are listed in Table 3. From Table 3, we can find that MSFST-Net achieves 97.65%, 81.27% and 78.68% for the AUC, F1-Score and MCC, respectively, which is generally better than the other methods. The abovementioned results prove that our proposed MSFST-Net outperforms the compared methods.

Table 3.  Comparison of segmentation results of different methods.

Method	AUC	F1-Score	MCC
DANN [55]	97.21%	77.89%	76.02%
DRCN [56]	97.17%	79.17%	77.37%
DRNN [27]	96.80%	79.62%	77.92%
FD [57]	96.88%	79.13%	77.48%
KDFD [57]	97.13%	80.30%	78.57%
AMCD [58]	96.91%	78.60%	-
MSFST-Net	97.65%	81.27%	78.68%

---

 | Show Table

DownLoad: CSV

It should be mentioned that although the evaluation metrics obtained by our proposed model are not significantly higher than those of other methods, our proposed model has fewer parameters than other models. Explicitly, most of the comparison models obtained good segmentation results by designing relatively complex network structures that have more parameters while we designed a relatively lightweight segmentation model, and our model can generally achieve an improved segmentation accuracy than that of these complex networks.

4.5.2.   Comparison of the number of parameters

Table 4 lists the number of parameters (Params) of the different methods and our proposed method. Fewer parameters indicate that the model is more lightweight. In practical applications, our proposed method trains Cycle-GAN offline before segmentation, so the main parameter number of our method in the training process is the parameter number of MSF-Net. From Table 4, we can find that our proposed method is the most lightweight. For example, the number of parameters of FD is 114.58 M, while the number of parameters of our proposed model is approximately 0.11 M. Although we propose a lightweight model, our model can generally achieve a better segmentation accuracy than other methods, as shown in Tables 1-3. In addition, for blood vessel segmentation, the average time to infer and save segmentation results per image is a total of 175 ms in our method, which can meet practical applications in medical scenarios.

Table 4.  Comparison of the number of parameters.

Method	Params
U-Net [7]	35 M
SegR-Net [41]	0.64 M
FD [57]	114.58 M
Ours	0.11 M

---

 | Show Table

DownLoad: CSV

4.5.3.   Ablation experiment results

To objectively verify the performance of each module in MSFST-Net, we test the influence of each module in MSFST-Net on the segmentation performance when handling retinal vessel segmentation with a domain shift problem. The specific results are listed in Table 5.

Table 5.  Ablation experiment results.

Method	AUC	Acc	F1-Score	MCC
Baseline	96.61%	94.89%	76.97%	74.65%
Baseline+Pseudo-label	96.63%	94.98%	78.20%	73.75%
Baseline+Cycle-GAN	96.99%	95.03%	77.79%	75.42%
Baseline+Cycle-GAN+Pseudo-label (MSFST-Net)	97.65%	95.45%	81.27%	78.68%

---

 | Show Table

DownLoad: CSV

In Table 5, the baseline is the optimal MSF-Net model trained on the CHASE_DB1 dataset, and Baseline+Pseudo-label refers to the model that introduces the pseudo-label learning strategy into the baseline. Baseline+Cycle-GAN refers to the model that introduces the style transfer module into the baseline. Baseline+Cycle-GAN+Pseudo-label is the model that introduces both style transfer and pseudo-label learning into the baseline, that is, the final proposed method MSFST-Net. From Table 4, we can summarize the following points. First, after adding the pseudo-label learning strategy to the baseline, most of the evaluation indicator values are increased, which demonstrates that the pseudo-label learning strategy can significantly strengthen the segmentation performance. Second, after adding the style transfer module to the baseline model, most of the evaluation indicators are significantly improved, which means that the style transfer module can effectively reduce the style differences between the different datasets, thus further increasing the segmentation accuracy. Finally, after simultaneously introducing the style transfer module and pseudo-label learning strategy into the baseline model, the values of all evaluation indicators are significantly increased, which verifies that the introduction of the style transfer module and the pseudo-label learning strategy simultaneously have obvious roles in improving the segmentation performance.

4.5.4.   Style transfer module parameter testing

The iteration times of the style transfer module in the training process have a great influence on the segmentation results. Therefore, we carry out different iterations when transferring the style of the image and compare the segmentation results obtained by the different iterations, as shown in Table 6. In Table 6, the baseline is the MSF-Net model.

Table 6.  Comparison of segmentation results under the different iteration times.

Method	AUC	Acc	F1-score	Se
Baseline	96.61%	94.89%	76.97%	69.79%
Baseline+Cycle-GAN (200)	96.44%	94.75%	76.84%	70.34%
Baseline+Cycle-GAN (300)	96.78%	94.94%	77.36%	70.64%
Baseline+Cycle-GAN (400)	96.99%	95.03%	77.79%	71.27%
Baseline+Cycle-GAN (600)	96.62%	94.22%	76.94%	69.33%
Baseline+Cycle-GAN (800)	96.64%	94.77%	76.56%	69.85%

---

 | Show Table

DownLoad: CSV

From Table 6, we can find the following:

1) Compared to the baseline, the values of AUC, Acc and F1-score slightly decrease when the iteration time is 200. The reason for this phenomenon is that Cycle-GAN cannot converge well under 200 iterations; thus, the quality of transferred images does not meet the demand, which further leads to the deterioration of the AUC, Acc and F1-score.

2) Baseline+Cycle-GAN performs better than the baseline when the number of iterations is 300 and 400, and Baseline+Cycle-GAN obtains optimal results in 400 iterations.

3) With an increase in the number of iterations (such as 600 and 800), the performance of Baseline+Cycle-GAN shows a slight decline again, which is due to the excessive number of iterations leading to overfitting of the network, which further leads to a decline in the segmentation performance.

Therefore, 400 is the optimal number of iterations and we select Cycle-GAN trained by 400 iterations as the style transfer module in our work.

To more intuitively show the influence of different iteration times, we further visualize the style transfer results, as shown in Figure 12. From Figure 12, we can observe that the segmentation accuracy gradually improves after gradually increasing the number of training iterations. The preliminary style transfer results are obtained after 200 iterations; however, at this time, the image after the style transfer is missing many details, and the blood vessels are blurred. Conversely, when the number of iterations increases to 400, the blood vessels in the image gradually become clear.

Figure 12.  Results of style transfer with different numbers of iterations.

DownLoad: Full-Size Img PowerPoint

5.   Conclusions

In this paper, a novel MSFST-NET was proposed for segmenting blood vessels, which first introduces SK to construct a new segmentation model MSF-Net to increase the model's segmentation ability for small blood vessels in fundus images. Then, to alleviate segmentation precision degradation caused by domain shift, we introduced a style transfer module and a pseudo-label learning strategy into MSFST-NET. The style transfer module is used to reduce the style difference between the different domain images to improve the segmentation performance. The pseudo-label learning strategy is combined with the style transfer method to further boost the generalization ability of the model. In the experiment, we adopted two datasets (DRIVE and CHASE_DB1) to test the performance of the proposed method. The experimental results verify that the proposed MSFST-NET has a good performance on the retinal vessel segmentation task, and style transfer and pseudo-label learning can strengthen the generalization ability of the model when dealing with cross-domain datasets that have the issue of domain shift.

In this paper, a limitation of our work is that it only focused on 2D medical images. In future work, we aim to explore and assess the performance of MSFST-NET in the context of 3D medical images. This extension will be a crucial step in evaluating the model's applicability in more complex medical imaging scenarios. Moreover, we plan to develop a network based on the large foundation model for medical image segmentation.

Use of AI tools declaration

The authors declare they have not used Artificial Intelligence (AI) tools in the creation of this article.

Acknowledgments

This work was supported by the Fund of the Jilin Provincial Science and Technology Department (No. 20220201157GX), the National Natural Science Foundation of China (No. 62272096), the Fund of Education Department of Jilin Province (No. JJKH20241729KJ), the Scientific Research Project of Jilin Animation Institute (Nos. KY22KZ01, KY21KZ04), the Outstanding Youth Project of Jiangxi Natural Science Foundation (No. 20212ACB212003), the Jiangxi Province Key Subject Academic and Technical Leader Funding Project (No. 20212BCJ23017) and the Qingpu District Industry University Research Cooperation Development Foundation (No. 202314).

Conflict of interest

The authors declare there is no conflict of interest.