Weakly supervised training for eye fundus lesion segmentation in patients with diabetic retinopathy

1.   Introduction

Diabetic retinopathy (DR) is a kind of specific change in fundus lesions and is a serious and syndromic manifestation of diabetes. Recent studies have also shown that DR affects not only the neural retina but also retinal pigment epithelium (RPE) cells in the outer layer of the retina, including changes in melanosomes and lipofuscin granules in RPE cytochrome granules ^[1,2]. According to a clinical meta-eye study ^[3], DR is a serious threat to human vision, leading to a large amount of blindness. Therefore, regular detection of retinopathy is of great significance for the treatment of patients with diabetic retinopathy. However, manual screening for diabetic retinopathy faces challenges such as an imbalance in the doctor-patient ratio and the varying experience of ophthalmologists.

The clinical manifestations of DR mainly include microaneurysm (MA), hemorrhage (HE), hard exudate (EX), soft exudate (SE) and other lesions ^[4]. Studies suggest that the accurate segmentation of key lesions plays a crucial role in the early detection and diagnosis of DR ^[5,6,7]. In recent years, with the continuous development of computer vision technology, more image processing algorithms have been widely used in analyzing medical images for detection, segmentation and classification tasks ^{[8,9,10,11,12,13]}. Shankar et al. applied histogram-based segmentation to fundus images and proposed a synergic deep learning model to grade the severity levels of DR ^[14]. Hire et al. proposed an exudate segmentation method based on ant colony optimization ^[15]. Imani et al. used a morphological component analysis algorithm to separate lesions from vessels and detect exudate regions ^[16]. However, the complexity of lesion appearance and the interference of vessels, noise and imaging artifacts challenge the robustness of previous lesion segmentation algorithms.

Recent deep learning technology development has further improved the performance of fundus segmentation and classification tasks, providing key information for the diagnosis of fundus diseases ^[17,18,19]. Tavakoli et al. studied the automatic detection of MA using a combination of a matching-based approach and a deep learning model and compared the effect of two different image preprocessing methods ^[20]. Yu et al. proposed an end-to-end deep semantic edge learning image segmentation architecture based on ResNet and a skip-layer architecture to address the problem of edge pixels belonging to more than one semantic class ^[21]. Mo et al. proposed cascaded deep residual networks that fuse multilevel hierarchical information to segment exudates accurately and efficiently to recognize DME ^[22]. Chen et al. proposed an encoder-decoder architecture with atrous separable convolution for semantic image segmentation, resulting in a faster and stronger segmentation result. Nevertheless, at this time, the number of fundus images of patients with pixel-level lesion markers is relatively insufficient, and the existing training samples cannot fully reflect the diversity of practical datasets, limiting the segmentation accuracy of a deep learning model to a certain extent. Therefore, methods are required to fully take advantage of the large amount of diverse unlabeled fundus images and improve the robustness of current deep learning-based fundus lesion segmentation algorithms ^[12,13].

In this study, we first designed an efficient lesion segmentation algorithm based on the grayscale and morphological features of fundus lesions and used this algorithm to perform coarse segmentation on unlabeled fundus datasets to expand the number of fundus image samples for training a fine lesion segmentation model based on deep learning. Then, we developed a UNet-based deep learning model that applied the residual structure and attention mechanism for fundus lesion segmentation. The performance of the model was improved by weakly supervised learning, which takes advantage of both a labeled preprocessed fundus image and an unlabeled fundus image with coarse segmentation results.

2.   Methodology

2.1.   Characteristics of fundus lesions

The fundus images used in our study were taken from patients with retinopathy using color fundus photography with a relatively complete fundus structure. In these images, MA usually appears as dark red round dots with distinct boundaries. HE refers to the leakage of blood from abnormal vessels, also mostly in the form of dark red spots, except some in the form of flames. The shape and size are irregular, and the density within the area is not uniform. The HE severity varies with its location. EX is the leakage of lipoprotein and other substances from abnormal blood vessels. It is usually bright yellow or yellowish-white and is usually clumped and ring-shaped, with obvious boundaries. SE is caused by fiber layer ischemia of the retinal nerve, which is white or yellow-white cotton-flocculent with a fuzzy boundary, also known as cotton wool spots ^[23]. Typical examples of the four lesions on a fundus image are shown in Figure 1.

2.2.   Image preprocessing

Each fundus image was first enhanced to improve the detectability of lesions. Based on the traditional adaptive histogram equalization (CLAHE) algorithm ^[24], which is widely used in the preprocessing of image segmentation tasks ^[25], algorithms including gamma correction, contrast transformation enhancement, intensity range adjustment, etc. were applied to enhance the local details and suppress noise in each fundus image.

2.3.   Lesion coarse segmentation algorithm based on grayscale and morphological features

Since the morphology and brightness features carry most lesion information, in this study, we use the grayscale image as the input for the proposed segmentation algorithm, which can not only retain the gradient information but also reduce the feature dimension. Fundus lesions can be classified into dark lesions and bright lesions according to their grayscale characteristics in enhanced fundus images ^[26]. The candidate areas of dark lesions mainly include HE and MA, as well as some fundus blood vessels and image noise/artifacts. The candidate areas of bright lesions mainly include EX and SE, as well as the optic disc region and noise/artifacts. The proposed lesion segmentation process is shown in Figure 2. The numbers in the diagram represent the key steps, which are described below.

1) Dual thresholds (TL and TH) were used to perform preliminary segmentation on the preprocessed grayscale images to obtain the candidate regions of dark and bright lesions, respectively:

where ${f(x, y)}_{rc}$ is a candidate dark lesion region and ${f(x, y)}_{wc}$ is a candidate bright lesion region.

2) A threshold (TB) was selected to binarize the grayscale image to extract all blood vessels and their approximate areas in the fundus image by:

Then, multiple sizes of kernels (MK) were selected to carry out continuous opening and closing morphological operations on the binarized images to connect the small breakpoints in the blood vessels and remove the discrete noise pixels:

The erosion operator ^[27]:

and the dilation operator ^[27]:

where $\left[{ \varepsilon }_{B}\left(X\right)\right]\left(x\right)$ represents the structural element B that corrodes from a child element x of set X, $\left[{\delta }_{B}\left(X\right)\right]\left(x\right)$ represents the expansion operation of structural element B on a child element x in set X, and ${X}_{B}$ represents the x value in the structural element B.

The opening operation ^[27]:

and the closing operation ^[27]:

Finally, to obtain more accurate blood vessel segmentation results, the center point of each remaining region was selected for polynomial fitting verification:

Polynomial fitting verification ^[28]:

where $\Delta$ represents the offset, ${x}_{i}$ and ${y}_{i}$ represent the central coordinates of each region, ${y}_{{x}_{i}}$ represents the output of the polynomial fitting verification with ${x}_{i}$ as its input, ${a}_{0}, {a}_{1}, {a}_{2}$ represent three constants in the polynomial fitting verification, and L represents the maximum allowable offset.

The objective function of polynomial fitting is minimizing the mean square error (MSE) ^[29]:

3) The approximate convergence point of the blood vessels in the image was obtained by calculating the horizontal and vertical histograms:

where i and j represent the row and column of the image, respectively, a and b represent the number of pixels in each row and column, respectively, and p represents the pixel value.

Then, to guarantee that in fundus images with different angles and directions, the proportion of space occupied by the optic disc is always fixed, we took the convergence point as the center and an empirical percent value (P%) of the image resolution as the radius to draw a circular region as the candidate location region of the optic disc.

where ${h(x, y)}_{circle}$ is a circle candidate region of the optic disc, ${x}_{i}{, y}_{i}$ represent the coordinates of each pixel in the image, and ${x}_{c}{, y}_{c}$ represent the coordinates of the convergence point.

Finally, an AND operation was performed to obtain the final segmentation results of the optic disc region:

where${h(x, y)}_{circle}$ is the circle candidate region of the optic disc.

4) The main blood vessels in the dark lesion candidate region were extracted and removed using a vessel segmentation algorithm. The noise was suppressed and removed through a morphological operation, including expansion and corrosion, connected domain calculations, including area calculations, contour calculations and edge processing.

5) The optic disc in the bright lesion candidate region was removed using an optic disc segmentation algorithm, and the noise was also reduced through a morphological operation, connected domain calculation and edge processing methods.

6) After acquiring each lesion's connected domains in an image, the area of the connected domains and the coordinates of the four vertices of the minimum enclosing rectangle were calculated. The screening and identification of MA were realized based on three limiting parameters. The first two limiting parameters filtered the shape of the candidate connected domain based on the characteristic that MA is mostly round or oval, and the third limiting parameter filtered the candidate area of the connected domain based on the assumption that MA is usually small.

The 1^st limiting parameter:

The 2^nd limiting parameter:

The 3^rd limiting parameter:

where ARA is the aspect ratio of the area, RA is the ratio of the area, I is a closed interval, $\lambda$, $\mu$, $\alpha, \beta$ are parameters, h and w represent the length and width of each connected domain, respectively, ${A}_{i}$ represents the area of the circular region drawn with the center of each connected domain as the midpoint and the mean distance between the center point and each contour as the radius, and ${B}_{i}$ represents the area of each connected domain.

Finally, the remaining connected domains were regarded as the MA segmentation results:

7) HE identification and segmentation was realized using an XOR operation:

8) Two groups of threshold values$\left(TG1\right(TG1A, TG1A^ \wedge$) and $TG2(TG2B, TG2B^ \wedge ))$ were selected to obtain the difference in the segmentation edges:

After that, an XOR operation was performed to obtain the SE contours:

Finally, the SE lesion segmentation result was obtained by filling the inner regions of the contours:

9) The EX lesion segmentation result was obtained by conducting an XOR operation:

2.4.   Lesion segmentation based on deep learning

2.4.1.   Framework of RAUNet

UNet is a conventional fully convolutional network for semantic segmentation, which was originally proposed for medical image segmentation ^[30]. UNet has a large number of feature channels in the upsampling part, which allows the network to propagate context information to higher resolution layers. By supplying the expansive path with finer spatial information, skip connections are applied to boost UNet's segmentation accuracy of target borders and improve gradient flow. Therefore, UNet can be trained on fewer data samples ^[31]. The encoder-decoder architecture and skip connection in UNet also help to capture multiscale information in images ^[32]. To further boost the training process of the deep neural network, we introduced a residual module to better solve the problem of network degradation, avoid gradient dispersion and improve the network fitting ability. At the same time, to better allocate the limited information processing resources to the important parts of the model and improve the detection ability of small target lesions, we introduced an attention mechanism. The proposed deep learning-based lesion segmentation network model was named Residual-Attention Unet (RAUNet).

2.4.2.   Finetuning the segmentation model based on weakly supervised training

To improve the robustness of the segmentation model, in this study, we proposed a framework to fine-tune the deep learning model by jointly using the labeled dataset and data with coarse labels obtained using the proposed grayscale and morphological feature-based segmentation method. Generally, weak supervision includes incomplete supervision, inexact supervision and inaccurate supervision ^[33]. The proposed training framework takes a mixed dataset as the input to RAUNet and fine-tunes the weights of the whole network through batch iteration using the Adam optimizer. Therefore, it can be considered a weak supervision structure. The hyperparameters of the deep learning model, including training duration, learning rate, step size, etc., were manually optimized to achieve better model robustness and generalization. The specific training process is shown in Figure 3.

2.4.3.   Evaluation indices

The accuracy and intersection over union (IOU) were selected as the two main evaluation indices for lesion segmentation:

The receiver operating characteristic (ROC) curve was also drawn, and the area under the curve (AUC) was calculated to better evaluate the actual performance of the model.

3.   Results and discussion

3.1.   Data samples

Three groups of data samples were used as input to the deep learning model:

1) A total of 528 labeled fundus images from Beijing Tongren Hospital manually labeled by professional ophthalmologists.

2) A total of 532 fundus images from the OIADDR dataset ^[34] extracted by professionals and annotated by ophthalmologists.

3) A total of 425 fundus images from the Messidor dataset ^[35] with coarse segmentation labels of lesions derived using the proposed segmentation method based on grayscale and morphological features.

We reduced the resolution of each image to 320 × 320 to accommodate images with poor resolution. All 1485 fundus images were randomly divided into a training set and verification set at a ratio of 4:1. All the images in the verification set were labeled manually by ophthalmologists.

3.2.   Coarse segmentation of fundus lesions based on grayscale and morphological characteristics

A total of 528 fundus images with a resolution of 1400 × 1200 labeled by doctors were used to evaluate the proposed grayscale and morphological feature-based segmentation algorithm. The processing time of a single image was approximately 7 s on a PC platform with an RTX 2080 graphics card. The parameters of the proposed coarse segmentation method were set empirically and are shown in Table 1. The overall segmentation accuracies for HE, MA, EX and SE were 27.80, 37.67, 36.30 and 45.32%, respectively, with IOUs of 18.87, 7.58, 29.90 and 41.71%, respectively. The effect of image preprocessing and the actual segmentation effect are shown in Figure 4.

After image preprocessing, the image details were greatly enhanced, and the influence of noise in each image was suppressed; therefore, the visual interpretation of an image was improved. Comparing the segmentation results and the ground truths, we can see that the proposed segmentation algorithm based on the grayscale and morphological features can segment most of the key lesions in an image effectively, and the coarse segmentation results derived were generally in accordance with the ground truth.

3.3.   Comparison with state-of-the-art models

The proposed RAUNet segmentation model was compared with existing state-of-the-art segmentation models. The IDRiD ^[36] open segmentation dataset was employed for the experiment (54 for training and 27 for testing). FCRN, CASENet, DeepLabv3+, LSeg and the proposed RAUNet model were used for training and testing. As seen in Table 2, RAUNet proposed in this paper was superior to the other four models in the segmentation AUC of all four types of lesions, especially for the two types of small target lesions, HE and MA.

3.4.   RAUNet ablation experiments

To verify the contribution of each improvement of the proposed segmentation model, ablation experiments were conducted. The original UNet, UNet with a residual module and the proposed RAUNet (UNet with a residual module and attention mechanism) were trained and evaluated with labeled dataset A (381 for training and 147 for testing).

It can be observed from Table 3 that compared with UNet as the benchmark, for all lesions, the IOU between the segmentation result and ground truth was gradually improved by introducing the residual module and attention mechanism. The IOU obtained by RAUNet was significantly higher than that of the other networks. For the mean segmentation accuracy of all lesions, the result derived by the proposed RAUNet is the highest. We also calculated the mean time required for each model to process a single image. As shown in the last column of Table 3, the processing time of the proposed segmentation algorithm is larger than that of UNet and Res-UNet due to the added attention mechanism. However, we believe this is acceptable since diagnosing DR usually does not require strict real-time processing, and the processing time of the proposed algorithm can be largely reduced by improving the hardware and computation efficiency.

The receiver operating characteristic (ROC) curves shown in Figure 5 are also in accordance with the above results. The proposed RAUNet obtained the largest AUC, followed by UNet combined with the residual and attention modules.

Some typical examples of the segmentation results of the four networks are shown in Figure 6. It can be observed that introducing residual and attention modules improved the UNet visual segmentation result. The residual module increased the model's sensitivity to HE and SE while reducing the false alarm rate of EX. The attention mechanism largely improved the segmentation for small lesions such as HE and MA. Therefore, the proposed RAUNet further obtained results closer to the ground truth labels than the compared networks for all kinds of lesions.

3.5.   Weakly supervised segmentation of fundus lesions based on deep learning

The weakly supervised training model was evaluated on the basis of controlling the validation set unchanged. The loss, accuracy and IOU varying with the number of iteration epochs of the weakly and fully supervised models (taking EX as an example) are shown in Figures 7 and 8, respectively, where red represents the result derived by the verification set and blue represents that of the training set.

As shown in Figure 7, for the proposed weakly supervised model, the evaluation indices on the validation and training datasets showed a generally synchronous upward trend with the increase in the number of training epochs. However, as shown in Figure 8, for the fully supervised learning model, when the number of training epochs reached a certain value, the evaluation indices of the validation set no longer increased with that of the training set, indicating a certain degree of overfitting.

A comparison of the segmentation results derived from the fully and weakly supervised training is shown in Table 4 (weakly supervised: 1200 for training and 285 for testing; fully supervised: 381 for training and 285 for testing). For HE, EX and SE, the segmentation accuracy and IOU of the segmentation results using weakly supervised training were higher than those using fully supervised training. However, the segmentation result of MA was not improved by weakly supervised learning. This may be attributed to the small size of MA, which may be caused by the effect of noise on grayscale and morphological feature-based segmentation. Overall, the segmentation performance was significantly improved by the proposed weakly supervised model. The coarse segmentation results derived from the proposed coarse segmentation model sufficiently take advantage of the large amount of unlabeled data samples and increase the diversity of the lesions by adding small turbulence on the segmentation label, which can avoid overfitting and improve the robustness of the proposed weakly supervised model.

The confusion matrices of the four lesions obtained using weakly supervised training are shown in Figure 9. It can be seen that most of the correctly segmented lesion areas are significantly higher than the misclassified and missed areas. The misclassification rate for HE is higher, resulting in the lowest accuracy and IOU, which is partly due to the complex appearance of this kind of lesion.

4.   Conclusions

Lesion segmentation in fundus images is beneficial to the diagnosis and treatment of patients. However, due to the low contrast, small size and variant appearance characteristics of the lesions, conventional image segmentation algorithms usually cannot obtain satisfactory segmentation results. In this study, a weakly supervised framework was proposed for fundus lesion segmentation using grayscale and morphological features of lesions and a deep neural network.

Experimental results proved that: 1) the CLAHE image enhancement algorithm effectively improved the contrast between lesions and background; 2) an attention mechanism can improve the segmentation accuracy by increasing the sensitivity of the model to small lesions, and a residual module can improve the multiscale feature extraction ability of the network and avoid the problem of gradient dispersion of a deep neural network; 3) the introduction of a coarse labeled training dataset derived from the proposed segmentation method based on the grayscale and morphological features effectively increased the diversity of lesion samples and further improved the generalization and robustness of the segmentation model using weakly supervised training.

Acknowledgments

This work was supported by grants from National Key R & D Program of China (2017YFC0909600), National Natural Science Foundation of China (8151101058, 11527801) and the Strategic Priority Research Program of Chinese Academy of Sciences (XDB41020104).

Conflict of interest

The authors declared that they have no conflicts of interest or competing interests to this work.