Figure 1.
The flowchart of data preprocessing.
In this paper, we introduce a novel deep learning method for dental panoramic image segmentation, which is crucial in oral medicine and orthodontics for accurate diagnosis and treatment planning. Traditional methods often fail to effectively combine global and local context, and struggle with unlabeled data, limiting performance in varied clinical settings. We address these issues with an advanced TransUNet architecture, enhancing feature retention and utilization by connecting the input and output layers directly. Our architecture further employs spatial and channel attention mechanisms in the decoder segments for targeted region focus, and deep supervision techniques to overcome the vanishing gradient problem for more efficient training. Additionally, our network includes a self-learning algorithm using unlabeled data, boosting generalization capabilities. Named the Semi-supervised Tooth Segmentation Transformer U-Net (STS-TransUNet), our method demonstrated superior performance on the MICCAI STS-2D dataset, proving its effectiveness and robustness in tooth segmentation tasks.
Citation: Duolin Sun, Jianqing Wang, Zhaoyu Zuo, Yixiong Jia, Yimou Wang. STS-TransUNet: Semi-supervised Tooth Segmentation Transformer U-Net for dental panoramic image[J]. Mathematical Biosciences and Engineering, 2024, 21(2): 2366-2384. doi: 10.3934/mbe.2024104
| [1] | G. Prethija, Jeevaa Katiravan . EAMR-Net: A multiscale effective spatial and cross-channel attention network for retinal vessel segmentation. Mathematical Biosciences and Engineering, 2024, 21(3): 4742-4761. doi: 10.3934/mbe.2024208 |
| [2] | Caixia Zheng, Huican Li, Yingying Ge, Yanlin He, Yugen Yi, Meili Zhu, Hui Sun, Jun Kong . Retinal vessel segmentation based on multi-scale feature and style transfer. Mathematical Biosciences and Engineering, 2024, 21(1): 49-74. doi: 10.3934/mbe.2024003 |
| [3] | Yanxia Sun, Xiang Li, Yuechang Liu, Zhongzheng Yuan, Jinke Wang, Changfa Shi . A lightweight dual-path cascaded network for vessel segmentation in fundus image. Mathematical Biosciences and Engineering, 2023, 20(6): 10790-10814. doi: 10.3934/mbe.2023479 |
| [4] | Meenu Garg, Sheifali Gupta, Soumya Ranjan Nayak, Janmenjoy Nayak, Danilo Pelusi . Modified pixel level snake using bottom hat transformation for evolution of retinal vasculature map. Mathematical Biosciences and Engineering, 2021, 18(5): 5737-5757. doi: 10.3934/mbe.2021290 |
| [5] | Jinzhu Yang, Meihan Fu, Ying Hu . Liver vessel segmentation based on inter-scale V-Net. Mathematical Biosciences and Engineering, 2021, 18(4): 4327-4340. doi: 10.3934/mbe.2021217 |
| [6] | Chen Yue, Mingquan Ye, Peipei Wang, Daobin Huang, Xiaojie Lu . SRV-GAN: A generative adversarial network for segmenting retinal vessels. Mathematical Biosciences and Engineering, 2022, 19(10): 9948-9965. doi: 10.3934/mbe.2022464 |
| [7] | Rafsanjany Kushol, Md. Hasanul Kabir, M. Abdullah-Al-Wadud, Md Saiful Islam . Retinal blood vessel segmentation from fundus image using an efficient multiscale directional representation technique Bendlets. Mathematical Biosciences and Engineering, 2020, 17(6): 7751-7771. doi: 10.3934/mbe.2020394 |
| [8] | Yinlin Cheng, Mengnan Ma, Liangjun Zhang, ChenJin Jin, Li Ma, Yi Zhou . Retinal blood vessel segmentation based on Densely Connected U-Net. Mathematical Biosciences and Engineering, 2020, 17(4): 3088-3108. doi: 10.3934/mbe.2020175 |
| [9] | Yun Jiang, Jie Chen, Wei Yan, Zequn Zhang, Hao Qiao, Meiqi Wang . MAG-Net : Multi-fusion network with grouped attention for retinal vessel segmentation. Mathematical Biosciences and Engineering, 2024, 21(2): 1938-1958. doi: 10.3934/mbe.2024086 |
| [10] | Xiaoli Zhang, Kunmeng Liu, Kuixing Zhang, Xiang Li, Zhaocai Sun, Benzheng Wei . SAMS-Net: Fusion of attention mechanism and multi-scale features network for tumor infiltrating lymphocytes segmentation. Mathematical Biosciences and Engineering, 2023, 20(2): 2964-2979. doi: 10.3934/mbe.2023140 |
In this paper, we introduce a novel deep learning method for dental panoramic image segmentation, which is crucial in oral medicine and orthodontics for accurate diagnosis and treatment planning. Traditional methods often fail to effectively combine global and local context, and struggle with unlabeled data, limiting performance in varied clinical settings. We address these issues with an advanced TransUNet architecture, enhancing feature retention and utilization by connecting the input and output layers directly. Our architecture further employs spatial and channel attention mechanisms in the decoder segments for targeted region focus, and deep supervision techniques to overcome the vanishing gradient problem for more efficient training. Additionally, our network includes a self-learning algorithm using unlabeled data, boosting generalization capabilities. Named the Semi-supervised Tooth Segmentation Transformer U-Net (STS-TransUNet), our method demonstrated superior performance on the MICCAI STS-2D dataset, proving its effectiveness and robustness in tooth segmentation tasks.
Fundus image diagnosis in the clinic can assist in screening various diseases, such as hypertension and diabetes. Ophthalmologists can make an initial diagnosis of the disease by observing the morphology of retinal blood vessels. However, the accuracy of automatic fundus vessel segmentation is still unsatisfactory, due to the complex morphology of the vessels and the observer dependence. Therefore, accurate retinal vessel segmentation technologies are extremely valuable in clinical environments.
Currently, the automatic segmentation methods of fundus blood vessels can be mainly divided into two categories: machine learning and deep learning. Furthermore, according to different strategies, machine learning-based approaches can be divided into unsupervised and supervised approaches.
For unsupervised machining methods, Chaudhuri et al. [1] introduced a Gaussian function in the segmentation task for the problem of low local contrast, and successfully designed a two-dimensional matched filter to detect blood vessel segments in images. Li et al. [2] constructed a simple and efficient multi-scale filtering method based on the response relationship of matched filters at three scales. After that, Sreejini et al. [3] introduced the particle swarm optimization algorithm in the multi-scale matched filter method, and discussed the process more comprehensively. The matched filter method is easy to implement, and the amount of calculation is relatively small. However, this method is highly restricted by factors, such as image contrast and noise, and its ability to distinguish vessel pixels from background ones is relatively poor.
In addition, Aibinu et al. [4] proposed a method for segmentation at the crossing and branching of vessels, which uses a hybrid crossing point method to identify the crossing and branching points of vessels, realize vessel tracking and extraction. Finally, Vlachos et al. [5] proposed a linear multi-scale tracking method, which tracks the gray-scale characteristics of blood vessel pixels from the initial seed node to form a gridded extraction of blood vessels. The blood vessel tracking method can obtain very accurate blood vessel width. However, the segmentation effect largely depends on the selection of the initial seed node. In addition, it is susceptible to noise interference, and the problem of segmenting blood vessel breakage occurs.
Moreover, Zana et al. [6] first determined the Gaussian-like contours of blood vessels, and then combined morphological processing with cross-curvature evaluation for segmentation. Fraz et al. [7] further obtained the blood vessel's skeleton based on detecting the centerline of the blood vessel. It received the direction map with the help of morphological plane slices, and generated the shape of the blood vessel at the same time. The vessel neutral line image is reconstructed through the orientation map and vessel shape, and, finally, the segmented vessel choroid map is obtained. Yang et al. [8] proposed a hybrid method to extract blood vessels based on mathematical morphology and fuzzy clustering. However, these traditional unsupervised-based image processing methods are less robust. Besides, they suffer from poor generalization ability due to customizing artificial features and expert annotations for specified datasets based on prior knowledge.
For supervised machine learning-based methods, its segmentation accuracy is higher than that of unsupervised ones, but with a possibility of overfitting. It has been widely used in retinal vessel segmentation. Staal et al. [9] used the K-nearest neighbor (KNN) algorithm to intercept the first k data to compare further and determine its pixel category, which is essentially a binary classification of each pixel. Soares et al. [10] used a two-dimensional Gabor filter to extract the overall features of retinal images, and then used a naive Bayesian classifier to classify retinal vessels and backgrounds. Osareh et al. [11] first computed feature vectors on a per-pixel basis, and then used a Gaussian mixture model combined with a support vector machine to classify the feature vectors. Khowaja et al. [12] proposed a framework based on a hybrid feature set and hierarchical classification approach. They first employed random forests for classification and evaluating the performance of each feature class for feature selection, and then combined the selected feature set with a hierarchical classification approach for vessel segmentation. Conventional machine learning methods is more suitable for scenarios with a small amount of data. However, deep learning performance will be more prominent if the amount of data increases rapidly.
Deep learning-based methods can automatically learn vessel features from the retinal image and avoid manual participation. Therefore, it has stronger robustness, higher frontal segmentation accuracy and more vital generalization ability. In recent years, deep learning has shown excellent performance in the field of medical image segmentation, and many researchers have conducted research in retinal blood vessel segmentation. Specifically, the proposal of U-Net [13] makes the U shape network a popular framework, and many improved models are proposed for retinal vessel segmentation. For example, Wu et al. [14] proposed a multi-scale network followed network (MS-NFN) to solve the small vessel segmentation problem. Zhuang et al. [15] introduced multiple encoding and decoding structures in their LadderNet, and increased the information flow path via skip connections. Alom et al. [16] proposed a recursive residual convolutional neural network (R2U-net) based on the U-shaped network model, which better retains feature information and achieves the effect of feature reuse. Finally, Li et al. [17] proposed a small U-Net segmentation method IterNet with multiple iterations, which expands the model's depth while considering the segmentation details. Gu et al. [18] proposed CE-Net, which introduces a cascaded upper and lower feature extraction module in the middle layer of the codec. It can ensure the acquisition of complete feature information, and extract deeper feature information. Lin et al. [19] proposed a multi-path high-resolution retinal vessel segmentation method combined with HR-Net. The feature map maintained high resolution in the feature extraction process, and enabled the information interaction between high and low-resolution branches, thus resulting in more accurate probability maps.
Although the models mentioned above achieved good results in retinal blood vessel segmentation, there are still some limitations:
➢ The codec structure transmits and receives information features in a single layer through skip connections, which aggravates the problem of information loss.
➢ The connecting layer of codec cannot thoroughly combine context information, and continuous pooling and convolution further cause a decrease in the recognition rate of the vessel ends.
To alleviate the above problems, this paper proposes a multi-scale integrated context network to improve blood vessel segmentation accuracy further. Aiming at the issues that existing algorithms have an insufficient feature extraction ability, serious feature information loss and low segmentation accuracy for color fundus retinal vascular images, an aggregated multi-scale integrated context model is proposed to further improve the accuracy of vascular segmentation. Its main new feature is that it has a multi-layer feature fusion mechanism, which can fully use the information in different scales. The main contributions of the network are as follows:
➢ A multi-layer feature fusion mechanism is proposed, combining the low-level details of feature maps at different scales with high-level semantic information through full-scale skip connections from the encoding path to the decoding path.
➢ In the encoder, a hybrid stride sampling (HSS) block is constructed to extract deeper semantic information while reducing the dimensionality of features.
➢ A Dense Hybrid Dilated Convolution (DHDC) block is designed between the encoder and the decoder to improve the accurate recovery of blood vessel details by obtaining richer contextual information.
➢ The Squeeze-and-Excitation module with residual connections is introduced into the decoder, and the weight of each scale feature is adaptively adjusted to strengthen the effective channel while suppressing redundant information.
The rest of this paper is organized as follows: Section Ⅱ introduces the proposed method in detail. Section Ⅲ describes the experimental implementation and illustrates the experimental results. Section Ⅳ gives the conclusions of this paper.
The retinal images in the fundus dataset have many samples with poor contrast and high noise. Therefore, proper preprocessing is critical for later training. This paper uses four preprocessing methods, including gray-scale transformation, data standardization, contrast limited adaptive histogram equalization (CLAHE) and gamma correction [18,20,21,22], to process each original retinal blood vessel image.
Figure 1 provides a schematic diagram of the staged processing results of the original color retinal image after gray-scale transformation, contrast-limited adaptive histogram equalization and gamma correction. It can be seen from the figure that the image texture after preprocessing is clear, the edge is prominent, and the detailed information is enhanced.
We use a patch extraction strategy to augment the experimental data and avoid the overfitting problem. There are three patch extraction ways: sequential crop, overlap crop and random crop.
This paper selects random cropping for data augmentation in the training phase. In addition, to maintain the consistency of the training data, we performed the same augmentation processing for the ground truth images manually segmented by experts.
Different from the cropping approach in the training phase, in the testing phase, each image block needs to be re-spliced into a complete image and then binarized to obtain the segmentation result map. All patches must be spliced to restore their separation ratios to the level of the original fundus image. However, if random cropping is used, the time and space complexity of splicing according to the index is extremely high. To avoid this problem, the overlap crop is chosen in the testing phase. We set the stride to 12 based on the trade-off of workstation performance.
As shown in Figure 2, each test image is divided into several patches by the overlapping cropping strategy, and Eq (1) calculates the number of patches for each image:
| N_patches_per_img=(⌊img_h−patch_hstride_h⌋+1)×(⌊img_w−patch_wstride_w⌋+1), | (1) |
where img_h and img_w represent the height and width of the test image, patch_h and patch_w represent the height and width of the image block, and stride_h and stride_w represent the step size of horizontal and vertical sliding, respectively. After obtaining the prediction results of overlapping patches, a reconstruction algorithm continuously reconstructs the final average segmentation results (final_avg) by Eq (2).
| final_avg=full_profull_sum, | (2) |
where full_pro and full_sum represent the sum of each pixel's prediction probability and extraction frequency in each patch, respectively.
Figure 3 presents the overall framework of the MIC-Net proposed in this paper (the source code is publicly available at https://github.com/Mamdanni/MIC-Net). The network preserves the two-layer end-to-end basic structure of U-Net. First,a hybrid stride sampling (HSS) block is designed in the encoder,which uses full-scale skip connections to replace the single one of U-Net. Second,the interconnection between encoding and decoding paths is redesigned as the dense hybrid dilated convolution (DHDC) block. Finally,the decoder utilized the Squeeze-and-Excitation module with residual connections. Each part is described in detail below.
The max pooling downsampling is mainly used to reduce the image's resolution. However, valuable information is lost while extracting features. To minimize the data loss caused by the downsampling, we designed an HSS block in the encoding process. It performs the downsampling process before two successive convolutions, which can reduce the feature dimension. Meanwhile, it alleviates the loss of helpful information as much as possible and thus extracts deeper semantic information.
As shown in Figure 4, the HSS module is implemented by paralleling a convolution operation with a convolution kernel size of 2 × 2 or 4 × 4, a stride of 2 or 4, and a pooling kernel size of 2 × 2 or 4 × 4. It is executed by max pooling downsampling with stride 2 or 4.
Compared with downsampling by a single max-pooling or a fixed stride convolution, the proposed sampling module could reduce the information loss caused by dimension reduction. Besides, it is worth noting that we also conducted the HSS block before the full-scale skip connection. On the other hand, upsampling uses transposed convolution to achieve the fusion of different scale features.
Skip connections can fuse the high-resolution information from the encoder with the decoder upsampled feature maps, thereby helping to refine the tiny features of the segmentation map. However, though the single skip connection is intuitive and straightforward, the single-layer transmission and reception information features lead to the inability to utilize the full-scale information fully.
However, the feature maps of different scales often contain extra information. Therefore, this paper introduces the full-scale skip connections mechanism, as shown in Figure 5. First, the feature map is re-sampled to a uniform size before downsampling at each layer. Then, we implement full-scale skip connections from the encoding path to the decoding path, thereby achieving a fusion of feature maps at different scales, that is, a complete fusion of low-level details and high-level semantic information.
To improve the receptive field without losing information and cascade convolutional convolutions with different dilation ratios to obtain multi-scale information gain, we developed the DHDC block (shown in Figure 6) between the encoder and the decoder inspired by DenseASPP [23]. A set of atrous convolutions are connected in the form of dense connections, and atrous convolutional layers share information through residual connections. Among them, d represents the expansion rate of the hole convolution. The convolution layers with different expansion rates are interdependent. The feedforward process will not only form a denser feature pyramid, but also increase the receptive field of the convolution kernel to perceive richer contextual information.
During the decoding process, we use twice transposed convolution with a kernel size of 3 × 3 to upsample the feature map. Besides, we introduce squeeze-and-excitation with residual connections (SERC) in the decoder (as shown in Figure 7).
We first fused the feature maps obtained by sampling with that of different scales, and then input the residual SERC block. As a result, the weight of each scale feature is adaptively adjusted. It, thus, strengthens the effective channel and suppresses redundant information. Finally, the channels are adjusted using a 1 × 1 convolution kernel.
Eight standard evaluation metrics for retinal vessel segmentation tasks include Accuracy (Acc), Specificity (Spe), Sensitivity (Sen), Precision (Pre), and F1_Score, Intersection over Union (IoU), Floating point operations (FLOPs) and Parameters (Params) [17,20,22,39].
In addition, we depicted the receiver operating characteristic curve (ROC), which was generated with TP as the ordinate and FP as the abscissa. We also provided the Area under the ROC curve (AUC), which considers the Sen and Spe under different thresholds, and is suitable for measuring retinal vessel segmentation.
We evaluated our proposed method on three public datasets of fundus images, including DRIVE, STARE and CHASE. Figure 8 shows some typical cases from the three datasets.
The DRIVE dataset (https://drive.grand-challenge.org/) contains 40 fundus retinal color images, seven of which are from patients with early diabetic retinopathy, with a resolution of 565 × 584 and stored in JPEG format. The original dataset uses 20 images for training and 20 for testing with masks, and two experts manually annotated the dataset. In this paper, we divide the dataset into a training set, a validation set, and a test set according to the ratio of 18:2:20, and choose the first expert's result as the ground truth. Specifically, 110,000 image patches were obtained based on the original dataset for the later training.
The STARE dataset (http://cecas.clemson.edu/~ahoover/stare/) provides 20 fundus color images with a resolution of 700 × 605. We use 15 of these images for training and five for testing. The original dataset is not divided into a validation set like the DRIVE dataset. Thus, we choose 10% of the training data for validation. The STARE dataset also provides annotated images of two experts, and we chose the first ones as the ground truth. Finally, a total of 130,000 image patches were obtained.
The CHASE dataset (https://blogs.kingston.ac.uk/retinal/chasedb1/) contains 28 color retinal images with a resolution of 996 × 960. It was taken from the left and right eyes of 14 children. We used 20 of these images for training and eight for testing, and a total of 230,000 patches were extracted for training.
All experiments were run on a GPU server with Intel Xeon Silver 4110 CPU, NVIDIA GeForce RTX 2080Ti GPU and 64GB RAM. The development environment is based on CUDA11.2 + cuDNN8.1 + TensorFlow2.6.0 + keras2.6.0, Python 3.7.13 and the Ubuntu 18.04 operating system.
In the training process, we set the maximum number of training epochs to 30, batch_size to 4, and initial learning rate to 0.0001. A binary cross-entropy loss (BCE) was used as the objective function to supervise the model's training process. The DRIVE, STARE and CHASE datasets followed the same data augmentation strategy: randomly extract image patches with a resolution of 48 × 48 from the preprocessed images.
Unlike the training phase, we performed overlap cropping in the original image with a fixed step size. Since these patches had overlapping areas (i.e., each pixel appears multiple times in different patches), we averaged the probability value of each pixel belonging to retinal blood vessels and set the threshold to obtain a binarized prediction map. In addition, we used an early stopping mechanism to prevent the occurrence of overfitting,
The proposed MIC-Net needed to convert the vessel segmentation task into pixel-level classification. Therefore, we chose the BCE loss function to complete the classification task in this paper. Its equation is defined as follows:
| Loss=−1NN∑i=1[gilogpi+(1−gi)log(1−pi)] | (3) |
In the above formula, g represents the label value. There are only two possible values of 0 and 1, and p represents the predicted value of the pixel. When g is 0, the first half of the formula equals 0. If you want the loss value to be smaller, p should be as close to 0 as possible; conversely, when g is 1, the second half of the formula is 0. For a minor loss, p should be as close to 1 as possible. Furthermore, the sigmoid activation function is necessary to ensure that the model output is in the range of (0, 1).
We conducted ablation experiments on the DRIVE dataset to verify each module's contribution to the entire model's performance. As can be seen from Table 1, the AUC, Acc, Spe and Sen of MIC-Net reached 98.62, 97.02, 98.80 and 80.02%, respectively. Compared with the baseline model in the DRIVE dataset, the performance of the final model is improved by 0.2, 0.22, 0.23 and 0.31%, respectively.
| Methods | AUC (%) | Acc (%) | Spe (%) | Sen (%) |
| Baseline | 98.42 | 96.80 | 98.57 | 78.50 |
| No FSC | 98.48 | 96.89 | 98.85 | 76.40 |
| No DHDC | 98.54 | 96.96 | 98.58 | 78.81 |
| No SERC | 98.58 | 96.99 | 98.69 | 79.23 |
| MIC-Net | 98.62 | 97.02 | 98.80 | 80.02 |
| *Note: For each metric, the bold value indicates that column's best result. | ||||
DownLoad:
CSV
Figure 9 shows the segmentation results of the MIC-Net model for different combinations of ablation models. It can be observed from the figure that the segmentation results of the complete MIC-Net model proposed by us have more minor errors than other combinations, and are closer to the standard segmentation images, thus verifying the rationality of the model combination.
Generally, the number of parameters of a model is directly proportional to its computational complexity, while fewer model parameters often degrade the performance of the network. Table 2 lists the number of parameters, FLOPs and AUC (on DRIVE dataset) of different methods. Compared with other existing models, the parameters and FLOPs of our proposed MIC-Net are comparable to those of Att-Unet, MultiResUNet, and FCN = 8s (all less than 10M). Besides, the segmentation performance of our proposed method is slightly higher than other models. Therefore, the proposed MIC-Net achieves high segmentation performance with lower computational complexity.
| Methods | Parameters (M) | FLOPs (M) | AUC |
| SegNet | 29.46 | 58.91 | 0.9294 |
| FCN_8s | 9.01 | 18.01 | 0.9410 |
| MultiResUNet | 7.26 | 14.55 | 0.9451 |
| LinkNet | 11.55 | 23.62 | 0.9492 |
| DeepLabV3+ | 41.06 | 82.23 | 0.9575 |
| Att-UNet | 8.91 | 17.82 | 0.9793 |
| R2U-Net | 17.65 | 51.03 | 0.9804 |
| MIC-Net | 9.13 | 18.23 | 0.9862 |
| *Note: For each metric, the bold value indicates that column's best result. | |||
DownLoad:
CSV
To verify the model's generalization performance, we trained on the DRIVE dataset, and applied the saved weights to testing on the STARE dataset. Figure 10 shows our segmentation results. The generalization visualization experiment shows that the detection of vessels is relatively complete, and the ends and bifurcations can also be completely segmented, which verifies the consistency of the proposed method in different data distributions, and its generalization ability is strong.
In addition, Table 3 lists the generalization performance comparison between the proposed method and other methods. Among them, the method proposed in this paper has obtained the optimal value in the two indicators of AUC and Sen, thus verifying that the proposed method has a good consistency and generalization ability.
| Methods | AUC (%) | Acc (%) | Spe (%) | Sen (%) |
| Yan et al. [41] | 97.08 | 95.69 | 98.40 | 72.11 |
| Jin et al. [42] | 94.45 | 96.90 | 97.59 | 70.00 |
| Wu et al. [22] | 96.35 | 95.44 | 97.85 | 73.78 |
| Proposed method | 97.70 | 96.44 | 97.51 | 76.02 |
DownLoad:
CSV
Figure 11 and Table 4 present our proposed method's partial segmentation results on three datasets. As can be seen from the table and figure, the segmentation result of our proposed MIC-Net is very close to the ground truth, which cannot only extract the main vessel from the background, but also correctly segment the vessel edge.
| Dataset | AUC (%) | Acc (%) | Spe (%) | Sen (%) | Pre (%) | F1-score (%) | IoU (%) |
| DRIVE | 98.62 | 97.02 | 98.80 | 80.02 | 86.32 | 82.20 | 68.32 |
| STARE | 98.60 | 97.76 | 98.61 | 87.72 | 93.70 | 85.99 | 69.51 |
| CHASE | 98.73 | 97.38 | 98.44 | 81.60 | 77.95 | 79.73 | 64.07 |
DownLoad:
CSV
In addition, we also give the ROC curves of the three datasets in Figure 12. It can be seen from the figure that the value of AUC is relatively close to 1, which proves the superior performance of our proposed method on retinal vessel segmentation.
Although the proposed MIC-Net can successfully segment blood vessels, some intractable abnormalities still occur. For example, as shown in Figure 13, in the segmentation results of the DRIVE dataset and the CHASE dataset, we found two cases where the optic disc boundary was identified as a blood vessel, which indicates that the specificity of the proposed method needs to be improved.
*Note: For each metric, bold values are the best result in that column.
We further compare the proposed method with existing state-of-the-art (SOTA) methods to verify the effectiveness of the proposed method. Table 5 presents our quantitative experimental results on the DRIVE, STARE and CHASE datasets. All compared methods refer to open-source codes on GitHub. The experiment's environment configuration and the specific parameter settings, including batch size, epoch, learning rate, etc., are consistent with the settings of our proposed method in this paper during training and testing.
| Methods | Year | DRIVE(%) | STARE(%) | CHASE(%) | |||||||||||
| AUC | Acc | Spe | Sen | AUC | Acc | Spe | Sen | AUC | Acc | Spe | Sen | ||||
| Azzopardi [24] | 2015 | 96.14 | 94.42 | 97.04 | 76.55 | 95.63 | 94.97 | 97.01 | 77.16 | 94.87 | 93.87 | 95.87 | 75.85 | ||
| Li et al. [25] | 2015 | 97.38 | 95.27 | 98.16 | 75.69 | 98.79 | 96.28 | 98.44 | 77.26 | 97.16 | 95.81 | 97.93 | 75.07 | ||
| Liskowski [26] | 2016 | 97.20 | 94.95 | 97.68 | 77.63 | 97.85 | 95.66 | 97.54 | 78.67 | - | - | - | - | ||
| Fu et al. [27] | 2016 | - | 95.23 | - | 76.03 | - | 95.85 | - | 74.12 | - | 94.89 | - | 71.30 | ||
| Dasgupta [28] | 2016 | 97.44 | 95.33 | 98.01 | 76.91 | - | - | - | - | - | - | - | - | ||
| Chen et al. [29] | 2017 | 95.16 | 94.53 | 97.35 | 74.26 | 95.57 | 94.49 | 96.96 | 72.95 | - | - | - | - | ||
| Yan et al. [30] | 2018 | 97.52 | 95.42 | 98.18 | 76.53 | 98.01 | 96.12 | 98.46 | 75.81 | 97.81 | 96.10 | 98.09 | 76.33 | ||
| Wu et al. [14] | 2018 | 98.07 | 95.67 | 98.19 | 78.44 | - | - | - | - | 98.25 | 96.37 | 98.47 | 75.38 | ||
| Yan et al. [31] | 2019 | 97.50 | 95.38 | 98.20 | 76.31 | 98.33 | 96.38 | 98.57 | 77.35 | 97.76 | 96.07 | 98.06 | 76.41 | ||
| Jin et al. [20] | 2019 | 98.02 | 95.66 | 98.00 | 79.63 | 98.32 | 96.41 | 98.78 | 75.95 | 98.04 | 96.10 | 97.52 | 81.55 | ||
| Wang et al. [32] | 2020 | 98.23 | 95.81 | 98.13 | 79.91 | 98.81 | 96.73 | 98.44 | 81.86 | - | - | - | - | ||
| Li et al. [17] | 2020 | 98.16 | 95.73 | 98.38 | 77.35 | 98.81 | 97.01 | 98.86 | 77.15 | 98.51 | 96.55 | 98.23 | 79.70 | ||
| Shi et al. [33] | 2021 | - | 96.76 | 98.26 | 80.65 | - | 97.32 | 98.66 | 82.90 | - | 97.31 | 98.89 | 75.04 | ||
| Guo et al. [34] | 2021 | 98.53 | 96.67 | 98.17 | 82.21 | 98.97 | 97.24 | 98.59 | 82.10 | 98.69 | 96.97 | 98.45 | 81.89 | ||
| Xu et al. [35] | 2015 | 96.70 | 96.30 | 98.23 | 87.45 | - | - | - | - | 96.77 | 96.94 | 97.94 | 89.16 | ||
| Zhang et al. [36] | 2022 | 88.95 | 97.01 | 97.99 | 77.19 | 83.91 | 96.91 | 99.11 | 69.12 | 91.42 | 98.11 | 99.81 | 85.06 | ||
| Deng et al. [37] | 2022 | 97.93 | 95.39 | 97.12 | 83.68 | 98.55 | 96.43 | 97.79 | 84.35 | 98.06 | 95.87 | 96.93 | 85.43 | ||
| Our MIC-Net | 2022 | 98.62 | 97.02 | 98.80 | 80.02 | 98.60 | 97.76 | 98.61 | 87.72 | 98.73 | 97.38 | 98.44 | 81.60 | ||
DownLoad:
CSV
In the comparative experiment, the statistical method of t-test is also used to verify whether the proposed method is significantly different from other methods on the accuracy. All statistical hypothesis tests are based on the representative metrics of Acc.
It can be seen that the overall performance of our proposed method achieves significant improvements on all performance metrics, outperforming existing SOTA methods on the three datasets. Besides, regarding AUC metrics, our AUC is 0.46% higher than the second on the DRIVE dataset, 0.52% higher than the second on the STARE dataset and 0.41% higher than the second on the CHASE dataset.
However, compared with the experiments of existing methods, the proposed module has improved the performance of retinal vessel detection. Still, Spe and Sen performances on the three datasets are not outstanding, which indicates that, although the performance improvement of our proposed method is achieved to a certain extent, there is still a significant error on FP and FN.
Figure 13 compares the visualization results of several SOTA methods on three datasets. Our method achieves satisfactory segmentation results from the locally zoom-in images. Compared with other improved approaches, such as scSEU-Net [38] and R2U-Net [16], our proposed MIC-Net can detect retinal vessels more correctly and reduce misclassified retinal vessel pixels. In addition, better recognition is achieved for tiny vessels and edge regions.
This paper proposes an end-to-end fundus retinal vessel segmentation network called MIC-Net. This multi-layer feature fusion mechanism can fully utilize the information on different scales to improve information flow. First, the HSS block we designed on the encoder side can minimize the loss of helpful information caused by the downsampling operation. Second, the DHDC block between the encoder and decoder can perceive richer contextual information without sacrificing feature resolution. Third, the SERC module at the decoder can strengthen the effective channel, while suppressing redundant information.
The experimental results show that the performance of our proposed method on the DRIVE, STARE and CHASE datasets could achieve comparable segmentation results to existing SOTA methods on retinal vessel segmentation. Thus, it has significant application prospects in the early screening of diabetic retinopathy. Nevertheless, the proposed MIC-Net still has some limitations for segmenting tiny blood vessels that cannot be effectively distinguished by direct observation of the human eye. Therefore, our future work will focus on the cascaded semantic segmentation framework for segmenting small blood vessels.
This work was supported by the National Nature Science Foundation (No. 61741106, 61701178).
We declare that there are no conflicts of interest.
| [1] |
P. Sanchez, B. Everett, Y. Salamonson, S. Ajwani, S. Bhole, J. Bishop, et al., Oral health and cardiovascular care: Perceptions of people with cardiovascular disease, PLoS One, 12 (2017), e0181189. https://doi.org/10.1371/journal.pone.0181189 doi: 10.1371/journal.pone.0181189
|
| [2] | M. P. Muresan, A. R. Barbura, S. Nedevschi, Teeth detection and dental problem classification in panoramic x-ray images using deep learning and image processing techniques, in 2020 IEEE 16th International Conference on Intelligent Computer Communication and Processing (ICCP), IEEE, (2020), 457–463. https://doi.org/10.1109/ICCP51029.2020.9266244 |
| [3] |
V. Hingst, M. A. Weber, Dental X-ray diagnostics with the orthopantomography–technique and typical imaging results, Der Radiologe, 60 (2020), 77–92. https://doi.org/10.1007/s00117-019-00620-1 doi: 10.1007/s00117-019-00620-1
|
| [4] |
J. C. M. Román, V. R. Fretes, C. G. Adorno, R. G. Silva, J. L. V. Noguera, H. Legal-Ayala, et al., Panoramic dental radiography image enhancement using multiscale mathematical morphology, Sensors, 21 (2021), 3110. https://doi.org/10.3390/s21093110 doi: 10.3390/s21093110
|
| [5] |
R. Izzetti, M. Nisi, G. Aringhieri, L. Crocetti, F. Graziani, C. Nardi, Basic knowledge and new advances in panoramic radiography imaging techniques: A narrative review on what dentists and radiologists should know, Appl. Sci., 11 (2021), 7858. https://doi.org/10.3390/app11177858 doi: 10.3390/app11177858
|
| [6] |
Y. Zhao, P. Li, C. Gao, Y. Liu, Q. Chen, F. Yang, et al., Tsasnet: Tooth segmentation on dental panoramic X-ray images by Two-Stage Attention Segmentation Network, Knowledge-Based Syst., 206 (2020), 106338. https://doi.org/10.1016/j.knosys.2020.106338 doi: 10.1016/j.knosys.2020.106338
|
| [7] |
A. E. Yüksel, S. Gültekin, E. Simsar, Ş. D. Özdemir, M. Gündoğar, S. B. Tokgöz, et al., Dental enumeration and multiple treatment detection on panoramic X-rays using deep learning, Sci. Rep., 11 (2021), 12342. https://doi.org/10.1038/s41598-021-90386-1 doi: 10.1038/s41598-021-90386-1
|
| [8] |
R. J. Lee, A. Weissheimer, J. Pham, L. Go, L. M. de Menezes, W. R. Redmond, et al., Three-dimensional monitoring of root movement during orthodontic treatment, Am. J. Orthod. Dentofacial Orthop., 147 (2015), 132–142. https://doi.org/10.1016/j.ajodo.2014.10.010 doi: 10.1016/j.ajodo.2014.10.010
|
| [9] | J. Keustermans, D. Vandermeulen, P. Suetens, Integrating statistical shape models into a graph cut framework for tooth segmentation, in Machine Learning in Medical Imaging, Springer, (2012), 242–249. https://doi.org/10.1007/978-3-642-35428-1_30 |
| [10] | O. Ronneberger, P. Fischer, T. Brox, U-Net: Convolutional networks for biomedical image segmentation, in Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015, Springer, (2015), 234–241. https://doi.org/10.1007/978-3-319-24574-4_28 |
| [11] |
W. Wang, X. Yu, B. Fang, Y. Zhao, Y. Chen, W. Wei, et al., Cross-modality LGE-CMR segmentation using image-to-image translation based data augmentation, IEEE/ACM Trans. Comput. Biol. Bioinf., 20 (2023), 2367–2375. https://doi.org/10.1109/tcbb.2022.3140306 doi: 10.1109/tcbb.2022.3140306
|
| [12] |
W. Wang, J. Chen, J. Wang, J. Chen, J. Liu, Z. Gong, Trust-enhanced collaborative filtering for personalized point of interests recommendation, IEEE Trans. Ind. Inf., 16 (2020), 6124–6132. https://doi.org/10.1109/tii.2019.2958696 doi: 10.1109/tii.2019.2958696
|
| [13] |
B. G. He, B. Lin, H. P. Li, S. Q. Zhu, Suggested method of utilizing soil arching for optimizing the design of strutted excavations, Tunnelling Underground Space Technol., 143 (2024), 105450. https://doi.org/10.1016/j.tust.2023.105450 doi: 10.1016/j.tust.2023.105450
|
| [14] |
J. Chen, S. Sun, L. Zhang, B. Yang, W. Wang, Compressed sensing framework for heart sound acquisition in internet of medical things, IEEE Trans. Ind. Inf., 18 (2022), 2000–2009. https://doi.org/10.1109/tii.2021.3088465 doi: 10.1109/tii.2021.3088465
|
| [15] |
J. Chen, W. Wang, B. Fang, Y. Liu, K. Yu, V. C. M. Leung, et al., Digital twin empowered wireless healthcare monitoring for smart home, IEEE J. Sel. Areas Commun., 41 (2023), 3662–3676. https://doi.org/10.1109/jsac.2023.3310097 doi: 10.1109/jsac.2023.3310097
|
| [16] |
Y. Zhang, X. Wu, S. Lu, H. Wang, P. Phillips, S. Wang, Smart detection on abnormal breasts in digital mammography based on contrast-limited adaptive histogram equalization and chaotic adaptive real-coded biogeography-based optimization, Simulation, 92 (2016), 873–885. https://doi.org/10.1177/0037549716667834 doi: 10.1177/0037549716667834
|
| [17] |
J. H. Lee, S. S. Han, Y. H. Kim, C. Lee, I. Kim, Application of a fully deep convolutional neural network to the automation of tooth segmentation on panoramic radiographs, Oral Surg. Oral Med. Oral Pathol. Oral Radiol., 129 (2020), 635–642. https://doi.org/10.1016/j.oooo.2019.11.007 doi: 10.1016/j.oooo.2019.11.007
|
| [18] |
J. Chen, Z. Guo, X. Xu, L. Zhang, Y. Teng, Y. Chen, et al., A robust deep learning framework based on spectrograms for heart sound classification, IEEE/ACM Trans. Comput. Biol. Bioinf., 2023 (2023), 1–12. https://doi.org/10.1109/TCBB.2023.3247433 doi: 10.1109/TCBB.2023.3247433
|
| [19] |
S. H. Wang, D. R. Nayak, D. S. Guttery, X. Zhang, Y. D. Zhang, COVID-19 classification by CCSHNet with deep fusion using transfer learning and discriminant correlation analysis, Inf. Fusion, 68 (2021), 131–148. https://doi.org/10.1016/j.inffus.2020.11.005 doi: 10.1016/j.inffus.2020.11.005
|
| [20] | H. Chen, X. Huang, Q. Li, J. Wang, B. Fang, J. Chen, Labanet: Lead-assisting backbone attention network for oral multi-pathology segmentation, in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, (2023), 1–5. https://doi.org/10.1109/ICASSP49357.2023.10094785 |
| [21] |
L. Wang, Y. Gao, F. Shi, G. Li, K. C. Chen, Z. Tang, et al., Automated segmentation of dental cbct image with prior-guided sequential random forests, Med. Phys., 43 (2016), 336–346. https://doi.org/10.1118/1.4938267 doi: 10.1118/1.4938267
|
| [22] |
S. Liao, S. Liu, B. Zou, X. Ding, Y. Liang, J. Huang, et al., Automatic tooth segmentation of dental mesh based on harmonic fields, Biomed Res. Int., 2015 (2015). https://doi.org/10.1155/2015/187173 doi: 10.1155/2015/187173
|
| [23] | R. Girshick, Fast R-CNN, in Proceedings of the IEEE International Conference on Computer Vision, (2015), 1440–1448. https://doi.org/10.1109/ICCV.2015.169 |
| [24] | K. He, G. Gkioxari, P. Dollár, R. Girshick, Mask R-CNN, in Proceedings of the IEEE International Conference on Computer Vision, (2017), 2961–2969. https://doi.org/10.1109/ICCV.2017.322 |
| [25] |
E. Y. Park, H. Cho, S. Kang, S. Jeong, E. Kim, Caries detection with tooth surface segmentation on intraoral photographic images using deep learning, BMC Oral Health, 22 (2022), 1–9. https://doi.org/10.1186/s12903-022-02589-1 doi: 10.1186/s12903-022-02589-1
|
| [26] | G. Zhu, Z. Piao, S. C. Kim, Tooth detection and segmentation with mask R-CNN, in 2020 International Conference on Artificial Intelligence in Information and Communication (ICAIIC), IEEE, (2020), 070–072. https://doi.org/10.1109/ICAIIC48513.2020.9065216 |
| [27] |
Q. Chen, Y. Zhao, Y. Liu, Y. Sun, C. Yang, P. Li, et al., Mslpnet: Multi-scale location perception network for dental panoramic X-ray image segmentation, Neural Comput. Appl., 33 (2021), 10277–10291. https://doi.org/10.1007/s00521-021-05790-5 doi: 10.1007/s00521-021-05790-5
|
| [28] |
P. Li, Y. Liu, Z. Cui, F. Yang, Y. Zhao, C. Lian, et al., Semantic graph attention with explicit anatomical association modeling for tooth segmentation from CBCT images, IEEE Trans. Med. Imaging, 41 (2022), 3116–3127. https://doi.org/10.1109/tmi.2022.3179128 doi: 10.1109/tmi.2022.3179128
|
| [29] |
E. Shaheen, A. Leite, K. A. Alqahtani, A. Smolders, A. Van Gerven, H. Willems, et al., A novel deep learning system for multi-class tooth segmentation and classification on Cone Beam Computed Tomography. A validation study, J. Dent., 115 (2021), 103865. https://doi.org/10.1016/j.jdent.2021.103865 doi: 10.1016/j.jdent.2021.103865
|
| [30] | M. Ezhov, A. Zakirov, M. Gusarev, Coarse-to-fine volumetric segmentation of teeth in cone-beam CT, in 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019), (2019), 52–56. https://doi.org/10.1109/ISBI.2019.8759310 |
| [31] | A. Alsheghri, F. Ghadiri, Y. Zhang, O. Lessard, J. Keren, F. Cheriet, et al., Semi-supervised segmentation of tooth from 3D scanned dental arches, in Medical Imaging 2022: Image Processing, (2022), 766–771. https://doi.org/10.1117/12.2612655 |
| [32] |
X. Liu, F. Zhang, Z. Hou, L. Mian, Z. Wang, J. Zhang, et al., Self-supervised learning: Generative or contrastive, IEEE Trans. Knowl. Data Eng., 35 (2021), 857–876. https://doi.org/10.1109/tkde.2021.3090866 doi: 10.1109/tkde.2021.3090866
|
| [33] | Q. Li, X. Huang, Z. Wan, L. Hu, S. Wu, J. Zhang, et al., Data-efficient masked video modeling for self-supervised action recognition, in Proceedings of the 31st ACM International Conference on Multimedia, (2023), 2723–2733. https://doi.org/10.1145/3581783.3612496 |
| [34] |
H. Lim, S. Jung, S. Kim, Y. Cho, I. Song, Deep semi-supervised learning for automatic segmentation of inferior alveolar nerve using a convolutional neural network, BMC Oral Health, 21 (2021), 1–9. https://doi.org/10.2196/preprints.32088 doi: 10.2196/preprints.32088
|
| [35] |
F. Isensee, P. F. Jaeger, S. A. Kohl, J. Petersen, K. H. Maier-Hein, nnU-Net: A self-configuring method for deep learning-based biomedical image segmentation, Nat. Methods, 18 (2021), 203–211. https://doi.org/10.1038/s41592-020-01008-z doi: 10.1038/s41592-020-01008-z
|
| [36] | A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, et al., An image is worth 16x16 words: Transformers for image recognition at scale, arXiv preprint, (2020), arXiv: 2010.11929. https://doi.org/10.48550/arXiv.2010.11929 |
| [37] | H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, H. Jégou, Training data-efficient image transformers & distillation through attention, in International Conference on Machine Learning, PMLR, (2021), 10347–10357. |
| [38] | N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, S. Zagoruyko, End-to-end object detection with transformers, in European Conference on Computer Vision, Springer, (2020), 213–229. https://doi.org/10.1007/978-3-030-58452-8_13 |
| [39] | X. Zhu, W. Su, L. Lu, B. Li, X. Wang, J. Dai, Deformable DETR: Deformable transformers for end-to-end object detection, arXiv preprint, (2020), arXiv: 2010.04159. https://doi.org/10.48550/arXiv.2010.04159 |
| [40] |
Q. Li, X. Huang, B. Fang, H. Chen, S. Ding, X. Liu, Embracing large natural data: Enhancing medical image analysis via cross-domain fine-tuning, IEEE J. Biomed. Health. Inf., 2023 (2023), 1–10. https://doi.org/10.1109/JBHI.2023.3343518 doi: 10.1109/JBHI.2023.3343518
|
| [41] | J. Chen, Y. Lu, Q. Yu, X. Luo, E. Adeli, Y. Wang, et al., Transunet: Transformers make strong encoders for medical image segmentation, arXiv preprint, arXiv: 2102.04306. https://doi.org/10.48550/arXiv.2102.04306 |
| [42] | A. Srinivas, T. Lin, N. Parmar, J. Shlens, P. Abbeel, A. Vaswani, Bottleneck transformers for visual recognition, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, (2021), 16519–16529. https://doi.org/10.1109/CVPR46437.2021.01625 |
| [43] | Y. Li, S. Wang, J. Wang, G. Zeng, W. Liu, Q. Zhang, et al., GT U-Net: A U-Net like group transformer network for tooth root segmentation, in Machine Learning in Medical Imaging, Springer, (2021), 386–395. https://doi.org/10.1007/978-3-030-87589-3_40 |
| [44] | W. Lin, Z. Wu, J. Chen, J. Huang, L. Jin, Scale-aware modulation meet transformer, arXiv preprint, (2023), arXiv: 2307.08579. https://doi.org/10.48550/arXiv.2307.08579 |
| [45] | S. Woo, J. Park, J. Lee, I. Kweon, CBAM: Convolutional block attention module, in Computer Vision–ECCV 2018, Springer, (2018), 3–19. https://doi.org/10.1007/978-3-030-01234-2_1 |
| [46] |
Y. Zhang, F. Ye, L. Chen, F. Xu, X. Chen, H. Wu, et al., Children's dental panoramic radiographs dataset for caries segmentation and dental disease detection, Sci. Data, 10 (2023), 380. https://doi.org/10.1038/s41597-023-02237-5 doi: 10.1038/s41597-023-02237-5
|
| [47] |
K. Chen, L. Yao, D. Zhang, X. Wang, X. Chang, F. Nie, A semisupervised recurrent convolutional attention model for human activity recognition, IEEE Trans. Neural Networks Learn. Syst., 31 (2019), 1747–1756. https://doi.org/10.1109/tnnls.2019.2927224 doi: 10.1109/tnnls.2019.2927224
|
| [48] |
G. Litjens, C. I. Sánchez, N. Timofeeva, M. Hermsen, I. Nagtegaal, I. Kovacs, et al., Deep learning as a tool for increased accuracy and efficiency of histopathological diagnosis, Sci. Rep., 6 (2016), 26286. https://doi.org/10.1038/srep26286 doi: 10.1038/srep26286
|
| [49] |
J. Chen, L. Chen, Y. Zhou, Cryptanalysis of a DNA-based image encryption scheme, Inf. Sci., 520 (2020), 130–141. https://doi.org/10.1016/j.ins.2020.02.024 doi: 10.1016/j.ins.2020.02.024
|
| [50] |
D. Yuan, X. Chang, P. Y. Huang, Q. Liu, Z. He, Self-supervised deep correlation tracking, IEEE Trans. Image Process., 30 (2020), 976–985. https://doi.org/10.1109/tip.2020.3037518 doi: 10.1109/tip.2020.3037518
|
| [51] |
Y. Tian, G. Yang, Z. Wang, H. Wang, E. Li, Z. Liang, Apple detection during different growth stages in orchards using the improved YOLO-V3 model, Comput. Electron. Agric., 157 (2019), 417–426. https://doi.org/10.1016/j.compag.2019.01.012 doi: 10.1016/j.compag.2019.01.012
|
| [52] |
D. Yuan, X. Chang, Q. Liu, Y. Yang, D. Wang, M. Shu, et al., Active learning for deep visual tracking, IEEE Trans. Neural Networks Learn. Syst., 2023 (2023), 1–13. https://doi.org/10.1109/TNNLS.2023.3266837 doi: 10.1109/TNNLS.2023.3266837
|
| [53] |
Y. Zhang, L. Deng, H. Zhu, W. Wang, Z. Ren, Q. Zhou, et al., Deep learning in food category recognition, Inf. Fusion, 98 (2023), 101859. https://doi.org/10.1016/j.inffus.2023.101859 doi: 10.1016/j.inffus.2023.101859
|
| [54] |
D. Cremers, M. Rousson, R. Deriche, A review of statistical approaches to level set segmentation: Integrating color, texture, motion and shape, Int. J. Comput. Vision, 72 (2007), 195–215. https://doi.org/10.1007/s11263-006-8711-1 doi: 10.1007/s11263-006-8711-1
|
| [55] |
X. Shu, Y. Yang, J. Liu, X. Chang, B. Wu, Alvls: Adaptive local variances-based levelset framework for medical images segmentation, Pattern Recognit., 136 (2023), 109257. https://doi.org/10.1016/j.patcog.2022.109257 doi: 10.1016/j.patcog.2022.109257
|
| [56] |
K. Ding, L. Xiao, G. Weng, Active contours driven by region-scalable fitting and optimized laplacian of gaussian energy for image segmentation, Signal Process., 134 (2017), 224–233. https://doi.org/10.1016/j.sigpro.2016.12.021 doi: 10.1016/j.sigpro.2016.12.021
|
| [57] | G. Jader, J. Fontineli, M. Ruiz, K. Abdalla, M. Pithon, L. Oliveira, Deep instance segmentation of teeth in panoramic X-ray images, in 2018 31st SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI), IEEE, (2018), 400–407. https://doi.org/10.1109/SIBGRAPI.2018.00058 |
| [58] | T. L. Koch, M. Perslev, C. Igel, S. S. Brandt, Accurate segmentation of dental panoramic radiographs with U-Nets, in 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019), IEEE, (2019), 15–19. https://doi.org/10.1109/ISBI.2019.8759563 |
| [59] |
Z. Cui, C. Li, N. Chen, G. Wei, R. Chen, Y. Zhou, et al., Tsegnet: An efficient and accurate tooth segmentation network on 3D dental model, Med. Image Anal., 69 (2021), 101949. https://doi.org/10.1016/j.media.2020.101949 doi: 10.1016/j.media.2020.101949
|
| [60] |
X. Wang, S. Gao, K. Jiang, H. Zhang, L. Wang, F. Chen, et al., Multi-level uncertainty aware learning for semi-supervised dental panoramic caries segmentation, Neurocomputing, 540 (2023), 126208. https://doi.org/10.1016/j.neucom.2023.03.069 doi: 10.1016/j.neucom.2023.03.069
|
| [61] |
A. Qayyum, A. Tahir, M. A. Butt, A. Luke, H. T. Abbas, J. Qadir, et al., Dental caries detection using a semi-supervised learning approach, Sci. Rep., 13 (2023), 749. https://doi.org/10.1038/s41598-023-27808-9 doi: 10.1038/s41598-023-27808-9
|
| [62] | Y. Zhang, H. Liu, Q. Hu, Transfuse: Fusing transformers and cnns for medical image segmentation, in Medical Image Computing and Computer Assisted Intervention–MICCAI 2021, Springer, (2021), 14–24. https://doi.org/10.1007/978-3-030-87193-2_2 |
| [63] | Y. Wang, T. Wang, H. Li, H. Wang, ACF-TransUNet: Attention-based coarse-fine transformer U-Net for automatic liver tumor segmentation in CT images, in 2023 4th International Conference on Big Data & Artificial Intelligence & Software Engineering (ICBASE), IEEE, (2023), 84–88. https://doi.org/10.1109/ICBASE59196.2023.10303169 |
| [64] |
B. Chen, Y. Liu, Z. Zhang, G. Lu, A. W. K. Kong, TransAttUnet: Multi-level attention-guided U-Net with transformer for medical image segmentation, IEEE Trans. Emerging Top. Comput. Intell., 2023 (2023), 1–14. https://doi.org/10.1109/TETCI.2023.3309626 doi: 10.1109/TETCI.2023.3309626
|
| [65] | A. Hatamizadeh, Y. Tang, V. Nath, D. Yang, A. Myronenko, B. Landman, et al., UNETR: Transformers for 3D medical image segmentation, in 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), (2022), 1748–1758. https://doi.org/10.1109/WACV51458.2022.00181 |
| [66] | H. Cao, Y. Wang, J. Chen, D. Jiang, X. Zhang, Q. Tian, et al., Swin-Unet: Unet-like pure transformer for medical image segmentation, in European Conference on Computer Vision, Springer, (2022), 205–218. https://doi.org/10.1007/978-3-031-25066-8_9 |
| [67] | S. Li, C. Li, Y. Du, L. Ye, Y. Fang, C. Wang, et al., Transformer-based tooth segmentation, identification and pulp calcification recognition in CBCT, in International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, (2023), 706–714. https://doi.org/10.1007/978-3-031-43904-9_68 |
| [68] |
M. Kanwal, M. M. Ur Rehman, M. U. Farooq, D. K. Chae, Mask-transformer-based networks for teeth segmentation in panoramic radiographs, Bioengineering, 10 (2023), 843. https://doi.org/10.3390/bioengineering10070843 doi: 10.3390/bioengineering10070843
|
| [69] | W. Chen, X. Du, F. Yang, L. Beyer, X. Zhai, T. Y. Lin, et al., A simple single-scale vision transformer for object detection and instance segmentation, in European Conference on Computer Vision, Springer, (2022), 711–727. https://doi.org/10.1007/978-3-031-20080-9_41 |
| [70] | M. R. Amini, V. Feofanov, L. Pauletto, E. Devijver, Y. Maximov, Self-training: A survey, arXiv preprint, (2023), arXix: 2202.12040. https://api.semanticscholar.org/CorpusID: 247084374 |
| [71] | Z. Zhou, M. M. R. Siddiquee, N. Tajbakhsh, J. Liang, UNet++: A nested U-Net architecture for medical image segmentation, in Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, Springer, (2018), 3–11. https://doi.org/10.1007/978-3-030-00889-5_1 |
| [72] | H. Huang, L. Lin, R. Tong, H. Hu, Q. Zhang, Y. Iwamoto, et al., Unet 3+: A full-scale connected unet for medical image segmentation, in 2020 IEEE International Conference on Acoustics, Speech and Signal Processing ICASSP, IEEE, (2020), 1055–1059. https://doi.org/10.1109/ICASSP40776.2020.9053405 |
| [73] |
Q. Zuo, S. Chen, Z. Wang, R2AU-Net: Attention recurrent residual convolutional neural network for multimodal medical image segmentation, Secur. Commun. Netw., 2021 (2021), 1–10. https://doi.org/10.1155/2021/6625688 doi: 10.1155/2021/6625688
|
| [74] |
C. Sheng, L. Wang, Z. Huang, T. Wang, Y. Guo, W. Hou, et al., Transformer-based deep learning network for tooth segmentation on panoramic radiographs, J. Syst. Sci. Complexity, 36 (2023), 257–272. https://doi.org/10.1007/s11424-022-2057-9 doi: 10.1007/s11424-022-2057-9
|
| [75] | R. Azad, R. Arimond, E. K. Aghdam, A. Kazerouni, D. Merhof, DAE-former: Dual attention-guided efficient transformer for medical image segmentation, in Predictive Intelligence in Medicine, Springer, (2023), 83–95. https://doi.org/10.1007/978-3-031-46005-0_8 |
| [76] | E. Xie, W. Wang, Z. Yu, A. Anandkumar, J. M. Alvarez, P. Luo, Segformer: Simple and efficient design for semantic segmentation with transformers, Adv. Neural Inf. Process. Syst., 34 (2021), 12077–12090. |
| 1. | Caixia Zheng, Huican Li, Yingying Ge, Yanlin He, Yugen Yi, Meili Zhu, Hui Sun, Jun Kong, Retinal vessel segmentation based on multi-scale feature and style transfer, 2023, 21, 1551-0018, 49, 10.3934/mbe.2024003 | |
| 2. | Mohd Zulfaezal Che Azemin, Mohd Izzuddin Mohd Tamrin, Firdaus Yusof, Adzura Salam, Nur Syazriena Ghazali, High-resolution retinal imaging system: diagnostic accuracy and usability, 2025, 6, 2735-0584, 69, 10.31436/ijohs.v6i1.357 |
Figures(4) / Tables(3)
Duolin Sun, Jianqing Wang, Zhaoyu Zuo, Yixiong Jia, Yimou Wang. STS-TransUNet: Semi-supervised Tooth Segmentation Transformer U-Net for dental panoramic image[J]. Mathematical Biosciences and Engineering, 2024, 21(2): 2366-2384. doi: 10.3934/mbe.2024104
| Methods | AUC (%) | Acc (%) | Spe (%) | Sen (%) |
| Baseline | 98.42 | 96.80 | 98.57 | 78.50 |
| No FSC | 98.48 | 96.89 | 98.85 | 76.40 |
| No DHDC | 98.54 | 96.96 | 98.58 | 78.81 |
| No SERC | 98.58 | 96.99 | 98.69 | 79.23 |
| MIC-Net | 98.62 | 97.02 | 98.80 | 80.02 |
| *Note: For each metric, the bold value indicates that column's best result. | ||||
DownLoad:
CSV
| Methods | Parameters (M) | FLOPs (M) | AUC |
| SegNet | 29.46 | 58.91 | 0.9294 |
| FCN_8s | 9.01 | 18.01 | 0.9410 |
| MultiResUNet | 7.26 | 14.55 | 0.9451 |
| LinkNet | 11.55 | 23.62 | 0.9492 |
| DeepLabV3+ | 41.06 | 82.23 | 0.9575 |
| Att-UNet | 8.91 | 17.82 | 0.9793 |
| R2U-Net | 17.65 | 51.03 | 0.9804 |
| MIC-Net | 9.13 | 18.23 | 0.9862 |
| *Note: For each metric, the bold value indicates that column's best result. | |||
DownLoad:
CSV
| Dataset | AUC (%) | Acc (%) | Spe (%) | Sen (%) | Pre (%) | F1-score (%) | IoU (%) |
| DRIVE | 98.62 | 97.02 | 98.80 | 80.02 | 86.32 | 82.20 | 68.32 |
| STARE | 98.60 | 97.76 | 98.61 | 87.72 | 93.70 | 85.99 | 69.51 |
| CHASE | 98.73 | 97.38 | 98.44 | 81.60 | 77.95 | 79.73 | 64.07 |
DownLoad:
CSV
| Methods | Year | DRIVE(%) | STARE(%) | CHASE(%) | |||||||||||
| AUC | Acc | Spe | Sen | AUC | Acc | Spe | Sen | AUC | Acc | Spe | Sen | ||||
| Azzopardi [24] | 2015 | 96.14 | 94.42 | 97.04 | 76.55 | 95.63 | 94.97 | 97.01 | 77.16 | 94.87 | 93.87 | 95.87 | 75.85 | ||
| Li et al. [25] | 2015 | 97.38 | 95.27 | 98.16 | 75.69 | 98.79 | 96.28 | 98.44 | 77.26 | 97.16 | 95.81 | 97.93 | 75.07 | ||
| Liskowski [26] | 2016 | 97.20 | 94.95 | 97.68 | 77.63 | 97.85 | 95.66 | 97.54 | 78.67 | - | - | - | - | ||
| Fu et al. [27] | 2016 | - | 95.23 | - | 76.03 | - | 95.85 | - | 74.12 | - | 94.89 | - | 71.30 | ||
| Dasgupta [28] | 2016 | 97.44 | 95.33 | 98.01 | 76.91 | - | - | - | - | - | - | - | - | ||
| Chen et al. [29] | 2017 | 95.16 | 94.53 | 97.35 | 74.26 | 95.57 | 94.49 | 96.96 | 72.95 | - | - | - | - | ||
| Yan et al. [30] | 2018 | 97.52 | 95.42 | 98.18 | 76.53 | 98.01 | 96.12 | 98.46 | 75.81 | 97.81 | 96.10 | 98.09 | 76.33 | ||
| Wu et al. [14] | 2018 | 98.07 | 95.67 | 98.19 | 78.44 | - | - | - | - | 98.25 | 96.37 | 98.47 | 75.38 | ||
| Yan et al. [31] | 2019 | 97.50 | 95.38 | 98.20 | 76.31 | 98.33 | 96.38 | 98.57 | 77.35 | 97.76 | 96.07 | 98.06 | 76.41 | ||
| Jin et al. [20] | 2019 | 98.02 | 95.66 | 98.00 | 79.63 | 98.32 | 96.41 | 98.78 | 75.95 | 98.04 | 96.10 | 97.52 | 81.55 | ||
| Wang et al. [32] | 2020 | 98.23 | 95.81 | 98.13 | 79.91 | 98.81 | 96.73 | 98.44 | 81.86 | - | - | - | - | ||
| Li et al. [17] | 2020 | 98.16 | 95.73 | 98.38 | 77.35 | 98.81 | 97.01 | 98.86 | 77.15 | 98.51 | 96.55 | 98.23 | 79.70 | ||
| Shi et al. [33] | 2021 | - | 96.76 | 98.26 | 80.65 | - | 97.32 | 98.66 | 82.90 | - | 97.31 | 98.89 | 75.04 | ||
| Guo et al. [34] | 2021 | 98.53 | 96.67 | 98.17 | 82.21 | 98.97 | 97.24 | 98.59 | 82.10 | 98.69 | 96.97 | 98.45 | 81.89 | ||
| Xu et al. [35] | 2015 | 96.70 | 96.30 | 98.23 | 87.45 | - | - | - | - | 96.77 | 96.94 | 97.94 | 89.16 | ||
| Zhang et al. [36] | 2022 | 88.95 | 97.01 | 97.99 | 77.19 | 83.91 | 96.91 | 99.11 | 69.12 | 91.42 | 98.11 | 99.81 | 85.06 | ||
| Deng et al. [37] | 2022 | 97.93 | 95.39 | 97.12 | 83.68 | 98.55 | 96.43 | 97.79 | 84.35 | 98.06 | 95.87 | 96.93 | 85.43 | ||
| Our MIC-Net | 2022 | 98.62 | 97.02 | 98.80 | 80.02 | 98.60 | 97.76 | 98.61 | 87.72 | 98.73 | 97.38 | 98.44 | 81.60 | ||
DownLoad:
CSV
| Methods | AUC (%) | Acc (%) | Spe (%) | Sen (%) |
| Baseline | 98.42 | 96.80 | 98.57 | 78.50 |
| No FSC | 98.48 | 96.89 | 98.85 | 76.40 |
| No DHDC | 98.54 | 96.96 | 98.58 | 78.81 |
| No SERC | 98.58 | 96.99 | 98.69 | 79.23 |
| MIC-Net | 98.62 | 97.02 | 98.80 | 80.02 |
| *Note: For each metric, the bold value indicates that column's best result. | ||||
| Methods | Parameters (M) | FLOPs (M) | AUC |
| SegNet | 29.46 | 58.91 | 0.9294 |
| FCN_8s | 9.01 | 18.01 | 0.9410 |
| MultiResUNet | 7.26 | 14.55 | 0.9451 |
| LinkNet | 11.55 | 23.62 | 0.9492 |
| DeepLabV3+ | 41.06 | 82.23 | 0.9575 |
| Att-UNet | 8.91 | 17.82 | 0.9793 |
| R2U-Net | 17.65 | 51.03 | 0.9804 |
| MIC-Net | 9.13 | 18.23 | 0.9862 |
| *Note: For each metric, the bold value indicates that column's best result. | |||
| Methods | AUC (%) | Acc (%) | Spe (%) | Sen (%) |
| Yan et al. [41] | 97.08 | 95.69 | 98.40 | 72.11 |
| Jin et al. [42] | 94.45 | 96.90 | 97.59 | 70.00 |
| Wu et al. [22] | 96.35 | 95.44 | 97.85 | 73.78 |
| Proposed method | 97.70 | 96.44 | 97.51 | 76.02 |
| Dataset | AUC (%) | Acc (%) | Spe (%) | Sen (%) | Pre (%) | F1-score (%) | IoU (%) |
| DRIVE | 98.62 | 97.02 | 98.80 | 80.02 | 86.32 | 82.20 | 68.32 |
| STARE | 98.60 | 97.76 | 98.61 | 87.72 | 93.70 | 85.99 | 69.51 |
| CHASE | 98.73 | 97.38 | 98.44 | 81.60 | 77.95 | 79.73 | 64.07 |
| Methods | Year | DRIVE(%) | STARE(%) | CHASE(%) | |||||||||||
| AUC | Acc | Spe | Sen | AUC | Acc | Spe | Sen | AUC | Acc | Spe | Sen | ||||
| Azzopardi [24] | 2015 | 96.14 | 94.42 | 97.04 | 76.55 | 95.63 | 94.97 | 97.01 | 77.16 | 94.87 | 93.87 | 95.87 | 75.85 | ||
| Li et al. [25] | 2015 | 97.38 | 95.27 | 98.16 | 75.69 | 98.79 | 96.28 | 98.44 | 77.26 | 97.16 | 95.81 | 97.93 | 75.07 | ||
| Liskowski [26] | 2016 | 97.20 | 94.95 | 97.68 | 77.63 | 97.85 | 95.66 | 97.54 | 78.67 | - | - | - | - | ||
| Fu et al. [27] | 2016 | - | 95.23 | - | 76.03 | - | 95.85 | - | 74.12 | - | 94.89 | - | 71.30 | ||
| Dasgupta [28] | 2016 | 97.44 | 95.33 | 98.01 | 76.91 | - | - | - | - | - | - | - | - | ||
| Chen et al. [29] | 2017 | 95.16 | 94.53 | 97.35 | 74.26 | 95.57 | 94.49 | 96.96 | 72.95 | - | - | - | - | ||
| Yan et al. [30] | 2018 | 97.52 | 95.42 | 98.18 | 76.53 | 98.01 | 96.12 | 98.46 | 75.81 | 97.81 | 96.10 | 98.09 | 76.33 | ||
| Wu et al. [14] | 2018 | 98.07 | 95.67 | 98.19 | 78.44 | - | - | - | - | 98.25 | 96.37 | 98.47 | 75.38 | ||
| Yan et al. [31] | 2019 | 97.50 | 95.38 | 98.20 | 76.31 | 98.33 | 96.38 | 98.57 | 77.35 | 97.76 | 96.07 | 98.06 | 76.41 | ||
| Jin et al. [20] | 2019 | 98.02 | 95.66 | 98.00 | 79.63 | 98.32 | 96.41 | 98.78 | 75.95 | 98.04 | 96.10 | 97.52 | 81.55 | ||
| Wang et al. [32] | 2020 | 98.23 | 95.81 | 98.13 | 79.91 | 98.81 | 96.73 | 98.44 | 81.86 | - | - | - | - | ||
| Li et al. [17] | 2020 | 98.16 | 95.73 | 98.38 | 77.35 | 98.81 | 97.01 | 98.86 | 77.15 | 98.51 | 96.55 | 98.23 | 79.70 | ||
| Shi et al. [33] | 2021 | - | 96.76 | 98.26 | 80.65 | - | 97.32 | 98.66 | 82.90 | - | 97.31 | 98.89 | 75.04 | ||
| Guo et al. [34] | 2021 | 98.53 | 96.67 | 98.17 | 82.21 | 98.97 | 97.24 | 98.59 | 82.10 | 98.69 | 96.97 | 98.45 | 81.89 | ||
| Xu et al. [35] | 2015 | 96.70 | 96.30 | 98.23 | 87.45 | - | - | - | - | 96.77 | 96.94 | 97.94 | 89.16 | ||
| Zhang et al. [36] | 2022 | 88.95 | 97.01 | 97.99 | 77.19 | 83.91 | 96.91 | 99.11 | 69.12 | 91.42 | 98.11 | 99.81 | 85.06 | ||
| Deng et al. [37] | 2022 | 97.93 | 95.39 | 97.12 | 83.68 | 98.55 | 96.43 | 97.79 | 84.35 | 98.06 | 95.87 | 96.93 | 85.43 | ||
| Our MIC-Net | 2022 | 98.62 | 97.02 | 98.80 | 80.02 | 98.60 | 97.76 | 98.61 | 87.72 | 98.73 | 97.38 | 98.44 | 81.60 | ||
