Improved U-Net based on cross-layer connection for pituitary adenoma MRI image segmentation

1.   Introduction

On the basis of the latest statistics from the World Health Organization, there were 19.29 million new cancers and 9.96 million deaths around the world in 2021. Among all tumor diseases, pituitary adenoma is one of the most common primary intracranial tumors, accounting for about 10-15% of all brain tumors ^[1]. Although most pituitary adenoma are benign and do not pose a threat to the life of patients, they can cause clinical problems such as visual impairment, infertility and metabolic syndrome. In recent years, countries around the world have put forward "precision medicine" plans, aiming to combine patients' living environment and medical data, and apply modern genetics, molecular imaging, artificial intelligence and other technologies to develop personalized prevention plans and precision diagnosis and treatment plans. Doctors can use non-invasive methods to monitor the structure and function of organs and tissues to make faster and more accurate assessments and predictions of conditions. Therefore, the research on the assisted pituitary adenoma diagnosis based on deep learning theory has a strong frontier and clinical application value.

In the medical Imaging diagnosis of pituitary adenoma, the conventional techniques mainly involve computed tomography (CT) and magnetic resonance imaging (MRI). Because MR images have good soft tissue resolution and can provide more detailed information on pituitary lesions, it is an ideal method for detecting pituitary adenomas, as shown in Figure 1. However, no matter which imaging technique is used, accurate localization and delineation of pituitary lesion are indispensable. In the clinical treatment implementation system, this work is still done manually, which heavily depends on the radiologist's reading experience and work status. In addition, image diagnosis is highly subjective and the repetition of segmentation results is poor, which limits the promotion and application of "precision medicine" policy to a certain extent. At present, a large number of researchers and scholars have devoted themselves to the segmentation method of tumor MR images. By using the gray scale, shape and boundary information, various diseased tissues are detected, and then the corresponding diagnosis results are given. However, due to the different locations, shapes and sizes of tumors in MR images, and the presence of noise, field shift effects, and partial volume effects, fully automatic segmentation of medical images is still a complex and challenging research.

With the continuous improvement of medical imaging technology and mathematical theory, lots of MR image segmentation approaches based on deep learning ^[2,3,4,5] have emerged. To automatically segment and detect the location of brain tumors from MRI images, Rai et al. ^[6] presented a depth-based network framework. By choosing U-Net ^[7] as the basic skeleton and incorporating ResNext-50 ^[8] as the encoder, it not only solves the problem of gradient descent, but also saves computational overhead. Lu et al. ^[9] presented a new end-to-end medical image segmentation model inspired by multi-scale cross-fusion encoding network. In this study, dual context aggregation and attention guided modules are used to construct the network, and the segmentation performance is further improved by optimizing the corresponding objective function. Rehman et al. ^[10] introduced feature enhancement modules at each encoder stage to achieve the purpose of improving the receiving domain. Meanwhile, the loss function is redesigned to solve the problem of unbalanced class. Tang et al. ^[11] proposed a dense frame algorithm based on dual attention for nasopharyngeal carcinoma image segmentation, which helps to obtain better performance while alleviating the increase of parameters, thus overcoming the problem of gradient disappearance.

Recently, inception module ^{[12,13,14,15]} has achieved significant success in medical image segmentation tasks. Zhou et al. ^[16] added inception layer and extended residual module into the framework of U-Net to achieve accurate segmentation of superficial muscle. By redesigning the inception module, Chen et al. ^[17] can obtain more features on multi-scale channels. At the same time, BConvLSTM and self-attention layers were added to achieve more accurate segmentation results. Hoorali et al. ^[18] proposed a detection approach for anthrax microscopic image, which alleviated the problem of different semantic features by exploiting the advantages of inception block and residual block. Zhang et al. ^[19] presented a new CNN approach by adding inception-res and dense convolution operations into U-Net. The width of its network is increased without need for additional parameters, and the possible gradient disappearance is mitigated. Bala et al. ^[20] presented a generalized encoder-decoder network by modifying U-Net. Firstly, dense connection is used to replace simple jump connection, and then a new inception module is proposed to replace traditional convolution. Finally, image enhancement and normalization are carried out to increase the generalization ability.

Inspired by the above models, we propose a new automatic segmentation framework for pituitary adenoma which takes advantage of cross-layer connection and inception module. The creative points of this work are summarized: 1) A newly cross-layer is designed to capture richer multi-scale features and contextual information. 2) Full-scale skip structure is utilized, thereby enhancing the learning of pituitary tumor shape. 3) We utilize the inception-dense module for block unit convolution, so that the network can expand the effective receptive field and increase the depth. 4) The training process is optimized by employing the function based on binary cross-entropy and Jaccard losses to obtain better performance.

The remainder of our paper is structured as follows: Section 2 gives the proposed method. Section 3 demonstrates the superiority of our algorithm on real pituitary adenoma images. Finally, the conclusion is discussed in Section 4.

2.   Materials and methods

Inspired by the original U-shaped encoder-decoder network in ^[7], we design a new end-to-end network and its overall framework is shown in Figure 2. The proposed network mainly includes cross-layer connection, inception-dense module and full-scale skip structure. For down-sampling in coding stage, cross-layer connection is introduced and inception-dense block is used instead of the original convolution block to extract the input image features effectively. During the decoding stage, the feature map is up-sampled by convolution transpose operation and then fused with the multi-layer features information obtained by the down-sampling. Finally, a mixed function is developed to help achieve better training results. The detailed technical of this network will be described in the following subsection.

2.1.   Cross-layer connection

For pituitary adenoma images, most of them have problems such as blurred edges, high noise and high similarity to the surrounding normal tissues. Therefore, it is difficult to retain target detail under a single scale. In general, multi-layer features can not only expand the receiving field, but also better extract semantic context information and spatial details of the image. From this point, we utilize cross-layer connection to effectively improve network performance ^[21,22]. Firstly, two multi-scale feature branches (Pool1a and Pool1b) are obtained by applying inception-dense block and Max-pooling layer with step size of 2 × 2 and 4 × 4, as shown in Figure 2. Then, the above operations were repeated to extract multi-scale features Pool2a and Pool2b. Obviously, Pool1b and Pool2a have the same image dimension. The research shows that Pool2a can obtain higher level of context information by small stride, while Pool1b can maintain a larger receptive domain by bigger stride ^[23]. Therefore, the designed cross-layer connections can achieve the aggregation of multi-scale features and multi-level contextual information.

2.2.   Inception-dense module

Inception module ^[24,25,26] is designed to improve performance by increasing network width. Its key point is to replace the classical convolution layer with pooling layer and convolution layer in different scales, and adopt 1 × 1 convolution layer for dimensionality reduction. Inception can make self-adapt to convolutional kernels of different sizes, and can obtain more complex image features compared to convolutional kernels of fixed size. As shown in Figure 3(a), the original Inception module includes 1 × 1, 3 × 3, and 5 × 5 convolutional size filters and a 3 × 3 maximum pooling layer. The advantage of this structure is that it can learn spatial features at different scales and give different weights, so as to achieve better segmentation results. Based on the original Inception architecture, this paper proposes a modified version, as shown in Figure 3(b). Our improved Inception model adopts the idea of asymmetric convolution decomposition and proposes a lightweight feature information extraction module. The key improvement is to replace the 3 × 3 convolutional filter in the original Inception module with a set of 1 × 3 and 3 × 1 asymmetric convolutional operation. Similarly, we can decompose the 5 × 5 convolutional filter into 1 × 5 and 5 × 1 sequence groups. This asymmetric convolution operation can ensure the receiving field and reduce the module dimension, so as to improve the over-fitting problem.

To strengthen the information transfer between layers, our modified Inception module is inserted into the dense connection blocks, as shown in Figure 4. Therefore, feature-maps $[{x_0}, \ldots, {x_{l -1}}]$ of all previous layers are used as input, and then the output can be expressed:

where ${x_l}$ represents the input of layer $l$, ${H_l}$ is a composite function that can affect information transfer. In this study, the advantages of dense connection and Inception module were used to obtain different scale features with convolution kernels of different sizes. At the same time, without increasing the depth and width of network, the features are fully extracted to obtain better generalization and representation ability.

2.3.   Full-scale skip structure

The classical U-Net ^[7] model fuses the feature map output by each layer of the encoder with the feature map generated by the corresponding decoder. It only combines the features of the same scale at the same height, so the global context information will be gradually weakened when it is gradually spread to the shallow layer. The U-Net++ ^[27] designs a nested and dense jump connection, and utilizes features from each layer of the network to automatically learn the importance level of different depth features. This model integrates the features of different receptive fields, but these features all come directly from the same layer or the next layer, so there are still phenomena such as insufficient fine granularity, loss of edge and location information, etc. To remedy the defects of the above models, a new full-scale skip structure is designed to improve segmentation accuracy. Modifications of this architecture can combine low level details with high level semantics from feature maps at different scales. The difference from ^[27] is that the modified full-scale skip structure can transmit the characteristics information of all previous layers, containing all layers in the encoder and deeper layers in the decoder, to a specific layer in the decoder. The advantage of this is that each decoder layer can introduce larger scale feature maps from different levels and scales, thus greatly mitigating the phenomenon of feature loss.

2.4.   Loss function

To show more balanced segmentation, we design a combination of binary cross-entropy ^[28,29] and Jaccard ^[30,31] as loss function. The formula is as follows:

where ${L_{BCE}}$ represents binary cross-entropy loss, it is defined as:

where $N$ denotes the sum of pixels, ${y_i}$ is the predicted mask, and ${\hat y_i}$ is the ground truth mask. However, when dealing with an imbalanced number of classes, the optimizer can easily get stuck in local minima. For the stability of training, Jaccard loss is introduced:

where ${L_{JA}}$ represents the Jaccard loss.

3.   Experiments and results

In this section, we mainly discuss the experimental results on real patient data from Quzhou People's Hospital. We further prove the superiority of our method by comparing with some classical learning approaches, employing the Dice, IoU, Mcc and precision as the evaluation indicators.

3.1.   Experimental settings

3.1.1.   Dataset description

The pituitary adenoma MRI dataset used in this paper consists of 500 lesion images, obtained from the rehabilitation department of Quzhou People's Hospital. All images were taken by doctors for the subject's routine examination, with a size of 320 × 320 pixels. Although the images are acquired with the same equipment, they show high variability due to the influence of field of view size, camera view, appearance and other aspects, as shown in Figure 1. Based on the interactive segmentation results of Labelme, the ground truths are labelled by junior annotators and verified by an experienced radiologist. Considering the limiting memory usage, we adjusted all the images to 256 × 256 pixels. To eliminate the problem of small samples and unbalanced data, augmentation techniques (such as random rotating, scaling, translating, shearing, left-right flipping, and mosaic augmentation) are employed to extend the training dataset. In the current work, the original MR image dataset is 500 images, which 20% (100 images) are retained for testing and 80% (400 images) are used for enhancement, so the total training dataset is (400 × 7) 2800 images. The enhanced images and their corresponding label are shown in Figure 5.

3.1.2.   Evaluation metrics

The Dice coefficient ^[32], Intersection over Union ^[33], Matthews correlation coefficient ^[34] and precision ^[35] were used to effectively evaluate the advantages of different methods. In addition, the larger these values, the better the segmentation performance. The evaluation matrices are defined as:

where TP, TN, FP, and FN denote numbers of true positives, true negatives, false positives, and false negatives on the pixels set, respectively.

3.1.3.   Implementation details

All methods are implemented using Python 3.7 based on the open-source framework Keras package with Tensorflow 2.0 as backend. The experimental environment is based on a Windows10 workstation configured with an Intel Xeon Gold 6154 CPU@3.00 GHz processor, 64 GB of 2933 MHz DDR4 ECC RDIMM, NVIDIA Quadro RTX 6000 GPU memory of 24 GB. Since the convolutional neural network will continuously iterate during the training process, it is necessary to estimate the impact of these hyper-parameters on the model performance, mainly including batch size, learning rate and channel. Based on the speed and environment configuration, Adam optimizer ^[36] is utilized to optimize the segmentation network. As shown in Tables 1 and 2, we can clearly see that the network performance is best when the batch size is set to 16 and the learning rate is set to 0.001. From Table 3, the network achieves the best results when the channel is set to 32.

3.2.   Comparison experiment

We firstly observe the accuracy and loss values of 300 epochs on the training and validation sets, and the results are shown in Figure 6. It can be shown that for all cases, the proposed method can quickly converge. This is mainly due to the advantages of Inception-dense module in feature extraction. In addition, it should be noted that, except for some special points, the deviation of the loss and accuracy curves on the training set and test set is very small, which indicates that our model is reliable and robust.

Figure 7 qualitatively evaluates the segmentation results of different methods with ground truth. The first column is the original images, the second column is the ground truth hand-sketched by a radiologist, the third to last columns are segmentation results of AttUNet ^[37], DenseUnet ^[38], ENet ^[39], HRNet ^[40], ICNet ^[41], R2U-net ^[42], SegNet ^[43], U-Net ^[7], XNet ^[44], U-Net+++ ^[45], UNet++ ^[27] and our method, respectively. According to the visualization results, it is obvious that U-net and XNet are easily disturbed by noise in the complex background, so these two models perform the worst. Through the multi-scale feature extraction techniques, U-Net+++ and UNet++ can increase the completeness of complex lesion segmentation, but the edge details are obviously insufficient. Although AttUNet and DenseUnet show some advantages in dealing with complex shadows, they still suffer from insufficient segmentation. The other methods will also misclassify some shadows as lesions, resulting in over-segmentation or under-segmentation. However, our proposed segmentation model can well complete the task of medical image segmentation, and has a good ability to extract details of lesions. This shows that our model is more stable than other methods during the evolution, and the network fluctuation is smaller.

To verify the proposed scheme, the above models are quantitatively evaluated and compared using the evaluation metrics of Dice, IoU, Mcc and precision, which is listed in Table 4. According to Table 4, the competitive performance of DenseUnet, ICNet, SegNet and XNet is generally lower than that of other networks. As the baseline, the Dice, IoU, Mcc and precision values of U-Net reached 0.8636, 0.7713, 0.8664 and 0.8636. UNet++ and UNet+++ have nested dense jump paths, so there are obvious improvements in false detection and missed detection. On the basis of the U-Net network architecture, the segmentation effect of pituitary tumor lesions can be significantly improved whether the attention module or Recurrent Residual module is added. Compared with the U-Net, the Dice, IoU, Mcc and precision values of our method are improved to 2.51, 3.54, 2.27 and 0.05%, respectively. This result can show the effectiveness of the proposed network in the localization and extraction of pituitary tumor lesions.

To verify the processing performance on complex data, the ISIC-2018 Challenge dataset ^[46] was used for further testing. The dataset is released by the international organization ISIC, and it has 2594 images of skin lesions with corresponding labels. Among them, 70% of these images are used for training, 10% for validation, and the rest for testing. The images in this database are characterized by large differences in the size of the lesion area, low contrast with the surrounding skin, hair interference, and irregular shape. Figure 8 shows seven examples of our proposed method and other models at ISIC-2018 dataset, where white represents the pixels of the lesion area, and black represents the normal skin pixels. It can be observed that our method can better cope with the influence of external interference, accurately distinguish lesions from surrounding tissues, and thus can completely segment skin lesions. Table 5 shows the experimental results of various methods. It can be seen that the proposed network can achieve Dice, IoU, Mcc and precision of 0.8683, 0.7686, 0.8344 and 0.9349, indicating that the performance of the modified model is very competitive.

3.3.   Ablation experiment

Next, the contribution of each module in the proposed model was performed for ablation study to verify their performance. For a fair comparison, the U-Net model is selected as the benchmark, and the results are shown in Table 6. It can be observed that under the same segmentation framework, the overall segmentation performance is slightly improved after the introduction of the cross-layer module, and the Dice, IoU, Mcc and prscision values are increased by 0.47, 0.51, 0.34 and 0.01% respectively. Under the framework of Baseline+Cross-layer, inception-dense module and full-scale skip structure are added respectively, and the segmentation performance is further improved. When all modules are fused into the U-Net framework, the values of Dice, IoU, Mcc and prscision all reach the optimal value. This indicates that the cross-layer connection, inception-dense module and full-scale skip structure proposed in our model can well strengthen the feature dependencies between channels and capture the lesion features of pituitary tumors, which can help the network to make better decisions.

Figure 9 shows the visualization results of the ablation experiment, the first two rows show pituitary tumor lesions and their corresponding label images, respectively. It can be found that due to the influence of other tissues and blurred boundary, the segmentation effect of Baseline for the lesion is poor, and the edge part is not smooth. After adding the cross-layer module, the performance of the model is improved, but there are still some classification errors. When the inception-dense module and full-scale skip structure is incorporated, our model can accurately segment the pituitary tumor lesion area, and the segmentation edge is smoother and has higher similarity with the gold standard.

To further demonstrate that the Inception-density module outperforms traditional convolutional modules for segmentation problems, the experiments are carried out with and without the Inception-dense module. Figure 10 shows the heatmaps of our model in the process of pituitary tumor image training. The heat map shows that by integrating multi-scale features and multi-level context information, the Inception-dense module can depict more and clearer lesion features, which is helpful to achieve better segmentation performance.

3.4.   Complexity analysis

The computational complexity and the number of parameters can evaluate the complexity of a model and limit its application in real scenarios to some extent. As shown in Table 7, due to the introduction of cross-layer connection, inception-Dense Module and full-scale skip structure, our model has a large number of parameters, but the processing time is in the middle among all methods. Therefore, our model achieves a good balance between detection accuracy and efficiency.

4.   Conclusions

In this paper, by modifying the original U-Net architecture, we propose a novel automatic segmentation framework for pituitary adenoma MRI image segmentation. The core idea is to design a newly cross-layer connection in the coding phase to capture richer multi-scale features and contextual information. Meanwhile, full-scale skip structure is employed to combine the output of each path, which can make full use of the above information obtained by different layers. Furthermore, in the whole framework, an improved inception-dense module is used to replace the standard convolution block of U-Net, which can expand the effective receptive field and increase the depth. Experimental results on pituitary adenoma images indicate that the proposed model can effectively and accurately achieve high-quality lesion segmentation when compared with other advanced deep learning-based method.

Acknowledgments

This work was supported by the National Natural Science Foundation of China (No. 62102227, 51805124, 62101206), Zhejiang Basic Public Welfare Research Project (No. LZY22E050001, LZY22D010001, LGG19E050013, LZY21E060001), Science and Technology Major Projects of Quzhou (2021K29).

Conflict of interest

The authors declare there is no conflict of interest.