Automatic segmentation of breast cancer histological images based on dual-path feature extraction network

1.   Introduction

According to the latest global cancer data released by International Agency for Research on Cancer (IARC), female breast cancer has surpassed lung cancer as the most commonly occurring cancer worldwide for the first time ^[1]. Even though, the mortality rate of breast cancer has been steadily decreasing in recent years, especially in developed countries. The contrast is due in large part to the gradual improvement of clinical treatment methods which depend on early detection and correct diagnosis. Histological examination of biopsy specimen is considered to be the golden standard for breast cancer detection and affirmation ^[2]. Traditionally pathologists perform operations such as focusing, zooming and panning on the slices of breast tissue under an electronic or light microscope in order to observe and analyze the density and shape of cells, and the structure and morphology of local tissues at different magnifications. Accordingly, whether cancer occurs and the location and extent of the lesions could be determined, and further treatments could be made ^[3].

With the increasing number of breast cancer patients, the traditional diagnosis method based on manual observation is facing huge challenges. Firstly, observation through microscopes consumes substantial time and effort owing to the complexity and diversity of breast tissue, thus leading to a percentage of misdiagnosis ^[4]. Furthermore, the average diagnostic concordance among different pathologists' individual interpretations based on a single breast biopsy slide reaches only 75.3% ^[5] affected by both objective factors such as different staining methods and equipment batches, and subjective factors including diversity of pathologists' personal clinical experience and evaluation criteria as well ^[6,7]. Such challenges are unfavorable for the determination of therapy courses, which may bring about overtreatment or undertreatment and further lead to harm or even death of patients. Therefore, there is urgent need for an objective, accurate and effective auxiliary system to automatically process histological images and assist the pathologists to make wiser diagnosis and treatment options.

More recently, thanks to the tremendous progress of digital pathology technology and machine learning, computer-aided diagnosis (CAD) has become possible ^[8]. This paper focuses on the semantic segmentation of whole-slide images (WSIs). In the light of the proliferation degree of breast tumor, each pixel in the WSIs is classified into 4 predominant cancer types: normal, benign, in situ carcinoma and invasive carcinoma, thus WSIs are automatically divided into four regions respectively. The automatic segmentation result could provide pathologists a pixel-wise understanding of WSIs and a scientific basis for correct and fast diagnosis.

In this paper, we propose an end-to-end automatic segmentation system for breast cancer histological images. The main contributions of this paper are as follows:

1) We adopt the dual-path structure to the semantic segmentation of histological images for the first time and modify it to solve the high-resolution and semantically complicated WSIs of breast cancer, consequently improving the segmentation accuracy and efficiency.

2) We introduce dilated convolution structure in the spatial path and draw the attention mechanism into the semantic path as well as the feature fusion module, reconciling the rich context of WSIs with detailed information, enabling the network with powerful feature acquisition and discrimination ability.

3) The pre-processing operation used alleviates the data capacity and the light-weight backbone structure of our network lows the model complexity, which greatly speeds up the segmentation of high-resolution WSIs and further makes automatic diagnosis entering the clinic possible.

The rest of the paper is organized as follows. Section 2 suggests and analyzes the related works of automatic segmentation of medical images. In Section 3, the dataset we used and the steps of automatic segmentation are declared in detail. Section 4 describes experimental details and displays the results. Section 5 makes a discussion on the results. Finally, Section 5 makes a summary on the paper.

2.   Related works

The development process of automatic segmentation of pathological images can be roughly divided into two stages, based on the different basic theory used and various methods of feature extraction ^[9].

In the first stage, researchers employed traditional machine learning algorithms along with handcrafted features of pathological images such as texture, spatial structure and colors. A lot of classical algorithms of feature extraction emerged during this period. Naik et al. ^[10] used a level-set algorithm to identify the feature of boundaries and a template matching algorithm with shape models to identify glands and nuclei. The accuracy obtained 87% when the algorithm was used to discriminate cancer form non-cancer in breast histology. Jung and Kim ^[11] proposed a watershed-based method for segmentation of cervical and breast cell images, and introduced ellipsoidal modeling of contours to adjust nuclei contours. Kowal et al. ^[12] used four different clustering approaches including K-means, fuzzy C-means, competitive learning neural networks and Gaussian mixture models to each extract 42 morphological, topological and texture features. Next these features were put into three different classifiers to decide whether cancer occurred. Belsare et al. ^[13] proposed a similarity based super pixel generation method and integrate it with texton representation to form spatio-color-texture map of breast histology image. This approach segmented nuclear arrangement in normal or malignant duct. Compared to manual segmentations, the above methods did demonstrate good accuracy on early pathological datasets. However, these operations are time-consuming and computationally expensive ^[14], and cannot work without professional pathological knowledge. Besides, as the resolution and scale of datasets continue to improve, the low-level visual feature information that is easy to be observed by human eyes no longer meet the requirements, and abstract and high-level features are difficult to be captured. Meanwhile, the limitations of conventional algorithms also lead to poor robustness of the model, which seriously restrict the evolution of medical image segmentation.

In the second stage, deep learning (DL), especially convolutional neural network (CNN) begins to play an important role in the field of computer vision ^[15]. CNNs can automatically learn useful features from training data by optimizing the loss function, which not only avoids the complexity and limitation of handcrafted features, but also enables image segmentation models better robustness and versatility. For the past few years, CNNs have been widely used in medical image segmentation and have made excellent achievements. Ciresan et al. ^[16] classified patches with each pixel as the center, and take the classification result as the category of the pixel to get a pixel-wise classification image so that the segmentation completed. This method ranked first in the ICPR2012 mitosis detection competition. But the pixel-by-pixel computation causes high overlapping and wasting, which is not suitable for giga-pixel WSIs. Litjens et al. ^[17] converted CNN to a fully convolutional network (FCN) and applied it to the WSI to get a likelihood map where each pixel has a continuous potential between 0 and 1 of containing cancer. Compared with pixel-by-pixel method ^[16], FCN employed transposed convolution instead of simple upsampling and achieved end-to-end segmentation in a more convenient and faster way. In 2015, U-net, one of the most frequently used models in medical field, was proposed by Ronneberger et al. ^[18]. U-net was an encoder-decoder model, which captured global features on the contraction path and realize accurate location on the extension path, so as to reserve both low-level and high-level information of images. The segmentation accuracy was greatly improved and a lot of state-of-art schemes were the modified version of the encoder-decoder structure. Ho et al. ^[19], using U-net as the single-magnification network, proposed a multi-encoder multi-decoder multi-concatenation architecture for breast cancer images at different magnifications and segmentated the WSIs into carcinoma, benign epithelial, background, stroma, necrotic and adipose. Anand et al. ^[20] raised a ResU-Net structure for breast tumor segmentation where residual connections were added to address vanishing gradient. Chen et al. ^[21] proposed Deeplab which introduced dilated convolution for the first time. Without increasing the amount of calculation, Deeplab can obtain more contextual information, which won the first place in PASCAL VOC competition. The improved Deeplab v3+ was introduced to recognize and segmentate gastric cancer ^[22], and has proven its worth in automatic medical image processing.

Despite the above studies to some extent achieved satisfactory results, semantic segmentation on breast cancer images still has a long way to go. Most of the studies to date focus on binary classification such as detection of cancer areas, and classification of benign and malignant. Multi-classification concerning the malignant degree of tumor is more clinically valuable because it provides more specific grading information, where interclass difference is much smaller and needs multiscale features integrating more detailed information. Moreover, most of the existing models have complicated structure and a large number of parameters, leading to high memory consumption and low inference speed, which is contrary to the original intention of aiding diagnosis in an efficient way. Considering the above problems, we refer to Bilateral Segmentation Network (BiSeNet) ^[23] with excellent performance on remote sensing images and proposed a dual-path feature extraction network for breast pathological image segmentation.

3.   Materials and methods

In this section, we first display the dataset used in this paper. Then, the pre-processing operations of the whole-slide images (WSIs) are stated. In the last subsection, the architecture of the proposed segmentation model is presented and each module of the model is described in detail.

3.1.   Dataset

This paper adopts the standard dataset from ICIAR 2018 Grand Challenge on BreAst Cancer Histology images (BACH) Part B ^[24]. All the whole-slide images (WSIs) of this dataset were collected from hematoxylin-eosin (H & E) stained slices of breast cancer patients during 2013–2015, in Castelo Beanco, Portugal. The acquisition system was Leica SCN400. WSIs are released in the form of. svs, with a pixel scale of 0.467/pixel. Each image has a different size, with width ∈ [39980, 62952] (pixels) and height ∈ [27972, 44889] (pixels). Ten WSIs out of 30 were annotated by a pathologist and revised by a second one, where there was disagreement were discarded. Benign, in situ carcinoma and invasive carcinoma regions are pixel-wise annotated, and the rest is considered as normal tissue. The corresponding ground truth of each image was provided in the form of.xml files, containing a corresponding set of labelled coordinates of each kind of region.

3.2.   Pre-processing

WSIs comprise breast tissue as well as none-tissue regions. To focus on the tissue area and speed up the following process, this paper designs a method to extract the foreground area from WSI. In the H & E-stained slices, the cell nuclei are purple and the cytoplasm is pink, while the non-tissue area is grayish-white ^[25]. It is obvious that the more colorful the region, the more likely it is to be a tissue region. In other words, the more inconsistent the RGB channel values of the image, the more it would be preserved.

The binary image ${I}_{binary}$ is obtained by thresholding the range of the RGB channels. As shown in Figure 2, the colorful area is filtered out so that the binary mask is made and the useless none-tissue areas are removed.

WSIs cannot be directly input to the neural network because the existing graphics card cannot support an array of giga-pixel images. What's more, the length and width of the images from ICIAR2018 are not equal. If WSIs are forced into squares, 1/4 of the information will be lost and distortion may occur. Accordingly, we crop WSIs into patches with a size of 512 × 512. The patch of this size is not only large enough to envelop plenty diagnostic information and prevent the imbalance of training categories that may cause overfitting, but also small enough to concentrate on cell-level details and avoid too much memory and time consuming ^[18]. Finally, considering the great gap in the proportion of samples (normal regions are much more than the other three), we performed data augmentation on the training set to ensure uniform distribution and prevent overfitting ^[26].

3.3.   Model building

The framework of our network is shown as Figure 3. Two paths are proposed to capture and encode different levels of feature information of breast cancer images, a feature fusion model (FFM) is followed. To realize the end-to-end segmentation, we adopt 8 unsampling operations on the joint spatial-semantic feature. At last, a Softmax function is conducted to supervise the output of the network. The rest of this section specifically illustrates each part of the network.

3.3.1.   Spatial path

The structure of spatial path is shown as Figure 4. Firstly, three ordinary convolutions were performed on the pre-processed patch with a kernel size of 3 × 3 and a stride of 2, producing a feature map whose size is 1/8 of the original patch, which directly composes 1/4 of the output of spatial path as identity mapping by residual connection.

We introduce three parallel branches based on the structure of RFB ^[27], which respectively consist of different size of kernels and dilated convolution layers with various dilation rates. From top to bottom: The first branch consists of a 1 × 1 convolution kernel, a 3 × 3 dilated convolution kernel with dilation rate set to 1, equivalent to a 1 × 1 receptive field; The middle branch compromises a 3 × 3 convolution kernel together with a 3 × 3 dilated convolution kernel with dilation rate set to 3, equivalent to a 1 × 1 receptive field. The third branch introduced two 3 × 3 convolutional layers instead of a 5 × 5 kernel, followed by a 3 × 3 dilated convolution kernel with dilation rate set to 3.

Eventually, the three feature maps extracted by the three parallel branches are concatenated together with the identity mapping, forming the output of spatial path. The output feature map with the size of (64, 64,128) includes multi-scale contextual information and further prepares high-resolution low-level spatial information for the segmentation task.

3.3.2.   Semantic path

In contrast to spatial path, semantic path aims at the extraction of high-level semantic information, as it is shown in Figure 5. Xception39 ^[28] is explored as the backbone network, which is a depthwise separable convolution network designed on the basis of InceptionV3. It is light-weighted compared with other networks ^{[17,18,19,21]} commonly used in medical images thus effectively reducing training costs. Xception39 was then is followed by a global average pool to integrate global context information.

Every channel of the feature map corresponds to a specific semantic feature response, while only discriminative features do help with segmentation results ^[29]. Therefore, this paper designs a feature attention module which aims to pick out discriminative features from the last two stages of Xception36. This module is based on the channel attention mechanism which can capture channel-wise dependencies and put emphasis on salient objects to suppress the redundant discriminative features.

The structure of feature attention module is shown in Figure 6. Compared to SE-Block ^[29], we conduct both global average and max pool on the input feature map and generating two 1D feature vectors. Then the two vectors are concatenated and then reduced the channel number to 1/16 by a 1 × 1 convolution layer. ReLU function introduces non-linearity after the channel reduction. Then the channel expanded via a 1 × 1 convolution layer to 128, and a sigmoid function is utilized to activate the convolution result so that 1D attentive weight W_FAM could be obtained. The value of each element of W_FAM is constrained between 0 and 1. The input feature map of feature attention module passes through a 1 × 1 projection layer which calibrates the number of channels to 128, and finally a channel-wise multiplication is performed with attentive weight W_FAM to generate the output feature map of feature attentive module.

Back to the bottom of Figure 5, semantic path step-wise combines upsampled output feature maps from the global pool layer and the last two stages of the semantic path.

3.3.3.   Feature fusion module

The features extracted by the two paths have different levels of information. The spatial path captures rich low-level detailed spatial information, while the semantic path encodes high level semantic information. If two different levels of information are integrated forcibly, the huge gap between them may lead to terrible segmentation results. Therefore, a feature fusion module must be proposed to narrow the gap. In our model, the feature fusion module is also based on the channel attention mechanism, as shown in Figure 7.

We first element-wise add two output feature maps from two paths as a preliminary fusion. Then a global average pooling is operated. Next two fully-connected convolution (FC) layers are performed on the 1D feature vector for channel reduction and expansion, which respectively follow the two activation functions of ReLU and Sigmoid. The 1D weight vector W_FFM is generated by the preliminary fusion, so it can effectively narrow the gap between high-level and low-level feature information, and guide feature selection. By the means of channel-wise multiplying W_FFM with output feature maps from spatial and semantic path, feature representations important for segmentation are adaptively selected and discriminative ones are inhibited. Eventually, the two reweighted feature map are concatenated to generate the final fusion output, which is the end of the whole dual-path feature extraction.

4.   Experiment

4.1.   Experimental details

The entire semantic segmentation experiment is completed under the Keras framework built under windows 10. The experimental platform established by Inter ® Xeon E5-2620 v4 CPU 2.4 GHz, and NVIDIA RTX2080 GPU with 8 G GPU memory.

There are 10 pixel-wise annotated WSIs in ICIAR2018 BACH dataset. We use 8 of them for training and 2 for verification. According to the label type of the image center pixel, 2000 normal, 2000 benign and 2000 in situ carcinoma, and 2000 invasive carcinoma patches were randomly extracted from each WSI to form the training dataset. In order to prevent model overfitting ^[30], all the patches are rotated by 90°, 180° and 270° each, so that the scale of training data is expanded to 4 times the original.

The initial learning rate of the model is set to 0.00 01, and the Adam optimizer is selected for iterative optimization. Small batch stochastic gradient descent algorithm is used. The batch size is set to 4, and the epoch is set to 500. In addition, in order to avoid model overfitting, the regularization method of early termination is used in the training process.

4.2.   Evaluation metrics

In the segmentation of breast cancer whole-slide images, we adopt three metrics to evaluate the accuracy including Intersection over Union (IoU), mean Intersection over Union (mIoU), and Pixel Accuracy (PA), together with two metrics to estimate effectiveness of the algorithm, involving calculation time and model complexity.

Pixel accuracy (PA) is the ratio of the sum of all correctly classified pixels to the total number of pixels, which is defined as follows:

The intersection over union (IoU) of each category is the ratio of the number of real samples to the sum of the numbers of real samples, false negative samples and false positive samples in this category. And mean Intersection over Union (mIoU) averages the IoU of all categories, which is defined as follows:

4.3.   Experimental results

To evaluate the proposed model, we compare the segmentation performance with three other methods based on deep learning, including FCN-32s ^[17], U-Net ^[18], and Deeplab ^[21].

4.3.1.   Results for accuracy evaluation

As shown in Table 1, The PA and mIoU scores of the semantic segmentation method proposed in this paper are higher than those of other networks. At the same time, the IoU score of our model in benign, in situ carcinoma and invasive carcinoma is the highest in all networks, and the IoU of normal tissue is slightly lower than that of the best Deeplab network. However, the basic purpose of CAD is to automatically identify the cancer area, which means it is more critical to accurately segment cancerous regions (benign, in situ, and invasive) than normal tissues. Therefore, in terms of overall accuracy, the experimental results on ICIAR2018 BACH dataset strongly verify the effectiveness of the proposed networks' feature capture ability and model performance.

4.3.2.   Results for efficiency evaluation

The fundamental purpose of computer-aided diagnosis is to improve the diagnosis efficiency of pathologists. The blind pursuit of segmentation accuracy while ignoring the calculation cost cannot put the algorithm into practical use. Therefore, efficiency is also an important evaluation index of the semantic segmentation model. As we can see in Table 2, the parameter amount of the FCN is large, and the training time is the longest. Although Deeplab and U-net have nice performance, due to the complex network structure, the running speed is also low. The light-weight network proposed in this paper reaches a better segmentation performance without increasing the number of parameters, and the calculation speed is obviously faster than the other three networks.

5.   Discussion

5.1.   Discussion on the segmentation results

5.1.1.   Analysis of the segmentation accuracy

Figure 8 shows the comparison between our network and other networks. The superiority of our structure is reflected in the following aspects:

1) The dual-path framework in feature extraction exactly meets the requirements of breast tissue classification. The semantic path can extract high-level semantic features, and the spatial path can supplement the underlying spatial information accordingly, which can fully extract the rich information at the cell level as well as tissue level in the pathological image of breast cancer.

2) Dilated convolution is utilized to enhance the capability of encoding multi-scale contexts and thus resolve high intra-class differences in breast tissue.

3) Feature attention module based on channel attention reinforces representational ability of semantics and thus address low inter-class variations in breast tissue.

4) Channel attention mechanism efficiently fusion the low-level and high-level features. Two levels of feature fused preliminarily and then direct the low-level feature to restore semantic predictions, and also provide high-level feature with contextual particulars. This fusion method closes the distance of feature maps from dual paths and combines them more wisely.

5.1.2.   Analysis of the segmentation efficiency

When applied in practice, the load of high-resolution histological images along with large model complexity not only cannot be an effective auxiliary diagnosis tool, but also may become a burden to GPU/CPU memory and put forward higher requirements on the performance of computers in hospitals. So, efficiency is definitely a vital factor when segmentation system is devised. Our method has obvious advantage over other state-of-time models on ICPR2018 dataset as it is suggested in Section 4.3.2. There are mainly three following key concepts attributing to the high efficiency of our segmentation system:

1) The foreground segmentation based on thresholding operated in the pre-processing stage removes nontissue areas in WSIs, which considerably reduces the data that needs to be processed later.

2) In the spatial path of the proposed network, shallow layers are adapted and channel capacity is cautiously arranged to ensure no added model complexity and no loss of accuracy at the same time.

3) The semantic path needs a deep-layer network to capture high-level feature maps. We utilized the light-weight backbone network Xception to alleviate the computational burden caused by deep networks.

5.2.   Ablation analysis on the segmentation network

To investigate the property of every single part in the proposed network and further maximize the benefits of the model, we conducted separate experiments targeting at different modules.

Firstly, in the spatial path, the three parallel branches involve dilated convolutional layers with three dilated rates. It is elaborated in chapter 3.3.1 that the eccentricity of the receptive field is controlled by dilated rates, which inevitably become the key parameters determining the model performance. Therefore, we decide three sets of dilation rates {1, 3, 5}, {1, 2, 3}, {5, 6, 7} to see their impact on our segmentation model. The results shown in Table.3 confirm that the dilated rate {1, 3, 5} is favorable for the proposed breast cancer segmentation algorithm.

While in the semantic path, the feature attention module plays the dominant role in extracting and selecting discriminative features. Therefore, the ablation study carried out by removing it and replacing it with an alternative module SE-block as shown in Table 4. The accuracy decreased significantly when the channel attention module is not introduced. Our feature attention module is inspired by SE-block ^[29] but presents a slightly better performance, probably owing to the addition of max pooling operation that aims at selecting the local distinctive feature and thus having a better channel-wise inference of feature mapping highlights.

In addition, we also display the ablation analysis using the feature fusion method. We compare the channel–attention-mechanism-based feature fusion module of our network with element-wise addition and concatenation as it is shown in Table 5. Element-wise addition and concatenation could to some extent reduce the model complexity, but show poor segmentation effects, especially in terms of pixel accuracy (PA) score. Our method suggests a significant improvement of model performance, which shows effectiveness in integrating different levels of information from different paths.

5.3.   Reprocessing

It can be seen from Figure 8 that although the method proposed in this paper is ahead of other algorithms in terms of segmentation effect, there are still many holes in particular regions, and the boundary is not smooth and clear enough. After analysis, it is found that the main reason is that the 512 × 512 image block cropping operation limits the available context of the convolutional neural network to a certain extent, making the network unable to obtain a larger range of organizational features, and the final segmentation result lacks global shape information. In view of the above problems, this paper implies a conditional random field (CRF) ^[20] to optimize the images, and the final results of mapping on the original WSIs are shown in Figure 9. The segmented area tends to be complete and smooth, which is closer to the ground truth. Table 6 shows the changes of accuracy and efficiency before and after reprocessing. It can be seen that the morphological pre-processing has a slightly positive impact on the segmentation accuracy, but there is a loss in costing time.

6.   Conclusions

The automatic segmentation of different regions of breast tissue pathological images is of great significance to the computer-aided diagnosis of breast cancer, and can provide reliable reference for the diagnosis and prognosis of breast cancer. This paper proposes a deep convolutional network based on dual-path feature extraction networks to solve the semantic segmentation task of high-resolution breast cancer WSIs. The light-weight dual-path structure is designed comprising dilated convolution layers in for encoding sufficient multi-scale contexts, feature attention module in semantic path for selecting discriminative semantic features, and feature fusion module for the efficient fusion of heterogeneous outputs of two different levels. This segmentation system focuses on improving the consistency of the same type of regions and alleviate high inter-class variation of breast cancer histological images, reaching a balance between segmentation accuracy and efficiency. On the ICPR2018 dataset, our method outperformed three other networks in segmentation accuracy, especially for the most lethal invasive carcinoma. The PA score of the model in this paper reached 87.02%, and the mIoU score reached 68.12%.

At the same time, the method proposed also has some limitations. Firstly, because the standard medical datasets are scarce and difficult to obtain, this model was only verified on only one dataset. In the future, it should be tested on more datasets and applied to other computer vision tasks based on histological images, such as the detection of mitosis and nuclei. Secondly, according to the Nottingham grading system ^[31], invasive breast cancer can be further graded with three independent indexes ^[32], each of which worth future exploration, because the further classification of subtypes and histological grading has better clinic value for breast cancer prognosis.

Acknowledgments

The authors declare there is no acknowledgments.

Conflict of interest

The authors declare there is no conflict of interest.