Gas-Net: A deep neural network for gastric tumor semantic segmentation

1.   Introduction

All around the world, cancer is the disease that is always scary. The gastric cancer is a type of malignant (cancer) cells that form in the lining of the stomach and must be treated by a team of experts from several disciplines. However, the diagnosis of gastric tumour mainly relies on clinical manifestation (abdominal pain and/or vomiting), pathological images and medical imaging [2]. Furthermore, medical imaging-based endoscopy captures, simplifies the interpretation of gastric cancer that is more available to be real, accessible and without effects for the patient [3]. The team of experts based on the diagnosis of the oncologists decide how to proceed and what is the best treatment for the patient. Here, new technologies play an important role.

The artificial intelligence (AI) assists systems in pathology to solve limitations of human in diagnostic errors. So, using AI via deep learning let the improvement of state-of-the-art approaches [4] in various medical fields. In imaging data analysis, the convolutional neural network (CNN) is the well adapted application including a deep learning architecture. Whatever, in current image processing, CNN has accomplished success in objects recognition and classification [5],[6]. Sakai et al. have been attracted by gastric endoscopic images; they developed a patch-based classification algorithm based convolutional neural network (CNN) in order to identify gastric cancer in endoscopic images with an accuracy of 87.6% [7]. Nevertheless, as gastric tumor types vary and is more complex, the segmentation of the object that is the tumor creates a challenging task in computer-aided diagnosis, that has to correct performance and instant progress. Deep learning technics performance are certain to conduct such studies of medical images diagnosis in order to have the best results and satisfaction [8]. In 2018, study in [9] had used CNN to classify the genomics data of genes on Pan-Cancer Atlas. In [10], authors had combined the support vector machine (SVM) with random forest (RF) and CNN in classification of the high-grade gliomas, the study showed gain in time of 3.81 than other studies in literature. Furthermore, studies in this concern observed that deep learning technics detect and classify tumors based on histopathological images, that it was been more efficient than that of manual recognition.

Furthermore, [11] propose a method that controls the boundaries of the lungs and heart to measure the CTR (a parameter for cardiac and pulmonary diseases). They are based a segmentation network of two classes, X-RayNet-1 and X-RayNet-2, that aim to provide fine segmentation performance through a minimum of trainable parameters. The precision of segmentation is relative to the precision of the CTR value. So, the most famous technics are Normal distribution, Uniform Approach, K-means. Moreover, the fuzzy logic and the decision tree is developed by [12] for the detection of Colorectal Cancer including 1162 cases in the Masoud internal clinic (Tehran, Iran). This study can recognize a Colorectal Cancer helping healing sooner, and which could decrease the Colorectal Cancer mortality degree.

In this concern, several common segmentation techniques have been developed. An FCN [13] substitutes the fully connected layer of a CNN through an up-sampling layer. Each pixel is considered with the class of object. This is known as “semantic segmentation”. However, the semantic segmentation methods process the entire image, this can cause more processing time. Works such as OVIS [14], are based classical methods for large volumes of the video (a set of images) in databases. In addition, individual objects in an image cannot be recognized. For semantic segmentation, the application of Mask R-CNN [15] to the diagnosis of gastric cancer had a success in several works [16]–[18] where, the detection of the tumor region can be surrounded by a bounding box and will be labeled in the mask layer.

In this work, an introduction to a new method of gastric cancer based Deep Learning technics is proposed. The second section displays the state of the art. The third section express problematic, and the fourth section describes the elementary proposal. The last section shows the different experiments and the obtained results confirming the effectiveness of the proposed method. At the end of the paper, a summary and future works are discussed.

2.   Related works

The aim of image classification is to attribute one or more label to images according to predefined categories. New technologies based deep learning for object recognition and segmentation prove the best results actually. It is because of the idea to create hidden feature representations based on the Convolutional Neural Networks. This technology is able to analyze and compute more quickly than traditional one. In medical imaging, systems are developed to aid doctors in diagnosis of early cancers.

More applications and studies for classification, detection and segmentation in gastric cancer ‎[19],[20], had been advanced where more annotated datasets are available for free or limited access. Authors in ‎[21] developed an automatic network able to learn features on two public datasets. But, most studies analysis of the evolution of gastric cancer progress ‎[22]–[24]. In this area of research, detection and localization of tumors is very important, it represents an instance of a predefined object that is a set of pixels of an image. Models like YOLO-V2 ‎[25], are used for multi-objects detector on the same image. Here in ‎[21], authors, develop a detector for a small object increasing feature information from the basic network model. Work in ‎[26] shows that bounding box can be annotated basing on label smoothing weighted loss enhancing the feature representation. Also, a new version of faster R-CNN had been presented in ‎[27] called RetinaNet, they use pavement surface images obtained from 2400 m of dual road units.

Another study from ‎[5] describes the sensitivity (SE) that is the detected number of correct gastric cancer lesion/actual number of gastric cancer lesions; And the Positive Predictive Value (PPV), is the detected number of correct gastric cancer lesions/number of lesions that were diagnosed as gastric cancer by a CNN. As a result of this study, the CNN takes 47 seconds to analyze the 2296 test images, it identifies 71 of 77 gastric cancer lesions with a PPV of 30.6% and a sensitivity of 92.2%. It diagnosed 232 total lesions as gastric cancer, 161 of which were non-cancerous and 70 of 71 lesions with diameter upper than 6 mm were correctly detected.

Also, in 2017, authors in‎ [28] introduce an architecture called SegNet that serves to develop a model by ‎[29] to detect real time polyps during colonoscopy as the same study in ‎[30]. Furthermore, for automatic gastric cancer recognition, Hirasawa et al. ‎[31] presented a system that transforms the output space of bounding boxes into a set of default boxes over various aspect ratios and scales per feature, with a sensitivity of 92.2% and a positive predictive value of 30.6%. Similarly, Sakai et al. ‎[7] developed a patch-based classification system-based CNN to identify stomach cancer in endoscopic images, achieving an accuracy of 87.6%. Hence, 2020 has known a project for early detection of gastric cancer from endoscopic images ‎[32], it is based mask R-CNN for instance segmentation areas, then, from different forms of morphologies missing detection of the early gastric cancer, they define 22.2% of the miss rate of gastric cancer.

Lesion based CNN was been developed in ‎[6], it is a form of deep learning algorithm that uses endoscopic images to identify the entire lesion. Endoscopy aids doctors in detecting the GI tract more easily, making it an important aspect of the gastrointestinal (GI) tract examination; yet, the accuracy of its diagnostic results is limited by the practitioner's expertise and the GI system's complex environmental circumstances. Furthermore, when specialists utilize EGD (esophagogastroduodenoscopy), which is a standard procedure for diagnosing stomach cancer, the false negative rate for identifying gastric cancer is 4.6–25.8% ‎[5]. They are based the cafe deep learning framework ‎[4], which is one of the most general and usually used framework. Here, the CNN effectively recognized 71 of 77 gastric cancer lesions with a sensitivity of 92.2 percent and a positive prediction value of 30.6 percent. It diagnosed 232 total lesions as gastric cancer, 161 of which were non-cancerous and 70 of 71 lesions with diameter upper than 6 mm were correctly detected.

Recently, in 2019, Guitao and Zhenwei achieved 61.2 of average prediction gastric cancer classification. These results were very promising compared to previous work ‎[32]. They can explore the improvement of the network structure of Mask R-CNN to make more suitable to detect medical images and using more data augmentation to perform their model. Also, other studies are based Deep Learning methods improved such studies in the areas of prediction and detection of gastric cancer ‎[31],[33],[34].

3.   Problematic

The objective of this work is to detect regions of lesions (abnormal tissue) that mean tumors in gastric organ. To this aim, we develop a method-based U-Net ‎[26] architecture. We named it as GAS-Net. We know that this one is the usually adopted architecture in medical image segmentation. U-Net was initially developed for medical image understanding and segmentation, it was been the core architecture in medical imaging. This architecture is divided into two major paths: a contracting path and an expansive path, the first one comprises several steps of processes related to convolution, normalization, max pooling, activation, concatenation. It aims to extract the key features of the input images and outputs a feature vector of a specific length. The second path extracts information from the first one as a collection, and based on feature vector, it operates in up-convolutions, and by a sequential process, it produces an output segmentation map. The main idea here is the relationship between the first and the second path of the architecture, it permits to the network to achieve highly correct information from the first path, so producing the segmentation mask that converge to the intended output.

The main concept of the GAS-Net architecture consists of establishing a network that begins in input by a set of x images and x groundtruth corresponding masks. From these groundtruth masks and based on probability and generalized function, the network learns the area of tumor from the x learning images, then it proceeds to create a submersion relationship between x learning images and x groundtruth corresponding masks. This is mathematically known as: Ω

F# is a continuous linear operator. T is the contracting function that determines the characteristic vectors of tumor areas to create a valid region. Similarly, F is the expansive function, it localizes the spatial features in order to detect the abnormal tissue region. Then, Ω corresponds to probabilities extracted from the x input images. These probabilities are comprised between 0 and 1 and correspond to the final convolutional layer of classification by the SoftMax function. Here, the main motivation is the spatiotemporal representation of key bounding boxes representing the tumor region.

4.   Our proposal

This section describes the GAS-Net method, it aims to detect and recognize gastric tumor area from endoscopic images. The GAS-Net method overview is shown in Figure 1. Typically, it is built on two major parts; the first one comprises the preprocessing step. While the second one, achieves the contracting and expansive processes in order to determine high level features from training set, and also, to detect the spatial information.

4.1.   Global process

Our work discusses an efficient approach to overcome the need of classification of endoscopic gastric cancer images. However, to boost the development of image recognition in gastric cancer, we experiment latest technologies relevant to gastric cancer detection and recognition. So, the aim was to detect and recognize tumors regions. Our method is based on two consecutives phases; the first phase is for pre-processing, it is more important because it represents the basis step of our approach, where we apply techniques to the chosen dataset suitable with the training algorithm; the second phase concerns training where we define a new method based on contracting/expansion path. This leads us to add a pseudo-code to eliminate or enlarge the area of the segmentation. Based on necessities, we adjust and add the hyper-parameters, after that, the combination of predictions and the application of the method to the predicted image and the mask, the outputs were enhanced. Furthermore, our experiments were based on Python language and Tensorflow libraries. This is satisfied by calculation done on GPU Nvidia GTX 1080. Additionally, Adam optimizer was used for training the network.

4.2.   Preprocessing

At the beginning, the size of the images varied from 150 × 150 to 1280 × 1080 pixels; and were saved in JPEG format. Moreover, all the images were downsampled to 256 × 256 pixels as of the image matrix size in relationship with the different cases. Via this operation, we normalize diverse images into the different five types of gastric cancer: Adenocarcinomas images, Gastrointestinal stromal tumors (GISTs)images, Neuroendocrine tumors (including carcinoids) images, Lymphomas images, and other cancers images, removing noise, same images, and improved accuracy. “Table 1” resumes the total images for the five types of cancer collected for the dataset1.

4.3.   GAS-Net architecture

In order to detect gastric tumor, from endoscopic images, we propose the GAS-Net method, based on the main architecture in medical images that is U-Net. The network of GAS-Net method is provided as input by a pre-processing step and a groundtruth masks. In order to locate feature map information, our method uses on a contracting path to extract all appropriate feature maps.

By increasing the depth of the network, the contracting path is generally composed of four residual blocks, each residual block involves two consecutives 3 × 3 convolutions complemented by a ReLU activation unit and a max-pooling layer, it approves understating vanishing gradient in order to save local information for enhancing precision and improving the conception of layers. These residual blocks form a feature map from an input vector to a residual unit's output. However, two inputs are introduced to the expansion path, the first one is from the output of the contracting path, the feature map is upsampled using 2 × 2 up-convolution, after that, the collected input from the parallel layer of the contracting path is linked to the upsampled feature map and complemented by two consecutives 3 × 3 convolutions and ReLU activation. Similarly, the second input is from the precedent layer of the expansion path. An applied 1 × 1 convolution is done to decrease the feature map, Softmax activation function is applied to predict the area of tumor (Function (2)).

After that, we propose to consider all neighbors pixels to the output contours in order to test them if they are able or not to be added to the area. Training the network was been satisfied, the common objects in the image were identified and classified correctly, however, these advances on the model output, leave a small margin to the objects with less quality of segmentation. To this end, with the spatial information of the input image, we propose to design prediction of original image and segmentation area around high overlap pixels. Algorithm1 describes all the process explained above.

In fact, Algorithm 1 describes the process of Gas-Net Method as designed in Figure 1, where the input of the Softmax function is z→(z₀, ... , z_k), note that z_i is a real value. The function e^zj serves to a positive value of each portion of the input vector. Therefore, k represents the number of classes in the multiclass classifier to associate an effective probability distribution. Next, we calculate the stochastic gradient in order to minimize the loss function explained in paragraph.4.4. The weighted combination of (BCL and SSL) in the loss function allowed a good convergence of the gradient and relevant learning. Adam optimizer selects estimations of gradient to adapt a single learning rate for all weights updated and the learning rate does not change during training. So, here, the Adam optimizer is responsible for updating the weights based on the backpropagation method.

The last step of the process of Algorithm1calculates the color variation between the predicted pixels of segmentation and their neighbors' pixels. Hence, if the variation is less than a threshold θ, we replace the old point by the newest ones. Then, we use a delay followed by loss to delete noisy pixels in the predicted area. At the end, we remove detected area that have a low score.

4.4.   The loss functions

In order to obtain precise mask and segmentation, we are based impacts of different loss functions in literature. The originality here is to combine two loss functions, the Binary Crossentropy Loss (BCL), and the Sensitivity Specificity Loss (SSL). In these loss functions, the aim is to measure the variance distributions in pixel level classification for segmentation. This is following by an output value that passes by the Softmax activation function. Here, α means the label and β means the predicted probability by the predicted model.

The BCL function is defined as follow:

And the Sensitivity Specificity Loss (SSL) is defined as follow:

Where both sensitivity and specificity are based metrics to predict segmentations. This loss function, is added to the first one in order to decrease the inequality of classes via the ω parameter. “Table 2” shows results of loss functions. The advantage of the union between BCL and SSL improves performance compared to calculating BCL alone or SSL alone. To explore these improvements, we show results on different Average precisions (AP, AP50, AP75) and the Average Recall (AR), where different loss functions impact area segmentations. BCL loss runs significantly limited with precision but uncertain pixels for segmentations due to the inequality of classes. However, we enhance equalities weights but this can lead us to an imprecise outline drawing of the mask and consequently to the instance segmentations and consistent masks. On the other hand, SSL loss function also overcome to inequalities of classes but still also imprecise in outline drawing of the area. Thus, joining BCL loss to SSL loss permitted a good convergence of the gradient and relevant learning. We got high performance in the area of gastric tumor localization on the test phase.

5.   Experiments

This section describes the whole experiments in order to consider the achievement of our proposed method. We define experimental results on two different datasets, the first one is from our basic collection (See Table 1) and the second one is a public available and open dataset named HyperKvasir ‎[1]. The using of both two datasets is a primordial idea to confirm the robustness of the method. Also, we show the importance of using hyper-parameters in order to control the learning process. At the end, we give results of the different performance metrics in the aim to perform the effectiveness of our method.

5.1.   The Dataset1 collection (Table 1)

For an automatic detection of gastric cancer, we had collected anonymously a complete dataset of EGD (Esophagogastroduodenoscopy) images of gastric cancer for our work from the cancer registry from 2011 to 2019 from a private medical-hospital clinic. These images served to establish automatic recognition of tumor based on GAS-Net method. Furthermore, a multidisciplinary team is organized to work together, it englobes gastroenterologist, surgeon, medical oncologist, pathologist, and dietitian. The pathologist observes some lesions from the stomach under a microscope, then, they assign a grade of 1 to 4 to stomach cancer. Low-grade cancer cells are well distinguished, whereas, High-grade cancer cells are poorly differentiated or undifferentiated. This grade plays a very important role for the team to decide on the results and to recognize the different cell kinds in each image for the training phase. On the other hand, as the world health organization, it exists five types of the stomach cancer: Adenocarcinomas, Gastrointestinal stromal tumors (GISTs), Neuroendocrine tumors (including carcinoids), Lymphomas, and other cancers.

“Table 1” shows all the data collection where the Adenocarcinomas type comprises 102 patients divided into 2 grades: grade 1,3 which comprise respectively 80 and 22 patients; The GISTs type comprise 46 patients divided into 2 grades: grade 2 and grade 3 which comprise respectively 34 and 12 patients; The Neuroendocrine type comprise 67 patients with grade 4; The Lymphomas type comprise 75 patients with grade 3; And 31 patients are non-identified tumors.

So, in our experiments, we aim to satisfy detection and recognition of the five types of stomach cancer cases. We have selected randomly 75% of the images for training and 25% for validation and test. Figure 2 shows the obtained masks on Dataset1.

5.2.   HyperKvasir dataset

This dataset is publicly available for research and educational purposes. It is divided into four distinct parts; Labeled image data, which contains 23 different classes of results, Unlabeled image data, that contains 99,417 unlabeled images, Segmented image data, that contains 2 folders, one for images and the other is for masks, each of them contains 1,000 JPEG-compressed images, and Annotated video data, that contains approximately 11.62 hours of videos and 1,059,519 video frames, it represents a total of 171 annotated results. Figure 3 shows both in the left original images as well as in the middle their corresponding ground truth masks and in the right their obtained masks.

5.3.   Performances of the network

All the implementations of the network are conducted on a workstation equipped with an Intel i7-6850K CPU with a 64 GB Ram and a single NVIDIA GTX Geforce 1080 Ti GPU and the operating system is Ubuntu 18.04.2., the batch is 64 and learning rate is initialized to 0.002 for all layers and reduced to 0.0002 every 10 epochs from the 200. Also, the aim of all systems in literature is to improve time in training and test phases to be an ergonomic system. In this respect, our study required time in execution at the training and test phases as a variation in accordance to the number of regions of interest to be processed per image, it was between 480 and 890 ms for an image sized 256 × 256.

Moreover, the performance of the network is based on using hyper-parameters. Therefore, we realized diverse experiments in order to select best values of the hyper-parameters and verify achievements of our method.

5.4.   Performance metrics

This step of study is fundamental to approve the behaviour of the model and whether the model is achieving its purposes. Diverse performance metrics are used in literature for object detection and image classification such as Accuracy, Precision, Sensitivity, Specificity, and other metrics. The formulas for evaluating these performance measures are represented respectively as below in Eqs 5–10.

Note that TP means the True Positive, TN means the True Negative, FP means the False Positive and FN means the False Negative. The parameters are estimated based on the confusion matrix, offering details about true and false classification of images from all categories. Furthermore, the sensitivity had an impact when an overlap was perceived between the obtained area of tumor and the ground truth created by the team discussed above.

6.   Results and discussion

The goal of our segmentation method is to simplify and/or transform the representation of the input image into a set of segments that are more expressive and easier to analyse.

6.1.   Comparison study on different backbones

In our study, we achieved several experiments using different architectures as a backbone (VGG16, VGG19, Resnet-50, Resnet-101 and ResNeXt-101), where the top outcomes were gained with the ResNeXt-101 architecture and the Adam optimizer. “Table 3” compares results on the different backbones on the proposed method. Precision, recall and F-measure, determine that our method with ResNeXt-101 as a backbone architecture gives the best result in precision and F-measure, in parallel for the recall, the architecture with ResNet-50 gives the best results compared to the others. “Table 4” resumes a comparison on the diverse training and test performances in order to enrich the study and improve the detection, the recognition and the classification of gastric tumor in images. In order to control the learning process, the impact of hyper-parameters is well seen, they were been used on the model's performance, and primarily at the test phase. It is evident to randomly select images from test dataset to achieve evaluation and consider performance of instance segmentation from the group of experts and the adopted method. This study has satisfied detection and recognition of objects with diverse sizes. Furthermore, this study is based on an improvement in the U-Net model. This improvement is able to draw contours around detected and recognized objects that have the same properties. Hence, experiments support the proposed method achievements and approve success of deep learning approaches for the detection and recognition of tumors.

Note that predictions with low score (less than 85%) have been eliminated. This is realized when a threshold must be fine-tuned. So, we use a larger threshold to confirm contours around the detected and recognized object. Moreover, identify all parts of images by considering the combination of the same number of each type, which induce an inequality in the dataset, and consequently a bad generalization. It can also be easily seen that predicted contours are clearer and conform to their adequate pixels. Our proposal proves the rapidity of deep learning approaches, in the detection and recognition of gastric tumors at all part of gastric system. Therefore, our proposal demonstrates that it is able to detect all relevant objects, thus, it is more efficient than the natives' models, which can be benefit for gastric diagnosis problems. Also, our proposal includes object detection in the segmentation, it consists originally of locating the tumor, after that, achieving the segmentation of each recognized object delimited by contours, which permits diverse parts according to the whole features instead of diverse data of each pixel.

Consequently, this study proves results realized through deep learning approaches. Our proposal is robust according to this kind of dataset, it offers a good vision and adoption of its pertinence. Figures 4,5 show the effectiveness of our proposal, where a trigger interrupts the training where the performance measure starts to decrease. Based on our discussions with team of experts at the unit of oncology, these obtained results based on our proposal will allow experts to have accurate results in record time and thus alleviate tedious tasks for oncologists and gastrologist in their daily practice. The main benefit of our proposal is the rapidity of reply compared to natives' methods. This approach can be added into a web application installed on a server in order to be used by several experts in parallel.

6.2.   Comparison between based models and our proposal on the two datasets

In this sub-section, Table 5 presents obtained results of evaluation on dataset 1 and dataset 2, of our method of endoscopic images compared to based models such as UNET [35], Attention U-Net [36], DenseUNet [37], Mask R-CNN [15], and UNet++ [38]. Dice, Sensitivity, and Accuracy metrics achieve 89.3%, 92.8% and 95.2% on dataset-1, and 89.3% 89% 94.7% on dataset-2.

7.   Conclusions

In this work, we introduced a smart proposal from new technologies in concern with deep learning. These approaches aim to automatically overcome to problems of detection, recognition and classification of diverse tumors in gastric medical images. A method was developed and adopted and different architectures as a backbone were tested such as VGG-16, VGG-19, Resnet-50, Resnet-101 and ResNeXt-101 where this last one proved the best results. This proposal is able to recognize abnormal tissue inline to the stomach. From the five types of tumors in different grades, a binary mask is calculated to achieve the segmentation. The time of training is minimized for a best generalization. We had used two functions for loss in order to measure the variance distributions and also to decrease the inequality of classes. We had collected a first dataset based on a set of images form a private clinic and a second dataset was introduced to confirm the proposal regarding to the ground truth. Results in experiments demonstrate that the proposal can distinguish lesions from any input image. Also, high precision in segmentation and classification of tumor proves its effectiveness. Our future works consists on developing an extension to the adopted backbone in order to ensure a best instance segmentation of the area of the tumor.