MARnet: multi-scale adaptive residual neural network for chest X-ray images recognition of lung diseases

1.   Introduction

According to statistical data from the World Health Organization, lung diseases rank the third among all causes of death worldwide ^[1]. Lung diseases lead to the death of more than five million people worldwide each year ^[2]. As an important organ of the respiratory system, the lungs connect with the outside and the whole blood flows, so the lungs are vulnerable to internal and external microbial attack. Pneumonia is an inflammation of the lung parenchyma caused by pathogenic microbiological, physical and chemical factors, immune damage, allergies and drugs ^[3]. Bacterial pneumonia is the most common one, and its chest X-ray image shows pulmonary exudation or solid shadow. Lung infiltration is there are some abnormal cells or foreign body in the lungs. Lung infiltration is caused by bacteria, fungi and viruses, and infiltrated shadow can be seen on chest X-ray images. Exudative lesions are acute inflammation, mostly caused by bacterial, viral or fungal infections, and its chest X-ray images will appear cloudy shadows. Atelectasis is a common complication of respiratory diseases which is divided into congenital atelectasis and secondary atelectasis. The symptoms include lung compression and decrease of ventilation-flow ratio in lungs ^[4]. The causes of congenital atelectasis are mostly childbirth injury, and the causes of secondary atelectasis include tuberculosis and drowning. The X-ray features of atelectasis include low-texture region, cavity, high density areas and fibrosis so on. For the lung nodule, there are some opaque nodular sections with clear edge, surrounded by air-bearing lung tissue, which diameter is less than or equal to 3 cm, without symptoms of atelectasis, hilar enlargement or pleural effusion. Tumor, infectious granuloma and congenital lesions can cause pulmonary nodules. In chest X-ray images, nodules show small, localized, round, and high-density shadows. Since malignant tumors have nodular lesions, effective detection of nodules plays an important role in the diagnosis of lung cancer ^[5].

Chest X-ray image is the most commonly medical imaging technology in clinical diagnosis of lung diseases, which is very effective in the identification and detection of cardiothoracic, pulmonary and interstitial diseases and plays an important role in the treatment of lung diseases ^[6]. Accurate analysis of patient's health information is a great challenge for radiologists. So computer-aided diagnosis system is proposed to control the differences among radiologists and provide reference for clinicians. In recent years, deep learning has been widely used in medical image field. Compared with manual feature extraction, deep learning technology can automatically learn features from data by training large-scale dataset, and has made significant progress in computer vision ^{[7,8,9,10,11]}. Deep learning has been increasingly applied in the field of medical image, especially image classification ^[12,13,14]. Related researches mainly include the automatic classification of lung tuberculosis ^[15], lung nodules detection ^[16,17], lung cancer detection ^[18,19,20], pneumonia detection ^[21,22,23], pneumothorax detection ^[24] and COVID-19 prediction ^[25,26,27]. Here, we summarize the related work of deep learning technology for lung disease detection.

Recent studies showed that deep neural networks can automatically learn features from training data, that show higher efficiency and discrimination accuracy. Kim et al. ^[28] used a six-layer convolutional neural network (CNN) consisting of four convolutional layers and two fully connected layers to classify interstitial lung diseases. Comparing with the results of support vector machine (SVM), they found that the accuracy of CNN classifier is 6–9% higher than SVM classifier. Sivaramakrishnan et al. ^[29] compared the effects of chest X-ray image in crowd screening, who found that the pre-trained and customized deep models are much better than the shallow models in performance. In order to detect nodules early, Pinheiro et al. ^[30] trained convolutional neural networks using swarm intelligence algorithms, and verified whether this method is more effective than traditional training algorithms such as back propagation and gradient descent. The results showed that in the lung image database consortium image collection (LIDC-IDRI), the model achieved an accuracy of 93.71%, which was superior to the back propagation model. In the application of deep neural network in lung tumor prediction, Xu et al. ^[31] evaluated the role of deep neural network in predicting clinical results by analyzing CT images of patients with locally advanced non-small cell lung cancer (NSCLC), and achieved good results. In order to screen lower respiratory tract diseases in children, Chen et al. ^[32] proposed a computer-aided chest X-ray diagnostic program for the classification of bronchiolitis/bronchitis, bronchopneumonia/interstitial pneumonia, lobar pneumonia, pneumothorax and normal children. What's more, Grad-GAM technique was used to show the results of the model. Since they used the existing ResNet34 and DenseNet169 as discriminant models, the recognition performance was expected to be improved.

Rajpurkar et al. ^[33] developed a model called CheXNet and tested 14 types of lung diseases. They compared the detection performance of CheXNet with 4 professional radiologists, and found that CheXNet exceeds average radiologist level on the F1-score. For Chest X-ray 14 dataset, Angelica et al. proposed GraphXNET ^[34] and achieved good results. Wang et al. proposed TieNet ^[35]. Compared with the previous classical neural networks, TieNet has significantly improved the recognition ability of chest X-ray images (AUCs increase by 6% on average). Xu et al. ^[36] designed CXNet-m1 and proposed a new loss function sin-loss. In addition, they optimized the convolution kernel of CXNet-m1 to achieve better classification accuracy. The experimental results showed that CXNet-m1 with sin-loss function achieves better promotion in the index of accuracy rate, recall rate, F1-score, and AUC value. Zhao et al. ^[37] proposed AM_DenseNet for chest X-ray image classification. This model used a dense connection network and added an attention module after each dense block to optimize the ability of the model to extract features. They used the Focal Loss function to solve the data imbalance problem. The average AUC of AM_DenseNet detection for 14 kinds of chest diseases was 0.8537. Ho et al. ^[38] proposed different knowledge distillation strategies for 14 diseases, and used heat map techniques to visualize the abnormal characteristics of X-ray images. As them mentioned in their paper, although the total amount of chest X-ray 14 dataset exceeds 100,000, it is extremely unbalanced among various categories and there are annotation errors. In addition, some diseases of different categories have great similarity or related characteristics, directly classifying these data will cause great interference to the model and lead to poor results.

At present, there are still several limitations in using deep learning to study chest X-ray images. First of all, most studies focus on a certain type of disease, and two-class classification lacks universality. In multi-classification problems, the similarity of pathological features of some diseases leads to poor recognition ^[32]. What's more, in previous researches about chest X-ray images, some models cannot well identify the morphological and scale characteristics of different lung diseases on chest X-ray images, so the recognition results are poor. In addition, although a large number of convolutional neural networks recognize chest X-ray images accurately, they do not explain how and why the algorithm gets the final results to the medical staff. In order to display the output results more intuitively, target detection technology has been widely used. This technology uses the bounding box to circle the diseased parts, so as to visualize the classification and positioning of chest X-ray images ^[39]. However, the use of target detection technology needs to establish a special training database which records the disease area coordinates marked by experts. This greatly limits the further development of deep learning technology in the field of X-ray image detection. Our contributions are as follows:

1) We construct a relatively large and balanced dataset, including four types of chest X-ray images of nodules, atelectasis, infection and healthy individuals. Firstly, we isolate six types of images which suffer from only one disease from chest X-ray 14. Since the three types of images have high similarity, we classify them as a class called infection. Then we add healthy individual chest X-ray images. Finally, we constitute the dataset for test MARnet performance.

2) According to the characteristics of different lung diseases on chest X-ray images, we construct MARnet to identify chest X-ray images with different symptoms. In order to identify four types of chest X-ray images better, we alternately transmit the feature information extracted by adaptive structure and residual block to different layers. Experimental studies have shown that MARnet has improved greatly the ability to identify chest X-ray images compared with other models. Finally, we use 5-fold cross-validation to test the generalization of MARnet.

3) In order to solve the problem that the identification of neural network is difficult to be understood by human beings, we use Grad-CAM technology to visualize the disease area in the form of heat map. Heat maps make the results of MARnet more intuitive, providing a more valuable reference for clinicians to diagnose pulmonary diseases.

2.   Materials and methods

2.1.   Design

This section describes the overall process of MARnet for chest X-ray image recognition. As shown in Figure 1, chest X-ray images are first sent to block A. Image features are extracted from block A and the information is introduced into a series of blocks B. Then feature information by block B is introduced into the following block A and another series of blocks B. The process of image processing through 1 block A and 4 or 6 blocks B is called one round feature extraction. At the overall process, the input image is extracted by 4 rounds of feature extraction. The heat map is used to display the disease area of the image. Finally, the result is archived.

In Figure 1, the parameter C in block A represents the number of features extracted from the convolution layer. The color squares in block A and B represent the feature information extracted from the convolution kernels.

2.2.   Data description

The chest X-ray 14 dataset selected in our experiment is the largest publicly available one in the world, which contains 112,120 X-ray images from 30,805 unique patients. Each image is marked with a single or multiple pathological labels, and radiological reports shows that the accuracy rate is more than 90% ^[38]. Considering the large number of patients with multiple pulmonary diseases in the dataset, we select images from the dataset with only infiltration, effusion, pneumonia, nodule, and atelectasis in order to eliminate interference. There is big similarity in the chest X-ray images among infiltration, effusion and pneumonia, so medical examination is required to confirm the diagnosis. We combine these three types into the infection. Finally, we study the chest X-ray images of 13,382 patients with nodules, atelectasis, infection and normal. Figure 2 shows the randomly selected chest X-ray images of nodule, atelectasis, normal and infection.

The datasets are divided into train sets, test sets and validation sets in a ratio of 7:2:1. Table 1 shows the number and category of chest X-ray images.

To inspect the generalization of MARnet, we use 5-fold cross-validation. We process the dataset in Table 1. First, we combine all the data, and then split the data into five equal pieces. One is selected as the test set and the rest as the train set in turn. Finally, MARnet is trained and tested five times. We resize images from 1024 × 1024 pixels to 224 × 224 pixels to meet the input size requirements of the model.

2.3.   Constitution of MARnet

This section introduces the structure of MARnet, which is a 137-layer network. In MARnet, the adaptive structure and residual block alternately transmit the extracted feature information to the subsequent layer with high efficiency.

2.3.1.   Residual structure

The deeper the neural network is, the more semantic information can be extracted from different levels and the more abstract meaning can be expressed. However, when the level of neural network increases to a certain amount, gradient dispersion or gradient explosion will occur, so that the accuracy is no longer improved or even decreased. The residual structure can be used to train a deeper neural network. Therefore, we added residual structures to the multi-scale and adaptive modules of MARnet.

2.3.2.   Adaptive structure

The feature map in convolutional neural network reflects the feature information extracted by convolutional kernel to a certain extent. Adaptive learning is to automatically obtain the importance degree of each feature. Then, according to this importance degree, enhancing the main feature and suppressing the secondary feature ^[40]. In the adaptive structure, the adaptive global average pooling layer (AdaptiveAvgPool) is used to compress the feature along the spatial dimension, and each two-dimensional feature map is transformed into a real number with global sensing ability, which can reflect the feature distribution. Then a 1 × 1 convolution kernel (input channel number is a, output channel number is a/4) is used to reduce dimension. ReLU activation function is used to increase nonlinearity. Another 1 × 1 convolution kernel (input channel number is a/4, output channel number is a) is used to increase dimension and reduce computational cost. In the last layer of the adaptive structure, Sigmoid function is used to generate a real number between 0 and 1 as the output result. The adaptive number is transmitted to the behind layer of the network, in fact which means a weight coefficient is multiplied to the feature matrix to enhance the main feature and suppress the secondary feature.

2.3.3.   Multi-scale residual block

The chest X-ray images of nodule, atelectasis, normal and infection have different feature scales. Considering that convolution kernels of different sizes have different receptive fields in neural networks, we use 3 parallel convolution kernels of different scales to extract feature information, and fuse them together. This multi-scale structure shows good performance in our experiments. The MARnet has three-group parallel multi-scale residual blocks and each group is composed of 4 convolution layers. We exploit the residual structure to transmit the information of the 0th layer to the end of the last layer. In the multi-scale residual block, we use 3 × 3 uniform convolution kernel in the first group. In the second group, we replace the first two 3 × 3 convolution layer with 5 × 5 convolution kernel on the basis of group 1. In the third group, we replace the first three convolution layer of 3 × 3 with 7 × 7 convolution kernel.

2.3.4.   Adaptive residual block

In order to better extract the image's feature information, we add the residual structure in the adaptive block to construct a deeper neural network. The adaptive structure generates a feature adaptive coefficient between 0 and 1, and the residual structure creates a matrix. If the adaptive and residual structures transmit the feature information from the same layer to the same subsequent layer, it means the feature matrix generated by the residual structure will be added to another matrix multiplied by the adaptive coefficient, that lead to the weakening of the adaptive function due to the residual structure. Based on the reason, in MARnet, the residual structure transmits the 0th layer feature to the end of the 3rd and 6th layers. Meanwhile, the adaptive structure transmits the 0th, 2nd and 4th layer feature to the end of the 2nd, 4th and last layer, respectively. This structure efficiently avoids the weakening effect of residual block on adaptive structure. Figure 3 demonstrates MARnet's adaptive residual block.

2.4.   Hyper-parameters tuning

In our exploration, we try to estimate the influence of various hyper-parameters on the performance of the neural network. The parameters include batch size, learning rate and activation functions, which have an important impact on the model performance. Five different batch size parameters, {24, 36, 32, 48, 64}, are balanced by lots of contrast experiments, and finally the model performance is best when this parameter is got as 32. The learning rate affects the convergence speed of the model, and excessive learning rate will reduce the accuracy. We attempt 8 kinds of learning rate, {1e-3, 1e-4, 1e-5, 3e-3, 3e-4, 3e-5, 3e-6, 3e-7}, to find the result is best when the learning rate is set to 1e-4 or 3e-4. In terms of learning algorithm, we find that Adam is significantly better than SGD or Adadelta. The internal parameters in the neural network include convolution kernel size, stride, kernel size of the pooling layer, channel and dropout rate. Table 2 shows the structure and parameters of MARnet.

2.5.   Grad-CAM visualization

In terms of image recognition, if the local part of the image determining the decision classification is highlighted, a good visual result will be given to improve the interpretation of recognition. Gradient-weighted class activation map (Grad-CAM) uses gradient information flowing into the last convolution layer of the neural network to infer the importance of each neuron, and gives the corresponding results in the form of heat map which helps to visualize the prediction results of the network ^[41]. We use Grad-CAM technology to display the disease area on chest X-ray images, which provides a more intuitive reference to doctors.

3.   Experiments

3.1.   Performance evaluation

In order to comprehensively evaluate the performance of our model, we introduce accuracy (ACC) ^[42], Precision, Recall and F1-score as the evaluation indicator ^[43]. The formulas for these indicators are shown below.

Here TP, FP, TN and FN represent the number of true positive, false positive, true negative and false negative, respectively. In addition, we draw the receiver operating characteristic (ROC) curve and calculate the area under curve (AUC) to further evaluate the performance of the model ^[44,45]. The higher the AUC value, the better the performance of the model.

3.2.   Ablation study

We conduct ablation study on MARnet to evaluate the effectiveness of such design. In ablation study, Multi-scale Residual convolutional neural network (MRnet), adaptive residual convolutional neural network (ARnet) and fusion adaptive convolutional neural network (FAnet) are used. MRnet is a 43-layer neural network that uses three sets of residual neural networks with similar structures in parallel. ARnet is a 43-layer neural network that links 13 adaptive residual modules. In the ARnet, each adaptive residual module contains three layers of convolution, and the residual and adaptive structure pass the 0th layer information to the end of the last layer respectively. The FAnet has 75 layers and consists of four groups of similar structures, and each group uses a block A followed by 4 block B that is the same as the block in ARnet.

3.2.   Results

Table 3 shows the test results of MARnet compared with MRnet, ARnet and FAnet for nodule, atelectasis, normal and infection. The results of ablative study show that the recognition performance of MARnet can not be achieved only by using multi-scale residual structure, only using adaptive residual structure or using adaptive residual structure (as ARnet). In general, the design of MARnet is effectiveness.

Table 4 shows the test results of MARnet compared with AlexNet ^[7], VGG16 ^[46], InceptionV2 ^[47], ResNet101 ^[12] and CliqueNet ^[48]. All experimental results are obtained by multiple experiments on the test dataset.

From Table 4, we can see that the shallow and simple AlexNet model cannot well identify four types of chest X-ray images. For example, when AlexNet identifies nodule, the recall, F1-score and AUC are 12.5, 14.8% and 0.69, respectively, indicating that the identification effect is not good. Compared with AlexNet, the overall recognition performance of the deeper network VGG16 for four types of images is improved, but the recall index is 34.0 ± 2.2% and the F1-score is 37.2 ± 1.4% when recognizing nodule, which still cannot meet the needs of practical application. ResNet101 with deeper structure and residual block, and CliqueNet with complex feature transfer mechanism improve furtherly the recognition effect of atelectasis, normal and infection. ResNet101 achieves the highest recall of 95.7% when identifying atelectasis, but these two models still fail to solve the problem of low recognition rate of nodule. InceptionV2 has the characteristics of multi-scale structure and improves the recognition ability of nodule, the precision, recall and F1-score reach 58.9, 42.5 and 42.0%, respectively. However, due to the shallow network layer, the recognition efficiency of InceptionV2 for the other three types of images cannot be improved. MARnet is a deep convolutional neural network with multi-scale characteristics. When MARnet recognizes nodule, the precision, recall and F1-score reach 69.8, 58.5 and 59.4%, respectively. The recognition ability of MARnet to atelectasis, normal and infection is also improved furtherly. Except that the recall index of atelectasis recognition is lower than ResNet101, the other indicators reach the highest level. Overall, MARnet performs best in all models. Figure 4 shows the ROC curves obtained by MARnet and five classical neural methods in identifying nodule, atelectasis, normal and infection on the test set. ROC curve shows that the recognition performance of MARnet on nodule is significantly better than that of other networks. In the identification of atelectasis, although the identification capabilities of these six models are relatively close, MARnet is still at a high level. When identifying normal, different types of neural networks show significant differences in recognition performance, and MARnet shows better recognition ability than other models. In the identification of infection, the traditional five neural networks have shown good recognition results, while the recognition performance of MARnet is significantly better than that of the others.

Figure 5 reveals the ACC and average AUC obtained by MARnet and other neural networks on the test set. From this figure, we can find that ACC obtained by MARnet reaches 83.3%, which is 7.9% higher than that of the second (VGG16), and the average AUC index reaches 0.970, which is 3.0% higher than that of the second (VGG16, InceptionV2 and ResNet101). It is clear that MARnet is significantly better than the classical five neural networks in ACC and average AUC.

Table 5 shows the results of comparing MARnet with 4 state-of-the-art convolutional neural networks: CheXNet ^[33], GraphXNet ^[34], TieNet ^[35] and AM_DenseNet ^[37]. It shows that MARnet works best in identifying four types of chest X-ray images. Since other datasets do not merge infiltration, effusion, pneumonia into infection as we do, the infection here takes the average value of infiltration, effusion and pneumonia.

To evaluate comprehensively the performance of MARnet we use 5-fold cross-validation to test the recognition effect of the model on chest X-ray images. Figure 6 shows the precision, recall and F1-score obtained by MARnet. The identification results of MARnet for normal and infection are relatively stable. The results of MARnet in identifying nodule and atelectasis fluctuate greatly, which may be due to the large interference of the data itself.

Figure 7 shows the ROC curve of MARnet at 5-fold cross-validation. In the identification of nodule and atelectasis, the results obtained by MARnet under different data partitions are slightly different. When it comes to identifying normal and infection, MARnet gets roughly the same results on different data partitions. Above all, how to part the data set has little effect on the classification performance of MARnet that show the model has good generalization ability. The ROC curve obtained by MARnet at 5-fold cross-validation shows strongly correlation with the precision, recall and F1-score. If accuracy, recall and F1 score fluctuate, the ROC curve is slightly different. While, when accuracy, recall and F1 score are relatively stable, ROC curve is relatively stable. Overall, ROC curves show that MARnet has relatively stable recognition performance.

Figure 8 shows the confusion matrix ^[49] obtained by using MARnet in 5-fold cross-validation. MARnet achieves the highest ACC value of 83.50%, the lowest ACC value of 80.99%, and the average ACC value of 82.46%. In general, MARnet has relatively stable recognition of chest X-ray images on all dataset.

In order to explain MARnet discriminant basis for chest X-ray images, we use Grad-CAM technology to display the discriminant results of MARnet in the form of heat map. Figure 9 shows the heat maps of MARnet for distinguishing four kinds of chest X-ray images: nodule, atelectasis, normal and infection. When identifying chest X-ray images with nodule, the disease area identified by the model contains a round-like high-density shadow at the bottom of the left lungs, which is the symptom of nodule. In the recognition of atelectasis, the model identifies the upper left lung area as location of atelectasis symptom. And when recognizing normal, MARnet does not find abnormalities in the bilateral lungs. In identifying infection, our model identifies the cloudy shadows on both sides of the lung, which is the lung infection caused by infiltration or effusion. Heat map makes the discriminant results of MARnet more directly and convenient to understand.

4.   Discussion and conclusions

In recent years, convolutional neural networks have been widely used in supporting medical staff to diagnose lung diseases. According to the characteristics of chest X-ray images, we construct MARnet to recognize 4 types of chest X-ray images including nodule, atelectasis, normal and infection, and achieve better recognition results than other models. In addition, we use the Grad-CAM technology to reveal the discriminant results of the MARnet, which provides a more intuitive reference to doctor.

However, MARnet is not effective in recognizing nodule. The confusion matrix in Figure 8 shows that MARnet misjudges over 40% of nodules as atelectasis, and misjudges approximately 15% of atelectasis as nodules. In practice, many patients diagnosed with nodule are accompanied by varying degrees of atelectasis. The factor makes nodule and atelectasis have similar characteristics on chest X-ray images and result in a large number of misjudgments.

In order to further study the reasons for MARnet's poor performance in identify nodule and atelectasis, we identify nodule, atelectasis, normal with ResNet-152. Figure 10 shows the confusion matrix obtained by using ResNet-152 to classify nodule and atelectasis, classify nodule and normal, and classify nodule and normal, respectively. The results reveal that there is a high misjudgment rate in identifying nodule and atelectasis. But in the classification of nodule and normal, all of nodule image are judged correctly, and there are only 4 normal are misjudged as nodule. In the classification of atelectasis and normal, there is only 1 atelectasis is misjudged as normal, and 2 normal are misjudged as atelectasis. Thus, the data itself is the cause that it is difficult to identify nodule and atelectasis.

Further studies find that in the chest X-ray 14 dataset, many patients suffer from multiple diseases such as nodule, atelectasis and even infection. To avoid interference, we extract chest X-ray images which has only nodule or atelectasis symptom from the original dataset. However, due to the specificity of the disease itself, the symptom is often complicated and associated. For example, the chest X-ray images marked as nodule sometimes have a certain degree of atelectasis symptoms, and the chest X-ray images marked as atelectasis sometimes have a certain degree of nodule symptoms. The factor causes great interference to MARnet for distinguishing nodule and atelectasis. In future work, we need to build more accurate dataset. A more reasonable classification scheme is needed to classify diseases with similar symptoms to avoid unnecessary interference to the model. What's more, in the dataset of chest X-ray 14 with more than 100,000 images, there are 65,000 healthy images, and the remaining 14 kinds of disease is extremely unbalanced, which seriously affects the application of deep learning in distinguish them. Finally, due to the lack of disease annotations from chest X-ray 14, we cannot determine whether the abnormal parts obtained by MARnet are accurate when using the heat map to display the discriminant results. It means that we cannot compare the discriminant results with the real situation. This limits our ability to reversely correct our model from the visual results. Therefore, we need to construct a certain number of datasets that accurately label the location of disease information to improve MARnet in future.

Acknowledgments

The work was supported by the Natural Science Foundation of Liaoning Province of China (No.20180551011).

Conflict of interest

The authors declare no conflict of interest.