COVID-19 disease identification network based on weakly supervised feature selection

1.   Introduction

Respiratory infections are a major cause of disease and death worldwide. Further, pneumonia affects millions of people every year, posing a significant risk to children, adults aged 65 years and older, and individuals with health problems such as diabetes, obesity, and hypertension. Pneumonia has more than 30 different causes, but its main etiological factors are viruses and bacteria. Coronavirus disease 2019 (COVID-19) spread rapidly, resulting in a large number of deaths among those infected, and affecting economies and health systems worldwide ^[1]. Although a larger number of people were vaccinated, 6,833,388 deaths were reported by February 10, 2023 ^[2]. Computed tomography (CT) images of the chest are usually used to determine the exact area and size of the lesion. At present, CT diagnosis relies heavily on physicians' superior skills. Radiologists can make mistakes even after long clinical training and professional guidance due to the complex pathology and subtle textural changes in the images of different lung lesions. Therefore, developing intelligent classification methods for CT images to support disease identification is extremely important. Usually, these classification methods are of three types: 1) attention-guided methods; 2) methods using convolutional neural networks; and 3) label correlation methods. Attention-guided methods usually direct attention to the primary lesion region, but lack attention to the suspicious region. Methods using convolutional neural networks extract different features mainly by changing the network structure, but these methods cannot reject the background information and the acquired features are redundant. Tag correlation methods can obtain the connection between multiple tags and their interdependencies, but the images often have some noisy information that affects the multi-label learning ability. Therefore, this study was significant in examining the images after removing the background, thus effectively avoiding the interference of background information.

In recent years, artificial intelligence and swarm intelligence optimization algorithms have been increasingly used to solve biological problems ^[3,4], perform disease diagnosis ^[5,6,7] and address combinatorial optimization problems ^[8]. The rapid development and application of deep learning have enabled its extensive use in medical image processing for lesion segmentation or detection ^[9,10,11,12], disease classification ^[13], noise induction ^[14], image annotation ^[15], alignment, regression, and so forth. The continuous improvement in computer hardware and software and the emergence of more functional and powerful neural network models have made deep learning models more capable of extracting image features for better image classification. In this study, we used deep learning to study the CT classification task of COVID-19.

Deep learning has developed rapidly in recent years in the field of artificial intelligence and played an important role in diagnosing COVID-19. Most researchers are using it for intelligent diagnosis ^[16,17,18]. Wang et al. ^[19] proposed a model based on ELU (exponential linear unit) called ELUCNN and developed a mobile application based on it. The backbone network of this model was a 10-layer CNN which could help diagnose COVID-19. The experimental results showed that the proposed ELUCNN performed better than 14 other advanced methods. Zhang et al. ^[20] proposed a deep learning model SNELM using SqueezeNet as the backbone network. This model was combined with extreme learning machine as the classifier. The results of 10 cross-validations on two datasets showed that the model outperformed seven state-of-the-art classification networks. Zhang et al. ^[21] proposed a new deep learning network called ANC. This model could visualize Gradient-weighted Class Activation Mapping (Grad-CAM) heat map for deep learning networks and expand the training set using 18 data augmentation methods. The validation results on two COVID-19 classification datasets showed that this model performed better than all nine currently popular classification networks. Lai et al. ^[22] used the segmentation network U-Net for disease diagnosis, which achieved 94.1% accuracy in diagnostic experiments on 1000 COVID-19 images due to the better performance of the U-Net network in segmentation and its less application in classification. Irmak et al. ^[23] proposed two new CNNs as backbone networks. One CNN could perform COVID-19 and non-COVID-19 binary classification, and the second CNN can perform triple classification. Experiments on COVID-19 chest x-ray images demonstrated its effectiveness. All the aforementioned studies used the whole image as the target of the study, and most of them used a single network to extract features without removing redundant data and extracting single features. While classifying lung diseases, the lesion region is usually located in the central region of the image, and the noise area of the edge background is extremely large, reducing the efficiency and accuracy of feature extraction. The CT images of patients with COVID-19 usually show ground glass–like changes. Also, the size of the lesion region varies depending on the disease, which requires removing the background and focusing on the lesion region. Hence, we performed weakly supervised preprocessing using the data. Also, we used two identical networks to extract complementary features and then combined them linearly to obtain better classification results considering the strong deep learning feature extraction capability. The main contributions of the method proposed in this study are as follows:

① A weakly supervised COVID-19 classification network (W-COVNet) was proposed, which was mainly divided into three modules: weakly supervised feature selection module (W-FS), deep learning bilinear feature fusion module (DBFF) and Grad-CAM++ based network visualization module (Grad-Ⅴ).

② W-FS mainly learned the lung area features by training Data2 to get the segmentation model and then performed feature selection on Data1 using the segmentation model. The new dataset Data1-Seg was obtained through multiple attempts of merit seeking. Five multiple-way data augmentation methods were used for data enhancement to expand the training set.

③ DBFF mainly used migration learning methods with the lightweight network VGG as the backbone network. The network used two identical groups of networks focusing on different regions to extract different features for linear fusion, improving the feature extraction capability. Also, four DBFF schemes were proposed.

④ Grag-V focused on the visual display of deep learning networks using Grad-CAM++. With this module, it was possible to show whether the network could accurately identify focal areas and interpret the classification results favorably.

⑤ The proposed network, W-COVNet, was compared with the seven most popular current classification networks using a fivefold cross-validation method on the same dataset, and the results showed that W-COVNet outperformed the other networks.

This manuscript is structured as follows. In Section 1, we describe diagnostic methods and deep learning–related research of COVID-19. In Section 2, recent studies on COVID-19 disease diagnosis are also discussed. In Section 3, weakly supervised dataset preprocessing methods are introduced, and new models for classification networks and Grad-CAM++ visualization methods are proposed. In Section 4, the specific steps, methods and results used for the experiments are explained and compared with other methods. In Section 5, the results are discussed and analyzed based on the results obtained from the model, and finally conclusions are presented.

2.   Related works

The research of the intelligent diagnosis of diseases can help doctors accurately diagnose various diseases. Xie et al. ^[24] found that the aberrations in certain genes affect adjacent genes, thus leading to the development of cancer. Further, the small size of cancer samples does not allow easy detection of cancer-causing genes. Therefore, a conceptual learning model was proposed to explore the characteristics of genes effectively via a dimensionality reduction process. Huang et al. ^[25] proposed an algorithm based on BI-RADS and CART using a decision tree algorithm to classify breast ultrasound data and identify malignant and benign tumors. This solved the problem of the inability of CAD systems to handle data from different sources and the problem of limited transparency to physicians. AI is not interpretable, and therefore AI-based drug recommendations are less credible, Xi et al. ^[7] used the proposed evaluation metric of traceability rate in an interpretable AI drug recommendation model, leading to an improvement in model performance. Dong et al. ^[26] proposed MorbidGCN, a GCN-based multi-disease prediction network, taking into account the differences between patients with single and multiple diseases and the relationship between multiple diseases. The network integrated population phenotype information and disease network information and performed better on two large datasets.

Usually, image classification and recognition are performed using global features. For example, Li and Liu ^[27] used a combination of wavelet packet Tsallis entropy, feedforward neural network and real number coding biogeography–based optimization (RCBBO) methods for the pathological examination of the brain and found optimal weights and biases. Fulton et al. ^[28] built a machine learning model using ResNet-50 to classify Alzheimer's disease. The prediction accuracy was 99% for the validation set triple classification and 99.34% for the training set.

However, different diseases and people have different sizes of diseased areas, and their locations are difficult to predict. Therefore, a large number of noisy areas exist when using global images for feature extraction. If the lesion area can be found accurately, its features can be extracted more easily, thus improving the classification accuracy. Hence, two aspects can be considered. First, methods such as the attention mechanism can be used to guide the network to focus on the lesion area; second, useless background areas can be removed so that the network can narrow its focus and improve its ability to obtain valid features. Guan et al. ^[29] proposed a three-branch attention-guided ResNet50 as the backbone of the convolution neural network (AG-CNN), which first acquired the mask of the focal area under the attention heat map guidance, cut it out as the branch feature extraction region and fused the global and local features. The final experimental result on the ChestX-ray14 dataset was an average Area under curve (AUC) value of 0.841. Ypsilantis et al. ^[30] mimicked the human visual attention mechanism using recurrent attention. Pesce et al. ^[31] proposed two network architectures, one consisting of a recurrent attention model and the other using weak markers and manually circling bounding boxes, which could accurately circle lung nodules. Togacar et al. ^[32] fused features from SqueezeNet and MobileNetV2 and used a new feature selection and combination method: social mimic optimization. Finally, the classification accuracy of COVID-19 reached 100% using a support vector machine for classification, and this model could be applied to mobile devices. Cohen et al. ^[33] proposed a COVID-19 virus severity network (CSSNet). First, seven large non-COVID-19 chest x-ray datasets were pretrained to obtain correlation features. The severity of infection was then evaluated on a small number of COVID-19 datasets. Ni et al. ^[34] used the proposed NiNet to segment and detect lesion regions on CT images of a few patients with COVID-19. Ko et al. ^[35] used a migration learning approach to find the optimal one from four commonly used deep learning networks as the backbone network. The proposed deep learning framework FCONet was used to tri-classify the enhanced images with an accuracy of 96.97%. Wang et al. ^[36] first obtained a lung segmentation model in an unsupervised manner and then proposed a new lightweight method based on the weakly supervised detection of COVID-19. Khan et al. ^[37] fused two different datasets, pretrained them using migration learning, and then performed the classification and detection of COVID-19 with 98.2% accuracy for triple classification and 95% accuracy for quadruple classification. They named the proposed network as Coro-Net. Hussain et al. ^[38] used a fivefold cross-validation for COVID-19 and normal CXR dichotomous classification with accuracy, sensitivity, and specificity results of 100%.

The aforementioned related studies for lung segmentation mainly focused on x-ray images, with fewer studies on the segmentation of CT images. Most of the new methods for the classification of COVID-19 extracted global features, but with the problem of feature redundancy. In this study, a weakly supervised method was used to obtain lung fields for COVID-19 CT images and perform feature selection. We improved the deep learning VGG network and proposed a new bilinear method to obtain effective features.

3.   Methodology

3.1.   Dataset and preprocessing

Two main datasets were used in this study, both of which were obtained from open-source websites. The first dataset (Data1) contained 397 non-COVID-19 CT images and 349 COVID-19 CT images. This dataset was validated by radiologists at Tongji Hospital in Wuhan, China ^[39]. The second dataset (Data2), COVID-19 CT segmentation ^[40], had 100 axial CT images from more than 40 patients. This dataset had COVID-19 lung CT masks that were labeled, it was mainly used for effective semi-supervised lung region segmentation. All CT images were collected by the Italian Society of Medical and Interventional Radiology.

In this study, we introduced the Inf-Net ^[41] segmentation model. The backbone of this model was Res2Net, which could be trained on a small number of labeled images via a reverse attention mechanism and showed edge detection attention. It could effectively perform lesion segmentation and showed the best performance compared with five currently popular segmentation networks. Six different metrics were used for performance evaluation in the COVID-19 lesion segmentation dataset, namely, $Dice$, Specificity ($Spec.$), Sensitivity ($Sen.$), Structure Measure (${S_\alpha }$), Enhanced-alignment Measure ($E_\varphi ^{mean}$) and Mean Absolute Error ($MAE$) (Table 1). As lung segmentation was required in this study, the first step involved obtaining the edge image from the ground truth (GT) map of the lung region of the Data2 dataset with lung region markers, and the second step used the GT map with the edge map to apply Inf-Net training to obtain the lung segmentation. The third step employed the obtained lung segmentation via the semi-supervised segmentation network Semi-Inf-Net ^[41] using the lung segmentation parameters to segment the lungs of Data1. In the fourth step, the weakly supervised feature selection module (W-FS) method was proposed because some bias existed in the training process, and some images had poor lung segmentation, as shown in Figure 1. Further, features within the lung region were removed. Data1 was trained three times for lung region segmentation, and the image with the best result was selected for data optimization. Finally, a new complete lung region image dataset Data1-Seg was obtained.

Table 1.  Performance comparison of Inf-Net with five popular segmentation networks ^[41].

Methods	$Dice$↑	$Sen.$↑	$Spec.$↑	${S_\alpha }$↑	$E_\varphi ^{mean}$↑	$MAE$↓
U-Net [36]	0.439	0.534	0.858	0.622	0.625	0.186
Attention-UNet [37]	0.583	0.637	0.921	0.744	0.739	0.112
Gated-UNet [38]	0.623	0.658	0.926	0.725	0.814	0.102
Dense-UNet [39]	0.515	0.594	0.840	0.655	0.662	0.184
U-Net++ [35]	0.581	0.672	0.902	0.722	0.720	0.120
Inf-Net [34]	0.682	0.692	0.943	0.781	0.838	0.082
Note:↑denotes that higher is better, and↓denotes that lower is better.

---

 | Show Table

DownLoad: CSV

Figure 1.  Flow chart of optimization-seeking weakly supervised segmentation preprocessing module.

DownLoad: Full-Size Img PowerPoint

3.2.   Data augmentation on the training set

Data1-Seg contained two types of CT images: COVID-19 and non-COVID. We resized the set D¹ of the Data1-Seg to a consistent size of 224 × 224 to obtain the new set D² of images, as shown in Eq (1).

The dataset was randomly divided into two subsets by the random hold-out method: the testing set (X: 20%) and the training set (Y: 80%). The relevant data partitioning information is shown in Table 2. The relevant sizes related to these subsets satisfied Eq (2). To address the issue of limited data, the MDA ^[42] technique was applied to expand the data and reduce the occurrence of overfitting. Five data enhancement methods were applied to the training set, turning it from the original one image to six images.

Table 2.  Distribution of Data1-Seg dataset in the model.

Dataset	COVID-19	non-COVID	Total
Training (80%)	280	318	598
Testing (20%)	69	79	148
Total (100%)	349	397	746

---

 | Show Table

DownLoad: CSV

where i is the number of X (testing set) images, j is the number of Y (training set) images, and |.| refers to the cardinality of a set.

It was assumed that K_MDA MDA techniques (K_MDA = {K₁, K₂, K₃, K₄, K₅}) were used. In this study, K_MDA MDA techniques included gamma correction, rotation, noise injection, brighter and darker. The N_tra training set images were generated using each MDA technique, and finally, K_MDA × N_tra images were generated for all MDAs. The following five MDA techniques were mainly used in this study:

① Gamma correction (Ga_Co)

The gamma correction factor r^G_C = 1.5 was used to produce new images as follows:

② Rotation (Rot)

The rotation angle θ^Rot = 90° was applied to the images:

③ Noise injection (No_In)

Gaussian noise was injected into the image to get a new image:

④ Brighter (Bri)

⑤ Darker (Dar)

where $\overrightarrow {{y^K}(i)}$ means that the data augmentation was the concatenation of five MDA results.

where $y(i)$ means that the training set consisted of the original and augmented images.

3.3.   Bilinear feature extraction

Transfer learning uses old knowledge to learn new knowledge, and the main goal is to transfer the learned knowledge to a new domain quickly. In this study, we used the idea of transfer learning to migrate the network after training in ImageNet, which could reduce the network parameters and converge quickly. Further, we used VGG for migration learning, a simple and robust network with four main structures: VGG11, VGG13, VGG16 and VGG19. We conducted experiments on each of them. Figure 2 shows the structure of VGG16.

Figure 2.  Diagram of the VGG16 structure.

DownLoad: Full-Size Img PowerPoint

The symmetric complementary bilinear classification network had two feature extractors, and the two sets of extracted features had local complementary characteristics for fine-grained images with scattered lesion areas such as COVID-19. Then, linear multiplication was performed to combine the features efficiently and obtain richer features. Figure 3 shows the symmetric complementary bilinear module using two VGG networks.

Figure 3.  Flow chart of the symmetric complementary bilinear module.

DownLoad: Full-Size Img PowerPoint

3.4.   Grad-CAM++ based network visualization

Deep learning is not interpretable and is usually understood as a black box. Grad-CAM (Gradient-weighted Class Activation Mapping) ^[43] is a heat map that can be loaded with network parameters and model structures to visualize deep learning networks. Usually, the network feature maps are processed by using global average pooling (Figure 4), and each feature map contained information of different classes.

Figure 4.  Global average pooling operation.

DownLoad: Full-Size Img PowerPoint

In Grad-CAM, the weights are calculated by averaging the gradients of the feature maps globally. The more accurate features are further back in the network. Grad-CAM usually shows the features of the last convolutional layer, as shown in Eq (10).

where $\omega _k^c$ denotes the weight of the kth feature map corresponding to category c, and by using the heat map, gradient is calculated. $A$ denotes the number of feature map pixels, ${g^c}$ denotes the gradient score of category c, and $B_{ij}^k$ denotes the pixel value at the (i, j) coordinate in the kth feature map.

The heat map is obtained by first obtaining the category weights corresponding to all the feature maps and then performing a weighted summation, as shown in Eq (11).

where ${B^k}$ denotes the kth feature map. Grad-CAM++ ^[44] is more effective than Grad-CAM, mainly because the weights are more complex and accurately obtained, and the positioning is precise.

3.5.   Proposed approach

In this study, we proposed a weakly supervised COVID-19 classification network (W-COVNet), which consisted of three main parts: Part 1, the weakly supervised feature selection module, which was mainly used for image lung segmentation; Part 2, the deep learning bilinear feature fusion module; and Part 3, the Grad-CAM++ based network visualization module. W-COVNet could perform feature selection and focus on effective regions, thus improving classification accuracy while visualizing focal areas. Figure 5 shows the overall framework. To elaborate further, pseudo code for the W-COVNet algorithm is given in Algorithm 1.

Figure 5.  Structure of the proposed W-COVNet.

DownLoad: Full-Size Img PowerPoint

Table Algorithm 1.  Pseudo code of our W-COVNet algorithm.

Phase Ⅰ: W-FS
Step 1	Input: Original image set Data1 and Data2.
Step 2	Train Data2 to get lung field segmentation set L.
Step 3	Use weakly supervised learning method to train the Data1 by L.
Step 4	Get the lung field of Data1 set L_F.
Step 5	Segment the lung field of the original image Data1 by L_F using crop.
Step 6	Obtain the Data1-Seg through the lung field with background removed set D1.
Step 7	Resizing: Resize the image D1 to [224, 224] and get dataset D2. See Eq. (1).
Step 8	YRHO: testing set (X: 20%) and training set (Y: 80%). See Eq. (2).
Step 9	MDA(Y): Ga_Co, No_In, Rot, Bri, and Dar to augment training set Y.
Phase Ⅱ: DBFF
Step 10	Read one raw pretrained model VGG16.
Step 11	Train two groups of VGG16 in parallel to obtain different features.
Step 12	Obtain two sets of features F1 and F2.
Step 13	Bmm (F1, F2).
Step 14	Generate DBFF.
Phase Ⅲ: Grag-Ⅴ
Step 15	Read a DBFF bilinear COVID-19 classification model set M.
Step 16	Read a COVID-19 image set I.
Step 17	Grad-CAM++ visualization of image I using model M.
Step 18	Obtain the heat map IGrad after visualization.
Step 19	Generate Grag-V.
Step 20	Output: The model W-COVNet and its performances.

---

 | Show Table

DownLoad: CSV

4.   Experiments and results

4.1.   Experimental platform

The study was executed in a Linux environment on a 32 GB Tesla V100 graphics card-equipped NVIDIA DGX Station deep learning workstation. The complete code, including data preprocessing and algorithm implementation, was implemented in Python. It made use of libraries such as NumPy and PyTorch, a deep learning toolset. The model's tuned hyper-parameters were the epochs, learning rate (LR), batch size (BS), and dropout rate (DR). The best experimental results were obtained when epochs = 50, LR = 0.003, BS = 8 and DR = 0.4.

4.2.   Classification performance

We used various metrics in the testing set, namely, accuracy ($Acc$), precision ($Pre$), sensitivity ($Sen$), specificity ($Spe$), recall ($Rec$) and F1-score ($F1 - sc$), to evaluate the performance of the proposed W-COVNet method. The corresponding equations were as follows.

For the four forms of VGG, we proposed four network schemes with VGG11, VGG13, VGG16 and VGG19 as the backbone networks, as shown in Table 3. We created a histogram display of the results, which are shown in Figure 6. Accuracy refers to the ratio of the number of correctly predicted samples to the total number of predicted samples. Precision is the proportion of positive samples that are actually predicted to be positive. The larger the value of sensitivity is, the greater the "sickness is judged as sickness" and the smaller the "missed detection" (FN). The larger the value of specificity is, the larger the "healthy are judged as healthy" and the smaller the "false detection" (FP). The f1-score (equilibrium average) is the result of the calculation that takes into account the accuracy and completeness of the model, and the value is more inclined to the smaller value of the index. The f1-score indicates how many of the data with the correct real value can be predicted correctly. Table 3 shows that VGG16 results were balanced and performed best in overall prediction ability, with the ability to predict positive samples correctly, and VGG16 also had the smallest false detection rate. The best results were obtained when VGG16 was used as the backbone network. In conclusion, the performance in terms of accuracy, sensitivity, recall and F1-score on the testing set was good. Therefore, the W-COVNet proposed in this study was effective.

Table 3.  Classification of W-COVNet networks after using two kinds of testing set (%).

Network	Class	Acc	Sen	Pre	Rec	Spe	F1-sc
[VGG11]2	COVID-19	84.5	82.6	83.8	82.6	86.1	83.2
	non-COVID	84.5	86.1	85.0	86.1	82.6	85.5
[VGG13]2	COVID-19	82.4	82.6	80.3	82.6	82.3	81.4
	non-COVID	82.4	82.3	84.4	82.3	82.6	83.3
[VGG16]2	COVID-19	86.5	87.0	84.5	87.0	86.1	85.7
	non-COVID	86.5	86.1	88.3	86.1	87.0	87.2
[VGG19]2	COVID-19	83.1	84.1	80.6	84.1	82.3	82.3
	non-COVID	83.1	82.3	85.5	82.3	84.1	83.9

---

 | Show Table

DownLoad: CSV

Figure 6.  Comparison of the experimental results of the four schemes.

DownLoad: Full-Size Img PowerPoint

Figure 6 depicts the W-COVNet training process. Scheme 3 with the best results was selected to plot the training loss and the testing accuracy under each epoch. The horizontal axis is the epoch, the left vertical axis is the test accuracy, and the right vertical axis is the training loss. Figure 7 shows that the training loss became smaller with the iteration of epoch until it reached 0.1 and no longer changed, while the test accuracy became larger with the increase in epoch until it reached 0.86 and no longer changed, indicating that the network converged around the tenth epoch and performed well with high test accuracy.

Figure 7.  Test accuracy and training loss of W-COVNet.

DownLoad: Full-Size Img PowerPoint

4.3.   Confusion matrix

We used the confusion matrix ^[45] to display the test results of the experimental data. For each class C = 1, 2 (1: COVID-19, 2: non-COVID), we set the COVID-19 class to "positive" and the other class to "negative". The true negatives, true positives, false negatives and false positives were used to identify the diagnosis of CT images with the model. Figure 8 shows the confusion matrix of the W-COVNet with VGG11, VGG13, VGG16 and VGG19 as the backbone proposed in this study.

Figure 8.  Confusion matrix of W-COVNet with VGG11 (a), VGG13 (b), VGG16 (c) and VGG19 (d) as the backbone.

DownLoad: Full-Size Img PowerPoint

4.4.   5-Folds cross-validation

We divided the data into a training set and a testing set due to the small amount of data trained and to prevent overfitting problems during the study. The data were divided into five equal parts; one part of each experiment was used for testing, and the remaining parts were used for training. The experiment was repeated five times to find the average. The first experiment used 20% of the data in the red area as the test set and the rest as the training set. The second experiment used 20% of the second red area as the test set and the rest as the training set, and so on, so that the experimental results were more accurate. A 5-Folds multiple partitioning format allowed the use of the full dataset. Finally, the averaging method could accurately represent the model performance and verify the generalization performance of the network.

We experimentally determined that option 3 was the best result. Therefore, W-COVNet selected VGG16 as the backbone and performed a 5-folds cross-validation using the new dataset Data1-Seg. The results are shown in Table 4.

Table 4.  Classification of W-COVNet networks after using two kinds of the testing set (%).

Run	Class	Acc	Sen	Pre	Rec	Spe	F1-sc
R1	COVID-19	85.8	85.5	84.3	85.5	86.1	84.9
	non-COVID	85.8	86.1	87.2	86.1	85.5	86.6
R2	COVID-19	86.5	87.0	84.5	87.0	86.1	85.7
	non-COVID	86.5	86.1	88.3	86.1	87.0	87.2
R3	COVID-19	83.3	80.0	83.6	80.0	86.3	81.8
	non-COVID	83.3	86.3	83.1	86.3	80.0	84.7
R4	COVID-19	84.6	87.1	81.3	87.1	82.3	84.1
	non-COVID	84.6	82.3	87.8	82.3	87.1	85.0
R5	COVID-19	86.5	87.0	84.5	87.0	86.1	85.7
	non-COVID	86.5	86.1	88.3	86.1	87.0	87.2
Mean ± SD	COVID-19	85.3 ± 1.37	85.3 ± 3.04	83.6 ± 1.35	85.3 ± 3.04	85.4 ± 1.72	84.4 ± 1.64
	non-COVID	85.3 ± 1.37	85.4 ± 1.72	87.0 ± 2.19	85.4 ± 1.72	85.3 ± 3.04	86.1 ± 1.22

---

 | Show Table

DownLoad: CSV

4.5.   Ablation study

We conducted ablation experiments to verify the effectiveness of the proposed W-COVNet method. We used VGG16 as the backbone network to conduct ablation experiments. Experiment No.1 used the backbone network VGG16 to classify the Data1-Seg data after feature selection. Experiment No.2 is the result after training the Data1-Seg using feature fusion. Experiment No.3 was the result after preprocessing the images and the training set had enhanced data. Experiment No.4 showed the result after performing feature selection, data enhancement and feature fusion. As shown in Table 5, experiment No.1, No.2, No.3 and No.4 validated the effectiveness of several improvements. Among these, Experiment No.2 and No.3 validate the effectiveness of DBFF data fusion and MDA data enhancement, respectively. Finally No.4 showed the performance of our proposed method W-COVNet. As shown in Table 5, the performance of W-COVNet was the best, and the method was effective.

Table 5.  Ablation studies of our W-COVNet (%).

Methods	Acc	Sen	Pre	Rec	Spe	F1-sc
(No.1) Backbone	74.47	65.22	78.95	65.22	83.33	71.43
(No.2) Backbone + DBFF	79.43	71.01	84.48	71.01	87.50	77.17
(No.3) Backbone + MDA	81.76	86.96	76.92	86.96	77.22	81.63
(No.4) Backbone + MDA + DBFF	86.49	86.96	84.51	86.96	86.08	85.71

---

 | Show Table

DownLoad: CSV

4.6.   Grad-CAM++ based network visualization results

The objective of this study was to use the image set labeled data without pixel labeling of the lesion area. However, the deep learning network could find the lesion by learning, allowing for accurate disease diagnosis. Therefore, the Grad-CAM++ technique was used for visualization operation. Three images were selected for each category, namely COVID-19 and non-COVID. Each image was divided into three groups: the first group was the original CT image; the second group was the image of the lung area after removing the background and keeping only the feature area; and the third group was the heat map visualized using Grad-CAM++. As shown in Figure 9, the red area is the feature area, and the blue area is the non-feature area. If the features are denser, the corresponding feature values increase, leading to a brighter color. The displayed results showed that using Grad-CAM++ could not only provide an effective interpretation of the deep learning network but also help doctors find the lesion areas.

Figure 9.  Use Grad-CAM++ + to visualize the feature area.

DownLoad: Full-Size Img PowerPoint

4.7.   Comparison experiments

We compared the W-COVNet method with seven types of better-performance classification methods: NCA-ResNet ^[46], Fused-DenseNet-Tiny ^[47], BCNN_SVM ^[48], ECOVNet-EfficientNetB3 base ^[49], COVNet ^[35], Xception ^[50] and DTL-V19 ^[51] to demonstrate its effectiveness. All experiments used the dataset Data1-Seg and the same data preprocessing and performed a fivefold cross-validation. The experimental results are shown in Table 6. Figure 10 and Figure 11 show the test accuracy and training loss trend of our method with these seven comparison experiments, and it can be seen that all converge within 20 epochs, and our proposed W-COVNet has the highest accuracy after convergence. It was observed that our proposed W-COVNet method performed the best, with an accuracy of 85.3%. The extracted features were more accurate using the images with the background removed. The use of two symmetric networks at the same time allowed focus on different features and improved the feature richness. The effectiveness of the network was verified by the experiments. The use of MDA data augmentation prevented the overfitting of the model, thus improving its performance. In addition, the BCNN_SVM backbone network was VGG16 and VGG19, and the DTL-V19 backbone network comprised VGG19. In this study, we also used the VGG network as the backbone network. Comparing the experimental results, we could see that VGG performed better than other networks on this dataset.

Table 6.  Performance comparison of the proposed W-COVNet with other studies (%).

Method	Class	Pre	Spec	F1-sc	Acc
NCA-ResNet	COVID-19	65.9 ± 4.35	72.7 ± 5.31	62.3 ± 2.53	66.4 ± 2.85
	non-COVID	67.0 ± 1.96	59.2 ± 2.89	69.7 ± 3.23
Fused-DenseNet-Tiny	COVID-19	64.1 ± 3.41	70.5 ± 5.18	61.9 ± 6.86	65.7 ± 3.95
	non-COVID	67.3 ± 4.61	60.4 ± 4.18	68.6 ± 3.11
BCNN_SVM	COVID-19	80.9 ± 2.99	83.1 ± 4.98	80.2 ± 4.84	81.7 ± 3.26
	non-COVID	83.3 ± 6.13	80.2 ± 9.86	82.9 ± 2.49
ECOVNet-EfficientNetB3 base	COVID-19	71.1 ± 2.70	74.0 ± 3.64	71.8 ± 3.64	73.4 ± 2.63
	non-COVID	75.8 ± 4.25	72.7 ± 6.70	74.9 ± 1.98
COVNet	COVID-19	62.7 ± 4.39	76.8 ± 10.10	49.1 ± 10.47	60.7 ± 2.24
	non-COVID	60.8 ± 3.66	42.6 ± 14.95	67.4 ± 2.78
Xception	COVID-19	64.1 ± 3.60	65.3 ± 5.34	66.9 ± 4.02	67.6 ± 3.70
	non-COVID	71.4 ± 4.03	70.1 ± 5.80	68.1 ± 4.03
DTL-V19	COVID-19	79.5 ± 4.63	82.6 ± 5.97	76.9 ± 3.96	79.1 ± 2.94
	non-COVID	79.4 ± 4.23	75.1 ± 7.74	80.8 ± 2.80
W-COVNet (this work)	COVID-19	83.6 ± 1.34	85.4 ± 1.72	84.4 ± 1.64	85.3 ± 1.37
	non-COVID	87.0 ± 2.19	85.3 ± 3.04	86.1 ± 1.21

---

 | Show Table

DownLoad: CSV

Figure 10.  Training loss of our method with other studies.

DownLoad: Full-Size Img PowerPoint

Figure 11.  Test accuracy of our method with other studie.

DownLoad: Full-Size Img PowerPoint

5.   Conclusions

Lung diseases are a major cause of death, and the emergence of COVID-19 in the last 3 years has taken a huge toll on humanity. The sudden outbreak has led to the failure of the medical system because of the high contagiousness of the disease. The doctors and equipment are not sufficient for rapid diagnosis and treatment, and hence artificial intelligence–assisted diagnosis is especially important. In this study, we proposed a weakly supervised COVID-19 classification network (W-COVNet). Given a large amount of redundant information in CT data, W-COVNet could perform feature selection to eliminate unnecessary information and retain only the core feature regions, avoiding interference in the deep learning training process. Meanwhile, bilinear fusion provided a stronger feature representation than a single linear model. Because of the poor visualization of deep learning, W-COVNet also used the currently popular Grad-CAM++ technique to visualize the deep learning process. We conducted extensive experiments to verify the performance of the network and achieved an average classification accuracy of 85.3% on the dataset with background removed, which was better than that of other COVID-19 classification networks and classical classification networks. W-COVNet could extract effective features to help physicians quickly diagnose and discover areas of lesions.

This study had certain limitations. Fewer data led to insufficient training. Moreover, the proposed W-COVNet had two disadvantages: 1) Since the segment was trained on the lung region of CT images, the network could not perform weakly supervised segmentation of CXR (chest X-ray) images. 2) Also, it could not judge the severity of the disease.

Acknowledgments

The data that support the findings of this study are openly at https://github.com/UCSD-AI4H/COVID-CT ^[39]. http://medicalsegmentation.com/covid19/ ^[40].

This study was supported by the Jilin Provincial Key Laboratory of Intelligent Computing in Medical Image (No. YDZJ202102CXJD026), the Innovation Team Projects of Universities in Guangdong (No. 2022KCXTD057), the Higher Education Natural Science Foundation of Anhui Province (No. 2022AH051099), the Key Project of Science Research in Universities of Anhui Province of China (KJ2021A1066) and the open Foundation of Anhui Engineering Research Center of Intelligent Perception and Elderly Care, Chuzhou University (No. 2022OPA03).

Conflict of interest

The authors report no declarations of interest.