Search Advanced

Mathematical Biosciences and Engineering

2021, Volume 18, Issue 6: 9264-9293. doi: 10.3934/mbe.2021456
Previous Article Next Article
Research article Special Issues

Factors determining generalization in deep learning models for scoring COVID-CT images

  • 1.
    Center for Advanced Modelling and Geospatial Information Systems (CAMGIS), Faculty of Engineering and Information Technology, University of Technology Sydney, Australia
  • 2.
    Center of Excellence for Climate Change Research, King Abdulaziz University, Jeddah 21589, Saudi Arabia
  • 3.
    Earth Observation Center, Institute of Climate Change, Universiti Kebangsaan Malaysia, Selangor 43600, Malaysia
  • 4.
    Fellowship of Interventional Radiology Imaging Center, IranMehr General Hospital, Iran
  • 5.
    Machine Vision and Digital Health (MaViDH), School of Computing, Mathematics, and Engineering, Charles Sturt University, Australia
  • Received: 04 August 2021 Accepted: 12 October 2021 Published: 27 October 2021

  • The COVID-19 pandemic has inspired unprecedented data collection and computer vision modelling efforts worldwide, focused on the diagnosis of COVID-19 from medical images. However, these models have found limited, if any, clinical application due in part to unproven generalization to data sets beyond their source training corpus. This study investigates the generalizability of deep learning models using publicly available COVID-19 Computed Tomography data through cross dataset validation. The predictive ability of these models for COVID-19 severity is assessed using an independent dataset that is stratified for COVID-19 lung involvement. Each inter-dataset study is performed using histogram equalization, and contrast limited adaptive histogram equalization with and without a learning Gabor filter. We show that under certain conditions, deep learning models can generalize well to an external dataset with F1 scores up to 86%. The best performing model shows predictive accuracy of between 75% and 96% for lung involvement scoring against an external expertly stratified dataset. From these results we identify key factors promoting deep learning generalization, being primarily the uniform acquisition of training images, and secondly diversity in CT slice position.

    Citation: Michael James Horry, Subrata Chakraborty, Biswajeet Pradhan, Maryam Fallahpoor, Hossein Chegeni, Manoranjan Paul. Factors determining generalization in deep learning models for scoring COVID-CT images[J]. Mathematical Biosciences and Engineering, 2021, 18(6): 9264-9293. doi: 10.3934/mbe.2021456

    Related Papers:

  • The COVID-19 pandemic has inspired unprecedented data collection and computer vision modelling efforts worldwide, focused on the diagnosis of COVID-19 from medical images. However, these models have found limited, if any, clinical application due in part to unproven generalization to data sets beyond their source training corpus. This study investigates the generalizability of deep learning models using publicly available COVID-19 Computed Tomography data through cross dataset validation. The predictive ability of these models for COVID-19 severity is assessed using an independent dataset that is stratified for COVID-19 lung involvement. Each inter-dataset study is performed using histogram equalization, and contrast limited adaptive histogram equalization with and without a learning Gabor filter. We show that under certain conditions, deep learning models can generalize well to an external dataset with F1 scores up to 86%. The best performing model shows predictive accuracy of between 75% and 96% for lung involvement scoring against an external expertly stratified dataset. From these results we identify key factors promoting deep learning generalization, being primarily the uniform acquisition of training images, and secondly diversity in CT slice position.


    加载中


    [1] STATISTA, Coronavirus Deaths Worldwide by Country, 2021. Available from: https://www.statista.com/statistics/1093256/novel-coronavirus-2019ncov-deaths-worldwide-by-country/.
    [2] U. S. CDC., About Variants of the Virus that Causes COVID-19, 2021. Available from: https://www.cdc.gov/coronavirus/2019-ncov/transmission/variant.html.
    [3] Global Preparedness Monitoring Board, A World in Disorder, 2021. Available from https://www.gpmb.org/annual-reports/overview/item/2020-a-world-in-disorder.
    [4] A. Ulhaq, J. Born, A. Khan, D. P. S. Gomes, S. Chakraborty, M. Paul, COVID-19 control by computer vision approaches: A survey, IEEE Access, 8 (2020), 179437-179456. doi: 10.1109/ACCESS.2020.3027685
    [5] C. Butt, J. Gill, D. Chun, B. A. Babu, Deep learning system to screen coronavirus disease 2019 pneumonia, Appl. Intell., 1 (2020), 1-7. doi: 10.48185/jaai.v1i1.30
    [6] J. Chen, L. Wu, J. Zhang, L. Zhang, D. Gong, Y. Zhao, et al., Deep learning-based model for detecting 2019 novel coronavirus pneumonia on high-resolution computed tomography, Sci. Rep., 10 (2020), 19196. doi: 10.1038/s41598-020-76282-0
    [7] H. Gunraj, L. Wang, A. Wong, COVIDNet-CT: A tailored deep convolutional neural network design for detection of COVID-19 cases from chest CT images, Front. Med., 7 (2020), 1025.
    [8] R. Kumar, S. Zhang, W. Wang, W. Amin, J. Kumar, Blockchain-federated-learning and deep learning models for COVID-19 detection using CT imaging, preprint, arXiv: 2007.06537.
    [9] Z. Li, Z. Zhong, Y. Li, T. Zhang, L. Gao, D. Jin, et al., From community-acquired pneumonia to COVID-19: A deep learning-based method for quantitative analysis of COVID-19 on thick-section CT scans, Eur. Radiol., 30 (2020), 6828-6837. doi: 10.1007/s00330-020-07042-x
    [10] Q. Ni, Z. Y. Sun, L. Qi, W. Chen, Y. Yang, L. Wang, et al., A deep learning approach to characterize 2019 coronavirus disease (COVID-19) pneumonia in chest CT images, Eur. Radiol., 30 (2020), 6517-6527. doi: 10.1007/s00330-020-07044-9
    [11] T. D. Pham, A comprehensive study on classification of COVID-19 on computed tomography with pretrained convolutional neural networks, Sci. Rep., 10 (2020), 16942-16942. doi: 10.1038/s41598-020-67245-6
    [12] M. Polsinelli, L. Cinque, G. Placidi, A light CNN for detecting COVID-19 from CT scans of the chest, Pattern Recognit. Lett., 140 (2020), 95-100. doi: 10.1016/j.patrec.2020.10.001
    [13] P. Silva, E. Luz, G. Silva, G. Moreira, R. Silva, D. Lucio, et al., COVID-19 detection in CT images with deep learning: A voting-based scheme and cross-datasets analysis, Inform. Med. Unlocked., 20 (2020), 100427-100427. doi: 10.1016/j.imu.2020.100427
    [14] S. Wang, B. Kang, J. Ma, X. Zeng, M. Xiao, J. Guo, et al., A deep learning algorithm using CT images to screen for Corona virus disease (COVID-19), Eur. Radiol., 31 (2021), 6096-6104. doi: 10.1007/s00330-021-07715-1
    [15] X. Wang, X. Deng, Q. Fu, Q. Zhou, J. Feng, H. Ma, et al., A Weakly-Supervised Framework for COVID-19 Classification and Lesion Localization From Chest CT, IEEE Trans. Med. Imaging, 39 (2020), 2615-2625. doi: 10.1109/TMI.2020.2995965
    [16] T. Akram, M. Attique, S. Gul, A. Shahzad, M. Altaf, S. S. R. Naqvi, et al., A novel framework for rapid diagnosis of COVID-19 on computed tomography scans, Pattern Anal. Appl., 24 (2021), 951-964. doi: 10.1007/s10044-020-00950-0
    [17] M. Mohammadpoor, M. S. Karizaki, M. S. Karizaki, A deep learning algorithm to detect coronavirus (COVID-19) disease using CT images, PeerJ. Comp. Sci., 7 (2021), e345. doi: 10.7717/peerj-cs.345
    [18] J. Zhang, Y. Xie, Y. Li, C. Shen, Y. Xia, COVID-19 screening on chest X-ray images using deep learning based anomaly detection, preprint, arXiv: 2003.12338.
    [19] F. Ucar, D. Korkmaz, COVIDiagnosis-Net: Deep Bayes-SqueezeNet based diagnosis of the coronavirus disease 2019 (COVID-19) from X-ray images, Med. Hypotheses, 140 (2020), 109761. doi: 10.1016/j.mehy.2020.109761
    [20] Y. Oh, S. Park, J. C. Ye, Deep learning COVID-19 features on CXR using limited training data sets, IEEE Trans. Med. Imaging, 39 (2020), 2688-2700. doi: 10.1109/TMI.2020.2993291
    [21] S. H. Yoo, H. Geng, T. L. Chiu, S. K. Yu, D. C. Cho, J. Heo, et al., Deep learning-based decision-tree classifier for COVID-19 diagnosis from chest X-ray imaging, Front. Med., 7 (2020), 427. doi: 10.3389/fmed.2020.00427
    [22] J. Civit-Masot, F. Luna-Perejón, A. Civit, Deep learning system for COVID-19 diagnosis aid using X-ray pulmonary images, Appl. Sci., 10 (2020), 4640. doi: 10.3390/app10134640
    [23] M. Blain, M. T Kassin, N. Varble, X. Wang, Z. Xu, D. Xu, et al., Determination of disease severity in COVID-19 patients using deep learning in chest X-ray images, Diagn. Interv. Radiol., 27 (2020), 20-27.
    [24] J. P. Cohen, L. Dao, K. Roth, P. Morrison, Y. Bengio, A. F. Abbasi, et al., Predicting COVID-19 pneumonia severity on chest X-ray with deep learning, Cureus, 12 (2020), e9448.
    [25] B. Liu, Y. Zhou, Y. Yang, Y. Zhang, Experiments of federated learning for COVID-19 chest X-ray images, preprint, arXiv: 2007.0559.
    [26] M. E. Karar, E. E. D. Hemdan, M. A. Shouman, Cascaded deep learning classifiers for computer-aided diagnosis of COVID-19 and pneumonia diseases in X-ray scans, Complex Intell. Syst., 7 (2021), 235-247. doi: 10.1007/s40747-020-00199-4
    [27] H. Amin, A. Darwish, A. E. Hassanien, Classification of COVID19 X-ray images based on transfer learning InceptionV3 deep learning model, in Digital Transformation and Emerging Technologies for Fighting COVID-19 Pandemic: Innovative Approaches, Springer International Publishing, (2021), 111-119.
    [28] K. Shankar, E. Perumal, A novel hand-crafted with deep learning features based fusion model for COVID-19 diagnosis and classification using chest X-ray images, Complex Intell. Syst., 7 (2020), 1277-1293.
    [29] O. M. Elzeki, M. Shams, S. Sarhan, M. Abd Elfattah, A. E. Hassanien, COVID-19: A new deep learning computer-aided model for classification, PeerJ. Comp. Sci., 7 (2021), e358. doi: 10.7717/peerj-cs.358
    [30] H. S. Alghamdi, G. Amoudi, S. Elhag, K. Saeedi, J. Nasser, Deep learning approaches for detecting COVID-19 from chest X-ray images: A survey, IEEE Access, 9 (2021), 20235-20254. doi: 10.1109/ACCESS.2021.3054484
    [31] J. Born, N. Wiedemann, M. Cossio, C. Buhre, G. Brändle, K. Leidermann, et al., Accelerating detection of lung pathologies with explainable ultrasound image analysis, Appl. Sci., 11 (2021), 672. doi: 10.3390/app11020672
    [32] S. Roy, W. Menapace, S. Oei, B. Luijten, E. Fini, C. Saltori, et al., Deep learning for classification and localization of COVID-19 markers in point-of-care lung ultrasound, IEEE Trans. Med. Imaging, 39 (2020), 2676-2687. doi: 10.1109/TMI.2020.2994459
    [33] T. Ai, Z. Yang, H. Hou, C. Zhan, C. Chen, W. Lv, et al., Correlation of chest CT and RT-PCR testing for coronavirus disease 2019 (COVID-19) in China: A report of 1014 cases, Radiology, 296 (2020), E32-E40. doi: 10.1148/radiol.2020200642
    [34] A. Bernheim, X. Mei, M. Huang, Y. Yang, Z. A. Fayad, N. Zhang, et al., Chest CT findings in coronavirus disease-19 (COVID-19): Relationship to duration of infection, Radiology, 295 (2020), 200463. doi: 10.1148/radiol.2020200463
    [35] M. Roberts, D. Driggs, M. Thorpe, J. Gilbey, M. Yeung, S. Ursprung, et al., Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans, Nat. Mach. Intell., 3 (2021), 199-217. doi: 10.1038/s42256-021-00307-0
    [36] R. F. Wolff, K. G. M. Moons, R. D. Riley, P. F. Whiting, M. Westwood, G. S. Collins, et al., Probast: A tool to assess the risk of bias and applicability of prediction model studies, Ann. Intern. Med., 170 (2019), 51-58. doi: 10.7326/M18-1376
    [37] Y. Ji, Z. Ma, M. P. Peppelenbosch, Q. Pan, Potential association between COVID-19 mortality and health-care resource availability, Lancet Glob. Health, 8 (2020), e480. doi: 10.1016/S2214-109X(20)30068-1
    [38] E. Tartaglione, C. A. Barbano, C. Berzovini, M. Calandri, M. Grangetto, Unveiling COVID-19 from chest X-ray with deep learning: A hurdles race with small data, Int. J. Environ. Res. Public Health, 17 (2020), 1-17.
    [39] OpenCV, OpenCV: Histograms-2: Histogram equalization, 2021. Available from: https://docs.opencv.org/master/d5/daf/tutorial_py_histogram_equalization.html.
    [40] K. Zuiderveld, Contrast limited adaptive histogram equalization, in Graphics gems IV: Academic Press Professional, Academic Press, (1994), 474-485.
    [41] Z. Al-Ameen, G. Sulong, A. Rehman, A. Al-Dhelaan, T. Saba, M. Al-Rodhaan, An innovative technique for contrast enhancement of computed tomography images using normalized gamma-corrected contrast-limited adaptive histogram equalization, Eurasip J. Adv. Sig. Pr., 2015 (2015), 32. doi: 10.1186/s13634-015-0214-1
    [42] A. Alekseev, A. Bobe, Gabornet: Gabor filters with learnable parameters in deep convolutional neural network, preprint, arXiv: 1904.13204.
    [43] S. P. Morozov, A. E. Andreychenko, I. A. Blokhin, P. B. Gelezhe, A. P. Gonchar, A. E. Nikolaev, et al., MosMedData: data set of 1110 chest CT scans performed during the COVID-19 epidemic, Dig. Diagnostics, 1 (2020), 49-59. doi: 10.17816/DD46826
    [44] V. Guarrasi, N. C. D'Amico, R. Sicilia, E. Cordelli, P. Soda, Pareto optimization of deep networks for COVID-19 diagnosis from chest X-rays, Pattern Recognit., 121 (2022), 108242. doi: 10.1016/j.patcog.2021.108242
    [45] J. R. Zech, M. A. Badgeley, M. Liu, A. B. Costa, J. J. Titano, E. K. Oermann, Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: A cross-sectional study, PLoS Med., 15 (2018), e1002683. doi: 10.1371/journal.pmed.1002683
    [46] P. Mooney, Chest X-ray images (pneumonia). Available from: https://www.kaggle.com/paultimothymooney/chest-xray-pneumonia.
    [47] G. Maguolo, L. Nanni, A critic evaluation of methods for COVID-19 automatic detection from X-ray images, Inf. Fusion, 76 (2021), 1-7. doi: 10.1016/j.inffus.2021.04.008
    [48] J. Cohen, P. Morrison, L. Dao, COVID-19 image data collection, preprint, arXiv: 2003.11597.
    [49] A. J. DeGrave, J. D. Janizek, S. I. Lee, AI for radiographic COVID-19 detection selects shortcuts over signal, Nat. Mach. Intell., 3 (2021), 610-619. doi: 10.1038/s42256-021-00338-7
    [50] X. Wang, Y. Peng, L. Lu, Z. Lu, M. Bagheri, R. M. Summers, ChestX-Ray8: Hospital-scale chest X-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases, in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), (2017), 3462-3471.
    [51] J. Saborit, J. Montell, A. Pertusa, A. Bustos, M. Cazorla, J. Galant, et al., BIMCV COVID-19+: A large annotated dataset of RX and CT images from COVID-19 patients, preprint, arXiv: 2006: 01174.
    [52] A. Bustos, A. Pertusa, J. M. Salinas, M. de la Iglesia-Vayá, Padchest: A large chest X-ray image dataset with multi-label annotated reports, Med. Imag. Anal., 66 (2020), 101797. doi: 10.1016/j.media.2020.101797
    [53] K. B. Ahmed, G. M. Goldgof, R. Paul, D. B. Goldgof, L. O. Hall, Discovery of a generalization gap of convolutional neural networks on COVID-19 X-rays classification, IEEE Access, 9 (2021), 72970-72979. doi: 10.1109/ACCESS.2021.3079716
    [54] P. R. Bassi, R. Attux, COVID-19 detection using chest X-rays: Is lung segmentation important for generalization?, preprint, arXiv: 2104.06176.
    [55] M. Elgendi, M. U. Nasir, Q. Tang, D. Smith, J.-P. Grenier, C. Batte, et al., The effectiveness of image augmentation in deep learning networks for detecting COVID-19: A geometric transformation perspective, Frontiers Med., 8 (2021).
    [56] J. Shuja, E. Alanazi, W. Alasmary, A. Alashaikh, COVID-19 open source data sets: A comprehensive survey, Appl. Intell., 51 (2020), 1296-1325.
    [57] M. Jun, G. Cheng, W. Yixin, A. Xingle, G. Jiantao, Y. Ziqi, et al., COVID-19 CT lung and infection segmentation dataset (verson 1.0), Zenodo, 2020. Available from https://doi.org/10.5281/zenodo.3757476.
    [58] F. Shan, Y. Gao, J. Wang, W. Shi, N. Shi, M. Han, et al., Lung infection quantification of COVID-19 in CT images with deep learning, preprint, arXiv: 2003.04655.
    [59] J. P. Cohen, P. Morrison, L. Dao, K. Roth, T. Q. Duong, M. Ghassemi, COVID-19 image data collection: Prospective predictions are the future, preprint, arXiv: 2006.11988.
    [60] J. Zhao, Y. Zhang, X. He, P. Xie, COVID-CT-dataset: A CT scan dataset about COVID-19, preprint, arXiv: 2003.13865.
    [61] MedRxiv, the Preprint Server for Health Sciences, Available from https://www.medrxiv.org.
    [62] BioRxiv, the Preprint Server for Biology, Available from https://www.biorxiv.org.
    [63] E. Soares, P. Angelov, S. Biaso, M. H. Froes, D. K. Abe, SARS-COV-2 CT-scan dataset: A large dataset of real patients CT scans for SARS-COV-2 identification, preprint, medRxiv: 2020.04.24.20078584.
    [64] K. Zhang, X. Liu, J. Shen, Z. Li, Y. Sang, X. Wu, et al., Clinically applicable AI system for accurate diagnosis, quantitative measurements, and prognosis of COVID-19 pneumonia using computed tomography, Cell, 181 (2020), 1423-1433. doi: 10.1016/j.cell.2020.04.045
    [65] S. A. Duzgun, G. Durhan, F. B. Demirkazik, M. G. Akpinar, O. M. Ariyurek, COVID-19 pneumonia: The great radiological mimicker, Insights Imaging, 11 (2020), 118-118. doi: 10.1186/s13244-020-00933-z
    [66] A. Krizhevsky, I. Sutskever, G. Hinton, Imagenet classification with deep convolutional neural networks, Commun. ACM, 60 (2017), 84-90. doi: 10.1145/3065386
    [67] S. K. Wajid, A. Hussain, K. Huang, W. Boulila, Lung cancer detection using local energy-based shape histogram (LESH) feature extraction and cognitive machine learning techniques, in 2016 IEEE 15th International Conference on Cognitive Informatics & Cognitive Computing (ICCI*CC), (2016), 359-366.
    [68] R. Sarkar, A. Hazra, K. Sadhu, P. Ghosh, A novel method for pneumonia diagnosis from chest X-ray images using deep residual learning with separable convolutional networks, in Computer Vision and Machine Intelligence in Medical Image Analysis, Springer, (2019), 1-12.
    [69] S. Marcel, Y. Rodriguez, Torchvision the machine-vision package of Torch, in Proceedings of the 18th ACM International Conference on Multimedia, (2010), 1485–1488.
    [70] G. Huang, Z. Liu, L. Van Der Maaten, K. Q. Weinberger, Densely connected convolutional networks, in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), (2017), 2261-2269.
    [71] J. Irvin, P. Rajpurkar, M. Ko, Y. Yu, S. Ciurea-Ilcus, C. Chute, et al., CheXpert: A large chest radiograph dataset with uncertainty labels and expert comparison, preprint, arXiv: 1901.07031.
    [72] H. Pham, T. Le, D. Ngo, D. Tran, H. Nguyen, Interpreting chest X-rays via CNNs that exploit hierarchical disease dependencies and uncertainty labels, Neurocomputing, 437 (2021), 186-194. doi: 10.1016/j.neucom.2020.03.127
    [73] I. Allaouzi, M. Ben Ahmed, A novel approach for multi-label chest X-ray classification of common thorax diseases, IEEE Access, 7 (2019), 64279-64288. doi: 10.1109/ACCESS.2019.2916849
    [74] H. Wang, S. Wang, Z. Qin, Y. Zhang, R. Li, Y. Xia, Triple attention learning for classification of 14 thoracic diseases using chest radiography, Med. Image Anal., 67 (2021), 64279-64288.
    [75] M. A. Morid, A. Borjali, G. Del Fiol, A scoping review of transfer learning research on medical image analysis using ImageNet, Comput. Biol. Med., 128 (2021).
    [76] Pytorch.org, Transfer Learning for Computer Vision Tutorial. Available from https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html.
    [77] D. Kingma, J. Ba, ADAM: A method for stochastic optimization, preprint. arXiv: 1412.6980.
    [78] L. Prechelt, Early stopping-but when?, in Lecture Notes in Computer Science, Springer Berlin, (2012), 53-67.
    [79] M. Horry, S. Chakraborty, M. Paul, A. Ulhaq, B. Pradhan, M. Saha, et al., COVID-19 detection through transfer learning using multimodal imaging data, IEEE Access, 8 (2020), 149808-149824. doi: 10.1109/ACCESS.2020.3016780
    [80] T. C. Kwee, R. M. Kwee, Chest CT in COVID-19: What the radiologist needs to know, Radiographics, 40 (2020), 1848-1865. doi: 10.1148/rg.2020200159
    [81] J. L. Lehr, P. Capek, Histogram equalization of CT images, Radiology, 154 (1985), 163-169. doi: 10.1148/radiology.154.1.3964935
    [82] O. Ronneberger, P. Fischer, T. Brox, U-Net: Convolutional networks for biomedical image segmentation, in Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015, Springer Verlag, (2015), 234-241.
  • Reader Comments
  • © 2021 the Author(s), licensee AIMS Press. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0)
通讯作者: 陈斌, bchen63@163.com
  • 1. 

    沈阳化工大学材料科学与工程学院 沈阳 110142

  1. 本站搜索
  2. 百度学术搜索
  3. 万方数据库搜索
  4. CNKI搜索
Mathematical Biosciences and Engineering
3.9

Metrics

Article views(3284) PDF downloads(123) Cited by(4)

Preview PDF
Download XML
Export Citation
Article outline

Figures and Tables

Figures(8)  /  Tables(15)

Other Articles By Authors

Related pages

Tools

/

DownLoad:  Full-Size Img  PowerPoint
Return
Return

Catalog