Identifying concepts from medical images via transfer learning and image retrieval

Xuwen Wang; Yu Zhang; Zhen Guo; Jiao Li; Xuwen Wang; Yu Zhang; Zhen Guo; Jiao Li

doi:10.3934/mbe.2019097

Mathematical Biosciences and Engineering

2019, Volume 16, Issue 4: 1978-1991. doi: 10.3934/mbe.2019097

Previous Article Next Article

Research article Special Issues

Identifying concepts from medical images via transfer learning and image retrieval

Institute of Medical Information and Library, Chinese Academy of Medical Sciences/Peking Union Medical College, Beijing 100020, China

Received: 18 December 2018 Accepted: 18 February 2019 Published: 08 March 2019

Automatically identifying semantic concepts from medical images provides multimodal insights for clinical research. To study the effectiveness of concept detection on large scale medical images, we reconstructed over 230,000 medical image-concepts pairs collected from the ImageCLEFcaption 2018 evaluation task. A transfer learning-based multi-label classification model was used to predict multiple high-frequency concepts for medical images. Semantically relevant concepts of visually similar medical images were identified by the image retrieval-based topic model. The results showed that the transfer learning method achieved F1 score of 0.1298, which was comparable with the state of art methods in the ImageCLEFcaption tasks. The image retrieval-based method contributed to the recall performance but reduced the overall F1 score, since the retrieval results of the search engine introduced irrelevant concepts. Although our proposed method achieved second-best performance in the concept detection subtask of ImageCLEFcaption 2018, there will be plenty of further work to improve the concept detection with better understanding the medical images.
- concept detection,
- transfer learning,
- multi-label classification,
- medical image retrieval,
- LDA
Citation: Xuwen Wang, Yu Zhang, Zhen Guo, Jiao Li. Identifying concepts from medical images via transfer learning and image retrieval[J]. Mathematical Biosciences and Engineering, 2019, 16(4): 1978-1991. doi: 10.3934/mbe.2019097

Related Papers:

Abstract

Automatically identifying semantic concepts from medical images provides multimodal insights for clinical research. To study the effectiveness of concept detection on large scale medical images, we reconstructed over 230,000 medical image-concepts pairs collected from the ImageCLEFcaption 2018 evaluation task. A transfer learning-based multi-label classification model was used to predict multiple high-frequency concepts for medical images. Semantically relevant concepts of visually similar medical images were identified by the image retrieval-based topic model. The results showed that the transfer learning method achieved F1 score of 0.1298, which was comparable with the state of art methods in the ImageCLEFcaption tasks. The image retrieval-based method contributed to the recall performance but reduced the overall F1 score, since the retrieval results of the search engine introduced irrelevant concepts. Although our proposed method achieved second-best performance in the concept detection subtask of ImageCLEFcaption 2018, there will be plenty of further work to improve the concept detection with better understanding the medical images.

References

[1]	G. J. Litjens, T. Kooi and B. E. Bejnordi, et al., A survey on deep learning in medical image analysis, Med. Image Anal., 42(2017), 60–88.
[2]	X. Kong, T. Tan and L. Bao, et al., Classification of breast mass in 3D ultrasound images with annotations based on convolutional neural networks. Chin. J. Biomed. Eng., 37(2018), 414–422.
[3]	S. J. Pan and Q. Yang, A survey on transfer learning, IEEE T. Knowl. Data. En., 22(2010), 1345–1359.
[4]	A. Esteva, B. Kuprel and R. A. Novoa, et al., Dermatologist-level classification of skin cancer with deep neural networks, Nature, 542(2017), 115–118.
[5]	Y. Yu, H. Lin and J. Meng, et al., Classification modeling and recognition for cross modal and multi-label biomedical image. J. Image Graph., 23(2018), 917–927.
[6]	C. Eickhoff, I. Schwall and A. G. Seco de Herrera, et al., Overview of ImageCLEFcaption 2017–image caption prediction and concept detection for biomedical images. In: G. J. F. Jones, S. Lawless and J. Gonzalo, et al., editors. Lect. Notes. Comput. SC.: Experimental IR meets multilinguality, multimodality, and interaction. 8th International Conference of the CLEF Association (CLEF 2017); September 11–14, 2017; Dublin, Ireland. Cham: Springer; 2017 Aug 17. 10456(2017), 315–337.
[7]	ImageCLEFcaption 2018: ImageCLEF/LifeCLEF–Multimedia Retrieval in CLEF [Internet]. Avignon, France: the CLEF initiative labs. 2018-[cited 2019 Feb 26]. Available from: http://www.imageclef.org/2018/caption.
[8]	UMLS: Unified Medical Language System [Internet]. Bethesda, Maryland: U.S. National Library of Medicine. 1986-[cited 2019 Feb 26]. Available from: https://www.nlm.nih.gov/research/umls/.
[9]	O. Bodenreider, The unified medical language system (umls): integrating biomedical terminology. Nucleic Acids Res., 32(2004), 267–270.
[10]	A. G. Seco de Herrera, C. Eickhoff and V. Andrearczyk, et al., Overview of the ImageCLEF 2018 caption prediction tasks. Paper presented at: CLEF 2018. Working Notes of CLEF 2018-Conference and Labs of the Evaluation Forum, CEUR Workshop Proceedings; 2018 Sep 10–14; Avignon, France.
[11]	E. Pinho, J. F. Silva and J. M. Silva, Towards representation learning for biomedical concept detection in medical images: UA.PT bioinformatics in ImageCLEF 2017. Paper presented at: CLEF 2017. Working notes of CLEF 2017-Conference and Labs of the Evaluation Forum, CEUR Workshop Proceedings; 2017 Sep 11–14; Dublin, Ireland.
[12]	D. Katsios and E. Kavallieratou, Concept detection on medical images using deep residual learning network. Paper presented at: CLEF 2017. Working notes of CLEF 2017-Conference and Labs of the Evaluation Forum, CEUR Workshop Proceedings; 2017 Sep 11–14; Dublin, Ireland.
[13]	N. N. Hoavy, J. Mothe and M.I. Randrianarivony, IRIT & MISA at ImageCLEF 2017-multi label classification. Paper presented at: CLEF 2017. Working notes of CLEF 2017-Conference and Labs of the Evaluation Forum, CEUR Workshop Proceedings; 2017 Sep 11–14; Dublin, Ireland.
[14]	L. Valavanis and T. Kalamboukis, IPL at ImageCLEF 2018: a KNN-based concept detection approach. Paper presented at: CLEF 2018. Working notes of CLEF 2018-Conference and Labs of the Evaluation Forum, CEUR Workshop Proceedings; 2018 Sep 10–14; Avignon, France.
[15]	M. M. Rahman, T. Lagree and M. Taylor, A cross-modal concept detection and caption prediction approach in ImageCLEFcaption track of ImageCLEF 2017. Paper presented at: CLEF 2017. Working notes of CLEF 2017-Conference and Labs of the Evaluation Forum, CEUR Workshop Proceedings; 2017 Sep 11–14; Dublin, Ireland.
[16]	Y. Zhang, X. Wang and Z. Guo, et al., ImageSem at ImageCLEF 2018 caption task: image retrieval and transfer learning, Paper presented at: CLEF 2018. Working notes of CLEF 2018-Conference and Labs of the Evaluation Forum, CEUR Workshop Proceedings; 2018 Sep 10–14; Avignon, France.
[17]	E. Pinho and C. Costa, Feature learning with adversarial networks for concept detection in medical images: UA.PT bioinformatics at ImageCLEF 2018. Paper presented at: CLEF 2018. Working notes of CLEF 2018-Conference and Labs of the Evaluation Forum, CEUR Workshop Proceedings; 2018 Sep 10–14; Avignon, France.
[18]	PMC: PubMed Central [Internet]. Bethesda, Maryland: National Center for Biotechnology Information (NCBI), U.S. National Institutes of Health's, National Library of Medicine. 2000-[cited 2019 Feb 26]. Available from: https://www.ncbi.nlm.nih.gov/pmc/.
[19]	QuickUMLS: System for Medical Concept Extraction [Internet]. Georgetown University, Washington: Luca Soldaini and Nazli Goharian. 2016-[cited 2019 Feb 26]. Available from: https://github.com/Georgetown-IR-Lab/QuickUMLS
[20]	MetaMap: A Tool for Recognizing UMLS Concepts in Text [Internet]. Bethesda, Maryland: U.S. National Institutes of Health's, National Library of Medicine. 1996-[cited 2019 Feb 26]. Available from: https://metamap.nlm.nih.gov/.
[21]	LIRE: Lucene Image Retrieval [Internet]. Klagenfurt University, AT: Mathias Lux. 2015-[cited 2019 Feb 26]. Available from: http://www.lire-project.net/.
[22]	R. Gan and J. Yin, Using LIRe to implement image retrieval system based on multi-feature descriptor. Proceedings of the Third International Conference on Digital Manufacturing & Automation; 2012 Jul 31–Aug 2; Guilin, China. IEEE; 2012 Sep 13. 1014–1017p.
[23]	C. Szegedy, V. Vanhoucke and S. Ioffe, et al., Rethinking the inception architecture for computer vision. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition; 2016 Jun 27–30; La Vegas, NV, USA. IEEE; 2016 Dec 12. 2818–2826p.
[24]	O. Russakovsky, J. Deng and H. Su, et al., ImageNet large scale visual recognition challenge. Int. J. Comput. Vis., 115(2015), 211–252.
[25]	D.M. Blei, A.Y. Ng and M. I. Jordan, Latent dirichlet allocation. J. Mach. Learn. Res., 3 (2003), 993–1022.
[26]	Gensim, topic modelling for humans [Internet]. Masaryk University, Czech: Radim Řehůřek. 2009-[cited 2019 Feb 26]. Available from: https://radimrehurek.com/gensim/.

Reader Comments

Your name:*

Email:*
© 2019 the Author(s), licensee AIMS Press. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0)