A review on multimodal machine learning in medical diagnostics

Keyue Yan; Tengyue Li; João Alexandre Lobo Marques; Juntao Gao; Simon James Fong; Keyue Yan; Tengyue Li; João Alexandre Lobo Marques; Juntao Gao; Simon James Fong

doi:10.3934/mbe.2023382

Mathematical Biosciences and Engineering

2023, Volume 20, Issue 5: 8708-8726. doi: 10.3934/mbe.2023382

Previous Article Next Article

Review Special Issues

A review on multimodal machine learning in medical diagnostics

1.
Department of Computer and Information Science, University of Macau, Macau SAR, China
2.
Laboratory of Applied Neurosciences, University of Saint Joseph, Macau SAR, China
3.
Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing 100084, China
4.
Institute of Artificial Intelligence, Chongqing Technology and Business University, Chongqing, China

Academic Editor: Mariano Torrisi

Received: 30 September 2022 Revised: 15 February 2023 Accepted: 15 February 2023 Published: 06 March 2023

Nowadays, the increasing number of medical diagnostic data and clinical data provide more complementary references for doctors to make diagnosis to patients. For example, with medical data, such as electrocardiography (ECG), machine learning algorithms can be used to identify and diagnose heart disease to reduce the workload of doctors. However, ECG data is always exposed to various kinds of noise and interference in reality, and medical diagnostics only based on one-dimensional ECG data is not trustable enough. By extracting new features from other types of medical data, we can implement enhanced recognition methods, called multimodal learning. Multimodal learning helps models to process data from a range of different sources, eliminate the requirement for training each single learning modality, and improve the robustness of models with the diversity of data. Growing number of articles in recent years have been devoted to investigating how to extract data from different sources and build accurate multimodal machine learning models, or deep learning models for medical diagnostics. This paper reviews and summarizes several recent papers that dealing with multimodal machine learning in disease detection, and identify topics for future research.
- multimodal learning,
- machine learning,
- deep learning,
- medical data
Citation: Keyue Yan, Tengyue Li, João Alexandre Lobo Marques, Juntao Gao, Simon James Fong. A review on multimodal machine learning in medical diagnostics[J]. Mathematical Biosciences and Engineering, 2023, 20(5): 8708-8726. doi: 10.3934/mbe.2023382

Related Papers:

Abstract

Nowadays, the increasing number of medical diagnostic data and clinical data provide more complementary references for doctors to make diagnosis to patients. For example, with medical data, such as electrocardiography (ECG), machine learning algorithms can be used to identify and diagnose heart disease to reduce the workload of doctors. However, ECG data is always exposed to various kinds of noise and interference in reality, and medical diagnostics only based on one-dimensional ECG data is not trustable enough. By extracting new features from other types of medical data, we can implement enhanced recognition methods, called multimodal learning. Multimodal learning helps models to process data from a range of different sources, eliminate the requirement for training each single learning modality, and improve the robustness of models with the diversity of data. Growing number of articles in recent years have been devoted to investigating how to extract data from different sources and build accurate multimodal machine learning models, or deep learning models for medical diagnostics. This paper reviews and summarizes several recent papers that dealing with multimodal machine learning in disease detection, and identify topics for future research.

References

[1]	J. Smith, Science and Technology for Development, Bloomsbury publishing, 2009.
[2]	J. Carbonell, R. Michalski, T. Mitchell, An overview of machine learning, Mach. Learn., 5 (1983), 3–23. https://doi.org/10.1016/B978-0-08-051054-5.50005-4 doi: 10.1016/B978-0-08-051054-5.50005-4
[3]	J. Tang, G. Liu, Q. Pan, A review on representative swarm intelligence algorithms for solving optimization problems: Applications and trends, IEEE/CAA J. Autom. Sin., 8 (2021), 1627–1643. https://doi.org/10.1109/JAS.2021.1004129 doi: 10.1109/JAS.2021.1004129
[4]	A. Triantafyllidis, A. Tsanas, Applications of machine learning in real-life digital health interventions: review of the literature, J. Med. Internet Res., 21 (2019), e12286. https://doi.org/10.2196/12286 doi: 10.2196/12286
[5]	W. Aziz, L. Hussain, I. Khan, J. Alowibdi, M. Alkinani, Machine learning based classification of normal, slow and fast walking by extracting multimodal features from stride interval time series, Math. Biosci. Eng., 18 (2021), 495–517. http://doi.org/10.3934/mbe.2021027 doi: 10.3934/mbe.2021027
[6]	L. Hussain, W. Aziz, I. Khan, M. Alkinani, J. Alowibdi, Machine learning based congestive heart failure detection using feature importance ranking of multimodal features, Math. Biosci. Eng., 18 (2021), 69–91. http://doi.org/10.3934/mbe.2021004 doi: 10.3934/mbe.2021004
[7]	Y. Xu, Y. Lin, R. Bell, S. Towe, J. Pearson, T. Nadeem, et al., Machine learning prediction of neurocognitive impairment among people with hiv using clinical and multimodal magnetic resonance imaging data, J. Neurovirol., 27 (2021), 1–11. https://doi.org/10.1007/s13365-020-00930-4 doi: 10.1007/s13365-020-00930-4
[8]	B. Naik, A. Mehta, M. Shah, Denouements of machine learning and multimodal diagnostic classification of alzheimer's disease, Visual Comput. Ind. Biomed. Art, 3 (2020), 1–18. https://doi.org/10.1186/s42492-020-00062-w doi: 10.1186/s42492-020-00062-w
[9]	R. Walambe, P. Nayak, A. Bhardwaj, K. Kotecha, Employing multimodal machine learning for stress detection, J. Healthcare Eng., 2021 (2021). https://doi.org/10.1155/2021/9356452
[10]	G. Battineni, M. Hossain, N. Chintalapudi, E. Traini, V. Dhulipalla, M. Ramasamy, et al., Improved alzheimer's disease detection by mri using multimodal machine learning algorithms, Diagnostics, 11 (2021), 2103. https://doi.org/10.3390/diagnostics11112103 doi: 10.3390/diagnostics11112103
[11]	L. Anand, K. Rane, L. Bewoor, J. Bangare, J. Surve, M. Raghunath, et al., Development of machine learning and medical enabled multimodal for segmentation and classification of brain tumor using MRI images, Comput. Intell. Neurosci., 2022 (2022). https://doi.org/10.1155/2022/7797094
[12]	M. Khan, I. Ashraf, M. Alhaisoni, R. Damaševičius, R. Scherer, A. Rehman, et al., Multimodal brain tumor classification using deep learning and robust feature selection: A machine learning application for radiologists, Diagnostics, 10 (2020), 565. https://doi.org/10.3390/diagnostics10080565 doi: 10.3390/diagnostics10080565
[13]	A. Tiulpin, S. Klein, S. Bierma-Zeinstra, J. Thevenot, E. Rahtu, J. Meurs, et al., Multimodal machine learning-based knee osteoarthritis progression prediction from plain radiographs and clinical data, Sci. Rep., 9 (2019), 1–11. https://doi.org/10.1038/s41598-019-56527-3 doi: 10.1038/s41598-019-56527-3
[14]	R. Prashanth, S. Roy, P. Mandal, S. Ghosh, High-accuracy detection of early parkinson's disease through multimodal features and machine learning, Int. J. Med. Inf., 90 (2016), 13–21. https://doi.org/10.1016/j.ijmedinf.2016.03.001 doi: 10.1016/j.ijmedinf.2016.03.001
[15]	C. Ieracitano, N. Mammone, A. Hussain, F. Morabito, A novel multi-modal machine learning based approach for automatic classification of eeg recordings in dementia, Neural Networks, 123 (2020), 176–190. https://doi.org/10.1016/j.neunet.2019.12.006 doi: 10.1016/j.neunet.2019.12.006
[16]	L. Zhao, M. Li, Z. He, S. Ye, H. Qin, X. Zhu, et al., Data-driven learning fatigue detection system: A multimodal fusion approach of ECG (electrocardiogram) and video signals, Measurement, 201 (2022), 111648. https://doi.org/10.1016/j.measurement.2022.111648 doi: 10.1016/j.measurement.2022.111648
[17]	S. Ma, J. Cui, W. Xiao, L. Liu, Deep learning-based data augmentation and model fusion for automatic arrhythmia identification and classification algorithms, Comput. Intell. Neurosci., 2022 (2022). https://doi.org/10.1155/2022/1577778
[18]	M. Ramkumar, R. Sarath Kumar, A. Manjunathan, M. Mathankumar, J. Pauliah, Auto-encoder and bidirectional long short-term memory based automated arrhythmia classification for ECG signal, Biomed. Signal Process. Control, 77 (2022), 103826. https://doi.org/10.1016/j.bspc.2022.103826 doi: 10.1016/j.bspc.2022.103826
[19]	J. Arteaga-Falconi, H. Al Osman, A. El Saddik, ECG and fingerprint bimodal authentication, Sustainable Cities Soc., 40 (2018), 274–283. https://doi.org/10.1016/j.scs.2017.12.023 doi: 10.1016/j.scs.2017.12.023
[20]	Z. Ahmad, A. Tabassum, L. Guan, N. Khan, ECG heartbeat classification using multimodal fusion, IEEE Access, 9 (2021), 100615–100626. https://doi.org/10.1109/ACCESS.2021.3097614 doi: 10.1109/ACCESS.2021.3097614
[21]	S. Irfan, N. Anjum, T. Althobaiti, A. Alotaibi, A. Siddiqui, N. Ramzan, Heartbeat classification and arrhythmia detection using a multi-model deep-learning technique, Sensors, 22 (2022), 5606. https://doi.org/10.3390/s22155606 doi: 10.3390/s22155606
[22]	Y. Zeng, S. Yang, X. Yu, W. Lin, W. Wang, J. Tong, et al., A multimodal parallel method for left ventricular dysfunction identification based on phonocardiogram and electrocardiogram signals synchronous analysis, Math. Biosci. Eng., 19 (2022), 9612–9635. https://doi.org/10.3934/mbe.2022447 doi: 10.3934/mbe.2022447
[23]	G. Song, J. Zhang, D. Mao, G. Chen, C. Pang, A multimodel fusion method for cardiovascular disease detection using ECG, Emerg. Med. Int., 2022 (2022). https://doi.org/10.1155/2022/3561147
[24]	K. Su, G. Yang, B. Wu, L. Yang, D. Li, P. Su, et al., Human identification using finger vein and ecg signals, Neurocomputing, 332 (2019), 111–118. https://doi.org/10.1016/j.neucom.2018.12.015 doi: 10.1016/j.neucom.2018.12.015
[25]	B. El-Rahiem, F. El-Samie, M. Amin, Multimodal biometric authentication based on deep fusion of electrocardiogram (ECG) and finger vein, Multimedia Syst., 28 (2022), 1325–1337. https://doi.org/10.1007/s00530-021-00810-9 doi: 10.1007/s00530-021-00810-9
[26]	M. Hammad, Y. Liu, K. Wang, Multimodal biometric authentication systems using convolution neural network based on different level fusion of ECG and fingerprint, IEEE Access, 7 (2018), 26527–26542. https://doi.org/10.1109/ACCESS.2018.2886573 doi: 10.1109/ACCESS.2018.2886573
[27]	M. Bugdol, A. Mitas, Multimodal biometric system combining ecg and sound signals, Pattern Recognit. Lett., 38 (2014), 107–112. https://doi.org/10.1016/j.patrec.2013.11.014 doi: 10.1016/j.patrec.2013.11.014
[28]	S. Ketu, P. Mishra, Empirical analysis of machine learning algorithms on imbalance electrocardiogram based arrhythmia dataset for heart disease detection, Arabian J. Sci. Eng., 47 (2022), 1447–1469. https://doi.org/10.1007/s13369-021-05972-2 doi: 10.1007/s13369-021-05972-2
[29]	E. Al Alkeem, C. Yeun, J. Yun, P. Yoo, M. Chae, A. Rahman, et al., Robust deep identification using ecg and multimodal biometrics for industrial internet of things, Ad Hoc Networks, 121 (2021), 102581. https://doi.org/10.1016/j.adhoc.2021.102581 doi: 10.1016/j.adhoc.2021.102581
[30]	J. Rahul, M. Sora, L. Sharma, V. Bohat, An improved cardiac arrhythmia classification using an rr interval-based approach, Biocybern. Biomed. Eng., 41 (2021), 656–666. https://doi.org/10.1016/j.bbe.2021.04.004 doi: 10.1016/j.bbe.2021.04.004
[31]	A. Kline, H. Wang, Y. Li, S. Dennis, M. Hutch, Z. Xu, et al., Multimodal machine learning in precision health: A scoping review, npj Digital Med., 5 (2022), 1–14. https://doi.org/10.1038/s41746-022-00712-8 doi: 10.1038/s41746-022-00712-8
[32]	Y. Wang, K. Yan, Prediction of significant bitcoin price changes based on deep learning, in 2022 5th International Conference on Data Science and Information Technology (DSIT), (2022), 1–5. https://doi.org/10.1109/DSIT55514.2022.9943971
[33]	C. Bock, M. Farlik, N. Sheffield, Multi-omics of single cells: strategies and applications, Trends Biotechnol., 34 (2016), 605–608. https://doi.org/10.1016/j.tibtech.2016.04.004 doi: 10.1016/j.tibtech.2016.04.004
[34]	H. Jung, Y. Sung, H. Kim, Omics and computational modeling approaches for the effective treatment of drug-resistant cancer cells, Front. Genet., 12 (2021), 742902. https://doi.org/10.3389/fgene.2021.742902 doi: 10.3389/fgene.2021.742902
[35]	Z. Yuan, Q. Zhou, L. Cai, L. Pan, W. Sun, S. Qumu, et al., Seam is a spatial single nuclear metabolomics method for dissecting tissue microenvironment, Nat. Methods, 18 (2021), 1223–1232. https://doi.org/10.1038/s41592-021-01276-3 doi: 10.1038/s41592-021-01276-3
[36]	H. Qiao, F. Wang, R. Xu, J. Sun, R. Zhu, D. Mao, et al., An efficient and multiple target transgenic rnai technique with low toxicity in drosophila, Nat. Commun., 9 (2018), 4160. https://doi.org/10.1038/s41467-018-06537-y doi: 10.1038/s41467-018-06537-y
[37]	F. Valenti, I. Falcone, S. Ungania, F. Desiderio, P. Giacomini, C. Bazzichetto, et al., Precision medicine and melanoma: multi-omics approaches to monitoring the immunotherapy response, Int. J. Mol. Sci., 22 (2021), 3837. https://doi.org/10.3390/ijms22083837 doi: 10.3390/ijms22083837
[38]	A. Wojtuszkiewicz, I. van der Werf, S. Hutter, W. Walter, C. Baer, W. Kern, et al., Maturation state-specific alternative splicing in FLT3-ITD and NPM1 mutated AML, Cancers, 13 (2021), 3929. https://doi.org/10.3390/cancers13163929 doi: 10.3390/cancers13163929
[39]	S. Stahlschmidt, B. Ulfenborg, J. Synnergren, Multimodal deep learning for biomedical data fusion: a review, Briefings Bioinf., 23 (2022). https://doi.org/10.1093/bib/bbab569
[40]	Z. Cao, G. Gao, Multi-omics single-cell data integration and regulatory inference with graph-linked embedding, Nat. Biotechnol., 40 (2022), 1458–1466. https://doi.org/10.1038/s41587-022-01284-4 doi: 10.1038/s41587-022-01284-4
[41]	Y. Lei, S. Li, Z. Liu, F. Wan, T. Tian, S. Li, et al., A deep-learning framework for multi-level peptide–protein interaction prediction, Nat. Commun., 12 (2021), 5465. https://doi.org/10.1038/s41467-021-25772-4 doi: 10.1038/s41467-021-25772-4
[42]	W. Zhou, K. Yang, J. Zeng, X. Lai, X. Wang, C. Ji, et al., FordNet: Recommending traditional Chinese medicine formula via deep neural network integrating phenotype and molecule, Pharmacol. Res., 173 (2021), 105752. https://doi.org/10.1016/j.phrs.2021.105752 doi: 10.1016/j.phrs.2021.105752
[43]	X. Lin, L. Hu, J. Gu, R. Wang, L. Li, J. Tang, et al., Choline kinase $\alpha$ mediates interactions between the epidermal growth factor receptor and mechanistic target of rapamycin complex 2 in hepatocellular carcinoma cells to promote drug resistance and xenograft tumor progression, Gastroenterology, 152 (2017), 1187–1202. https://doi.org/10.1053/j.gastro.2016.12.033 doi: 10.1053/j.gastro.2016.12.033

Reader Comments

Your name:*

Email:*
© 2023 the Author(s), licensee AIMS Press. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0)