Predicting the prognosis of HER2-positive breast cancer patients by fusing pathological whole slide images and clinical features using multiple instance learning

Yifan Wang; Lu Zhang; Yan Li; Fei Wu; Shiyu Cao; Feng Ye; Yifan Wang; Lu Zhang; Yan Li; Fei Wu; Shiyu Cao; Feng Ye

doi:10.3934/mbe.2023496

Mathematical Biosciences and Engineering

2023, Volume 20, Issue 6: 11196-11211. doi: 10.3934/mbe.2023496

Previous Article Next Article

Research article Special Issues

Predicting the prognosis of HER2-positive breast cancer patients by fusing pathological whole slide images and clinical features using multiple instance learning

Yifan Wang ^{1
,
,},
Lu Zhang ¹,
Yan Li ^{1
,
,},
Fei Wu ²,
Shiyu Cao ³,
Feng Ye ³

1.
School of Computer Science, Southwest Petroleum University, Chengdu 610500, China
2.
Institute of Applied Physics and Computational Mathematics, China Academy of Engineering Physics, Beijing 100094, China
3.
Institute of Clinical Pathology, Key Laboratory of Transplant Engineering and Immunology, NHC, West China Hospital, Sichuan University, Chengdu 610041, China

Received: 15 February 2023 Revised: 26 March 2023 Accepted: 03 April 2023 Published: 25 April 2023

In 2022, breast cancer will become an important factor affecting women's public health and HER2 positivity for approximately 15–20$ \% $ invasive breast cancer cases. Follow-up data for HER2-positive patients are rare, and research on prognosis and auxiliary diagnosis is still limited. In light of the findings obtained from the analysis of clinical features, we have developed a novel multiple instance learning (MIL) fusion model that integrates hematoxylin-eosin (HE) pathological images and clinical features to accurately predict the prognostic risk of patients. Specifically, we segmented the HE pathology images of patients into patches, clustered them by K-means, aggregated them into a bag feature-level representation through graph attention networks (GATs) and multihead attention networks, and fused them with clinical features to predict the prognosis of patients. We divided West China Hospital (WCH) patients (n = 1069) into a training cohort and internal validation cohort and used The Cancer Genome Atlas (TCGA) patients (n = 160) as an external test cohort. The 3-fold average C-index of the proposed OS-based model was 0.668, the C-index of the WCH test set was 0.765, and the C-index of the TCGA independent test set was 0.726. By plotting the Kaplan-Meier curve, the fusion feature (P = 0.034) model distinguished high- and low-risk groups more accurately than clinical features (P = 0.19). The MIL model can directly analyze a large number of unlabeled pathological images, and the multimodal model is more accurate than the unimodal models in predicting Her2-positive breast cancer prognosis based on large amounts of data.
- breast cancer,
- HER2,
- multiple instance learning,
- deep learning,
- survival analysis,
- prognosis
Citation: Yifan Wang, Lu Zhang, Yan Li, Fei Wu, Shiyu Cao, Feng Ye. Predicting the prognosis of HER2-positive breast cancer patients by fusing pathological whole slide images and clinical features using multiple instance learning[J]. Mathematical Biosciences and Engineering, 2023, 20(6): 11196-11211. doi: 10.3934/mbe.2023496

Related Papers:

Abstract

In 2022, breast cancer will become an important factor affecting women's public health and HER2 positivity for approximately 15–20$ \% $ invasive breast cancer cases. Follow-up data for HER2-positive patients are rare, and research on prognosis and auxiliary diagnosis is still limited. In light of the findings obtained from the analysis of clinical features, we have developed a novel multiple instance learning (MIL) fusion model that integrates hematoxylin-eosin (HE) pathological images and clinical features to accurately predict the prognostic risk of patients. Specifically, we segmented the HE pathology images of patients into patches, clustered them by K-means, aggregated them into a bag feature-level representation through graph attention networks (GATs) and multihead attention networks, and fused them with clinical features to predict the prognosis of patients. We divided West China Hospital (WCH) patients (n = 1069) into a training cohort and internal validation cohort and used The Cancer Genome Atlas (TCGA) patients (n = 160) as an external test cohort. The 3-fold average C-index of the proposed OS-based model was 0.668, the C-index of the WCH test set was 0.765, and the C-index of the TCGA independent test set was 0.726. By plotting the Kaplan-Meier curve, the fusion feature (P = 0.034) model distinguished high- and low-risk groups more accurately than clinical features (P = 0.19). The MIL model can directly analyze a large number of unlabeled pathological images, and the multimodal model is more accurate than the unimodal models in predicting Her2-positive breast cancer prognosis based on large amounts of data.

References

[1]	R. L. Siegel, K. D. Miller, H. E. Fuchs, A. Jemal, Cancer statistics, 2022, CA Cancer J. Clin., 72 (2022), 7–33. https://doi.org/10.3322/caac.21708
[2]	E. A. Perez, E. H. Romond, V. J. Suman, J. Jeong, G. Sledge, C. E. Geyer Jr, et al., Trastuzumab plus adjuvant chemotherapy for human epidermal growth factor receptor 2–Positive breast cancer: Planned joint analysis of overall survival from NSABP B-31 and NCCTG N9831, JCO, 32 (2014), 3744–3752. https://doi.org/10.1200/JCO.2014.55.5730 doi: 10.1200/JCO.2014.55.5730
[3]	C. L. Arteaga, M. X. Sliwkowski, C. K. Osborne, E. A. Perez, F. Puglisi, L. Gianni, Treatment of HER2-positive breast cancer: current status and future perspectives, Nat. Rev. Clin. Oncol., 9 (2012), 16–32. https://doi.org/10.1038/nrclinonc.2011.177 doi: 10.1038/nrclinonc.2011.177
[4]	J. N. Wang, B. H. Xu, Targeted therapeutic options and future perspectives for HER2-positive breast cancer, Sig. Transduct. Target Ther., 4 (2019), 34. https://doi.org/10.1038/s41392-019-0069-2 doi: 10.1038/s41392-019-0069-2
[5]	D. Cameron, M. J. Piccart-Gebhart, R. D. Gelber, M. Procter, A. Goldhirsch, E. de Azambuja, et al., 11 years' follow-up of trastuzumab after adjuvant chemotherapy in HER2-positive early breast cancer: Final analysis of the HERceptin Adjuvant (HERA) trial, Lancet, 389 (2017), 1195–1205. https://doi.org/10.1016/S0140-6736(16)32616-2 doi: 10.1016/S0140-6736(16)32616-2
[6]	Director's challenge consortium for the molecular classification of lung adenocarcinoma, Gene expression–based survival prediction in lung adenocarcinoma: a multi-site, blinded validation study, Nat. Med., 14 (2008), 822–827. https://doi.org/10.1038/nm.1790 doi: 10.1038/nm.1790
[7]	M. Y. Park, T. Hastie, L1-regularization path algorithm for generalized linear models, J Royal Statistical Soc B, 69 (2007), 659–677. https://doi.org/10.1111/j.1467-9868.2007.00607.x doi: 10.1111/j.1467-9868.2007.00607.x
[8]	E. Bair, R. Tibshirani, Semi-Supervised methods to predict patient survival from gene expression data, PLoS Biol., 2 (2004), 512–522. https://doi.org/10.1371/journal.pbio.0020108 doi: 10.1371/journal.pbio.0020108
[9]	A. Warth, T. Muley, M. Meister, A. Stenzinger, M. Thomas, P. Schirmacher, et al., The novel histologic international association for the study of lung cancer/American thoracic society/European respiratory society classification system of lung adenocarcinoma is a Stage-Independent predictor of survival, JCO, 30 (2012), 1438–1446. https://doi.org/10.1200/JCO.2011.37.2185 doi: 10.1200/JCO.2011.37.2185
[10]	B. Ehteshami Bejnordi, M. Veta, P. Johannes van Diest, B. van Ginneken, N. Karssemeijer, G. Litjens, et al., Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer, JAMA, 318 (2017), 2199. https://doi.org/10.1001/jama.2017.14585 doi: 10.1001/jama.2017.14585
[11]	Y. Yuan, H. Failmezger, O. M. Rueda, H. R. Ali, S. Gräf, S. Chin, et al., Quantitative image analysis of cellular heterogeneity in breast tumors complements genomic profiling, Sci. Transl. Med., 4 (2012). https://doi.org/10.1126/scitranslmed.3004330
[12]	J. Xu, Y. Cao, Y. Sun, J. Tang, Absolute exponential stability of recurrent neural networks with generalized activation function, IEEE Trans. Neural Networks, 19 (2008), 1075–1089, . https://doi.org/10.1109/TNN.2007.2000060 doi: 10.1109/TNN.2007.2000060
[13]	J. Tang, X. Liu, H. Cheng, K. M. Robinette, Gender recognition using 3-D human body shapes, IEEE Trans. Syst. Man Cybern. C, 41 (2011), 898–908. https://doi.org/10.1109/TSMCC.2011.2104950
[14]	X. Liu, J. Liu, X. Xu, L. Chun, J. Tang, Y. Deng, A robust detail preserving anisotropic diffusion for speckle reduction in ultrasound images, BMC Genom., 12 (2011), S14. https://doi.org/10.1186/1471-2164-12-S5-S14 doi: 10.1186/1471-2164-12-S5-S14
[15]	J. Tang, S. Millington, S. T. Acton, J. Crandall, S. Hurwitz, Ankle cartilage surface segmentation using directional gradient vector flow snakes 2004. IEEE Int. Conf. Inf. Process., 4 (2004), 2745–2748. https://doi.org/10.1109/ICIP.2004.1421672
[16]	J. Tang, S. Acton, An image retrieval algorithm using multiple query images, ISSPA 2003, 1 (2003), 193–196. https://doi.org/10.1109/ISSPA.2003.1224673 doi: 10.1109/ISSPA.2003.1224673
[17]	E. H. Cain, A. Saha, M. R. Harowicz, J. R. Marks, P. K. Marcom, M. A. Mazurowski, Multivariate machine learning models for prediction of pathologic response to neoadjuvant therapy in breast cancer using MRI features: a study using an independent validation set, Breast Cancer Res. Treat., 173 (2019), 455–463. https://doi.org/10.1007/s10549-018-4990-9 doi: 10.1007/s10549-018-4990-9
[18]	H. Wang, F. Xing, H. Su, A. Stromberg, L. Yang, Novel image markers for non-small cell lung cancer classification and survival prediction, BMC Bioinform., 15 (2014), 310. https://doi.org/10.1186/1471-2105-15-310 doi: 10.1186/1471-2105-15-310
[19]	Z. Hu, J. Tang, Z. Wang, K. Zhang, L. Zhang, Q. Sun Jr, Deep learning for image-based cancer detection and diagnosis-A survey, Pattern Recognition, 83 (2018), 134–149. https://doi.org/10.1016/j.patcog.2018.05.014 doi: 10.1016/j.patcog.2018.05.014
[20]	J. Yang, J. Ju, L. Guo, B. Ji, S. Shi, Z. Yang, et al., Prediction of HER2-positive breast cancer recurrence and metastasis risk from histopathological images and clinical information via multimodal deep learning, Comput. Struct. Biotel., 20 (2022), 333–342. https://doi.org/10.1016/j.csbj.2021.12.028 doi: 10.1016/j.csbj.2021.12.028
[21]	K. Yu, C. Zhang, G. J. Berry, R. B. Altman, C. Ré, D. L. Rubin, et al., Predicting non-small cell lung cancer prognosis by fully automated microscopic pathology image features, Nat. Commun., 7 (2016), 12474. https://doi.org/10.1038/ncomms12474 doi: 10.1038/ncomms12474
[22]	X. Liu, Z. Guo, J. Cao, J. Tang, MDC-net: A new convolutional neural network for nucleus segmentation in histopathology images with distance maps and contour information, Comput. Biol. Med., 135 (2021), 104543. https://doi.org/10.1016/j.compbiomed.2021.104543 doi: 10.1016/j.compbiomed.2021.104543
[23]	R. Yan, F. Ren, Z. Wang, L. Wang, T. Zhang, Y. Liu, Breast cancer histopathological image classification using a hybrid deep neural network, Methods, 173 (2020), 52–60. https://doi.org/10.1016/j.ymeth.2019.06.014 doi: 10.1016/j.ymeth.2019.06.014
[24]	X. Zhu, J. Yao, F. Zhu, J. Huang, Wsisa: Making survival prediction from whole slide histopathological images, in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), (2017), 7234–7242. https://doi.org/10.1109/CVPR.2017.725
[25]	R. Li, Graph CNN for survival analysis on whole slide pathological images, in Medical Image Computing and Computer Assisted Intervention, Springer International Publishing, (2018), 174–182. https://doi.org/10.1007/978-3-030-00934-2_20
[26]	J. Yao, X. Zhu, J. Jonnagaddala, N. Hawkins, J. Huang, Whole slide images based cancer survival prediction using attention guided deep multiple instance learning networks, Med. Image Anal., 65 (2022), 101789. https://doi.org/10.1016/j.media.2020.101789 doi: 10.1016/j.media.2020.101789
[27]	M. Y. Lu, D. F. K. Williamson, T. Y. Chen, R. J. Chen, M. Barbieri, F. Mahmood, Data-efficient and weakly supervised computational pathology on whole-slide images, Nat. Biomed. Eng., 5 (2021), 555–570. https://doi.org/10.1038/s41551-020-00682-w doi: 10.1038/s41551-020-00682-w
[28]	F. Wu, P. Liu, B. Fu, Y. Ye, DeepGCNMIL: Multi-head attention guided multi-instance learning approach for whole-slide images survival analysis using graph convolutional networks, ICMLC 2022, (2022), 67–-73. https://doi.org/10.1145/3529836.3529942
[29]	G. Campanella, M. G. Hanna, L. Geneslaw, A. Miraflor, V. Silva, K. J. Busam, et al., Clinical-grade computational pathology using weakly supervised deep learning on whole slide images, Nat. Med., 25 (2019), 1301–1309. https://doi.org/10.1038/s41591-019-0508-1 doi: 10.1038/s41591-019-0508-1
[30]	R. J. Chen, M. Y. Lu, J. Wang, D. F. K. Williamson, S. J. Rodig, N. I. Lindeman, et al., Pathomic fusion: An integrated framework for fusing histopathology and genomic features for cancer diagnosis and prognosis., IEEE Trans. Med. Imaging, 4 (2022), 757–770. https://doi.org/10.1109/TMI.2020.3021387 doi: 10.1109/TMI.2020.3021387
[31]	C. Kandoth, M. D. McLellan, F. Vandin, K. Ye, B. Niu, C. Lu, et al., Mutational landscape and significance across 12 major cancer types, Nature, 502 (2013), 333–339. https://doi.org/10.1038/nature12634 doi: 10.1038/nature12634
[32]	K. Simonyan, A. Zisserman, Very deep convolutional networks for large-scale image recognition, preprint, arXiv: 1409.1556.
[33]	K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), (2016), 770–778. https://doi.org/10.1109/CVPR.2016.90
[34]	T. N. Kipf, M. Welling, Semi-supervised classification with graph convolutional networks, preprint, arXiv: 1609.02907.
[35]	W. Hamilton, Z. Ying, J. Leskovec, Inductive representation learning on large graphs, Adv. Neural Inform. Proc. Syst., (2017), 30. https://doi.org/10.5555/3294771.3294869
[36]	P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Liò, Y. Bengio, Graph attention networks, preprint, arXiv: 1710.10903.
[37]	A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, et al., Attention is all you need, preprint, arXiv: 1706.03762.
[38]	J. Devlin, M. W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, preprint, arXiv: 1810.04805.
[39]	P. J. Heagerty, T. Lumley, M. S. Pepe, Time-dependent ROC curves for censored survival data and a diagnostic marker, Biometrics, 56 (2000), 337–344. https://doi.org/10.1111/j.0006-341X.2000.00337.x doi: 10.1111/j.0006-341X.2000.00337.x

Reader Comments

Your name:*

Email:*
© 2023 the Author(s), licensee AIMS Press. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0)