SAMS-Net: Fusion of attention mechanism and multi-scale features network for tumor infiltrating lymphocytes segmentation

Xiaoli Zhang; Kunmeng Liu; Kuixing Zhang; Xiang Li; Zhaocai Sun; Benzheng Wei; Xiaoli Zhang; Kunmeng Liu; Kuixing Zhang; Xiang Li; Zhaocai Sun; Benzheng Wei

doi:10.3934/mbe.2023140

Mathematical Biosciences and Engineering

2023, Volume 20, Issue 2: 2964-2979. doi: 10.3934/mbe.2023140

Previous Article Next Article

Research article Special Issues

SAMS-Net: Fusion of attention mechanism and multi-scale features network for tumor infiltrating lymphocytes segmentation

1.
College of Intelligence and Information Engineering, Shandong University of Traditional Chinese Medicine, Jinan 250355, China
2.
Center for Medical Artificial Intelligence, Shandong University of Traditional Chinese Medicine, Qingdao 266112, China
3.
Qingdao Academy of Chinese Medical Sciences, Shandong University of Traditional Chinese Medicine, Qingdao 266112, China

Academic Editor: Vicente García-Díaz

Received: 30 July 2022 Revised: 16 November 2022 Accepted: 21 November 2022 Published: 01 December 2022

Automatic segmentation of tumor-infiltrating lymphocytes (TILs) from pathological images is essential for the prognosis and treatment of cancer. Deep learning technology has achieved great success in the segmentation task. It is still a challenge to realize accurate segmentation of TILs due to the phenomenon of blurred edges and adhesion of cells. To alleviate these problems, a squeeze-and-attention and multi-scale feature fusion network (SAMS-Net) based on codec structure, namely SAMS-Net, is proposed for the segmentation of TILs. Specifically, SAMS-Net utilizes the squeeze-and-attention module with the residual structure to fuse local and global context features and boost the spatial relevance of TILs images. Besides, a multi-scale feature fusion module is designed to capture TILs with large size differences by combining context information. The residual structure module integrates feature maps from different resolutions to strengthen the spatial resolution and offset the loss of spatial details. SAMS-Net is evaluated on the public TILs dataset and achieved dice similarity coefficient (DSC) of 87.2% and Intersection of Union (IoU) of 77.5%, which improved by 2.5% and 3.8% compared with UNet. These results demonstrate the great potential of SAMS-Net in TILs analysis and can further provide important evidence for the prognosis and treatment of cancer.
- tumor infiltrating lymphocytes,
- prognosis,
- segmentation,
- multi-scale,
- attention
Citation: Xiaoli Zhang, Kunmeng Liu, Kuixing Zhang, Xiang Li, Zhaocai Sun, Benzheng Wei. SAMS-Net: Fusion of attention mechanism and multi-scale features network for tumor infiltrating lymphocytes segmentation[J]. Mathematical Biosciences and Engineering, 2023, 20(2): 2964-2979. doi: 10.3934/mbe.2023140

Related Papers:

Abstract

Automatic segmentation of tumor-infiltrating lymphocytes (TILs) from pathological images is essential for the prognosis and treatment of cancer. Deep learning technology has achieved great success in the segmentation task. It is still a challenge to realize accurate segmentation of TILs due to the phenomenon of blurred edges and adhesion of cells. To alleviate these problems, a squeeze-and-attention and multi-scale feature fusion network (SAMS-Net) based on codec structure, namely SAMS-Net, is proposed for the segmentation of TILs. Specifically, SAMS-Net utilizes the squeeze-and-attention module with the residual structure to fuse local and global context features and boost the spatial relevance of TILs images. Besides, a multi-scale feature fusion module is designed to capture TILs with large size differences by combining context information. The residual structure module integrates feature maps from different resolutions to strengthen the spatial resolution and offset the loss of spatial details. SAMS-Net is evaluated on the public TILs dataset and achieved dice similarity coefficient (DSC) of 87.2% and Intersection of Union (IoU) of 77.5%, which improved by 2.5% and 3.8% compared with UNet. These results demonstrate the great potential of SAMS-Net in TILs analysis and can further provide important evidence for the prognosis and treatment of cancer.

References

[1]	C. Kolberg-Liedtke, F. Feuerhake, M. Garke, M. Christgen, R. Kates, E. M. Grischke, et al., Impact of stromal tumor-infiltrating lymphocytes (sTILs) on response to neoadjuvant chemotherapy in triple-negative early breast cancer in the WSG-ADAPT TN trial, Breast Cancer Res., 24 (2022), 1–13. https://doi.org/10.1186/s13058-022-01552-w doi: 10.1186/s13058-022-01552-w
[2]	T. Nguyen, M. V. Ngo, V. P. Nguyen, Histopathological imaging classification of breast tissue for cancer diagnosis support using deep learning models, in International Conference on Industrial Networks and Intelligent Systems, 444 (2022), 152–164. https://doi.org/10.1007/978-3-031-08878-0_11
[3]	G. Floris, G. Broeckx, A. Antoranz, M. D. Schepper, R. Salgado, C. Desmedt, et al., Tumor infiltrating lymphocytes in breast cancer: Implementation of a new histopathological biomarker, in Biomarkers of the Tumor Microenvironment, Springer, (2022), 207–243. https://doi.org/10.1007/978-3-030-98950-7_13
[4]	H. Kuroda, T. Jamiyan, R. Yamaguchi, A. Kakumoto, A. Abe, O. Harada, et al., Tumor microenvironment in triple-negative breast cancer: The correlation of tumor-associated macrophages and tumor-infiltrating lymphocytes, Clin. Transl. Oncol., 23 (2021), 2513–2525. https://doi.org/10.1007/s12094-021-02652-3 doi: 10.1007/s12094-021-02652-3
[5]	T. Odate, M. K. Le, M. Kawai, M. Kubota, Y. Yamaguchi, T. Kondo, Tumor-infiltrating lymphocytes in breast FNA biopsy cytology: A predictor of tumor-infiltrating lymphocytes in histologic evaluation, Cancer Cytopathol., 130 (2022), 336–343. https://doi.org/10.1002/cncy.22551 doi: 10.1002/cncy.22551
[6]	S. Wang, J. Sun, K. Chen, P. Ma, N. Li, Perspectives of tumor-infiltrating lymphocyte treatment in solid tumors, BMC Med., 140 (2021), 1–7. https://doi.org/10.1186/s12916-021-02006-4 doi: 10.1186/s12916-021-02006-4
[7]	Y. Li, Z. Yang, Y. Wang, X. Cao, X. Xu, A neural network approach to analyze cross-sections of muscle fibers in pathological images, Comput. Biol. Med., 104 (2019), 97–104. https://doi.org/10.1016/j.compbiomed.2018.11.007 doi: 10.1016/j.compbiomed.2018.11.007
[8]	X. Wu, Y. Zheng, C. H. Chu, L. Cheng, J. Kim, Applying deep learning technology for automatic fall detection using mobile sensors, Biomed. Signal Process. Control, 72 (2022), 103355. https://doi.org/10.1016/j.bspc.2021.103355 doi: 10.1016/j.bspc.2021.103355
[9]	J. Cheng, S. Tian, L. Yu, C. Gao, X. Kang, X. Ma, et al., ResGANet: Residual group attention network for medical image classification and segmentation, Med. Image Anal., 76 (2022), 102313. https://doi.org/10.1016/j.media.2021.102313 doi: 10.1016/j.media.2021.102313
[10]	D. Müller, I. Soto-Rey, F. Kramer, An analysis on ensemble learning optimized medical image classification with deep convolutional neural networks, IEEE Access, 10 (2022), 66467–66480. https://doi.org/10.1109/ACCESS.2022.3182399 doi: 10.1109/ACCESS.2022.3182399
[11]	W. Pinaya, P. D. Tudosiu, R. Gray, G. Rees, P. Nachev, S. Ourselin, et al., Unsupervised brain imaging 3d anomaly detection and segmentation with transformers, Med. Image Anal., 79 (2022), 102475. https://doi.org/10.1016/j.media.2022.102475 doi: 10.1016/j.media.2022.102475
[12]	S. Javed, A. Mahmood, J. Dias, N. Werghi, N. Rajpoot, Spatially constrained context-aware hierarchical deep correlation filters for nucleus detection in histology images, Med. Image Anal., 72 (2021), 102104. https://doi.org/10.1016/j.media.2021.102104 doi: 10.1016/j.media.2021.102104
[13]	Z. Tan, J. Feng, J. Zhou, SGNet: Structure-aware graph-based network for airway semantic segmentation, in International Conference on Medical Image Computing and Computer-Assisted Intervention, (2021), 153–163. https://doi.org/10.1007/978-3-030-87193-2_15
[14]	Mehdi, S. Örjan, W. Chunliang, Prior-aware autoencoders for lung pathology segmentation, Med. Image Anal., 80 (2022), 102491. https://doi.org/10.1016/j.media.2022.102491 doi: 10.1016/j.media.2022.102491
[15]	T. Vicar, J. Chmelik, R. Kolar, Cell segmentation in quantitative phase images with improved iterative thresholding method, in European Medical and Biological Engineering Conference, (2020), 233–239. https://doi.org/10.1007/978-3-030-64610-3_27
[16]	M. Gamarra, E. Zurek, H. J. Escalante, L. Hurtado, H. San-Juan-Vergara, Split and merge watershed: A two-step method for cell segmentation in fluorescence microscopy images, Biomed. Signal Process. Control, 53 (2019), 101575. https://doi.org/10.1016/j.bspc.2019.101575 doi: 10.1016/j.bspc.2019.101575
[17]	E. Shelhamer, J. Long, T. Darrell, Fully convolutional networks for semantic segmentation, IEEE Trans. Pattern Anal. Mach. Intell., 39 (2017), 640–651. https://doi.org/10.1109/TPAMI.2016.2572683 doi: 10.1109/TPAMI.2016.2572683
[18]	O. Ronneberger, P. Fischer, T. Brox, U-Net: Convolutional networks for biomedical image segmentation, in Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, (2015), 234–241. https://doi.org/10.1007/978-3-319-24574-4_28
[19]	L. C. Chen, Y. Zhu, G. Papandreou, F. Schroff, H. Adam, Encoder-decoder with atrous separable convolution for semantic image segmentation, in Proceedings of the European Conference on Computer Vision, 11211 (2018), 833–851. https://doi.org/10.1007/978-3-030-01234-2_49
[20]	C. E. Akbas, M. Kozubek, Condensed U-Net (Cu-Net): An improved u-net architecture for cell segmentation powered by 4×4 max-pooling layers, in Proceedings of the 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI), (2020), 446–450. https://doi.org/10.1109/ISBI45749.2020.9098351
[21]	C. E. Akbaş, M. Kozubek, Weakly supervised multi-task learning for cell detection and segmentation, in Proceedings of the 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI), (2020), 513–516. https://doi.org/10.1109/ISBI45749.2020.9098518
[22]	X. Zhang, X. Zhu, K. Tang, Y. Zhao, Z. Lu, Q. Feng, DDTNet: A dense dual-task network for tumor-infiltrating lymphocyte detection and segmentation in histopathological images of breast cancer, Med. Image Anal., 78 (2022), 102415. https://doi.org/10.1016/j.media.2022.102415 doi: 10.1016/j.media.2022.102415
[23]	H. Wang, Y. Jiang, B. Li, Y. Cui, R. Li, Single-cell spatial analysis of tumor and immune microenvironment on whole-slide image reveals hepatocellular carcinoma subtypes, Cancers, 12 (2020), 3562. https://doi.org/10.3390/cancers12123562 doi: 10.3390/cancers12123562
[24]	E. Budginait, M. A. Morkūnas, Laurinaviius, P. Treigys, Deep learning model for cell nuclei segmentation and lymphocyte identification in whole slide histology images, Informatica, 1 (2021), 1–18. https://doi.org/10.15388/20-INFOR442 doi: 10.15388/20-INFOR442
[25]	J. Li, K. Jin, D. Zhou, N. Kubota, Z. Ju, Attention mechanism-based cnn for facial expression recognition, Neurocomputing, 411 (2020). https://doi.org/10.1016/j.neucom.2020.06.014 doi: 10.1016/j.neucom.2020.06.014
[26]	Z. Li, Z. Peng, S. Tang, C. Zhang, H. Ma, Text summarization method based on double attention pointer network, IEEE Access, 8 (2020). 11279–11288. https://doi.org/10.1109/ACCESS.2020.2965575 doi: 10.1109/ACCESS.2020.2965575
[27]	H. Jie, S. Li, S. Gang, Squeeze-and-excitation networks, in Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), (2018), 7132–7141. https://doi.org/10.1109/CVPR.2018.00745
[28]	W. Fei, M. Jiang, Q. Chen, S. Yang, X. Tang, Residual attention network for image classification, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), (2017), 6450–6458. https://doi.org/10.1109/CVPR.2017.683
[29]	C. Yin, S. Liu, R. Shao, P. C. Yuen, Focusing on clinically interpretable features: selective attention regularization for liver biopsy image classification, in Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), 12905 (2021), 153–162. https://doi.org/10.1007/978-3-030-87240-3_15
[30]	Y. Gao, M. Zhou, D. Metaxas, UTNet: A hybrid transformer architecture for medical image segmentation, in Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), 12903 (2021), 61–71. https://doi.org/10.1007/978-3-030-87199-4_6
[31]	Z. Zhong, Z. Q. Lin, R. Bidart, X. Hu, A. Wong, Squeeze-and-attention networks for semantic segmentation, in Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), (2020), 13062–13071. https://doi.org/10.1109/CVPR42600.2020.01308
[32]	T. Y. Lin, P. Dollar, R. Girshick, K. He, B. Hariharan, S. Belongie, Feature pyramid networks for object detection, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), (2017), 936–944. https://doi.org/10.1109/CVPR.2017.106
[33]	Z. Zhou, M. Siddiquee, N. Tajbakhsh, J. Liang, Unet++: Redesigning skip connections to exploit multiscale features in image segmentation, IEEE Trans. Med. Imaging, 39 (2020), 1856–1867. https://doi.org/10.1109/TMI.2019.2959609 doi: 10.1109/TMI.2019.2959609
[34]	H. Huang, L. Lin, R. Tong, H. Hu, J. Wu, UNet 3+: A full-scale connected unet for medical image segmentation, in Proceedings of the ICASSP 2020-2020 IEEE International Conference on Acoustics, (2020), 1055–1059. https://doi.org/10.1109/ICASSP40776.2020.9053405
[35]	K. M. He, X. Y. Zhang, S. Q. Ren, J. Sun, Deep residual learning for image recognition, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), (2016), 770–778. https://doi.org/10.1109/CVPR.2016.90
[36]	C. Zhao, M. Hu, F. Ju, Z. Chen, Y. Li, Y. Feng, Convolutional neural network with spatio-temporal-channel attention for remote heart rate estimation, Visual Comput., 2022 (2022), 1–19. https://doi.org/10.1007/s00371-022-02624-w doi: 10.1007/s00371-022-02624-w
[37]	A. Janowczyk, A. Madabhushi, Deep learning for digital pathology image analysis: A comprehensive tutorial with selected use cases, J. Pathol. Inf., 7 (2016), 1–18. https://doi.org/10.4103/2153-3539.186902 doi: 10.4103/2153-3539.186902
[38]	V. Badrinarayanan, A. Kendall, R. Cipolla, Segnet: A deep convolutional encoder-decoder architecture for image segmentation, IEEE Trans. Pattern Anal. Mach. Intel., 39 (2017), 2481–2495. https://doi.org/10.1109/TPAMI.2016.2644615 doi: 10.1109/TPAMI.2016.2644615
[39]	A. Paszke, A. Chaurasia, S. Kim, E. Culurciello, Enet: A deep neural network architecture for real-time semantic segmentation, preprint, arXiv: 1606.02147. https://doi.org/10.48550/arXiv.1606.02147
[40]	M. Z. Alom, C. Yakopcic, M. Hasan, T. M. Taha, V. K. Asari, Recurrent residual u-net for medical image segmentation, J. Med. Imaging, 6 (2019), 1–16. https://doi.org/10.1117/1.JMI.6.1.014006 doi: 10.1117/1.JMI.6.1.014006
[41]	Y. Wu, W. Cao, Y. Liu, Z. Ming, J. Li, B. Lu, Semantic auto-encoder with l2-norm constraint for zero-shot learning, in 2021 13th International Conference on Machine Learning and Computing, (2021), 101–105. https://doi.org/10.1145/3457682.3457699
[42]	F. Li, Y. Zhao, Y. Wei, Y. Xi, H. Bu, Tumor-infiltrating lymphocytes improve magee equation–based prediction of pathologic complete response in HR-Positive/HER2-Negative breast cancer, Am. J. Clin. Oncol., 158 (2022), 291–299. https://doi.org/10.1093/ajcp/aqac041 doi: 10.1093/ajcp/aqac041
[43]	K. M. Ratheesh, L. K. Seah, V. M Murukeshan, Spectral phase-based automatic calibration scheme for swept source-based optical coherence tomography systems, Phys. Med. Biol., 61 (2016) 7652–7663. https://doi.org/10.1088/0031-9155/61/21/7652 doi: 10.1088/0031-9155/61/21/7652
[44]	R. K. Meleppat, C. R. Fortenbach, Y. Jian, K. Wagner, B. S. Modjtahedi, M. J. Motta, et al., In Vivo imaging of retinal and choroidal morphology and vascular plexuses of vertebrates using swept-source optical coherence tomography, Transl. Vision Sci. Technol., 11 (2022), 1–21. https://doi.org/10.1167/tvst.11.8.11 doi: 10.1167/tvst.11.8.11
[45]	P. Udayaraju, P. Jeyanthi, Early diagnosis of age-related macular degeneration (ARMD) using deep learning, Intell. Syst. Sustainable Comput., 289 (2022), 657–663. https://doi.org/10.1007/978-981-19-0011-2_59. doi: 10.1007/978-981-19-0011-2_59

Reader Comments

Your name:*

Email:*
© 2023 the Author(s), licensee AIMS Press. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0)