A half jaw panoramic stitching method of intraoral endoscopy images based on dental arch arrangement

Tian Ma; Boyang Meng; Jiayi Yang; Nana Gou; Weilu Shi; Tian Ma; Boyang Meng; Jiayi Yang; Nana Gou; Weilu Shi

doi:10.3934/mbe.2024022

Mathematical Biosciences and Engineering

2024, Volume 21, Issue 1: 494-522. doi: 10.3934/mbe.2024022

Previous Article Next Article

Research article Special Issues

A half jaw panoramic stitching method of intraoral endoscopy images based on dental arch arrangement

College of Computer Science and Technology, Xi'an University of Science and Technology, Xi'an Shaanxi 710054, China

Received: 28 October 2023 Revised: 03 December 2023 Accepted: 09 December 2023 Published: 15 December 2023

To address the challenges of repetitive and low-texture features in intraoral endoscopic images, a novel methodology for stitching panoramic half jaw images of the oral cavity is proposed. Initially, an enhanced self-attention mechanism guided by Time-Weighting concepts is employed to augment the clustering potential of feature points, thereby increasing the number of matched features. Subsequently, a combination of the Sinkhorn algorithm and Random Sample Consensus (RANSAC) is utilized to maximize the count of matched feature pairs, accurately remove outliers and minimize error. Last, to address the unique spatial alignment among intraoral endoscopic images, a wavelet transform and weighted fusion algorithm based on dental arch arrangement in intraoral endoscopic images have been developed, specifically for use in the fusion stage of intraoral endoscopic images. This enables the local oral images to be precisely positioned along the dental arch, and seamless stitching is achieved through wavelet transformation and a gradual weighted fusion technique. Experimental results demonstrate that this method yields promising outcomes in panoramic stitching tasks for intraoral endoscopic images, achieving a matching accuracy of 84.6% and a recall rate of 78.4% in a dataset with an average overlap of 35%. A novel solution for panoramic stitching of intraoral endoscopic images is provided by this method.
- image stitching,
- panoramic image generation,
- intraoral endoscopic imagery,
- attention mechanism,
- RANSAC,
- weighted Fusion
Citation: Tian Ma, Boyang Meng, Jiayi Yang, Nana Gou, Weilu Shi. A half jaw panoramic stitching method of intraoral endoscopy images based on dental arch arrangement[J]. Mathematical Biosciences and Engineering, 2024, 21(1): 494-522. doi: 10.3934/mbe.2024022

Related Papers:

Abstract

To address the challenges of repetitive and low-texture features in intraoral endoscopic images, a novel methodology for stitching panoramic half jaw images of the oral cavity is proposed. Initially, an enhanced self-attention mechanism guided by Time-Weighting concepts is employed to augment the clustering potential of feature points, thereby increasing the number of matched features. Subsequently, a combination of the Sinkhorn algorithm and Random Sample Consensus (RANSAC) is utilized to maximize the count of matched feature pairs, accurately remove outliers and minimize error. Last, to address the unique spatial alignment among intraoral endoscopic images, a wavelet transform and weighted fusion algorithm based on dental arch arrangement in intraoral endoscopic images have been developed, specifically for use in the fusion stage of intraoral endoscopic images. This enables the local oral images to be precisely positioned along the dental arch, and seamless stitching is achieved through wavelet transformation and a gradual weighted fusion technique. Experimental results demonstrate that this method yields promising outcomes in panoramic stitching tasks for intraoral endoscopic images, achieving a matching accuracy of 84.6% and a recall rate of 78.4% in a dataset with an average overlap of 35%. A novel solution for panoramic stitching of intraoral endoscopic images is provided by this method.

References

[1]	M. Tian, X. Zhou, J. Yang, B. Meng, J. Qian, J. Zhang, et al., Dental lesion segmentation using an improved ICNet network with attention, Micromachines, 13 (2022), 1920. https://doi.org/10.3390/mi13111920 doi: 10.3390/mi13111920
[2]	H. Bay, A. Ess, T. Tuytelaars, L. V. Gool, Speeded-up Robust Features (SURF), Comput. Vision Image Understand., 110 (2008), 346–359. https://doi.org/10.1016/j.cviu.2007.09.014 doi: 10.1016/j.cviu.2007.09.014
[3]	E. Rublee, V. Rabaud, K. Konolige, G. Bradski, ORB: An efficient alternative to SIFT or SURF. 2011 International Conference on Computer Vision (ICCV), (2011), 2564–2571. https://doi.org/10.1109/ICCV.2011.6126544
[4]	L. Zhu, Y. Wang, B. Zhao, X. Zhang, A fast image stitching algorithm based on improved SURF, in 2014 Tenth International Conference on Computational Intelligence and Security (CIS'2014), (2014), 171–175. https://doi.org/10.1109/CIS.2014.67
[5]	Z. Y. Zhang, L. X. Wang, W. F. Zheng, L. R. Yin, R. R. Hu, B. Yang, Endoscope image mosaic based on pyramid ORB, Biomed. Signal Process. Control, 71 (2022). https://doi.org/10.1016/j.bspc.2021.103261
[6]	S. Peter, J. Uszkoreit, A. Vaswani, Self-attention with relative position representations, in North American Chapter of the Association for Computational Linguistics (NAACL2018), (2018). https://doi.org/10.48550/arXiv.1803.02155
[7]	P. Gabriel, M. Cuturi, Computational optimal transport, Found. Trends Mach. Learn, 11 (2018), 355–607. https://doi.org/10.48550/arXiv.1803.00567 doi: 10.48550/arXiv.1803.00567
[8]	R. Raguram, O. Chum, M. Pollefeys, J. Matas, J. -M. Frahm, USAC: A universal framework for random sample consensus, IEEE Transact. Pattern Anal. Mach. Intell., 35 (2013), 2022–2038. https://doi.org/10.1109/TPAMI.2012.257 doi: 10.1109/TPAMI.2012.257
[9]	T. Bergen, T. Wittenberg, Stitching and surface reconstruction from endoscopic image sequences: A review of applications and methods, IEEE J. Biomed. Health Inform., 20 (2016), 304–321. https://doi.org/10.1109/JBHI.2014.2384134 doi: 10.1109/JBHI.2014.2384134
[10]	Y. H. Mier, W. Blondel, C. Daul, D. Wolf, G. B. Heckly, 2D panoramas from Cystoscopic image sequences and potential application to fluorescence imaging, IFAC Proceed. Volumes, 39 (2006), 291–296. https://doi.org/10.3182/20060920-3-FR-2912.00054 doi: 10.3182/20060920-3-FR-2912.00054
[11]	T. Bergen, T. Wittenberg, C. Münzenmayer, C. Chen, G. D. Hager, A graph-based approach for local and global panorama imaging in cystoscopy, in Medical Imaging 2013: Image-Guided Procedures, Robotic Interventions, and Modeling (MI2013), 8671 (2013). https://doi.org/10.1117/12.2008174
[12]	T. Ishii, Computer-based endoscopic image-processing technology for endourology and laparoscopic surgery, Int. J. Urol., 16 (2009), 533–543. https://doi.org/10.1111/j.1442-2042.2009.02258.x doi: 10.1111/j.1442-2042.2009.02258.x
[13]	T. Ishii, Three-dimensional image processing system for the ureter and urethra using endoscopic video, J. Endoutol., 22 (2008), 1569–1572. https://doi.org/10.1089/end.2008.0150 doi: 10.1089/end.2008.0150
[14]	T. Ishii, S. Zenbutsu, T. Nakaguchi, M. Sekine, Y. Naya, T. Igarashi, Novel points of view for endoscopy: Panoramized intraluminal opened image and 3D shape reconstruction, J. Med. Imag. Health Inform., 1 (2011), 13–20. https://doi.org/10.1166/jmihi.2011.1002 doi: 10.1166/jmihi.2011.1002
[15]	A. Can, C. V. Stewart, B. Roysam, H. L. Tanenbaum, A feature-based, robust, hierarchical algorithm for registering pairs of images of the curved human retina, IEEE Transact. Pattern Anal. Mach. Intell., 24 (2002), 347–364. https://doi.org/10.1109/34.990136 doi: 10.1109/34.990136
[16]	D. E. Becker, A. Can, J. N. Turner, H. L. Tanenbaum, B. Roysam, Image processing algorithms for retinal montage synthesis, mapping, and real-time location determination, IEEE Transact. Biomed. Eng., 45 (1998), 105–118. https://doi.org/10.1109/10.650362
[17]	S. Yi, J. Xie, P. Mui, J. A. Leighton, Achieving real-time capsule endoscopy (CE) video visualization through panoramic imaging, in Real-Time Image and Video Processing 2013(JRTIP), 8656 (2013). https://doi.org/10.1117/12.2005243
[18]	M. Schuster, T. Bergen, M. Reiter, C. Münzenmayer, S. Friedl, T. Wittenberg, Laryngoscopic Image Stitching for View Enhancement and Documentation—First Experiences, Biomedical Engineering/Biomedizinische Technik (BMT), 57 (2012), 704–707. https://doi.org/10.1515/bmt-2012-4471 doi: 10.1515/bmt-2012-4471
[19]	H. R. Qing, G. L. Qing, X. Wen, L. Jun, Y. Hai, Y. Jun, et al., A teeth occlusal surface panoramic image stitching technique based on local optimization algorithms, China, patent, CN, CN201810428969.7, 2018, September 28.
[20]	H. R. Qing, G. L. Qing, X. Wen, L. Jun, Y. Hai, Y. Jun, et al., A teeth buccal side panoramic image stitching technique based on local optimization algorithms, China, patent, CN, CN201810435287.9, 2018, October 2.
[21]	L. Nie, L. Y. Chun, K. Liao, L. Q. Mei, Z. Yao, A view-free image stitching network based on global homography, J. Visual Commun. Image Represent., 73 (2020), 102950. https://doi.org/10.1016/j.jvcir.2020.102950 doi: 10.1016/j.jvcir.2020.102950
[22]	L. Nie, C. Lin, K. Liao, S. Liu, Y. Zhao, Unsupervised deep image stitching: Reconstructing stitched features to images, IEEE Transact. Image Process., 30 (2021), 6184–6197. https://doi.org/10.1109/TIP.2021.3092828 doi: 10.1109/TIP.2021.3092828
[23]	D. D. Tone, D. T. Malisiewicz, A. Rabinovich, SuperPoint: Self-supervised interest point detection and description, in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), (2017), 33712. https://doi.org/10.48550/arXiv.1712.07629
[24]	P. E. Sarlin, D. D. Tone, T. Malisiewicz, A. Rabinovich, SuperGlue: Learning feature matching with graph neural networks, in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), (2019), 4937–4946. https://doi.org/10.48550/arXiv.1911.11763
[25]	X. X. Yang, Y. S. Shan, W. Jun, D. Y. Ping, An image stitching method that integrates both global and local features, J. Beijing Institute Technol., 42 (2022), 9. https://doi.org/10.15918/j.tbit1001-0645.2021.093 doi: 10.15918/j.tbit1001-0645.2021.093
[26]	J. Bian, W. Y. Lin, Y. Matsushita, S. K. Yeung, T. D. Nguyen, M. M. Cheng, GMS: Grid-based motion statistics for fast, ultra-robust feature correspondence, in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), (2017), 2828–2837. https://doi.org/10.1109/CVPR.2017.302
[27]	K. M. Yi, E. Trulls, Y. Ono, V. Lepetit, M. Salzmann, P. Fua, Learning to find good correspondences, in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), (2018), 2666–2674. https://doi.org/10.1109/CVPR.2018.00282
[28]	J. Zhang, D. Sun, Z. Luo, A. Yao, L. Zhou, T. Shen, et al., Learning two-view correspondences and geometry using order-aware network, in 2019 IEEE/CVF International Conference on Computer Vision (ICCV), (2019), 5844–5853. https://doi.org/10.1109/ICCV.2019.00594
[29]	D. D. Tone, T. Malisiewicz, A. Rabinovich, Deep image homography estimation, (2016), arXiv, abs/1606.03798. https://doi.org/10.48550/arXiv.1606.03798
[30]	T. Ma, C. Fu, J. Yang, J. Zhang, C. Yang, RF-Net: Unsupervised low-light image enhancement based on retinex and exposure fusion, Comput. Mater. Cont., 77 (2023), 1103–1122. https://doi.org/10.32604/cmc.2023.042416 doi: 10.32604/cmc.2023.042416

Reader Comments

Your name:*

Email:*
© 2024 the Author(s), licensee AIMS Press. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0)