A global optimization generation method of stitching dental panorama with anti-perspective transformation

Ning He; Hongmei Jin; Hong'an Li; Zhanli Li; Ning He; Hongmei Jin; Hong'an Li; Zhanli Li

doi:10.3934/mbe.2023772

Mathematical Biosciences and Engineering

2023, Volume 20, Issue 9: 17356-17383. doi: 10.3934/mbe.2023772

Previous Article Next Article

Research article Special Issues

A global optimization generation method of stitching dental panorama with anti-perspective transformation

College of Computer Science and Technology, Xi'an University of Science and Technology, Xi'an 710054, China

Received: 04 July 2023 Revised: 05 August 2023 Accepted: 27 August 2023 Published: 08 September 2023

To address the limitation of narrow field-of-view in local oral cavity images that fail to capture large-area targets at once, this paper designs a method for generating natural dental panoramas based on oral endoscopic imaging that consists of two main stages: the anti-perspective transformation feature extraction and the coarse-to-fine global optimization matching. In the first stage, we increase the number of matched pairs and improve the robustness of the algorithm to viewpoint transformation by normalizing the anti-affine transformation region extracted from the Gaussian scale space and using log-polar coordinates to compute the gradient histogram of the octagonal region to obtain the set of perspective transformation resistant feature points. In the second stage, we design a coarse-to-fine global optimization matching strategy. Initially, we incorporate motion smoothing constraints and improve the Fast Library for Approximate Nearest Neighbors (FLANN) algorithm by utilizing neighborhood information for coarse matching. Then, we eliminate mismatches via homography-guided Random Sample Consensus (RANSAC) and further refine the matching using the Levenberg-Marquardt (L-M) algorithm to reduce cumulative errors and achieve global optimization. Finally, multi-band blending is used to eliminate the ghosting due to unalignment and make the image transition more natural. Experiments show that the visual effect of dental panoramas generated by the proposed method is significantly better than that of other methods, addressing the problems of sparse splicing discontinuities caused by sparse keypoints, ghosting due to parallax, and distortion caused by the accumulation of errors in multi-image splicing in oral endoscopic image stitching.
- panorama generation,
- image stitching,
- affine invariance,
- perspective transformation,
- global optimization
Citation: Ning He, Hongmei Jin, Hong'an Li, Zhanli Li. A global optimization generation method of stitching dental panorama with anti-perspective transformation[J]. Mathematical Biosciences and Engineering, 2023, 20(9): 17356-17383. doi: 10.3934/mbe.2023772

Related Papers:

Abstract

To address the limitation of narrow field-of-view in local oral cavity images that fail to capture large-area targets at once, this paper designs a method for generating natural dental panoramas based on oral endoscopic imaging that consists of two main stages: the anti-perspective transformation feature extraction and the coarse-to-fine global optimization matching. In the first stage, we increase the number of matched pairs and improve the robustness of the algorithm to viewpoint transformation by normalizing the anti-affine transformation region extracted from the Gaussian scale space and using log-polar coordinates to compute the gradient histogram of the octagonal region to obtain the set of perspective transformation resistant feature points. In the second stage, we design a coarse-to-fine global optimization matching strategy. Initially, we incorporate motion smoothing constraints and improve the Fast Library for Approximate Nearest Neighbors (FLANN) algorithm by utilizing neighborhood information for coarse matching. Then, we eliminate mismatches via homography-guided Random Sample Consensus (RANSAC) and further refine the matching using the Levenberg-Marquardt (L-M) algorithm to reduce cumulative errors and achieve global optimization. Finally, multi-band blending is used to eliminate the ghosting due to unalignment and make the image transition more natural. Experiments show that the visual effect of dental panoramas generated by the proposed method is significantly better than that of other methods, addressing the problems of sparse splicing discontinuities caused by sparse keypoints, ghosting due to parallax, and distortion caused by the accumulation of errors in multi-image splicing in oral endoscopic image stitching.

References

[1]	Y. Yan, L. Xu, Y. Y. Liu, X. Su, J. L. Gao, M. X. Wan, Larynx ultrasound image stitching based on multiconstraint super-pixel feature, IEEE Trans. Instrum. Meas., 72 (2023), 1–11. https://doi.org/10.1109/TIM.2023.3243675 doi: 10.1109/TIM.2023.3243675
[2]	M. Ghanoum, A. M. Ali, S. Elshazly, I. Alkabbany, A. A. Farag, Frame stitching in human oral cavity environment using intraoral camera, in IEEE International Conference on Image Processing (ICIP), (2019), 1327–1331. https://doi.org/10.1109/ICIP.2019.8803071
[3]	M. Ghanoum, A. M. Ali, S. Elshazly, I. Alkabbany, A. A. Farag, Panoramic view of human jaw under ambiguity intraoral camera movement, in IEEE 17th International Symposium on Biomedical Imaging (ISBI), (2020), 1–4. https://doi.org/10.1109/ISBI45749.2020.9098562
[4]	T. Jin, F. Z. Li, H. J. Peng, B. Li, D. P. Jiang, Uncertain barrier swaption pricing problem based on the fractional differential equation in Caputo sense, Soft Comput., 27 (2023), 11587–11602. https://doi.org/10.1007/s00500-023-08153-5 doi: 10.1007/s00500-023-08153-5
[5]	T. Jin, H. X. Xia, Lookback option pricing models based on the uncertain fractional-order differential equation with Caputo type, J. Ambient Intell. Human. Comput. (JAIHC), 14 (2023), 6435–6448. https://doi.org/10.1007/s12652-021-03516-y doi: 10.1007/s12652-021-03516-y
[6]	C. W. Shen, X. Y. Ji, C. L. Miao, Real-Time image stitching with convolutional neural networks, 2019 IEEE International Conference on Real-time Computing and Robotics (RCAR), (2019), 192–197. https://doi.org/10.1109/RCAR47638.2019.9044010 doi: 10.1109/RCAR47638.2019.9044010
[7]	L. Nie, C. Y. Lin, K. Liao, M. Q. Liu, Y. Zhao, A view-free image stitching network based on global homography, J. Vis. Comun. Imag. R, 73 (2019), 102950. https://doi.org/10.1016/j.jvcir.2020.102950 doi: 10.1016/j.jvcir.2020.102950
[8]	L. Nie, C. Y. Lin, K. Liao, Y. Zhao, Learning edge-preserved image stitching from multi-scale deep homography, Neurocomputing, 491 (2022), 533–543. https://doi.org/10.1016/j.neucom.2021.12.032 doi: 10.1016/j.neucom.2021.12.032
[9]	D. Y. Song, G. M. Um, H. K. Lee, D. Cho, End-to-end image stitching network via multi-homography estimation, IEEE Signal Process Lett., 28 (2021), 763–767. https://doi.org/10.1109/LSP.2021.3070525 doi: 10.1109/LSP.2021.3070525
[10]	D. Y. Song, G. Lee, H. K. Lee, G. M. Um, D. Cho, Weakly-Supervised stitching network for real-world panoramic image generation, Lect. Notes Comput. Sci., 13676 (2022), 54–71. https://doi.org/10.1007/978-3-031-19787-1_4 doi: 10.1007/978-3-031-19787-1_4
[11]	L. Nie, C. Y. Lin, K. Liao, S. C. Liu, Y. Zhao, Unsupervised deep image stitching: Reconstructing stitched features to images, IEEE Trans. Image Process, 30 (2021), 6184–6197. https://doi.org/10.1109/TIP.2021.3092828 doi: 10.1109/TIP.2021.3092828
[12]	J. R. Zhang, C. Wang, S. C. Liu, L. P. Jia, N. J. Ye, J. Wang, et al., Content-Aware unsupervised deep homography estimation, Lect. Notes Comput. Sci., 12346 (2020), 653–669. https://doi.org/10.1007/978-3-030-58452-8_38 doi: 10.1007/978-3-030-58452-8_38
[13]	S. C. Liu, N. J. Ye, C. Wang, J. R. Zhang, L. P. Jia, K. M. Luo, et al., Content-Aware unsupervised deep homography estimation and its extensions, IEEE Trans. Pattern Anal. Mach. Intell., 45 (2022), 2849–2863. https://doi.org/10.1109/TPAMI.2022.3174130 doi: 10.1109/TPAMI.2022.3174130
[14]	R. Huang, Q. Chang, Y. Zhang, Unsupervised oral endoscope image stitching algorithm, J. Shanghai Jiaotong University (Science), (2022), 1–10. https://doi.org/10.1007/s12204-022-2513-7 doi: 10.1007/s12204-022-2513-7
[15]	S. Wang, F. Y. Yuan, B. Chen, H. F. Jiang, W. Q. Chen, Y. Wang, Deep homography estimation based on attention mechanism, in 2021 7th International Conference on Systems and Informatics (ICSAI), (2021), 1–6. https://doi.org/10.1109/ICSAI53574.2021.9664027
[16]	L. David, Distinctive image features from scale-invariant keypoints, Int. J. Comput. Vis., 60 (2004), 91–110. https://doi.org/10.1023/B:VISI.0000029664.99615.94 doi: 10.1023/B:VISI.0000029664.99615.94
[17]	H. Bay, T. Tuytelaars, L. Van Gool, SURF: speeded up robust features, Lect. Notes Comput. Sci., 3951 (2006), 404–417. https://doi.org/10.1007/11744023_32 doi: 10.1007/11744023_32
[18]	E. Rublee, V. Rabaud, K. Konolige, ORB: An efficient alternative to SIFT or SURF, in 2011 International Conference on Computer Vision (ICCV), (2011), 2564–2571. https://doi.org/10.1109/ICCV.2011.6126544
[19]	G. F. Zhang, Y. He, W. F. Chen, J. Y. Jia, H. J. Bao Multi-viewpoint panorama construction with wide-baseline images, IEEE Trans. Image Process, 25 (2016), 3099–3111. https://doi.org/10.1109/TIP.2016.2535225 doi: 10.1109/TIP.2016.2535225
[20]	J. W. Bian, W. Y. Lin, Y. Liu, L. Zhang, S. K. Yeung, M. M. Cheng, et al., GMS: Grid-Based motion statistics for fast, ultra-robust feature correspondence, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), (2017), 4181–4190. https://doi.org/10.1109/CVPR.2017.302
[21]	R. Sprengel, K. Rohr, H.S. Stiehl, Thin-plate spline approximation for image registration, in Proceedings of 18th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, (1996), 1190–1191. https://doi.org/10.1109/IEMBS.1996.652767
[22]	K. Rohr, H. S. Stiehl, R. Sprengel, W. Beil, T. M. Buzug, J. Weese, et al., Point-based elastic registration of medical image data using approximating thin-plate splines, Lect. Notes Comput. Sci., 1131 (1996), 297–306. https://doi.org/10.1007/BFb0046967 doi: 10.1007/BFb0046967
[23]	J. Li, Z.M. Wang, S.M. Lai, Y. P. Zhai, M. J. Zhang, Parallax-tolerant image stitching based on robust elastic warping, IEEE Trans. Mult., 20 (2018), 1672–1687. https://doi.org/10.1109/TMM.2017.2777461 doi: 10.1109/TMM.2017.2777461
[24]	L. Nie, C. Y. Lin, K. Liao, S. C. Liu, Y. Zhao, Learning Thin-Plate spline motion and seamless composition for parallax-tolerant unsupervised deep image stitching, Comput. Vision Pattern Recogn. (CVPR), (2023), https://doi.org/10.48550/arXiv.2302.08207 doi: 10.48550/arXiv.2302.08207
[25]	M. Brown, D. G. Lowe, Automatic panoramic image stitching using invariant features, Int J Comput Vis, (2007), 59–73. https://doi.org/10.1007/s11263-006-0002-3 doi: 10.1007/s11263-006-0002-3
[26]	J. H. Gao, S. J. Kim, M. S. Brown, Constructing image panoramas using dual-homography warping, Comput. Vision Pattern Recogn. (CVPR), (2011), 40–56. https://doi.org/10.1109/CVPR.2011.5995433 doi: 10.1109/CVPR.2011.5995433
[27]	W. Y. Lin, S. Y. Liu, Y. Matsushita, T. T. Ng, L. F. cheong, Smoothly varying affine stitching, Comput. Vision Pattern Recogn. (CVPR), (2011), 345–352. https://doi.org/10.1109/CVPR.2011.5995314 doi: 10.1109/CVPR.2011.5995314
[28]	J. Zaragoza, T. J. Chin, M. S. Brown, D. Suter, As-projective-as-possible image stitching with moving DLT, Comput. Vision Pattern Recogn. (CVPR), (2013), 2339–2346. https://doi.org/10.1109/CVPR.2013.303 doi: 10.1109/CVPR.2013.303
[29]	C. H. Chang, Y. Sato, Y. Y. Chuang, Shape-preserving half-projective warps for image stitching, Comput. Vision Pattern Recogn. (CVPR), (2014), 3254–3261. https://doi.org/10.1109/CVPR.2014.422 doi: 10.1109/CVPR.2014.422
[30]	C. C. Lin, S. Pankanti, K. Ramamurthy, A. Aravkin, Adaptive as-natural-as-possible image stitching, Comput. Vision Pattern Recogn. (CVPR), (2015), 1155–1163. https://doi.org/10.1109/CVPR.2015.7298719 doi: 10.1109/CVPR.2015.7298719
[31]	Y. S. Chen, Y. Y. Chuang, Natural Image Stitching with the Global Similarity Prior, Lect. Notes Comput. Sci., 9909 (2016), 186–201. https://doi.org/10.1007/978-3-319-46454-1_12 doi: 10.1007/978-3-319-46454-1_12
[32]	T. Z. Xiang, G. S. Xia, L. P. Zhang, N. N. Huang, Locally warping-based image stitching by imposing line constraints, in 2016 23rd International Conference on Pattern Recognition (ICPR), (2016), 4178–4183. https://doi.org/10.1109/ICPR.2016.7900289
[33]	C. He, J. Zhou, Mesh-based image stitching algorithm with linear structure protection, J. Image Graph., (2018), 973–983. https://doi.org/10.11834/jig.170653 doi: 10.11834/jig.170653
[34]	P. Du, J. F. Ning, J. G. Cui, S. L. Huang, X. C. Wang, J. X. Wang, Geometric Structure Preserving Warp for Natural Image Stitching, in 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), (2022), 3688–3696 https://doi.org/10.1109/CVPR52688.2022.00367

Reader Comments

Your name:*

Email:*
© 2023 the Author(s), licensee AIMS Press. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0)