A KD-tree and random sample consensus-based 3D reconstruction model for 2D sports stadium images

1.   Introduction

Due to the disconnection between the long-term construction and structure, the design of sports venues and pavilions often comes before the structure or structure. Therefore, it is necessary to promote the integration of stadium design and structure, and the integration of stadium form, space and structure technology ^[1]. With the progress of science and technology and the rapid development of the economy, the popularization of multimedia capture equipment, the internet is producing tens of thousands of pictures all the time, and these pictures are becoming more and more the main way for people to perceive and understand the world ^[2].

For a long time, people like to know the world through their eyes and record what they see by painting or taking pictures, because images are informative and easy to obtain. Vision is a crucial tool for humans because it helps them feel and comprehend their surroundings ^[3]. The field of computer science, of course, has a lot to do with vision, namely, computer vision. Computer vision is used to simulate human vision through the machine to obtain external information, and through computer data processing, and finally give similar visual feedback to the human visual system to help or replace the human perception of the outside world. However, the world that humans perceive through their eyes is three-dimensional, while the images that can be acquired by machines are two-dimensional ^[4].

At present, the research results of 3D reconstruction have been widely used in the fields of cultural relic protection, architectural design, e-commerce, military operations and AR maps, making the development of the industry more diversified and people's life more convenient. In terms of technology, computer hardware and software go hand in hand, with breakthrough development in operation efficiency, information processing ability and other aspects. Related reconstruction algorithms and processes are also more perfect, providing infinite possibilities for the future. The emergence of deep learning provides a new possibility for the field of 3D reconstruction, and image-based reconstruction takes on a new look ^[5]. Inspired by the principle of human vision, based on the better computer hardware level and a large amount of data support, 3D reconstruction and deep learning methods are combined to reduce the complex process of data calibration and mathematical operation, and the 3D model of objects can be directly reconstructed from a single or multiple two-dimensional images.

Extracting data from two-dimensional images to reconstruct a three-dimensional model of a sports stadium scene, mostly using stereo vision, Time of Flight (ToF) and structured light methods. The following is a comparison of the advantages and disadvantages of these methods: This method can accurately calculate depth information by analyzing the differences between two or more images, and then reconstruct a 3D model. In addition, the stereo vision method has the advantage of low sensitivity to light and texture. The effectiveness of this method largely depends on the baseline length, which is often difficult to adjust. In addition, stereo vision methods have weak processing ability for mixed pixels, which may lead to inaccurate depth information. Not limited by baseline length, independent of texture, fast imaging speed. This method obtains distance information by measuring the flight time interval between the transmitted signal and the received signal, and then reconstructs a three-dimensional model. As it measures flight time, it is not limited by the baseline length and is independent of texture information. In addition, the imaging speed of the ToF method is usually faster. The resolution of the ToF method is limited by hardware devices and is usually low. In addition, this method is susceptible to environmental factors such as mixed pixels, external light sources, etc., resulting in inaccurate depth information. Furthermore, system error and random error also have a significant impact on the measurement results.

In the research of computer vision, most methods only focus on the generation of two-dimensional images and ignore the nature of the three-dimensional world ^[6]. Thus, something that seems obvious to humans is difficult for computers ^[7]. As long as a simple two-dimensional image is input and the number of images is not specified, the reconstruction result can be quickly obtained ^[8]. Moreover, in order to enhance the generalization of the model and expand the scope of application, some methods such as dropout are also used ^[9]. Image-based 3D modeling methods are divided into multi-view image modeling and single-view image modeling according to the different number of input images ^[10]. Based on the 3D reconstruction of images, the professional theory of computer vision and graphics can be fully used to restore the 3D model from a single or multiple images, even video ^[11]. Moreover, image-based modeling techniques can be used not only for the reconstruction of small objects, but also for the reconstruction of large outdoor scenes ^[12]. Because of its low image requirements, it can use pictures taken by common cameras and mobile phones, so the cost of reconstruction on the device is also greatly reduced ^[13].

Therefore, image-based modeling technology has rapidly gained a lot of research and development, which is mainly used in appearance inspection and machine navigation ^[14]. According to the relevant theories of computer vision, an image is a two-dimensional image information obtained by optical projection transformation of the object in the real scene through illumination. As a result, there are a lot of visual information in the image, such as brightness, shape and texture, and based on image modeling technology is the study of visual information, the combination of camera parameters and illumination condition information on the inverse transformation of optical projection operations. Thus, the image of a two-dimensional visual information is retrieved into reality scene 3D information.

Among them, the motion-based modeling method is to use the feature information detected and matched in the image, and use numerical methods to recover the three-dimensional information of camera parameters and matching information. This method has low image requirements, and various feature information extraction and matching methods have been developed, so it is very suitable for large-scale outdoor scenes, especially for the reconstruction of sports venues. In summary, I take two-dimensional images as the main body, optimizes the existing image data acquisition methods and proposes a more convenient and efficient reconstruction method for three-dimensional models. This method can well solve the common problems in the process of 3D re-modeling of sports venues, so as to get accurate 3D models of complex stadiums.

In this article, I mainly study how to extract data from two-dimensional images to reconstruct a three-dimensional model of a sports stadium scene. This model has the following contributions:

1) I provide a new method for 3D model reconstruction. This study proposes a new method that can extract data from two-dimensional images to reconstruct a three-dimensional model of a sports stadium scene.

2) The method proposed in this article can overcome the limitations of traditional 3D modeling methods and more accurately reconstruct the 3D model of the stadium scene.

3) I enhance the visualization effect of the model. By reconstructing the three-dimensional model of the stadium scene, I can provide more intuitive and realistic renderings, thereby better displaying the characteristics and details of the stadium scene.

Section 1 provides a background description of the generation of two-dimensional images in computer vision research. Section 2 elaborates on the data collection and preprocessing process. The feature parameters were extracted and optimized for search and matching using the nearest neighbor query algorithm based on KD tree. Section 3 analyzed the 3D model reconstruction method and triangulated the 2D image. The input 2D image features were obtained through a feature extraction network. Section 4 summarizes the entire text. The error of the 3D model reconstruction method proposed in this study is significantly lower than that of traditional methods, proving its applicability in sports field scenes.

2.   Data acquisition and preprocessing

2.1.   Extraction of feature parameters

To recover the pose of a camera from multiple images, it is necessary to establish the matching pixels in multiple images. Direct pairwise matching for each pixel will result in a very massive computation. The best strategy to strike a compromise between effectiveness and accuracy is to choose a few distinctive pixels that stand out from the surrounding pixels enough to be considered representative. These points—often referred to as feature points or corner points–are the feature points to be used for matching. The corner point can be defined as the point where the gray level in the neighborhood is compared with the pixels in each direction, and the gray level gradient meets certain conditions. The feature parameters in this research are extracted using the OFTR technique.

The key point of the OFTR feature extraction algorithm is an improved FAST corner with orientation information, and the description point is rBRIEF with rotation characteristics. The algorithm first extracts feature points quickly by oFAST and then uses BRIEF to construct feature description at corresponding feature points. FAST extraction steps are as follows:

1) From the image to select a pixel point S, assume that the color information is l.

2) Set a threshold W.

3) There are 25 pixels on the discretized circle with radius 3 centered on the pixel.

4) In the process of information extraction, a filtering operation is usually done on the neighborhood pixels in advance to quickly exclude most pixels that are not corner points. It is directly compared to the first, fifth, ninth and thirteenth pixels on the neighborhood circle. Only when three of the four pixels have gray values greater than L + W or less than L – W is the current pixel likely to be a corner; otherwise, it is excluded directly.

In order to improve the efficiency of feature point extraction, a box filter is used to optimize the second-order differential template. In each dimension of each pixel point for testing at the same time, I can get the value of the matrix and the matrix determinant:

where ${D}_{xx}$ and ${D}_{xy}$ are the approximate convolution values obtained by the box filter. Whether a pixel is a candidate key point depends on whether $c\left(x, y, \sigma \right)$ is greater than the preset threshold. Select 5 different groups of stadium images, each group of 30 images, and conduct the experiment. Partial results are shown in Figure 1 and Table 1.

The first category is the venues with simple structure and obvious architectural features, namely, the basketball arena and the gymnastics arena. Many feature points can be detected in these two scenes, because there are fewer reflective facilities in these two scenes, and the overall building presents a closed character. The external structure does not have too much complex construction, and more matching pairs can be obtained in the matching process, so more time is consumed in the running time. Additionally, the second type of scene is reflective facilities more stadiums, mainly swimming pool and ice hockey arena these two scenes.

As such venues contain more sunshades or smooth top surfaces, there are not many feature points to be obtained in the picture, so the number of matching pairs is not as large as that of the first type scene, and the running time is also shorter. The third type of scene is the open-air stadium, mainly represented by the athletics tube. Due to the complex structure of this kind of scene, it is not a simple closed structure, which contains more sports facilities and divided areas. As a result, the five scenarios might yield the most feature points when it comes to feature point detection. Due to the numerous interference objects, however, the number of matching pairs is low, and the operation time is comparable to that of the second category of venues.

2.2.   Cipal algorithm

There are numerous ways to match feature points at the moment, which can be loosely split into two categories: Exhaustive search methods and index matching methods. In this study, the search and matching of feature parameters are optimized using the nearest neighbor query algorithm based on KD-tree. The basic concept of KD-tree comes from binary Tree M, which is often used when searching and querying data in high-dimensional space. Using the concept of the binary tree, all points are distributed on the nodes of the tree. Using depth-first distancing, the search starts at the root of the tree and works its way down. Figure 2 shows a simple KD-tree construction process. The calculation process is shown in Figure 3.

In general, the classified data are sorted and the median is selected as the decision point. Corresponding to the KD-tree, the point is taken as the parent node and the remaining data is divided into the left and right subtree Spaces. However, for the conventional matching algorithm, most matching pairs have the same slope of the connecting line, which is prone to mismatching feature pairs. As a result, the accuracy of the position obtained by calculation is low and the problem of position estimation failure is easy to occur. Three types of sports venues discussed in Section 2.1 were selected for experiments, and 30 images were selected for each scene. Feature points were extracted from all images and matched in pairs. STT algorithm and SURF algorithm were used for experiments. The experimental results are shown in Figure 4.

2.3.   Mapping of feature parameters

The process of assigning a real texture to the model mainly includes the determination of texture mapping function and the selection of texture image. The best image from among several tilted images taken in the same area is chosen for texture mapping when the mapping function between texture coordinates and object coordinates is developed ^[15]. Texture coordinates (μ, ν) in image coordinate system need to be transformed into rectangular space coordinates (x, y, z) in object coordinate system. Confirm that the texture mapping function ƒ satisfies the equation:

The projection relation is:

The camera's calibration parameter matrix. The $x$ and $y$ are, respectively, the camera's horizontal and vertical focal lengths. The major picture spots' displacement in the horizontal and vertical directions are denoted by ${l}_{x}$ and ${l}_{y}$, respectively. $s$ is the distortion factor. The matrix of outside factors is called $v$.

2.4.   Test and results

To evaluate whether the extracted data can be used to build a 3D model, IoU value is introduced. IoU is defined as the following equation:

where T is voxel threshold, the I (.) is the indicator function, 𝑦 (𝑖, 𝑗, 𝑘) is the ground really solid element, 𝑝 (𝑖, 𝑗, 𝑘) is predicted occupied voxel, the greater the value of ious, on behalf of the reconstruction results, the better ^[16]. The test results of the suggested two-dimensional image data collecting method on various datasets of stadium facilities are mostly shown in Table 2. The table shows that the algorithm in this research distributes the majority of the data with the best IoU performance, demonstrating that my improved data collecting method is appropriate for 3D model reconstruction.

3.   3D model reconstruction method

3.1.   Triangulation of two-dimensional images

The camera parameters of the reconstructed image through the projection relationship from the image matching, as shown in Figure 5. After extracting features between image pairs and matching feature points, the matching relationship of feature points between each pair of images is obtained ^[17]. For binocular stereo vision, since only the position relationship of the camera when the camera shoots the image is considered, the projection matrices of the two cameras are:

In this scenario, K stands for the camera's internal parameter, R and t for its external parameter and I for its identity matrix. The 3D coordinate information of the corresponding space points of all matching point pairs in the image sequence is calculated, and the 3D information of the 3D reconstructed object can be obtained through the 3D point information ^[18].

3.2.   Model discretization

Before the reconstruction of 3D model, it is necessary to discretize the model. Each region after discretization is a surface, which is represented by S. The center point of the region is c (s), and the through-center point of the whole surface piece S is represented by N (s), as shown in Figure 6. The projection of the discretized surface piece on the picture is represented by c (i, j) ^[19]. When constructing the discrete model, two conditions must be met:

1) Local luminosity consistency: The projection of any discrete region S is consistent in at least Y images;

2) Global visibility consistency constraint, the sub-region will not be occluded by the remaining discrete regions in other images ^[20].

The experiment was carried out on sports venues. Three scenes, namely natatorium, basketball stadium and track and field hall, were selected, and 10, 15, 20, 25, 30, 35 and 40 pictures were selected, respectively, for model discretization construction and analysis. Table 3 displays the experimental outcomes for the obtained discrete number of regions.

The image shows that as the number of photos increases, so do the number of distinct locations in each of the three gymnasium scenes, and that as the number of photos increases, so does the panoramic range of the scene that has been recreated. The track and field hall is the scene with the least number of discrete areas among the three scenes. The fundamental reason is that there are many interfering objects in the track and field, and the shooting range is large while the target is too small. As a result, it is not easy to meet the requirements of local photometric consistency and global visibility consistency in the construction of discretization. For the other two scenes, most of the pictures are actual scenes of the building within the shooting range.

Since the shooting method of the natatorium is parallel to the building roof, and the shooting method of the basketball stadium is 90 degrees around the stadium, the overall visibility consistency is. More discrete areas in the natatorium will not be blocked by the discrete areas in other pictures, and the basketball stadium scene is slightly worse than the natatorium scene. In order to optimize the discretization of the model, the grid regions after discretization are further screened, which mostly contain two types ^[21].

1) When the outer point is outside the real surface, it is assumed that there is a discrete region U containing an incorrect matching ${p}_{0}$. If the following equation is satisfied, the matching ${p}_{0}$ is removed.

2) When the outer point is inside the real surface, the values of $S\left({p}_{0}\right)$ and $T\left({p}_{0}\right)$ are calculated for each ${p}_{0}$ point.

The previous gymnasium scene was screened, and the experimental results are shown in Figure 7. As can be seen from the figure, sparse 3D models can be transformed into dense 3D models to a certain extent using the screening method. In the three scenarios, continuous screening can continuously improve the experimental accuracy. According to the results, the scene with the largest number of newly added patches after screening is the natatorium, followed by the basketball stadium and finally the track and field stadium.

3.3.   Overall architecture of 3D grid

The feature extraction network of the model in this chapter mainly aims to obtain input two-dimensional image features, mainly using VGG16 network model (as shown in Figure 8). Different from that, the feature extraction network in this chapter does not use the complete VGG16 network, but removes the last fully connected layer of the network and uses only the network model from CONV1 to CONV5 to obtain the required two-dimensional image information. The image elements are extracted from the Conv3_3, Conv4_3 and Conv5_3 layers, respectively, and connected to provide the image collection features for the cascade deformation mesh ^[22]. This section mainly improves the structure of VGG16 feature extraction network. The improvement inspiration comes from the fact that U-NET network can fuse shallow and deep image feature information through jumping structure. Add a new network branch to the original VGG16 network.

Increased after the branch, the image can be shallow and deep information fusion, image characteristics are reused and can increase the image characteristics of information, and for lattice deformation level networking information network to provide more accurate image features, the reconstruction model has better performance in detail ^[23]. The branch added in this section is similar to the "Add" operation in ResNet network, which is to superimpose the information values before and after the network to enrich the information describing image features, while the number of channels of the model will not increase, so as to ensure that the number of channels and image size of the feature extraction network in the original experiment are consistent.

The same dataset provided in Section 2.4 is utilized to verify the method's logic, which contains 10 object categories and a total of more than 51,000 data models. The evaluation is mainly carried out by the soil moving distance, which is mainly used to measure the distance between two vertices ^[24]. The formula is as follows:

In 3D reconstruction, it is used to compare the similarity between predicted points and real points. The smaller the EMD, the higher the similarity between vertices and the better the reconstruction effect. EMD should focus on calculating the shortest distance between points of one-to-one correspondence between vertices to better measure the uniformity of vertex distribution ^[25]. The experiments in this chapter are based on Ubantu16.04 operating system and python3.6+Tensorflow1.12.0+cuda9.2 environment. The test results are shown in Table 4.

The mesh reconstruction method in this study is superior to the method used before reconstruction, as evidenced by the EMD values on various types of objects.

3.4.   Evaluation and quantification of the reconstruction model

Thirty images of the track and field hall were modeled in order to assess the validity and benefits of the model reconstruction process used in the gymnasium scenario ^[26]. The comparison results for all indices with various model reconstruction procedures are displayed in Table 5. This phenomenon verifies the applicability of the proposed 3D model reconstruction method in the stadium scene. Furthermore, Figure 9 lists the relationship between plane extraction efficiency and image block selection size. It can be found that with the increase of image block size, the computation time decreases continuously ^[27]. This shows that the proposed method can achieve real-time extraction by using image blocks as intermediate information.

In the subjective part, 3D models and 2D photos of different gymnasium scenes were provided respectively, and 40 investigators were asked to make choices, and the reconstruction results of different scenes were scored on a 10-point scale ^[28]. The results shown in Figure 10 show that the researchers gave relatively high ratings to the reconstructed model, indicating a better overall restoration level. However, some researchers are not satisfied with the reconstruction of details and have proposed suggestions for reducing the degree. For example, in a swimming pool, due to the large fence in front of the pool, the image of the fence is a repetitive unit, and there are many processes for feature point matching. Therefore, the results of reconstructing the information as part of the well front swimming pool indicate that the degree of score reduction is not very high. ^[29].

However, the recovered scene was able to tell me 100% that it was the scene of the swimming pool reconstruction. The gymnasium also has the same problem as the natatorium, because the gymnasium is more crowded, so it has a strong interference effect on the reconstruction result. By calculating the number of restored 3D point clouds in unit volume, the reconstruction model is quantified and evaluated, so as to determine the accuracy of restoration ^[30]. The number of 3D reconstructed point clouds in a unit volume is evaluated, and the test results are shown in Table 6.

It can be said that the ice hockey arena has the fewest 3D point clouds per unit volume, whereas the basketball arena has the most reconstructed 3D point clouds. According to the scene discussion, when the scene structure is simple, the building has obvious features, and there is no repetitive pattern because there is no occlusion interference, so the reconstruction effect is the best. Moreover, it can be found that because there are more reflective facilities in the ice hockey arena. In general, the reconstruction results of each scene are subjective and good. The reason why the value of gymnastics hall and basketball hall is higher than other buildings is that the two buildings are of the same architectural style. The roof of the gymnastics hall and basketball hall is a shell-like roof, so the reconstruction effect is good, and there are more three-dimensional point clouds in the unit volume of this part. When the scene is an open-air building, such as a track and field hall, the reconstruction result of the bottom surface is not good due to the great interference of the light and the track.

Based on the generated 3D model, spatial analysis and visualization can be performed. This can help venue managers comprehensively understand the structure and characteristics of the venue, such as detecting quality issues in the structure and analyzing audience flow patterns. By utilizing 3D model data, various functional extensions can be achieved. For example, model data can be used for venue maintenance and management, such as detecting parts that need to be repaired, calculating parts that need to be replaced and so on. Model data can be used for venue renovation planning, such as simulating new building schemes and evaluating the impact of the renovation. Model data can also be used to optimize audience seat allocation to improve the viewing experience of the audience.

3.5.   Inspection methods

The learning rate is a key parameter in the training process of neural networks, which determines the degree to which the model updates weights during each iteration. If the learning rate is set too high, it may cause the model to oscillate or fail to converge during the optimization process; if the learning rate is set too low, it may lead to slow model training speed and may not reach the optimal solution. This may cause the step size of gradient descent to be too small, resulting in slow convergence of loss values during the training process and increasing training time. The selection of learning rate is equally important in the task of constructing 3D models of sports field scenes. We usually need to find the optimal choice by trying different learning rate values, which can be achieved by trying different learning rate values during the training process and observing the training effect of the model. In this article, I evaluate whether the selection of learning rate is appropriate by observing the loss value change curve and the accuracy change curve of the model during the training process. During the training process, as the number of iterations increases, the loss value of the model gradually converges, and the model parameters are finely adjusted by gradually reducing the learning rate. The test contents mostly include a spatial reference frame, position accuracy, expression fineness, logic consistency, scene effect and accessory quality. The specific inspection contents are as follows.

1) Spatial reference frame. The weight is 0.01, and the geodetic datum, elevation datum and map projection are mostly investigated. The contents include the conformity of geodetic datum adopted, the conformity of elevation datum adopted and the conformity of map projection parameters adopted.

2) Position accuracy. The weight is 0.10, and the plane accuracy and elevation accuracy are mainly investigated. The approximation degree between the elevation value and the true value of the model feature points, as well as between the plane coordinate value and the true value of the model feature points, are included in the content.

3) Expression fineness. The weight is 0.45, and the model fineness and texture fineness are mostly investigated. For the model, the contents of investigation include the model's fineness and compliance with design requirements, whether the model structure has deformation and distortion, broken surface, leakage surface and double surface, whether the components and details are complete and accurate, whether the model has adhesion and whether the connectivity of viaducts, roads and rivers is reasonable. For the texture, the inspection content includes the texture fineness and the conformity with the design requirements, and whether the texture is blurred, stretched and distorted, mapping dislocation, splicing color difference and so on.

4) Logical consistency. The weight is 0.10. The format and topology are mostly investigated from two aspects. As for the format, the contents including data organization, storage structure, data format, file naming and so on meet the requirements, as well as whether the data file is missing or redundant, and whether the data is readable. For topology, the content is the accuracy of the reflection of the topological relationship between the monomeric 3D models.

5) Scene effect. The weight is 0.25. It focuses on completeness and coordination. For completeness, the investigation includes whether the coverage of the model in the scene meets the design requirements and whether the model is redundant or omitted. For coordination, the content includes the overall color of the scene, the lighting effect, and whether there are loopholes, suspended objects, outliers, etc.

4.   Conclusions

In this article, I mainly study how to extract data from 2D images to reconstruct 3D models. Based on previous research results and related algorithms, data extraction methods and model construction methods have been optimized. The error of the 3D model reconstruction method proposed in this study is significantly lower than that of traditional methods, proving its applicability in sports field scenes. As the size of the image block increases, the calculation time also decreases. This indicates that using image blocks as intermediate information, the proposed method can achieve real-time extraction. However, once the size of the image block reaches a specific threshold, the steady decrease in computational time will stabilize. By utilizing 3D model data, various functional extensions can be achieved. For example, model data can be used for venue maintenance and management, such as detecting components that need to be repaired, calculating components that need to be replaced and so on. Model data can be used for venue renovation planning, such as simulating new building schemes and evaluating the impact of renovation. Model data can also be used to optimize audience seat allocation to improve the viewing experience.

In the context of this article, the module I propose may have a positive impact on extracting data from 2D images and reconstructing 3D models of sports field scenes. However, due to the lack of ablation experiments, it is not possible to accurately understand how this module improves the final reconstruction results. Future research can design and conduct ablation experiments to further validate and quantify the improvement of the reconstruction results by the module proposed by the author. Future research can attempt to incorporate more details and textures into 3D models to enhance their realism. This can be achieved by collecting more images or using professional 3D scanning equipment. Furthermore, it is also possible to consider how to export the model to different file formats for the convenience of researchers in other fields.

This work takes the stadium as the object scenario, and a 3D reconstruction model is investigated. On the basis of this research work, it is important to explore a possibility to use the proposed research for other topics. It is also practical to explore the utilization of new methods like neural networks in 3D reconstruction of 2D images. Certainly, the 3D reconstruction is a very classical technical method that can be employed by many business scenes. Thus, I will investigate more potential scenes and develop specific methods to deal with more application demand.

Use of AI tools declaration

I declare that I have not used Artificial Intelligence (AI) tools in the creation of this article.

Acknowledgments

This work was supported by Science and Technology Research Project of Henan Province under grant 222102320255.

Conflict of interest

I declare no conflicts of interest.