A retrieval and ranking method of mathematical documents based on CA-YOLOv5 and HFS

1.   Introduction

Mathematical documents are important media that store information about science and technology, and mathematical expressions play an indispensable role in expressing the information in such documents. Using mathematical expressions to quickly and effectively find relevant mathematical documents is an important way for scientific and technological workers to obtain necessary information. At present, mathematical document retrieval based on mathematical expressions is still faced with two key challenges in practical applications: the fast and easy input of mathematical query expressions and mathematical expression matching considering variations in syntax and semantics.

Research on the detection of mathematical expressions in document images has achieved valuable results ^[1,2,3,4,5]. Gao et al. ^[6] used the LIBSVM algorithm to classify text lines as independent expression lines and non-independent expression lines and then detected the embedded expressions and independent expressions. To address the misclassification of text lines, the team proposed a learning-based merging strategy to merge incorrectly split text lines on the basis of the projected contour cutting results. In the merging strategy, they used the layout of text lines, textual features and the features in consecutive lines to detect lines that were misclassified ^[7]. Lin et al. ^[8] combined four novel features of PDF documents and proposed a method to directly use data extracted from PDF documents to detect mathematical expressions. Gao et al. ^[9] proposed a solution based on AlexNet and Bi-LSTM for the detection and recognition of mathematical expressions in PDF document images. Phong et al. ^[10] proposed a mathematical variable classification method based on CNNs (AlexNet and ResNet-50) for the detection of mathematical variables in embedded expressions. Then, the team proposed a unified mathematical expression detection system ^[11] to detect mathematical expressions in document images. First, this method was used for layout analysis involving entire document images, and it improved the accuracy of text line segmentation and word segmentation. Then, the features extracted by FFT and CNN (AlexNet and ResNet-18) models were used to detect independent expressions and embedded expressions in document images, respectively. This method combined manually extracted features and deep learning features for mathematical expression detection, and the detection accuracy was greatly improved compared with that of previous methods.

In studies of scientific document retrieval based on mathematical expressions, a variety of results have been obtained ^{[12,13,14,15,16,17,18]}. Considering the importance of the contextual features of mathematical expressions, Wang and Tian ^[19] proposed a method based on BERT to calculate the contextual similarity of mathematical expressions for scientific document retrieval. First, NTCIR data were preprocessed, and the correspondence between mathematical expressions and scientific documents was saved. Then, BERT was used to calculate the context similarity of the results returned by the mathematical expression similarity calculation module and finally output the retrieval results of scientific documents based on the similarity scores. This method was evaluated based on the NTCIR dataset with Chinese scientific documents added, and it performed reasonably well. Hussain and Khoja ^[20] proposed a method for retrieving scientific documents based on the semantic information from mathematical expressions to optimize the sorting of scientific documents. The variables, constants and operators in the expressions with unified symbols were replaced, and weights were assigned to the semantic subtrees of the expressions to enhance the retrieval results. This method was evaluated based on the NTCIR-12 and arXiv corpora, and for the top-5 documents, the precision of Wikipedia formula queries reached 47% and 44%, respectively. Xu et al. ^[21], in an effort to overcome the shortcomings of similarity calculations using only text information, proposed a method to calculate the similarity of scientific documents by combining text information and mathematical expression information. This method used the formula coverage in document pairs to measure formula similarity and the distances between feature words in the documents to measure text similarity. Finally, text similarity and formula similarity were used to calculate the similarity of the scientific documents. The experimental results showed that compared with the traditional vector space method, this method improved the precision of document similarity calculations and was more suitable for cross-language document similarity calculations. Pathak et al. ^[22] proposed a method to retrieve mathematical expressions based on the context of scientific documents. First, "context-formula" pairs were extracted by using a pattern-based method and stored in a knowledge base. Then, Apache Lucene was used to create an inverted index for the context in the knowledge base and to coordinate with the index and the knowledge base to obtain the retrieval results for expressions related to the text query. This method used the context of mathematical expressions to aid in retrieval, which could improve the precision of matching to a certain extent. By combining the methods of natural language processing (NLP) and mathematical language processing (MLP), Scharpf et al. ^[23] realized the classification and clustering of documents containing mathematical content, thus laying a foundation for the efficient retrieval of mathematical documents. The study aims to assess the impact of choice and combined encoding of natural and mathematical languages on the classification and clustering of documents containing mathematical content. For the coarse-grained classification of the primary MSC subject number (pMSCn), Schubotz et al. ^[24] proposed a method combined with machine learning to automate this process. The method reduces the effort while maintaining classification accuracy, contributing to research in mathematical documents retrieval.

This paper proposes a mathematical document retrieval and ranking method based on CA-YOLOv5 and HFS ^[25] to improve the performance of mathematical document retrieval systems by combining the ability of CA-YOLOv5 to quickly and accurately detect targets and HFS multiattribute decision-making. The main contributions of this research are as follows:

(1) In the proposed mathematical query interface, the automatic input of mathematical query expressions is achieved by using YOLOv5 ^[26] for the mathematical expression detection task and utilizing CA ^[27] model to obtain the target location information.

(2) In the mathematical matching stage, FDS is used to normalize mathematical expressions, and HFS algorithm is introduced to calculate the similarity between pairs of mathematical expressions, thus enabling our method to adapt to variable forms of mathematical expressions and improving the performance of mathematical document retrieval.

The remainder of this paper is organized as follows. A system overview is given in Section 2. In Section 3, the mathematical query interface module is proposed. In Section 4, the mathematical matching module is introduced. In Section 5, we present the experimental results and discuss them. Finally, conclusions are summarized in Section 6.

2.   Overview of the mathematical document retrieval system based on CA-YOLOv5 and HFS

The workflow of the mathematical document retrieval system is shown in Figure 1. The mathematical query interface uses the pretrained CA-YOLOv5 to automatically detect and recognize mathematical expressions in layout images. The mathematical matching module parses each symbol in a mathematical query expression into an n-tuple attribute feature by using the FDS algorithm. Then, a mathematical query feature index is established to match the mathematical expressions in the dataset. Finally, the results of mathematical document retrieval and sorting are obtained.

3.   Mathematical query interface

The structure of the mathematical query interface is shown in Figure 2. YOLOv5 is an end-to-end object detection network based on the YOLO ^[28] series of neural networks. The CA model is used to capture the positions of mathematical expressions in layout images, and by embedding the module into the YOLOv5 network architecture, the ability of YOLOv5 to extract the positional features of mathematical expressions is enhanced.

3.1.   Backbone

The backbone of the system mainly includes five modules: Focus, Conv, BCSP, SPP, and CA. Among them, the CA module aggregates the positional features of mathematical expressions in the horizontal and vertical spatial directions so that the attention block captures long-distance dependencies in one direction while retaining the positional information in the other direction. The module structure is shown in Figure 3.

First, based on the feature matrix ${\text{F}} \in {\mathbb{R}^{\left({C \times H \times W} \right)}}$ of the input document image, each channel is calculated in the horizontal and vertical directions by using two pooling kernels with spatial ranges $\left({H, 1} \right)$ and $\left({1, W} \right)$, as shown in Eqs (1) and (2).

Then, ${{\rm{z}}^h}$ and ${{\rm{z}}^w}$ are concatenated in the spatial dimension and transformed through $1 \times 1$ convolution, as shown in Eq (3), where ${Concat} ({{\rm{z}}^h}, {{\rm{z}}^w})$ represents the concatenation of the features ${{\rm{z}}^h}$ and ${{\rm{z}}^w}$ in the spatial dimension, ${\mathit{Conv2d}}$ is the $1 \times 1$ convolutional layer, and $\varTheta$ is the nonlinear activation function.

And then, ${\rm{f}}$ is split into two separate tensors ${{\rm{f}} ^h} \in {\mathbb{R}^{C/r \times H}}$ and ${{\rm{f}} ^w} \in {\mathbb{R}^{C/r \times W}}$ along the spatial dimension, $r$ is the reduction ratio for controlling the block size. The $1 \times 1$ convolutional layer ${\mathit{Conv2d}}$ is used to separately transform ${{\rm{f}} ^h}$ and ${{\rm{f}} ^w}$ to tensors with the same channel number to the input ${\rm{F}}$. As shown in Eqs (4) and (5).

Finally, the output is shown in Eq (6), where ${\rm{F}}' \in {\mathbb{R}^{\left({C \times H \times W} \right)}}$ is the output, ${\rm{F}}$ is the input feature, $\otimes$ represents elementwise multiplication.

3.2.   Neck

The neck part of the system mainly uses a PANet structure. Through bottom-up path augmentation, accurate localization signals in lower layers are used to enhance the entire feature hierarchy, thereby shortening the information path between lower layers and topmost features and enhancing the flow of pixel information in mathematical expressions.

3.3.   Detection

The detection module includes the CA module and the Conv module, which are mainly used to output the position of the mathematical expression. In order to better locate mathematical expressions of different sizes, the CA module and Conv module are added to detect the corresponding mathematical expressions adaptively.

4.   Mathematical matching module

The module workflow is shown in Figure 4 and mainly includes two parts: mathematical expression analysis and similarity calculation. First, an expression entered into the mathematical query interface is parsed by FDS and stored in the database in the form of five tuple attributes. Then, the HFS is used to calculate the similarity between mathematical expressions, and the relevant mathematical documents are matched and sorted according to the similarity scores.

4.1.   Mathematical expression analysis

Considering the syntactical and semantic variations in mathematical expressions, the FDS ^[29] algorithm is used to analyze mathematical expressions. The analysis process is shown in Figure 5. First, the relevant symbols are separately obtained in LaTeX. Then, if a symbol is ordinary, we use the BaseAnalysis module to integrate numbers and letters and extract information; otherwise, according to the matching results based on the special symbol database, different types of special symbols are processed in different ways with the FunctionAnalysis module. In this paper, each symbol in mathematical expressions is parsed into a five-tuple attribute (level, flag, count, ratio, operator).

The meanings of the five-tuple attributes are described as follows:

(1) "level" is the level of the current mathematical symbol, which is based on the position of the horizontal baseline. For example, in the mathematical expression $a + {{{c^d}} \mathord{\left/ {\vphantom {{{c^d}} b}} \right. } b}$, the level values of $a$, $+$, ${\square \mathord{\left/ {\vphantom {\square \square }} \right. } \square }$, $b$, $c$ and $d$ are 0, 0, 0, 1, 1, 2 respectively.

(2) "flag" refers to the relationship of the current mathematical symbol to its nearest prior in the higher level. Its value is from 1 to 7 respectively represents the up, superscript, subscript, down, inclusion, left superscript and left subscript. And the flag values of the symbols in main baseline are 0.

(3) "count" is the sequential position of the current mathematical symbol in the mathematical expression.

(4) "ratio" represents the frequency of the operator in the mathematical expression.

(5) "operator" refers to whether the current symbol is an operator or not. If it is, the operator value is 1, otherwise it is 0.

4.2.   Mathematical expression similarity calculations

In this section, the HFS is used to calculate the similarity between the expressions entered by the user and the expressions in the reference dataset. The definitions of relevant parameters and membership degrees are shown in Table 1. The implementation strategy is shown in Algorithm 1.

5.   Results and discussion

5.1.   Experimental datasets

Our experiment used the TFD-ICDAR2019v2 dataset¹, NTCIR dataset and Chinese scientific documents (CSD) dataset for CA-YOLOv5² pretraining and mathematical document retrieval, respectively. The TFD-ICDAR2019v2 dataset contains 795 English PDF document images and a total of 38,181 annotated mathematical expressions. The NTCIR dataset contains 31,742 English documents, with a total of 518,929 mathematical expressions. Furthermore, to make the experimental data more convincing, we also add CSD dataset to expand the NTCIR dataset, which contains 10,372 documents and 121,495 mathematical expressions.

¹ https://github.com/fireae/TFD-ICDAR2019/tree/master/TFD-ICDAR2019v2

² https://pan.baidu.com/s/17y4Cg-MDhpBLmZ-Xuoxfpg?pwd=spcv

5.2.   Experiment results

5.2.1.   Results of mathematical query expression positioning

The results of mathematical query expression positioning are shown in Figure 6. Notably, CA-YOLOv5 can fairly accurately detect the mathematical expressions contained in the layout images. However, there are also some problems in the detection process. For example, "$c = 0$" in the image is detected as "$e$ $c = 0$" because the font of the character $c$ is similar to that of the preceding word "$case$", resulting in overdetection. Moreover, incomplete detection (only a part of an expression is detected) occurs in some cases. Based on the above analysis, most of the detection errors are caused by the failure to effectively split or merge some expressions during the detection process.

As a result, complete ${\rm{IoU}}$(${\rm{CIoU}}$) was used as the evaluation metric for the results of the mathematical query expression positioning analysis. The description of the ${\rm{CIoU}}$ evaluation metric is shown in Eq (7).

In the equation, ${\rm{IoU}}$ represents the intersection over union of the ground truth and the anchors, ${\rho ^2}(b, {b^{gt}})$ represents the Euclidean distance between the central points of the anchors and the ground truth, and $c$ represents the diagonal length of the smallest enclosed area that can contain both the anchors and the ground truth. $\nu$ represents the blending degree of the aspect ratio of the anchors and the ground truth (for the expression in Eq (8)), $\alpha$ is the balance factor (formula is shown in Eq (9)).

where ${w^{gt}}$ is the width of the ground truth, ${h^{gt}}$ is the height of the ground truth, $w$ is the width of the corresponding anchor, and $h$ is the height of the corresponding anchor.

In mathematical expression detection tasks, both the precision of detection and the recall rate must be considered. Therefore, this paper uses the F1-score to evaluate the system's detection performance. In Table 2, the detection performance of the proposed method is compared to that of the RIT 2 system ^[30], RIT 1 system ^[30] and Michiking system ^[30] used in the TFD-ICDAR2019 competition.

Ablation study: When the CA model was introduced into the YOLOv5 network structure for mathematical expression detection, the detection performance was greatly improved. Compared with that of YOLOv5, the recall rate of our method increased by 3.52%, and the F1-score increased by 2.04%. A detailed comparison of results is shown in Figure 7. Therefore, the CA model can improve the performance of mathematical expression detection to a certain extent. Specifically, the CA model can capture the long-term dependence of mathematical symbol pixels in one spatial direction and retain important positional information in the other direction, thereby improving the detection performance for long mathematical expressions and multiline mathematical expressions.

5.2.2.   Mathematical document retrieval and ranking results

To increase the accuracy of the retrieval results from mathematical documents, ten mathematical query expressions obtained in the mathematical expression detection experiment are selected for retrieval. The mathematical query expressions and their LaTeX forms are listed in Table 3.

Since the mathematical document results retrieved by mathematical query expressions usually return a multi-result sequence, and the result sequence is in order, the DCG index is used to measure the ranking result. Eq (10) shows the corresponding formula, and nDCG is used to standardize the retrieval results based on Eq (11).

$i$ represents the ordinal number of the retrieval result. $re{l_i}$ represents the classification of the $i$th retrieval result as excellent, good or bad; these classifications are associated with scores of 3, 2, and 1, respectively. $P$ is the total number of retrieval results. IDCG represents DCG under ideal conditions.

In this experiment, mathematical document retrieval is performed using the mathematical query expressions in Table 3, and different retrieval results are obtained. The average nDCG of all mathematical documents is 80.89%, and the nDCG of mathematical document retrieval results under different mathematical query expressions is shown in Figure 8.

To explore the retrieval performance of our method, we use SearchOnMath ^[31] to conduct a comparative experiment. SearchOnMath is a mathematical document retrieval system based on mathematical expressions, but compared with our method, this system requires the manual input of mathematical query expressions in LaTeX format for retrieval, which is inconvenient. This paper uses the mathematical query expressions in Table 3 to compare the nDCG values obtained for our method and SearchOnMath, and the results are shown in Figure 9.

6.   Conclusions

This paper proposes a mathematical document retrieval and ranking method based on CA-YOLOv5 and HFS for the input and matching of mathematical expressions. First, we use CA-YOLOv5 to automatically detect mathematical expressions in layout images and input them into the relevant retrieval module. Then, the membership degrees of the symbol attributes in each mathematical expression are calculated, and the similarity of the mathematical expressions is calculated based on the HFS. Finally, mathematical documents are sorted according to the similarity of mathematical expressions. This method integrates the advantages of CA-YOLOv5 and HFS, and improves the input efficiency and matching accuracy of the mathematical document retrieval system to a certain extent.

However, this method has some limitations. In the future, we will improve the method from the following three perspectives:

(1) Continue to improve the CA-YOLOv5 network architecture, reduce the time required for mathematical expression detection, and improve the precision of mathematical expression detection;

(2) Add a Chinese dataset to the training set of CA-YOLOv5 to improve detection performance and enhance model applicability;

(3) Include information based on mathematical expressions and context, title, author, etc. in retrieval to further improve the accuracy of mathematical document retrieval and the relevance of the retrieval results.

Acknowledgments

This work is supported by the Natural Science Foundation of Hebei Province, China (No. F2019201329), and the Key Project of the Science and Technology Research Program in University of Hebei Province, China (No. ZD2019131).

Conflicts of interest

The author declares that there are no conflicts of interest in the publication of this article.