Reasoning arithmetic word problems entailing implicit relations based on the chain-of-thought model

1.   Introduction

For decades, the challenge of automatically solving arithmetic word problems (AWPs) has been a struggle for artificial intelligence. However, in recent years, deep learning methods have emerged as powerful tools for improving success rates in this area ^[1]. Deep learning has improved natural language processing tasks, such as question-answering and machine translation ^[2,3], and has also made significant progress in mathematical reasoning ^[4,5,6,7]. However, current deep-learning solvers focus primarily on solving problems rather than teaching step-by-step reasoning. Language models have shown great promise in mathematical reasoning, but there are concerns regarding the accuracy of their outputs. Researchers have noted the potential for generating ungrounded answers ^[8], and users often have to verify predicted outcomes with extra effort. While recent prompting strategies have been developed to provide rationales before making predictions ^[7], language models can still produce hallucinated statements and wrong answers. Therefore, there is a pressing need for novel approaches that enable more reliable reasoning.

Earlier research on the development of reasoning chains, conducted by ^[7], or relied on a solitary human-annotated prompt. However, manually constructing reasoning chains has two primary drawbacks. First, current models may need the capability to learn and execute all necessary reasoning steps as tasks become more complex. Additionally, these models may need help in generalizing different tasks. Second, depending on a single decoding process leaves it susceptible to incorrect inference steps, which can result in an incorrect final answer. To overcome these limitations, recent studies have focused on two primary approaches. The first approach is process-based, which involves creating more intricate demonstrations by hand. Zhou et al. ^[9] and Chen et al. ^[10] are examples of studies that have used this approach. The second approach is outcome-based, which involves utilizing ensemble-like methods. Wang et al. ^[11] and Li et al. ^[12] are examples of studies that have adopted this approach.

The study presents a qualia-based template prompt for the modifier-to-matrix (MTM) model, inspired by a qualia structure ^[13] to represent the derived properties. The study is unique in that it utilizes implicit mathematical knowledge through its reasoning approach, a general framework based on a state-action paradigm. Additionally, the paper introduces the missing entity recovery (MER) model, resulting in improved solution accuracy and reasoning interoperability. This groundbreaking approach combines explicit and implicit knowledge learning to enhance reasoning abilities, making it possible to comprehend complex problems and develop more effective solutions.

2.   Related work

2.1.   Mathematical trustworthy reasoning

Large language models (LLMs) are incredibly powerful in modeling natural language, but they can face significant obstacles when considering mathematical reasoning. One major issue is that pre-trained language models need more specific mathematical training, which can hinder their ability to perform math-related tasks as proficiently as natural language tasks. Large pre-trained models can be costly to train from scratch for specific tasks. Additionally, LLMs may need help with downstream tasks involving various input formats or modalities, such as structured tables ^[14].

Fortunately, researchers have developed a solution to these issues: The Chain-of-Thought (CoT) Prompting method ^[7]. CoT uses intermediate natural language rationales as prompts to enable LLMs to generate reasoning chains and predict answers for input questions. CoT prompts can be especially helpful in solving math word problems by encouraging a step-by-step thought process ^[15]. Recent research has focused on enhancing CoT reasoning under the few-shot setting by selecting better in-context examples and creating more effective reasoning steps ^[16].

2.2.   Process-based approaches

Process-based approaches optimize CoT reasoning for complex tasks. The least-to-most prompting method, proposed by ^[13], breaks down problems into sub-problems to solve them one by one. Khot et al. ^[17] used different prompts and diverse decomposition structures. Chen et al. ^[10] and Gao et al. ^[18] proposed a program-of-thoughts method that uses large language models to express reasoning as a program. Lu et al. ^[19] integrated different tools in Chameleon to improve the abilities of LLMs for compositional reasons.

Above all, while LLMs may face certain challenges with mathematical reasoning, researchers have developed effective methods to overcome them. With the help of CoT prompts and other techniques, LLMs can perform math-related tasks as well as natural language tasks.

3.   The proposed method

The framework comprises two modules collaborating seamlessly to produce prompts, which assists the model in effectively reasoning complex problems: The MTM model utilizes a matrix of nouns and an adjective modifiers library to create a graph $NDG$ linking to a knowledge base to solve problems. The knowledge base utilizes a known quantity and filters out the knowledge. The Figure 1 process is repeated until the objective uses a known quantity as a node. Finally, the solution engine is utilized to filter out the needed knowledge.

Figure 1.  Flowchart of the method.

DownLoad: Full-Size Img PowerPoint

3.1.   Modifier to matrix (MTM) model

The AWPs are annotated into a knowledge matrix with an explicit entity and a solution goal, and the method used to generate entity modifiers of AWPs has been effectively recategorized. The MTM extracts AWP implicit knowledge features and selects a commonsense and domain knowledge vocabulary from various learning resources. An MTM is centered information of an AWP syntax-semantic relation and connects the entities and attributes to the matrix. Through the graph matrix, we aggregate the contextual information of knowledge attribute words via Qualia role relation to form the knowledge representation of AWP. Finally, the AWP knowledge representation links implicit knowledge space using a learnable matrix.

Figure 2 shows that the modifying component is adjectival. It accesses the adjective library to find adjectives and creates a collection of adjectival modifier classes for AWPs. The modifier has a distinguishing effect entity to understand the related knowledge attribute words and different scenarios. The knowledge attributes words in specific scenarios to others. The interrelation association weights, denoted as $w_{ij}$, calculate the connection between words $a_i$ and scenarios $s_j$. These weights create a scene association matrix $A$, including the association weights between knowledge attribute words and scenarios. The number of knowledge scenarios $k$ equals the number of knowledge points.

Figure 2.  The proportion of each lexeme and the proportion of high-frequency adjectives.

DownLoad: Full-Size Img PowerPoint

Algorithm 1 shows that the MTM model generated a word matrix, and each noun entity can also be generated with its corresponding word matrix representation according to the Qulia and syntax-semantic relation. The entities are consistent with previously generated noun entities. The modifications of these noun entities are redundant. Therefore, problem-solving could be modeled as path searching to connect the known nodes $N$ to the unknown nodes in $NDG$ until the goal $g$ node is solved. Here, the goal node denotes the unknown variable asked to be solved in the question portion. The path searching determines nodes that could be selected to calculate an unknown node, especially in the case of implicit nodes in the graph. In the field of $AWP$ solving, it can build a static $NDG$ that contains domain knowledge and commonsense knowledge to store and represent the attribute relations of entities.

---

 | Show Table

DownLoad: CSV

Besides, $NDG$ can also represent the dynamic relations of entities given by the problem text. Ultimately, all the static and dynamic relations are integrated into a single $NDG$ represented and stored the quantity relations.

For example: "There are three yellow apple trees planted by Xiao Ming, nine red apple trees planted by Xiao Ming and Xiao Li, eight apple trees planted by Xiao Ming, and the number of red apple trees planted by Xiao Li." The matrix of the "apple tree" is shown in the following table:

Table 1.  Embedding matrix example.

apple trees	Xiao Ming	Xiao Li
yellow	X11	X12
red	X21	X22

---

 | Show Table

DownLoad: CSV

The answers to the questions can be obtained in the order of reasoning:

Red apple tree planted by Xiao Ming = apple tree planted by Xiao Ming - yellow apple tree planted by Xiao Ming.

Red apple tree planted by Xiao Li = red apple tree planted by Xiao Ming and Xiao Li - red apple tree planted by Xiao Ming.

Extending to the general case, the data is handled in the following manner: The two modified components are denoted as $M_{1n}$ and $M_{2m}$. The modified entities are then transformed into matrix form, which is illustrated in the table below. Here, $n$ represents the row vector and $m$ represents the column vector.

The iterative equation system is comprised of three parts: Row sum, column sum, and total sum. It assigns the variables a to the row sum, b to the column sum, and c to the total sum. The equation can then be constructed using these variables as follows:

---

 | Show Table

DownLoad: CSV

3.2.   Missing entity recovery (MER)

The $MTM$ is created through statistical techniques to reclassify adjectives. The missing entity recovery pertains to the reasoning of attribute values that are necessary to comprehend the implicit knowledge within formulas and concepts of semantic features. The MER model categorizes relations into different types of entities and operations. By utilizing the $NDG$ generation algorithm, mathematical entities can better infer implicit expressions and comprehend complex concepts and theories in mathematics more extensively.

The $NDG$ represents entities and attributes values as nodes while organizing qualia relations as links in math word problem-solving. The $NDG$ also contains domain knowledge and commonsense knowledge, which are essential in storing and representing attribute expressions of entities. As illustrated in Figure 3 and Figure 4, entity relationships can be classified into three fundamental types. This classification enables the organization of entity relationships into a network named $NDG$, which is composed of the three relationship networks mentioned above.

Figure 3.  The NDG-back of interest problem.

DownLoad: Full-Size Img PowerPoint

Figure 4.  The NDG-back of traffic problem.

DownLoad: Full-Size Img PowerPoint

In the implicit knowledge space, the feature acquisition process is strengthened by the vector knowledge attention mechanism represented by AWPs, which assigns adaptive weights by combining semantic and knowledge features. Our proposed MTM provides a robust solution for modeling from acquiring to generating auxiliary implicit relations and overcoming current drawbacks, such as the completeness of entity NDG relation generation and the assumed knowledge graph. This process is commonly referred to as an ontology-based auxiliary knowledge system during the code execution phase. Figures 3 and 4 provide compelling examples that demonstrate the effectiveness of the auxiliary generation of our model.

4.   Experiment

This section presents the AWPs that correspond to the dataset type of high-profile AWPs. Following our proposed algorithmic model, these AWPs are submitted to the large language model to generate inferences for relations. The accuracy of the reasoning is then compared, considering whether it includes the logic generated by the recommendations from the chain of thought.

To evaluate the performance of our method, we collected a subset of AWPs from the most widely used and publicly available datasets in the field of machine-solving study. We then compared our results with other methods in the same field to demonstrate the effectiveness of our approach. Our evaluation was based on nine datasets, namely PEP, BNU, EEP, M23K, Ape5kT, MWP, USC, Dol2K, and Arith. Among these, three datasets - PEP, BNU, and USC - are derived from real textbooks. PEP contains AWPs from the 2018 edition of the textbook for elementary school students published by People's Education Press. BNU comprises arithmetic word problems from the 2018 edition of the textbook for elementary school students published by Beijing Normal University. The USC dataset includes arithmetic word problems from the California Elementary Mathematics Textbook. On the other hand, the EEP dataset comprises arithmetic word problems from the elementary school entrance examination papers of 34 provinces in China between 2010 and 2019. The remaining five datasets - M23K, Ape5kT, MWP, Dol2K, and Arith - are publicly available datasets in solution research.

The selection of a large model for our experiment has been limited, and we have chosen the ChatGLM-6B int4 version. This is an open-source and bilingual conversational language model based on the General Language Model (GLM) architecture with 6.2 billion parameters. With the help of model quantization techniques, it can be deployed locally on consumer-grade graphics cards. At the INT4 quantization level, it can be deployed on graphics cards with as low as 6GB of video memory. ChatGLM-6B uses a similar technology to ChatGPT, which is optimized for Chinese AWPs.

The inference ability of the large model can significantly vary based on various factors such as parameter magnitude and learning level. However, our proposed model has generated an effective prompt, which is evident from Table 2. We have found that including CoT in the large model for AWP problem reasoning can significantly enhance its reasoning ability. Furthermore, our improved methodology can further boost the reasoning ability of the humanoid. Based on the experimental results, we have found that the model's quality plays a crucial role in determining the reasoning ability of the large model. Our proposed method is a step forward in improving the reasoning ability of the model while keeping the model itself unchanged.

Table 2.  The distribution of dataset over two types of problems.

Datasets		Explicit AWPs	Implicit AWPs	Problem collection source
name	quantity	quantity	quantity
PEP	504	226	278	Elementary Math Textbook
BNU	436	293	143	Elementary Math Textbook
EEP	2722	1641	1081	Primary Question Papers
M23K	20910	16568	4342	Data cleaned from math23k
ApeSKT	4570	3431	1139	Ape210k's test set
MWP	2373	2299	74	From MAWPS
USC	497	436	61	California Elementary Textbook
Dol2K	1878	1809	69	From Dolphin 1878
Arith	1541	1053	488	From AllArith

---

 | Show Table

DownLoad: CSV

Table 3.  Large model inference comparison results.

Types of Large Models	Reasoning accuracy
	base	Zero-shot-CoT	Our proposed prompt
ChatGLM-6b(int4)	3%	3.5%	4.5%
ChatGTP	63.5%	65%	68.5%

---

 | Show Table

DownLoad: CSV

5.   Conclusions

This paper proposed a generative model for explaining adjectival entity modifications and missing entity complementation to the large model prompt. The experiments conducted in this study show that the added prompt can effectively enhance the reasoning ability of the big model and help elementary and middle school students understand the questions and automatically generate a logical answer process.

While the model's implicit knowledge embedding has shown promise, there is still room for improvement to enhance its accuracy. Our plan is to continually expand the knowledge base and utilize vectorization techniques to improve word sense understanding. Currently, our model faces a specific error rate in selecting knowledge embeddings, which can be attributed to issues with retrieving. Nevertheless, with the development of a larger model and improvements to our existing one, we can significantly enhance the machine's reasoning expression ability. The latter will serve as the core technology for developing intelligent tutoring systems because it can potentially generate tutoring solutions.

Use of AI tools declaration

The authors declare they have not used Artificial Intelligence (AI) tools in the creation of this article.

Acknowledgments (All sources of funding of the study must be disclosed)

We would like to thank the constructive feedback provided by the reviewers. I gratefully acknowledge the financial support for my doctoral study provided by the China Scholarship Council (CSC). The Science and Technology Research Program of Chongqing supports this project. Municipal Education Commission (Grant No. KJZD-M202201801) and (Grant No. KJZD-K202001801).

Conflict of interest

The authors declare no conflicts of interest to report regarding the present study.

Ethics declaration

The author declared that the ethics committee approval was waived for the study.