Extractive text summarization model based on advantage actor-critic and graph matrix methodology

1.   Introduction

In natural language processing, automatic text summarization is widely used, e.g., to help users navigate web content and assist in building question-and-answer systems. Text summarization methods can be divided into two categories: extractive summarization and abstractive summarization. Extractive summarization models form summaries by selecting and copying text snippets from the document, while abstractive methods aim to generate concise summaries with paraphrasing. The commonly used abstractive methods include graph-based methods (e.g., graph-based lexical centrality as salience in text summarization (LexRank)) ^[1] and centrality-based methods (e.g., the centroid method) ^[2]. This work is primarily concerned with extractive summarization. Though abstractive summarization models have made strides in recent years, extractive techniques are still desirable, as they are simpler and more reliably yield semantically and grammatically correct sentences. Some authors have also made this point. For example, Li et al. used an abstractive method ^[3] and created the distributional semantics reward (DSR) model. The final experimental results showed that the summarization generated by DSR-facing long documents could not cover all of the critical information in the original text. See et al. used a hybrid pointer generator network and coverage for abstractive summaries ^[4]. Therefore, here, we used the extractive method to extract summarization.

The earliest extractive summarization method was introduced by Luhn in 1958. Luhn et al. used statistical information on words to calculate the importance of sentences ^[5] and selected high-scoring sentences to form the summarization. In 2000, Radev et al. built the multi-document summarization processor MEAD ^[6]. The model can reconstruct documents with summarization sentences and minimize the reconstruction error. Around 2004, extractive methods began to shift from "statistical methods" to "graph methods". Mihalcea et al. introduced the concept of bringing order into text with TextRank ^[7]. The core idea of TextRank is to determine the coverage of text units on the source document to get the score of sentence units. However, the algorithm does not consider the similarity between the sentences, resulting in information redundancy in the selected sentences.

In 2016, Ma et al. introduced an unsupervised multi-document summarization framework based on neural documents model (DocRebuild) to be able to refactor documents using summarization sentences ^[8] and selected summarization sentences to minimize refactoring errors. Cheng et al. proposed NN-SE and NN-WE ^[9], which are data-driven approaches based on neural networks and continuous sentence features. Nallapati et al. introduced a recurrent neural network-based sequence model for the extractive summarization of documents (SummaRuNNer) ^[10], and the model achieved state-of-the-art status within a year. Jadhav et al. built the Sentences and Words from Alternating Pointer Networks (SWAP-NET) model ^[11], which can establish the relationship between keywords and salient sentences. The model not only identified salient sentences and keywords, but it also combined them to form summaries. Zhou et al. have achieved neural extractive document summarization (NEUSUM) ^[12]. NEUSUM's integration of "selection" and "scoring" together solved the "sentence scoring" and "sentence selection" fragmentation in the extractive method. It achieved state-of-the-art results for the CNN/Daily Mail in that year. In 2020, Wang et al. created a model to dynamically update the weights of nodes by using a graph attention network iteratively ^[13] to extract the three highest-scoring sentences to form the summarization. Zhong et al. described the extractive summarization task as a semantic text matching task ^[14].

Although many scholars have recently achieved remarkable results in extracting summarization by using deep learning, it still has some limitations, particularly, the existence of exposure bias in neural network models ^[15]. Ranzato et al. introduced the exposure bias of neural networks ^[16] and pointed out that the cross-entropy is challenging to solve. He used reinforcement learning (RL) to improve exposure bias. Bahdanau et al. posted the exposure bias of neural networks ^[17] and alleviated this problem by using the actor-critic (AC) algorithm. Paulus et al. introduced the idea that models trained by supervised learning tend to exhibit exposure bias ^[18]. He used RL to circumvent this problem by focusing separately on the input and the continuously generated output. Narayan et al. pointed out that neural network training is based on cross-entropy ^[19], and that cross-entropy tends to produce redundant information when performing extractive summaries. Meanwhile, they suggested using RL to sequence sentences to overcome this difficulty. Mao et al. introduced the concept of multi-document summarization with maximal marginal relevance-guided RL ^[20] to address the redundancy of multi-document summarization in a unified manner.

To contribute to this field, we propose the graph matrix and advantage AC (GA2C) model to address the exposure bias of deep learning methods. GA2C translates "how to do extractive summarization" into "how to make action decisions" by uniting the graphs matrix with RL. Our contributions in this work could be summarized as follows: i) this is a novel application of advantage AC (A2C) on sentence selection for extractive summarization; ii) the analysis and empirical results show that GA2C alleviates the exposure bias problem; iii) the results disclosed that GA2C is better than previous models on CNN/Daily Mail data.

2.   Related work

GA2C's graph matrix generator uses the idea of TextRank. And, the similarity between nodes is calculated using Jaccard similarity and cosine (COS) similarity. In addition, the TextCNN model is used in the implementation of the A2C algorithm.

2.1.   TextRank

TextRank, proposed by Mihalcea et al. in 2004, is a graph-based keyword extractive method. In addition, this method refers to the idea of PageRank ^[21]. Indeed, TextRank is a widely used keyword extraction method ^[7]. The method splits the text into many keywords and uses the keywords to form nodes and the similarities between the nodes to form edges, thus forming a graph. The score is higher when a keyword is more similar to other keywords.

In general, the overall algorithmic process is divided into the following four steps.

i) The text is processed and added to the graph to form nodes;

ii) The similarity between nodes is calculated and added to the graph to form edges;

iii) The algorithm is iterated until convergence;

iv) The nodes are ranked based on their scores at final convergence.

The weight iteration algorithm for the node Vi is shown in Eq (1).

where $WS\left(Vi\right)$ represents the weight of sentence i. $W\text{ji}$ denotes the weight of the edge between the nodes Vj and Vi and d is the damping factor.

2.2.   Jaccard similarity

Jaccard similarity is used to establish the relationship between sentences to form the similarity matrix. The algorithm is mainly used to calculate the similarity between samples of a symbolic measure or a Boolean measure ^[22]. The Jaccard similarity is concerned with the characteristics that are common between samples. When the characteristic attributes between samples are identified by symbols and Boolean values, the magnitude of the specific weight of the difference cannot be measured, as one can only obtain a result such as "whether it is the same". In two samples, the Jaccard is equal to the concurrent set minus the intersections sets, and then divided by the concurrent sets as shown in Eq (2).

where S1 and S2 denote the vector representation of the two sentences.

2.3.   Cosine similarity

Cosine similarity is used to establish the relationship between sentences to form the similarity matrix. The algorithm measures the similarity between two vectors by calculating the cosine of the angle between them ^[23]. The value of cosine similarity is between 0 and 1. When two vectors have the same pointing, the cosine similarity is 1, which means that the similarity is 100%. When the angle of two vectors is 90°, the cosine similarity is 0, which implies that the similarity is 0. The cosine between two vectors can be found using the Euclidean dot product formula, as shown in Eq (3).

where S1 and S2 are the vector representations of the two sentences, and n is the dimension of the vector.

2.4.   TextCNN

The evaluation network and decision-making network of GA2C were implemented in this study by using TextCNN. Both of them share the parameters of TextCNN. The summarization is selected by TextCNN. And, the selected summarization is evaluated by TextCNN. In 2014, Kim applied a convolutional neural network (CNN) model to the text classification tasks ^[24] and proposed TextCNN. The method is a variant of a CNN ^[25] and it aims to extract different local features by defining different filter convolutional kernel sizes. The model is mainly used in the field of text classification. Extractive summarization can be viewed as a text classification task divided into two categories.

As shown in Figure 1, the input was a 6 × 5 sentence matrix with a dimension of 5 for each word. The sentence matrix is passed through six convolutional kernels of size (3, 4, 5). The (3, 4, 5) convolutional kernels correspond to the extracted features size of (5, 4, 3). The extracted features are pooled and combined to get the representation after convolution.

The convolutional layer is the core of TextCNN. The three filters in this work have convolutional kernel sizes of 3, 4 and 5, which leads to the extraction of different features on the three levels. Its calculation formula is shown in Eq (4).

$f$ represents the activation function, commonly used as the rectified linear unit (ReLU). ${w}_{i(x, y)}$ denotes the weight of the i-th node in the output matrix corresponding to the filter input node (x, y). ${\mathrm{C}}_{(x, y)}$ is the value of the node (x, y) in the filter. ${b}_{i}$ denotes the bias term corresponding to the i-th node. Local feature extraction is achieved by setting three filters with convolutional kernels of (3, 4, 5). ${h}_{i}$ is the result of the convolution layer.

3.   Extractive summarization model based on graph matrix and A2C methodology

GA2C, which combines the A2C algorithm with graph structure composition, can reduce the impact of exposure bias. The methods used are mainly composed of A2C, TextRank and TextCNN models. The GA2C model mainly consists of a graph matrix generator, an evaluation network and a decision-making algorithm. The text is input to the graph matrix generator, after which the evaluation network and decision network are used for training. The framework structure diagram is shown in Figure 2.

The articles are fed into the graph matrix generator to generate the graph matrix, and the evaluation network and the decision-making network use the graph matrix to get Rewards, the next state and the Action. The evaluation network evaluates the decision-making network based on the Action. The decision-making network then improves the decision process based on the evaluation of the evaluation network.

3.1.   Graph matrix generator

TextRank provides a reference for the graph matrix generator. The process of using a graph matrix generator to generate a graph matrix is shown in Figure 3. The articles are input to the graph matrix generator, and after segmenting the articles into word vectors, WORDAVG is used to represent the sentence vectors and calculate the similarity between the sentence vectors to get the similarity matrix of the articles. The last graph is constructed using the similarity matrix and sentences node.

3.2.   Evaluation network

In this model, the A2C algorithm is used. The evaluation network corresponds to the critic of A2C. The network mainly calculates the actual values of the nodes. The optional actions are fed to the network and then provided to the hidden layer for full connectivity to obtain the value of the current state. The schematic diagram of the evaluation network is shown in Figure 4.

The figure starts from the node s9. Using the graph matrix, the action nodes connected to s9 can be derived as s1, s4 and s5. These three nodes are sent to the TextCNN and fully connected through the hidden layer, and the actual values are obtained. The actual values are calculated using the predicted values to obtain the loss of the evaluated network.

At each action step, the current state of the decision-making network is the set D of moves connected by the selected actions. The decision-making network determines the next sentence di from Set D. The evaluation network compares the selected di with the reference summarization bi to calculate the reward. In this paper, two reward calculation methods are provided.

1) ROUGE-1, ROUGE-2 and ROUGE-L as a reward:

which is equal to

The label is the standard summarization of this method. $Countmatch\left(gram-1\right)$ is the number of gram-1 matches. $Count\left(gram-1\right)$ is the total number of gram-1. ROUGE-2 is the same. $PLCS$ is the accuracy rate of longest common subsequence (LCS). $RLCS$ is the recall rate of LCS, and $\text{b = }{P}_{LCS}/{R}_{LCS}$.

2) Another method is to use simple accumulation for reward calculation. A positive reward is given for a correct decision, and there is no reward for an error.

where JUDGE is a reward calculation function that compares di with bi. If it is a summarization, JUDGE returns a positive reward. If it is not a summarization, there is no reward. Evaluate the network to get the total reward by discounting and adding up.

where k is the number of times the selection action is executed. γ is the discount factor, and it is a constant.

In this method, we use the A2C algorithm. The advantage function uses TD-error.

This leads to the loss function of the evaluation network.

3.3.   Decision-making network

The decision-making network performs the selection of sentences. The sentences are fed to the decision network, and the probability of each current action is output after the hidden layer and SOFTMAX, and the action with the highest probability is selected as a result this time. The most basic TextCNN was used for action selection in this study to prove the framework's effectiveness.

The figure starts from the node s9. Using the graph matrix, the action nodes connected to s9 can be derived as s1, s4 and s5. These three nodes are fed to the TextCNN and passed through the hidden layer. The probabilities of the three actions are obtained, and the results are selected after SOFTMAX.

The decision-making network is the policy function πθ (s, a). TD-error records the error between the actual and predicted values of the current action. The decision-making network can use TD-error to calculate the update gradient. The formula is shown in Eq (11).

where the update of $\theta$ is $\theta \leftarrow \theta +\alpha {\nabla }_{\theta }G(\theta)$.

The loss function of the decision-making network is loss_actor.

4.   Results and discussion

4.1.   Dataset

CNN and Daily Mail were used to evaluate GA2C ^[26]. This dataset contains news documents and their corresponding summarizations from CNN and Daily Mail websites. It contains 287, 227 documents for training, 13, 368 documents for validation and 11, 490 documents for testing. We followed ^[13] to preprocess the dataset.

4.2.   Evaluation metrics

There are more evaluation methods for text summarization tasks. In this study, we used recall-oriented understudy for gisting evaluation (ROUGE), which Lin and Och constructed in 2004 ^[26]. It is a frequently used evaluation metric in various text summarization models ^[27,28,29]. ROUGE is used to calculate the F1 scores of ROUGE-1, ROUGE-2 and ROUGE-L between the sentences selected by GA2C and the standard sentences for evaluation.

In the formula for ROUGE-N, N-gram refers to a contiguous substring of length N in the text. The denominator is the number of N-grams of all reference summaries. The numerator is the number of N-grams of both the model-predicted and the reference summarizations. In this study, N was taken as 1 and 2, which means that the evaluation methods are ROUGE-1 and ROUGE-2.

where RS denotes the reference summarization.

The formula for ROUGE-L is shown in Eqs (14)-(16).

The LCS function is the length of the longest common substring of X and Y. m and n means the size of the reference summarization and the GA2C summarization, respectively. R_lcs and P_lcs respectively represent the recall and accuracy.

F_lcs indicates the F1 score for ROUGE-L. When β = 1, precision and recall are weighted equally. In some cases, if precision is more critical, β should be adjusted to less than 1. If recall is more important, β should be adjusted to be greater than 1. In this study, β was taken as 1.

4.3.   Implementation details

For these experiments, the parameter settings were mainly divided into a graph matrix generator, an evaluation network and a decision-making network.

1) Graph matrix generator

The sentences were cropped, and the average length of the sentences was set to 22 after calculation. The similarity between sentences and sentences was set to 0.05 to have the correct action in each candidate action. When the similarity is more significant than 0.05, the connection is created in the graph.

2) Evaluation network

When fixed-value rewards are used, this paper sets the ADD reward value to 0.1.

In this study, eight sets of experiments were conducted using the values of fixed rewards, as shown in Figure 6.

Figure 6 shows that, for the fixed-reward setting, the highest value is reached when the reward value is 0.1. The leads were 0.52 and 1.67 for ROUGE-1 and ROUGE-2, respectively.

3) Common parameters of the evaluation and decision-making networks

Where the discount factor of information entropy is 0.001, the discount factor of the reward was set to 0.99. The lexicon was 32341. The convolutional kernel size was (22, 2). The learning rate was 0.01.

4.4.   Update of decision-making network and evaluation network

GA2C's loss function consists of three parts, i.e., loss_value, loss_actor and entropy.

The total loss is loss_Total:

where $\beta$ and $\eta$ are discount factors. The loss_Total function used in this study is consistent with many A2C applications. For example, Li et al. used A2C to diversify recommendation results ^[30]. The method uses a loss calculation that also adds up the actor and critic parts to perform the update. In this study, the selection summary problem was considered as an action decision problem, which means that the intelligence is trained by putting them into a graph composed of sentences, somewhat like a path-planning problem. For this reason, we applied the loss function used in the A3C Super Mario implementation and added additional $entropy$ ^[31]. To demonstrate the effect of information entropy on the final effect, a set of comparative experiments was conducted. The values of ROUGE-1, ROUGE-2 and ROUGE-L for the case without $\eta *entropy$ were tested when the reward value was 0.1.

The bold in the table represents the best result. As shown in Table 1, GA2C leads to 0.45 and 0.07 for Rouge-1 and Rouge-2, respectively, for the case of added $\eta *entropy$. This result proves that adding $\eta *entropy$ is more helpful for the convergence of training.

4.5.   Experimental results

To verify the validity of the models in this paper, GA2C was compared with some typical methods on CNN/Daily Mail data, including Lead-3, BANDITSUM ^[15], Refresh ^[19], SummaRuNNer ^[10], Pointer-generator + coverage ^[32] and deep Q-network (DQN) ^[33]. Refresh, which combines maximum-likelihood cross-entropy loss with RL, was chosen as the baseline of this study. The model abstracted the single-document summarization task into a sentence ranking task, and global optimization ROUGE was used for extractive summarization. Meanwhile, several classical models were used as comparison models. SummaRuNNer is an extractive model based on a recurrent neural network. The model uses a novel training mechanism that allows the introduction of labels without the training phase. Lead-3 simply uses the first three sentences of the document as a summarization, but it achieved great results. Pointer-generator + coverage incorporates the seq2seq model, a pointer-generator network, a coverage mechanism, mitigating out-of-vocabulary words and information redundancy. The DQN is an RL algorithm based on the DQN method for extractive summarization. The method learns strategies to maximize the ROUGE score in Q-value approximation. BANDITSUM uses the optimized ROUGE score to treat the extractive summarization as a contextual bandit problem.

In addition, we conducted ablation experiments from many angles, including using four activation functions, using different evaluation network AC methods and uniting different reward functions with different mapping methods.

The bold in the table represents the best result. GA2C was compared with several classical models, as shown in Table 2, with the best results indicated by bolded numbers. Compared to Refresh, GA2C led on ROUGE-1, ROUGE-2 and ROUGE-AVG by 0.70, 9.01 and 2.73, respectively. Compared with Refresh, GA2C utilized article-building graphs to better establish the connection between individual sentences and speed up the training. And, GA2C incorporates the A2C algorithm, which is superior to Refresh that only draws on the RL framework. GA2C not only avoids the problems caused by maximum likelihood similarity, it also minimizes the reliance on real tags. It can be seen that the GA2C performance achieved some improvement. This indicates that the critical sentence content extracted by the model in this study was more accurate and fluent.

Moreover, four activation functions were used to test the GA2C model. The default here was to use the cumulative reward method and COS calculation to build the graph. The four activation functions included ReLU ^[34], Leaky ReLU ^[35], parametric ReLU (PReLU) ^[36] and randomized ReLU (RReLU) ^[37]. The four activation functions are shown in Figure 7.

ReLU: The ReLU activation function was a simple calculation. If the input was greater than 0, it directly returned the value provided as input; if the information was 0 or less, it returned the value 0.

Leaky ReLU: Leaky ReLU was used to assign a non-zero slope to all negative values, and the formula is shown in Eq (19).

where a_i denotes a fixed parameter in a (1, +∞) interval.

PReLU: PReLU can be considered as a variant of Leaky ReLU. In PReLU, the slope of the negative part is based on the data and not predefined.

RReLU: RReLU is also a variant of Leaky ReLU. In RReLU, the slope of the negative value is obtained randomly in training and becomes fixed in subsequent tests.

where a_ji is a randomly drawn value from a uniform distribution U (I, u) and I < u; u takes values in the range of [0, 1).

The bold in the table represents the best result. By comparison, RELU achieved the best results against the other four activation functions. Compared to Leaky RELU, GA2C led on ROUGE-1, ROUGE-2 and ROUGE-AVG by 0.70, 9.01 and 2.73, respectively. RELU was a simple way to achieve good results on GA2C. In this study, we finally decided to use the RELU activation function as the default function.

Meanwhile, to compare the different evaluation network algorithms (AC and A2C) ^[38], experiments were conducted in this study using AC combined with graph matrices. The experimental results are shown in Table 4.

The bold in the table represents the best result. The AC algorithm is slightly worse than the A2C algorithm in terms of the combined effect. Compared to AC, A2C led on ROUGE-1, ROUGE-L and Rouge-AVG by 0.70, 9.01 and 2.73, respectively. The AC algorithm has no baseline, and the feedback was positive, which can lead to some actions not converging. However, the A2C algorithm includes a baseline, making the feedback positive and negative. This made it easier to converge and train ^[39]. The A2C was more suitable for the model in this study.

In addition, to find the best pairing of a reward function and graph building method, we provided two ways of reward calculation: 1) cumulative way to calculate the reward of action, and 2) use of ROUGE to calculate the reward of action. And, two ways of text similarity when building graphs have been provided: (i) Jaccard similarity to calculate sentence similarity and (ii) cosine similarity to calculate sentence similarity. In this study, to try out the best combination, these two were combined separately for experiments.

The bold in the table represents the best result. The GA2C model used ROUGE to calculate rewards and COS to calculate sentence similarity to build graphs, which can yield the best results. The combination of ROUGE and COS performed the best. Compared to ADD and COS, it led on ROUGE-1, ROUGE-2, ROUGE-L and ROUGE-AVG by 0.24, 0.36, 0.30 and 0.31, respectively. The Jaccard was not satisfactory in combination with ROUGE or ADD, which was 6% worse than the combination with COS. This is mainly because Jaccard compared the intersection and concurrence of two sentences, which does not reflect the similar relationships between texts well. The results of the COS experiments were all higher than those of the Jaccard experiments, where ADD was 0.24, 0.36, 0.40 and 0.31 lower than ROUGE for the four metrics when the COS constructed graphs were applied, indicating that ROUGE returned a more accurate reward.

5.   Conclusions

In order to solve the problems of exposure bias and low precision in extractive summarization. This paper introduced the extractive summarization model GA2C. The algorithm transforms the data into a graph matrix and trains the model using an evaluation and a decision-making network. GA2C uses an RL network for feature extraction and achieves the selection of actions by using a decision-making network. The actions selected by the decision-making network are evaluated using the evaluation network and a reward is returned. GA2C utilizes RL to reduce exposure bias greatly. The experimental results showed that the summarization content extracted by GA2C was more accurate, and that the language was more fluent. Compared to Refresh, GA2C led on ROUGE-1, ROUGE-2 and Rouge-AVG by 0.70, 9.01 and 2.73, respectively.

The model in this paper could be further improved, as follows: i) the word vector representation can be trained using Bert, Word2vec, etc.; ii) several more types of connections between sentences can be tried, such as ROUGE or other methods of establishing relationships; and iii) GA2C can be applied to datasets in different domains, such as meeting minutes, emails, etc. In addition, GA2C training is time-consuming. The training time can be reduced in the future by learning from some techniques from graph CNN training. The reward design of GA2C can also be further optimized to improve the accuracy of summary extraction.

Conflict of interest

The authors declare that there are no conflicts of interest.