Combining statistical, structural, and linguistic features for keyword extraction from web pages

1.   Introduction

Keywords are widely used to summarize text documents. They can be manually annotated by humans or automatically generated by computer. Automatic keyword extraction from web pages refers to the selection of a set of words from the document that best describe its content. The keywords can further be used for information retrieval, document retrieval, document clustering, document classifying, indexing, summarization and topic detection ^[1].

On one hand, extracting keywords from web pages is more difficult than from plain text because the additional information like menu and navigational bars, comments, adverts and all the formatting codes present in an HTML document can disturb the process. On the other hand, the HTML code provides additional clues about which words are more important than others. There exist many techniques for keyword extraction from plain text, but they usually do not pay attention to the structure of the page. In this paper, we focus on keyword extraction from web pages.

Figure 1 shows an example of a webpage with a complex, free form structure containing heterogeneous text from multiple languages scattered across the page. In this work, we focus on the candidate word selection and features used to score these candidates. Most existing techniques utilize three different types of features:

1. Statistical

2. Structural

3. Linguistic

Statistical features can be simple frequencies of the words with the idea that more frequent words are more important than less frequent ones. However, this would lead to choosing common words such as the, and, for. A normalization is therefore needed by giving more weight to words that are frequent in the given document but not so in other documents. A corpus like Wikipedia and frequencies of the word use in search engines can be used, but also simple stop word lists can be rather powerful.

Structural features utilize emphasis, links and meta information in the HTML code. For example, words included in the header tags are more likely to be good keywords. Simple formatting like boldface, italic and Capitalization may also reveal good keywords. Meta information itself may already include manually annotated keywords, but keywords are often included in the title and even part of the link of the web page (URL). The structure and spread of the words across the web page can provide further clues about the importance of the words.

Linguistic features utilize the semantic meanings of the words and their roles in the sentences. For example, nouns are more likely to be keywords than verbs and adjectives. There are also good tools available to analyze popular languages like English. However, finding language processing tools such as part-of-speech (POS) taggers can be challenging for smaller and grammatically more complex languages like Finnish. Simple solutions like stop words are easier, as they can be found for many languages. Language-independence would make the method more general.

In this paper, we perform systematic comparison of the most common features used in keyword extraction. We evaluate them on twelve datasets containing 2935 web pages. We start with the statistical features and construct a simple baseline from the best combination. We then study the effects of different structural and linguistic features on the accuracy. We propose a new method called ACI‑rank from the best working combination while aiming at keeping the method simple and general. It is compared to the existing methods in the literature.

The rest of the paper is organized as follows. In Section 2, we define the problem and review the most relevant literature. We then study each of the three features as follows: statistical features in Section 3, structural features in Section 4 and linguistic features in Section 5. Experiments are presented in Section 6, and conclusions are drawn in Section 7.

2.   Keyword extraction

Keyword extraction has numerous challenges:

● Diversity of the words

● Keywords may not always appear in the page as such

● Keywords vs. Key phrases

● Multi-lingual pages

● Multiple topics on the same pages

● Structure of the webpage

We focus on single keywords even if there are examples where key-phrases (formula one) would be more appropriate. Nevertheless, most of the methods would generalize to key-phrases via n-grams. The questions of how many keywords and the issues of having multiple-languages and multiple topics in the same page are also not considered.

The overall framework of the studied keyword extraction framework is summarized in Figure 2. The main parts are the cleaning and extraction of the text, selection of the candidate words, calculating the features and scoring. Natural language processing (NLP) can be highly useful but also time-consuming and limited to specific languages. It might even be difficult to decide which is the base language. From various NLP tools, we therefore consider only POS tagging and stop word lists.

2.1.   Pre-processing

Pre-processing is one of the most critical steps in NLP because it shapes the results based on how we transform the input document into features ^[2]. Most typical pre-processing techniques are summarized in Table 1. A document is a collection of sentences. To extract the keywords, we need the natural language words inside the document. A typical pre-processing step is therefore to remove unnecessary information such as numbers and punctuation marks ^[1,2,8,18].

The remaining words can be further processed by stemming and lemmatization. A lemma is a chosen convention to represent a set of words (lexeme) originating from the same root and having the same meaning. For example, break, breaks, broke, broken and breaking all have roots to the same lemma, break. Lemmatization is the process or converting words into their lemmas.

Stem represents the root of a word carrying its lexical meaning. Unlike a lemma, a stem is always part of the original word and may not be a meaningful word itself. For example, the lemmas of the words produced and producing is produce, but their stem is produc because it is included in both words as such. In English, lemma and stem are often the same, but in other languages like Finnish they can differ more often.

Both stemming and lemmatization depend on language, and there exists plenty of different stemming algorithms. Stemming recognizes known suffixes of the words (e.g., -ing), and then chops the suffix off to obtain the stem. It has been widely used in keyword extraction ^{[1,2,6,11,15,16,18]} despite the drawback that the stem is not always a real word. This does not matter for algorithms but makes it less appealing for humans. According to ^[3], about 10% of English words would become non-real by stemming. The reasons for using stemming are that it is fast and easy to implement and does not require any dictionary. Many stemming algorithms for English exist, including Porter stemmer, Snowball stemmer and Lancaster stemmer ^[3]. Despite its better accuracy, lemmatization is less commonly used ^[1,14] than stemming mainly because it requires a dictionary.

Another common pre-processing method is stop word removal. Stop words are the most common words in the language, and they should therefore not be selected as keywords even if their frequency were high. Stop word lists must be built for each language separately. However, they are usually short (from a few dozen to a few hundred), and lists for many languages exist on the web¹.

¹ https://www.ranks.nl/stopwords

2.2.   Candidate selection

Extracted keywords are words found in the web page, but which words should we consider as the candidates in the first place? Table 2 summarizes typical methods. Tokenization breaks each sentence into smaller units called tokens, which are usually the words. It is by far the most used method. Exceptions might be languages like Chinese, in which words describing a specific meaning are usually composed of two or more Chinese characters. Mere tokenization is therefore not enough. Unlike English and most Western languages, Chinese text lacks clear word separators. The Jieba tool has been used for Chinese text segmentation in ^[5].

It is also possible to use a pre-defined dictionary of possible words called a bag of words ^[6]. Then, only those candidate words existing in this dictionary are considered. The method in ^[7] relies on Wikipedia and string matching without the need for explicit tokenization.

POS tags and patterns have also been utilized. Typical keywords are nouns, and for this reason, many methods select only nouns as keywords. Some methods also allow adjectives ^{[1,8,9,12,13,16]} and some verbs ^[9,16]. To limit the keywords according to its POS tags is a bit naïve, but it can improve the accuracy of simple baseline methods according to ^[10].

The problem is often considered as extraction of key phrases instead of single words. Combinations of noun + noun, noun + adjective and noun + verb have been considered ^[8,16,19,36]. N-grams are any fixed-length sequences of words and used for key phrase extraction in ^{[3,6,7,8,15,19,21,22,36]}. NP-chunks are variable length sequences of words and differ from n-grams in that only pre-prepared combinations extracted using regular expressions are allowed ^[8]; see Figure 3.

Figure 4 summarizes the pre-processing and tokenization steps for a sample web page producing 23 candidate words. Four candidates have frequency of 2: accessibility, BBC, homepage and victim. Simple frequency is not enough to select the keywords, so the next step is to evaluate these candidates.

3.   Statistical features

The most common feature is term frequency (TF), which simply selects the most common words in the web page. It has been used by many ^{[2,4,5,8,13,14,15,19,27]} because it is easy to implement by counting the appearances of the words in the page. Its main drawback is that the same words tend to be popular in all documents.

Removing stop words can compensate this deficiency, but this problem can also be attacked statistically using the so-called inverse document frequency (IDF). It counts how many documents contain the word. It helps to estimate the importance of the word so that a word that is frequent in all documents is less likely to be chosen. Vice versa, a word that is frequent only in the current web page is more likely to be a keyword. TF-IDF refers the joint use of TF and IDF.

For the BBC example in Figure 4, we get TF = 2 values for BBC, Homepage, Accessibility and Victim; and TF = 1 for the other candidates. We estimate their IDF-values by the number of Google search results they generate; see Table 4. Victim and BBC are the highest scoring words among those with TF = 2, and clergy and indifference among those with TF = 1. They all are potential keywords for this example. Wikipedia ^{[11,12,13,14]} and Bing search terms ^[19] have also been used for determining the IDF.

A more complex example using the Formula 1 Wikipedia page is summarized in Table 5. TF-IDF helps, but we can also see that the combination Formula one would be a more meaningful key phrase instead of the single word formula. It would provide the highest scores: TF = 751, IDF-freq = 31, TF-IDF = 7262. The example also shows that the role of pre-processing is crucial. Overall, term frequency with IDF seems to work reasonably well with these examples.

Another statistical feature found in literature is the first occurrence, which is just the running index of the first appearance of a word in the document. The idea is that more important words appear earlier than the less important ones. The statistical features are summarized in Table 3.

4.   Structural features

Structural features consider how the words are presented in the web page. Keywords are expected to have a stronger visual emphasis than normal words and therefore more often be used with the header tags (<h1 > to < h6 > ) and within the title tag (<title > ). A title tag is important for search bots, and therefore keywords are often added inside for that purpose. Keywords are often capitalized, either just the first letter or the entire word.

Keywords tend to appear also in the URL of the web page. We separate it into three meaningful parts: host, path and query. The host is the name of the web site, path is the directory structure used in the link, and query is the name of the actual web page. For example, the page² has candidate words University Herald in the host, articles in the path and the words poor, britain, salt, rich, warvick, socio, economic in the query part.

² http://www.universityherald.com/articles/11104/20140827/poor-britain-salt-rich-warvick-socio-economic.htm

A document object model (DOM) is a tree-structured representation of the web page based on tags like < head > , < div > , < a > and < h1 > . It divides the page into segments that can provide additional clues about the importance of the words; see Figure 5. The method in ^[16] assumes that the most important information appears in the beginning of the document and therefore analyzes only the first twenty DOM nodes. The method in ^[1] counts the number of DOM nodes in which the word appears. This assumes that less important words appear only locally in one node, whereas the keywords are more widely spread in the page.

Other signs of importance are anchor tags (<a > ) and meta tags (<meta > ). Anchor tags are links to other web pages, while meta tags include additional information about the technical content of the page such as the character set and page description, but they also used for storing keywords. The most common structural features are summarized in Table 6.

5.   Linguistic features

Language can be a very powerful tool to guide the keyword extraction, and linguistic features have been shown to significantly improve frequency-based methods ^[6,13]. For example, synonyms of the words have been utilized in ^[1,3,9,15]. A simple approach is to merge the counts of synonyms to get more reliable estimation of the important concepts in the document. The method in ^[1,12] does the opposite and assumes that important concepts are presented by the same keywords throughout the page for consistency, whereas synonyms are used more often for less important concepts to create variation. Chinese synonyms were used in ^[5], and FarsNet for Persian language was used in ^[2].

Semantic similarity takes the idea of synonyms further. The words do not need to have exactly the same meaning, but also words with similar meanings (car, taxi, truck) can increase their joint importance. However, the semantic meaning may differ depending on the context like "call me a cab" and "My name is Cab", so the use of semantics is not trivial. Semantic relatedness is also considered based on co-occurrence of the words. In ^[17], two words that occur frequently together can make a key-phrase.

The approach in ^[19] recognizes named entities, which are given higher emphasis in the evaluation. This is understandable, as named entities such as persons and places are often used as keywords in newspaper articles. The method in ^[7] builds a domain-specific concept hierarchy based on Wikipedia, and keywords are matched to these concepts. Starting from seed keywords, the method in ^[11] constructs a concept graph iteratively using Wikipedia's internal links. This graph is used when not enough keywords are found on a short-text page.

Parts of speech were listed in Section 2.2 already as a candidate word selection method. However, instead of a binary choice (to include or not), POS tags can also be used as a feature in the scoring process. They can be useful especially with trained classifiers but would require a good tagger. Several good taggers exist for the English language, but the accuracy for languages like Finnish with complex grammar is much weaker. The main drawback of using POS tags is that it makes the keyword extraction language dependent ^[20]. The most common linguistic features are listed in Table 7.

6.   Experiments

We next study the performances of the different components to find out which of them matters most. We use f-score based on standard precision and recall:

Precision and recall are counts of correctly extracted keywords relative to all ground truth keywords (precision) and relative to all extracted keywords (recall). The higher the precision, the more correct keywords were found; and vice versa, the higher the recall, the less incorrect keywords were given. We refer to the f-score as hard measure in the rest of this paper. In all experiments, we extract 5 keywords for Mopsi datasets and 10 keywords for the rest.

The hard measure recognizes only exact matches and may not give a realistic picture of the performance. For example, consider the ground truth {students, university, tuition, opportunities} and the extracted keywords {study, university, lecture, chances, fees}. Not only do the number of keywords differ, but there is only one exact match despite the result otherwise being good. For this reason, we also use the soft variant of precision and recall ^[34]. We refer to this as soft measure and use it for the final comparison in Table 13 to get better understanding of the real accuracy level.

6.1.   Datasets used

We used twelve datasets, summarized in Table 8. They are mainly collected from English and Finnish newspaper web sites but also German web sites and user-collected web pages in the Mopsi services platform. The newspaper web pages have ground truth keywords stored in their meta tags, annotated by the media itself for journalistic use. These web pages have uniform structure, which makes them easier to process. The German and Mopsi have more variations. The ground truth keywords in the Mopsi datasets have been manually annotated and sometimes do not even exist in the web page as such. The keywords may also use both English and Finnish in a mixed manner, which makes the datasets more challenging.

³http://cs.uef.fi/mopsi/MopsiSet/, http://cs.uef.fi/mopsi/newspaper/, http://cs.uef.fi/mopsi/newspaper/GermanSet/

Statistics of the data are summarized in Table 8. The average numbers of keywords in the cases of the newspaper datasets are 9.5 (English) and 7.8 (Finnish), with 16.2 in the case of the German datasets, but only 2.5 in the case of the Mopsi datasets. The latter two datasets also have a lot of annotated keywords that do not appear in the web page: German (64%), Mopsi (48%). In the newspaper web pages, the number of non-present keywords is low, usually below 10%.

6.2.   Statistical features

Results for term frequency are summarized in Table 9, with and without pre-processing and stop word removal. Our first observation is that the pre-processing is essential to useful results by the statistical features alone. The results are still rather modest though. Our second observation is that stop word removal works equally well to TF-IDF. For the Finnish and German datasets, we used two stop word lists: English and the primary language of the web page (Finnish or German).

6.3.   Candidate selection and other features

Next, we test the impact of the other features. Results are summarized in Table 10. Some variants are tested only with English datasets. Here, we observe that the individual formatting features have only a minor effect on the result; but when used together, they improve the f-score from 0.16 to 0.19. Title and URL seems to have the biggest impact among the individual features.

Linguistic features improved the accuracy on English datasets remarkably, from 0.27 to 0.33, on average. Among different features, using only nouns and no-synonyms improve the most. Stemming and lemmatization were counter-productive and decreased the performance. However, this might be partly due to the evaluation method (hard measure) requiring exact match. As soon as the words are stemmed or lemmatized, their original forms change. In the case of the English datasets, we also tested the named entity feature. We determined whether the word refers to a place, person or organization. This feature improved the baseline method but not when combined with other features.

6.4.   Summary of results

Based on the results, we construct two combinations in this paper: baseline (see Table 9) and the best performing combination, called ACI-rank (see Table 11). Frequency is used as such (baseline) and with IDF-value from Wikipedia (ACI-rank). Then no-synonym feature is a binary feature with only values 0 and 1. The rest of the features are the counts of the appearance of the feature. The scoring is simply the sum of the counts as such. The results are compared against the existing method, summarized in Table 12.

We compare our baseline and the proposed ACI-rank against existing methods shown in Table 12. The results in Table 13 are summarized so that News1 is the average of the four English newspaper datasets, and News2 the average of the six Finnish newspaper datasets. Soft evaluation results are also provided, as they provide a more realistic view of how good (or bad) the methods really are.

According to the soft measure, the proposed ACI-rank works best among the unsupervised methods (0.47) and close to the supervised approach, WebRank (0.44). In case of well-structured newspaper datasets, WebRank is better, whereas the proposed method is clearly superior on the most heterogenous Mopsi datasets. We also see that the difference from the mere frequency-based baseline method (0.37) is significant. It shows that the web HTML-based structural features are important.

It is also worth noting that the results with Finnish newspaper datasets were significantly worse than those of English newspaper data because the linguistic features were not used. Notable differences were seen with results of KeyBert, Yake and WebRank. However, the result of the frequency-based baseline deteriorated even it had access to stop words of Finnish and German and did not use any other linguistic feature.

7.   Conclusions

We have studied keyword extraction from web pages. Simple term frequency with stop word removal works reasonably, but pre-processing is important. Average results of the frequency-based baseline were 0.16 (hard) and 0.37 (soft). Further improvement was achieved by adding formatting and linguistic features, with the average results of 0.20 (hard) and 0.47 (soft). The new method, called ACI-rank, reaches the best results and is rather close to a supervised method (0.23 and 0.44). We expect that it can be improved even further by adding some of the more sophisticated ideas like concept graphs.

Future work includes applying clustering based on semantic or syntactic similarity ^[31] instead of the simple no-synonyms approach. Ideas from other summarization tasks, like title extraction ^[32] and representative image selection ^[33], could also be adopted to improve keyword extraction or to construct a complete content summarization that would cover all these three tasks. Many components used were rather simple, and the scoring of their combination was a bit naïve. We simply did not find significantly better combinations, and significant further improving seemed to require a machine learning based training approach. This is also a point of future work.

Conflict of interest

All authors declare that there is no conflict of interests in this paper.