Journal of information and
communication convergence engineering
(J. Inf. Commun. Converg. Eng.)

Automatic text summarization is an exciting research field in computer science. There are various publications on both extractive and abstractive methods [1]. In addition, there are some large-scale datasets for evaluating summary models in English, such as GigaWord [2,3] and CNN/Daily Mail [4]. However, there is little summarization research on largescale Vietnamese datasets [5,6,7] for a single document.

Therefore, this study focuses on the automatic text summarization of a single document for the Vietnamese dataset. The three main contributions of this work are as follows. First, we introduce a new Vietnamese large-scale dataset for automated text summarization research. Second, we propose a new robust and straightforward extractive text-summarization algorithm with O(n,k,p) = O(n^{(k + 2/p)}) + O(nlog₂n) + O(np) + O(nk²) + O(k) time complexity. Moreover, the output summary of our model is a grammatical document owing to the extractive technique. The third is we trained three word-embedding models: Word-to-Vector, Glove, and FastText, which can be used for other word-representation tasks.

The numerical test results showed that our extractive model performed better than the state-of-the-art abstractive summary model, not only in ROUGE-2, but also in training time, summarizing time, and the minimum required resources for experimentation.

The remainder of this paper is organized as follows: Section II describes the work related to our extractive summary model. The methodology is presented in Section III. Section IV discusses the experiment and the results of the summary models. Finally, the conclusions and future work are presented in Section V.

Text summarization was first introduced by Luhn [8], the task of creating a smaller version of a document or a set of documents while retaining the primary information. Two summarization methods are based on the output technique [1]: extractive and abstract. Abstractive methods attempt to rewrite a summary with a new vocabulary and sentence structure. In contrast, extractive techniques attempt to find the highestranking sentences in the original text to produce a summary.

However, based on the number of input documents used to produce a summary, there are two types of text summarization models: single-document and multi-document summary [1]. The single-document model uses one document as the input and generates a summary of the document. The multi-document summary model uses a set of documents as the input and produces a summary of the contents of these documents.

The proposed summarization model is extractive for summarizing a single document. This approach attempts to find sentences that are most similar to the main content of the input document in order to create a summary.

Our extractive summary model is similar to the extractive model proposed by Radev et al. 2004 [9] in using centroids to extract summary sentences. However, the Radev model and our model differ in terms of constructing the centroids and the number of input documents. Thus, Radev used the TF-IDF (term frequency – inverse document frequency) score of words in the corpus to determine the centroids, but we used a clustering algorithm. The next difference is that Radev proposed a model for summarizing multiple documents, whereas we proposed a model for summarizing a single document.

Our model is similar to Rossiello’s model [10] in that it uses word embedding to represent the input text as the vector feature and extracts the highest-ranking sentence for the summary based on the centroids. However, our model differs from Rossiello’s in creating centroids and the number of input documents. At that point, Rossiello’s model is similar to Radev’s model because both are multi-document summarization models that use TF-IDF to assign centroids.

For comparison, we evaluated our dataset on our extractive model and pointer generator network, which is a state-of-theart abstractive text summarization model.

A. Extractive summarization

1) Summary Model

A summary of the tasks is shown in Fig. 1. In the extractive model, the training set was used to train the word embedding. In the abstractive model, the training set was used to train the summary model. Finally, the testing set was used to evaluate both summary models.

Figure 1. The process of the summary models.

In our extractive model, we use a clustering model to specify the clusters of sentences in the input document, and then pick the nearest sentence with each cluster’s centroid to produce a summary. We used five types of word representations as the input models to find the best representation for our dataset. These are the (bag-of-words) BoW, TF-IDF, Word2Vec, Glove, and FastText models. The extractive summary, Algorithm 1, processes a single-document to produce the summary.

1	Given: document d, number of output sentences k
2	Initialize S is an empty set of sentences, X is an empty set of vectors,

	V is an empty set of vectors, summary is a zero-length text

3	S ← s sentences is split from d
4	For s ∈ S do
5		X ← x vector of s
6	end for

7	kmeans_model ← kmeans(X, k) // Clustering X into k clusters by k-means algorithm

8	C ← c cluster centroid in kmeans_model
9	For c ∈ C do
10	x ← f(c, X) // f(c, X) is the function for finding closest x ∈ X with c
11	V ← (x_idx, x_dis) // x_idx is index of x ∈ X, x_dis is distance of c and x
12	end for

13	sort V by increasing x_dis
14	For x_idx ∈ V do
15		summary = summary + S[x_idx] + “.”
16	end for

17	return summary

In Algorithm 1, |C| = k, and vectors x of s depend on the type of input model: BoW, TF-IDF, Word2Vec, Glove, and FastText. When using the BoW and TF-IDF input models, the vocabulary included all words in document d. However, the vocabulary was based on the training set when using Word2Vec, Glove, and FastText input models. The f(c, X) function is implemented using cosine similarity.

Given the number of sentences in the input text n, number of clustering centers k, and dimension of the feature vector p, the time complexity of Algorithm 1 is O(n,k,p) = O(n^{(k + 2/p)}) + O(nlog₂n) + O(np) + O(nk²) + O(k). where the time complexity of k-means clustering is n^{(k + 2/p)} [11] and the time complexity of Timsort [12] is nlog₂n. The step calculation of the time complexity for each line in Algorithm 1 is

Line 1 = C1 * 1
Line 2 = C2 * 1
Line 3 = C3 * 1
Line 4, 5, 6 = C4 * n * p
Line 7 = C5 * n(k + 2/p)
Line 8 = C6 * 1
Line 9, 10, 11, 12 = k * (C7 * k * n + C8 * 1)
Line 13 = C9 * nlog2n
Line 14, 15, 16 = C10 * k
Line 17 = C11 * 1
Total run time = C1 * 1 + C2 * 1 + C3 * 1 + C4 * n * p +
C5 * n(k + 2/p) + C6 * 1 + k * (C7 * k * n + C8 * 1) + C9 *
nlog2n + C10 * k + C11 * 1
= C + C + C + C * np + C * n(k + 2/p) + C + C * n * k2 +
C * k + C * nlog2n + C * k + C
= 5C + C * np + C * n(k + 2/p) + C * n * k2 + 2C * k + C *
nlog2n
= O(n(k + 2/p)) + O(nlog2n) + O(np) + O(n * k2) + O(k) + 0
= O(n(k + 2/p)) + O(nlog2n) + O(np) + O(nk2) + O(k)

In the evaluated dataset of this study, the average value of n is 21.27 (Sents/Art value of test set in Table 1), and the maximum value of k is 5. The value of p is 100 when using Word-to-Vector and FastText, and p is 300 when using Glove; the average value of p is 24.15 (Words/Sent value of test set in Table 1) when using BoW and TF-IDF.

Table 1 . Dataset information for VNText.

Information	Train	Test	Test - ref
Articles	880,895.00	110,103.00	110,103.00
Sentences	18,738,333.00	2,342,296.00	155,573.00
Words	452,686,377.00	56,563,630.00	4,163,723.00
Words/Sent	24.16	24.15	26.76
Words/Art	513.89	513.73	37.82
Sents/Art	21.27	21.27	1.41

2) The k-means Model

k-means [13,14] is one of the most useful algorithms in clustering. In this work, we use mini-batch k-means [15] (Algorithm 2), a variant of k-means that improves performance when working on large-scale datasets to find clusters of sentences in the input document. These clusters were then used to infer a summary.

In Algorithm 2, C is the set of cluster centers, c ∈ R^m is the cluster center, |C| = k, X is the collection of vectors x, f(C, x) returns the nearest cluster center c ∈ C to x by using the Euclidean distance.

1	Given: k, mini-batch size b, iterations t, data set X
2	Initialize each c ∈ C with an x selected randomly from X
3	v ← 0

4	For i = 1 to t do
5		M ← b examples selected randomly from X
6		for x ∈ M do
7			d[x] ← f(C, x)			// Cache the center nearest to x
8		end for
9		for x ∈ M do
10			c ← d[x]			// Get cached center for x
11			v[c] ← v[c] + 1			// Update per-center counts

12			η 1/ v[c]			// Get per-center learning rate

13			c ← (1 - η) c + ηx		// Take gradient step

14		end for
15	end for

3) Input Model

The inputs of the k-means model and the f(c, X) function in Algorithm 1 are vector types. Therefore, we must use the word representation model to transform the text document into a vector to feed the model.

a) Bag-of-words:

BoW [16] is one of the first methods of text representation based on the study of word probability in language, which was introduced in 1954 by Harris in the study “Distributional Structure”. In a traditional BoW, vocabulary is built using all words in the corpus. A document is typically represented by a vector. The vector element is the word frequency in that document, and the vector length is the number of vocabularies. In our study, to extract the sentences of the input document to construct a summary, the vocabulary includes all words in the input document. Each sentence is then represented by a vector BoW.

b) TF-IDF:

In BoW, the importance of a term (word) is based on its frequency. However, some terms, such as rare words, have a low frequency in the document, but that word is essential for expressing the content of that document. Therefore, in 1972, Jones introduced the TF-IDF [17], which uses weight to present the importance of a word of a record in the corpus to solve this problem. In our study, we treat the sentence as a document and the document as a corpus to calculate the TF-IDF value for all words in the input document.

c) Word-to-vector:

Instead of presenting the term frequency as an element of a vector, as in the BoW model, or presenting the weight of a term as an element of a vector, as in the TF-IDF model, word embedding is the technique used to learn the relation of a word in the corpus to present a term as a vector with short dimension but high quality for natural language processing tasks. The word-to-vector (Word2Vec) [18] was introduced in 2013 by Mikolov and can be used to learn the similarity between words from massive datasets with billions of terms and millions of words in the vocabulary. As it is built on top of the DistBelief architecture [19], Word2Vec can utilize computing clusters with thousands of machines to train large models. In our study, we used the training set as the corpus for training the Word2Vec model. Then, in the summary process, each sentence in the input document is a vector that is the sum of the word vectors according to the word (term) in that sentence.

d) Glove:

Word2Vec learns the relation between word terms using the local window through the content of the training text; this approach does not utilize the statistics of the corpus. In 2014, Pennington introduced the Glove model [20], which proposes a specific weighted least squares model, trains on global word-word co-occurrence counts, and thus makes efficient use of statistics. In our study, similar to Word2Vec, we train a Glove model with the training set; then, in the summary process, we sum the word vectors of all words in the sentence to construct a sentence vector.

e) FastText:

In the Word2Vec and Glove models, each vocabulary word is presented by a distinct vector without parameter sharing. Therefore, in 2016, Joulin introduced FastText [21], a word representation model that focuses on the morphology of words. In Fast Text, each word is represented as a bag of character n-grams. Each vector representation is associated with a character based on the n-gram model, and a word vector is expressed as the sum of all character representations in that word. In our study, FastText is used in the same way as other word representation methods to improve results in the case of rare words.

B. Abstractive Summarization

The Pointer-Generator network [4] is a state-of-the-art abstractive summarization model. This model is a sequenceto- sequence model [22] consisting of an encoder and decoder. The encoder is a single-layer bidirectional LSTM [23], and the decoder is a single-layer unidirectional LSTM.

Bahdanau’s attention [24] was applied to the decoder to calculate the probability distribution over source words. In addition, the author added a pointer network [25] to the decoder, which allows both copying words by pointing and generating words from a fixed vocabulary. In addition, a coverage vector is maintained to solve the problem of repetition, which is a common problem in sequence-to-sequence models.

Therefore, we chose this model as the standard for comparing our extractive summarization model and evaluating our large-scale Vietnamese dataset.

A. Experimentation

We evaluated our dataset with both our summary model and the Pointer-Generator network summary model, and the results are based on ROUGE [26] metrics.

1) Dataset

We name our new dataset the VNText dataset. Similar to the method used to build the CNN/Daily Mail dataset [4,27], our dataset was built by collecting articles from information websites in Vietnam. These articles were collected in HTML form and then cleaned by eliminating HTML tags and unrelated information, such as links and advertising. After preprocessing, the VNText dataset contained 1,101,101 plain text articles, and every article included the title, subtitle, and main content. The subtitle of each article is used as a reference summary, and its content is used as the input document for the summary models.

Next, VNText was split into three subsets: the training set with 880,895 records, the validation set with 110,103 records, and the testing set with 110,103 records, the details of which are shown in Table 1. The training set was used for training Word2Vec, Glove, FastText, and the Pointer-generator network model, the validation set was used for tuning parameters, and the testing set was used for evaluation.

In Table 1, Words/Sent represents the average number of words per sentence, Words/Art represents the average number of words per article, and Sents/Art represents the average number of sentences per article. In addition, the Test – ref column contains information on the reference summary in the testing set.

2) Extractive Model Training

Table 2 shows the parameters used in our extractive model and the training word embedding, which represent the vector features in our extractive summary model. In this table, the prefix “w2v” stands for the parameters for training Word2Vec and FastText models; the prefix “glove” parameters are used to train the Glove model.

Table 2 . Parameters of our extractive model.

Name	Value/Range	Use for/Meaning
k	1, 2, 3, 4, 5	Number of k-means clusters, and number of sentences in the output summary
w2v_embedding_len	100	Embedding length
w2v_epoch	5	Number of epochs
w2v_window_size	10	Context windows size
w2v_related	skip_gram	Related word model
glove_embedding_len	300	Embedding length
glove_epoch	15	Number of epochs
glove_window_size	15	Context window size
glove_related	skip_gram	Related word model

In addition, all the processes in our extractive model were run on one computer. Its CPU is an ARM Neoverse-N1 with four cores, 2.8 GHz, one thread per core, 64 GB RAM, and 150.34 MB/sec read and write disk speed.

3) Training Pointer-generator Network Model

Similar to the training strategy of the author of the Pointer-generator networks, we first trained this model on our training set in point-gen mode. We then used the last checkpoint to continue the training in the coverage mode.

However, because the VNText has approximately three times more records than the CNN/Daily Mail, we trained this model 10 epochs in point-gen mode and continued training 2,000 iterations for the coverage mode.

In addition, we trained this model on a V100 GPU using default parameters.

A. Results

1) Summary Length

Table 3 presents the average number of words in the summary produced by the summary model. This result provides a general overview of the length of the output summary.

Table 3 . Number of words per summary.

Input model/Mode	k = 1	k = 2	k = 3	k = 4	k = 5
BoW	46.81	83.75	116.31	146.47	174.98
TF-IDF	36.88	58.31	81.71	101.58	124.14
Word2Vec	45.95	81.54	115.05	142.24	171.96
Glove	47.26	83.02	116.51	143.59	173.57
FastText	46.05	81.75	115.21	142.11	171.81
Point-gen						41.38
Coverage						41.69

In this table, k is the number of sentences in the summary generated by the extractive model, and we use the value of k in {1, 2, 3, 4, 5}, as mentioned in the parameters section. In more detail, we can see that the average number of words per summary when using the TF-IDF input model is less than that of the other input models.

Point-gen is the point-gen mode, and coverage is the coverage mode of the abstractive model. The maximum length of the output of the abstractive summary model was 100 words.

As shown in Table 3, the average output length of our extractive summary model is longer than that of the abstractive summarization model, except when TF-IDF is used at k = 1.

2) ROUGE Score

a) ROUGE score of our extractive model:

Tables 4, 5, and 6 show the F1-score of our extractive summary model based on ROUGE-1, ROUGE-2, and ROUGE-L when using various types of input models.

Table 4 . F1-score (%) of extractive model based on ROUGE-1.

Input model	k = 1	k = 2	k = 3	k = 4	k = 5
BoW	50.97	47.09	40.97	36.06	32.31
TF-IDF	49.36	50.53	47.44	43.77	39.64
Word2Vec	51.61	46.60	40.87	36.72	32.81
Glove	51.91	46.23	40.49	36.46	32.59
FastText	51.64	46.50	40.77	36.70	32.82

Table 5 . F1-score (%) of extractive model based on ROUGE-2.

Input model	k = 1	k = 2	k = 3	k = 4	k = 5
BoW	17.88	18.45	17.92	17.18	16.45
TF-IDF	17.52	18.62	18.77	18.53	18.09
Word2Vec	17.22	17.08	16.66	16.28	15.80
Glove	17.45	16.97	16.47	16.09	15.60
FastText	17.27	17.03	16.56	16.17	15.70

Table 6 . F1-score (%) of extractive model based on ROUGE-L.

Input model	k = 1	k = 2	k = 3	k = 4	k = 5
BoW	29.72	27.68	25.31	23.27	21.62
TF-IDF	29.39	29.21	27.75	26.27	24.66
Word2Vec	29.55	27.62	25.31	23.54	21.84
Glove	29.65	27.58	25.27	23.51	21.79
FastText	29.57	27.61	25.31	23.57	21.87

In ROUGE-1, with Glove, the highest F1-score was 51.91%, with Word2Vec and FastText, and the F1-score was slightly lower at 51.61% and 51.64% when k = 1. The TFIDF F1-score was only 49.36% when k = 1 and increased to 50.53% when k = 2. If we look back to Table 3, we can see that the average length of the summary at the peak point of the F1-score based on ROUGE-1 is not high. In addition, the F1-score exhibits a downward trend when k increases to 3, 4, and 5.

In ROUGE-2 and ROUGE-L, with TF-IDF, the best F1-score is 18.77% when k = 3 and remains a highly stable trend when the k is changing.

Moreover, in ROUGE-L, with BoW input, the F1-score was the highest at 29.72% but decreased when k was changed. In ROUGE-2 and ROUGE-L, with Word2Vec, Glove, and FastText, the F1-score was only slightly different for various values of k.

b) ROUGE score of Pointer-generator networks model:

Table 7 presents the results of the Pointer-generator networks model, which shows that the coverage mode had the highest F1-score in both ROUGE-1 and ROUGE-L with 52.33 and 33.09%, respectively, which are higher than the F1-score of our extractive summary model.

Table 7 . F1-score (%) of the Pointer-generator networks.

Model	ROUGE-1	ROUGE-2	ROUGE-L
Point-gen	49.60	15.36	32.59
Coverage	52.33	16.17	33.09

However, the F1-score in ROUGE-2 was only 16.17%, which was not higher than that in our extractive model.

3) Time

Table 8 shows the time spent in training the Word2Vec, Glove, FastText, and Pointer-generator networks on the training set and the time to evaluate the testing set with both summary models.

Table 8 . Computing time (in hour).

Input model/Mode	Training	Evaluating
BoW	0.00	1.11
TF-IDF	0.00	1.12
Word2Vec	4.73	1.28
Glove	1.32	1.16
FastText	7.24	1.30
Point-gen	129.63	41.02
Coverage	134.30	35.78

With our extractive summary model, the evaluation time was not significantly different when using the difference in the word representation model for presenting the input. It required no more than 1.3 hours to summarize our testing set five times and calculate the ROUGE score of the output summaries. In particular, in the training step, we did not spend time using the BoW and TF-IDF. In addition, Glove requires less training time than Word2Vec and FastText.

With the Pointer-generator network model, the training time was not too high, and the evaluation time was also acceptable in both modes. The evaluation time of this model is the time to summarize our testing set at one time and the time to calculate the ROUGE score.

A new and efficient extractive text summarization model was proposed to summarize a single document. The approach uses BoW, TF-IDF, Word2Vec, Glove, and FastText to represent the sentences of the input document, and then the k-means algorithm is applied to these sentences to create the clusters for extracting the highest-ranked sentences in the summary. The test results on our Vietnamese large-scale dataset show that our extractive model achieved better performance than the state-of-the-art abstractive text summarization model in ROUGE-2, while saving computational time and resources. Further experimental investigations are recommended to combine the word representation model with the input of our extractive summary model to improve the ROUGE score.

Text summarization was first introduced by Luhn [8], the task of creating a smaller version of a document or a set of documents while retaining the primary information. Two summarization methods are based on the output technique [1]: extractive and abstract. Abstractive methods attempt to rewrite a summary with a new vocabulary and sentence structure. In contrast, extractive techniques attempt to find the highestranking sentences in the original text to produce a summary.

However, based on the number of input documents used to produce a summary, there are two types of text summarization models: single-document and multi-document summary [1]. The single-document model uses one document as the input and generates a summary of the document. The multi-document summary model uses a set of documents as the input and produces a summary of the contents of these documents.

The proposed summarization model is extractive for summarizing a single document. This approach attempts to find sentences that are most similar to the main content of the input document in order to create a summary.

Our extractive summary model is similar to the extractive model proposed by Radev et al. 2004 [9] in using centroids to extract summary sentences. However, the Radev model and our model differ in terms of constructing the centroids and the number of input documents. Thus, Radev used the TF-IDF (term frequency – inverse document frequency) score of words in the corpus to determine the centroids, but we used a clustering algorithm. The next difference is that Radev proposed a model for summarizing multiple documents, whereas we proposed a model for summarizing a single document.

Our model is similar to Rossiello’s model [10] in that it uses word embedding to represent the input text as the vector feature and extracts the highest-ranking sentence for the summary based on the centroids. However, our model differs from Rossiello’s in creating centroids and the number of input documents. At that point, Rossiello’s model is similar to Radev’s model because both are multi-document summarization models that use TF-IDF to assign centroids.