Gated Convolutional Neural Networks with Deep Layer Fusion for Abstractive Document Summarization

Abstract

Text summarization is one of the central tasks in Natural Language Processing. Recent advances in deep neural networks and representation learning have substantially improved text summarization technology. There are largely two approaches to text summarization: extractive and abstractive. The extractive approach generate a summary by extracting salient linguistic constitutes from the document and assembling them to make grammatical sentences. In contrast, the abstractive approach write summaries using words that may or may not exist in the document using sophisticated techniques such as meaning representation, content organization and surface realization.

In this thesis, we focus on abstractive summarization, and propose a model to represent and recognize salient content better from a document that is one of the major abilities to better text summarization. Furthermore, we introduce a large-scale Korean dataset for document summarization.

First of all, we adopt a hierarchical structure to capture various ranges of the representation. Moreover, we propose a gating mechanism to make better intermediate representations and we utilize POS (Part-of-Speech) tags to use morphological and syntactic features. Lastly, we propose a simple and efficient deep layer fusion to extract and merge salient information from the encoder layers. We evaluate our model using ROUGE metrics on three different datasets: CNN-DM, NEWSROOM-ABS, and XSUM. Experimental results show that the proposed model outperforms the state-of-the-art abstractive models on NEWSROOM-ABS and XSUM and shows comparable scores on CNN-DM.

These data-driven approaches require a large amount of data for model training. However, large-scale datasets do not exist for less well-known languages such as Korean, and building such a dataset is very labor-intensive and time-consuming. In this thesis, we propose Korean summarization datasets that are acquired automatically by leveraging the characteristics of news articles. The dataset consists of 206,822 article-summary pairs in which summaries are written in headline-style with multiple sentences. With analysis of our dataset and experimental results, we showed that the proposed dataset is being fairly large to train an abstractive summarization model, comparable to existing English news datasets and suitable for develop abstractive summarization models.