New Evaluation Metric for Large Language Models in Summarization

Abstract

ABSTRACT This thesis introduces and evaluates a new text summarization evaluation metric, SynSemScore, to fill the gap between the simple lexical-based metrics such as BLEU, ROUGE, and METEOR, and the complex embedding-based methods. While the previous methods are based mainly on surface level word matching or are costly in terms of computational complexity, SynSemScore lies in between by using syntactic matching together with semantic similarity measured using WordNet-like structures and a redundancy penalty to identify redundant content. The purpose is to offer a simple but effective way of measuring the quality of the machine produced summaries in terms of coherence, consistency, fluency and relevance. To validate SynSemScore empirically, this research employs the SummEval dataset which offers a variety of human annotated summaries for CNN/Daily Mail articles. The data is cleaned up thoroughly to ensure data integrity and correlation analyses are made between SynSemScore and other metrics. The results of the experiments reveal that SynSemScore is more relevant to the human rating, outperforming traditional approaches in several aspects. Hence, the proposed metric can be considered as a feasible solution for assessing the quality of summarization models on a large scale without the need for expensive training of transformer-based models. Besides the technical aspects of SynSemScore, the work describes the complete process of the research, including data collection and preparation, hyperparameter optimization, and correlation analysis of the metric. Finally, SynSemSemScore presents a flexible, easy-to-interpret, and semantic-aware method for assessing the performance of current summarization systems and can therefore help to enhance the quality of the generated text in both academic and business applications. Keywords: Text Summarization, SynSemScore, Semantic Similarity, Redundancy Penalty, SummEval, Evaluation Metrics, Large Language Models.