Date of Award
Spring 5-2-2025
Document Type
Thesis
Degree Name
Master of Arts (MA)
Department
Computer Science
First Advisor
Sarah Ita Levitan
Second Advisor
Elena Filatova
Academic Program Adviser
Subash Shankar
Abstract
This thesis investigates computational approaches to discourse coherence through two interconnected studies. First, we evaluate the capabilities of Large Language Models (LLMs) as coherence evaluation models across real-world textual data domains. Our experiments with GPT-4o, GPT-3.5, Llama, and Mistral models reveal that while LLMs significantly outperform traditional entity grid methods (55% improvement in F1 scores), their performance varies substantially across domains and coherence levels. Chain-of-thought prompting unexpectedly decreased performance, and fine-tuning experiments demonstrated improved detection of low and high coherence texts but diminished performance on medium coherence cases. In our second study, we explore the practical application of entity grid-based coherence models to detect red herring fallacies in argumentative text. Using features derived from entity transition patterns, our transformer-based classifier achieved a 78.42% F1-score in distinguishing texts containing red herrings from coherent Wikipedia passages. Together, these studies advance our understanding of computational coherence evaluation by systematically assessing state-of-the-art LLMs and demonstrating how coherence metrics can address challenging NLP tasks like fallacy detection.
Recommended Citation
Amin, Elyas A., "Evaluating and Applying Coherence Models: LLMs as Coherence Evaluators and Neural Entity Grids as Noun Entity Shift Indicators" (2025). CUNY Academic Works.
https://academicworks.cuny.edu/hc_sas_etds/1342
