Can AI Understand Literature? Columbia Researchers Put It to the Test

 Despite significant advances in large language models (LLMs) such as ChatGPT—particularly in assisting with thinking, research, summarization, and the comprehension of complex technical texts—their ability to interpret storytelling and literature remains an open question. Issues related to interpretive nuance and depth continue to persist.

Researchers at Columbia Engineering have addressed these challenges through a novel and ethically grounded evaluation framework. Their work received the Best Paper Award (2025) at Transactions of the Association for Computational Linguistics (TACL), highlighting its methodological rigor and contribution to the field.

“Before we can place real trust in LLMs’ analytical abilities, we need careful evidence of what they can and cannot do,” stated Kathleen McKeown, the Henry and Gertrude Rothschild Professor of Computer Science at Columbia Engineering. The research was led by Professor McKeown and Associate Professor Lydia Chilton.

“If LLMs are to serve as tools for human inquiry, we must first understand both the depth and the limitations of their analytical capabilities, particularly in domains such as narrative and literature.”

A Novel Evaluation Framework

The study assessed state-of-the-art language models—GPT-4, Claude-2.1, and LLaMA-2-70B—on their ability to summarize short fiction. Unlike prior studies that relied on publicly available texts (which may already exist in training datasets), this research introduced a controlled and original dataset.

Published authors contributed previously unpublished short stories and subsequently evaluated the summaries generated by the models. Using both quantitative and qualitative approaches informed by narrative theory, the study revealed that all three models produced faithfulness errors in over 50% of cases and consistently struggled with specificity, subtext interpretation, and nonlinear narrative structures.

“Models may appear to understand a story, but their outputs are inherently probabilistic and therefore unpredictable,” explained Melanie Subbiah, lead author of the study and a PhD researcher at Columbia. “While a trained human literary analyst provides consistent insights, even the best-performing model achieves only about a 50% reliability rate—essentially equivalent to a coin toss.”

Key Insights and Implications

The findings highlight significant limitations of current LLMs in intellectual and creative domains that require close reading and interpretive sensitivity. Although these systems can serve as useful tools, the researchers caution against relying on them for nuanced literary analysis or tasks demanding deep contextual understanding.

The study also reinforces the importance of human-centered, expert-informed evaluation in AI development.

Ethical and Methodological Contributions

Ethical considerations were central to the research design. Contributing authors were fully informed about the use of their work, appropriately compensated, and assured protection of their intellectual property. The study intentionally focused on evaluation rather than text generation, reflecting a commitment to responsible research practices.

Importantly, the project introduces a replicable framework for evaluating AI models using data guaranteed to be absent from training datasets. By collaborating directly with domain experts, the researchers provide a more reliable approach to assessing interpretive and analytical capabilities.

“The goal is to ensure that expert human insight guides how we evaluate LLMs, keeping people at the center of technological development,” Subbiah concluded.

#ArtificialIntelligence #LLMs #ChatGPT #AIResearch #ComputationalLinguistics #DigitalHumanities #AIethics #MachineLearning #LiteraryAnalysis #ResearchInnovation #ColumbiaEngineering #FutureOfAI