用小说摘要追踪 LLM 的概念参与度 Attention Flows: Tracing LLM Conceptual Engagement via Story Summaries

精读摘要

LLM 的上下文长度不断增长，但整合长篇文本信息的能力是否同步提升仍存疑。研究选取「写小说摘要」这一理解任务：人类作者压缩故事时，取舍本身暴露了他们眼中叙事重要的部分。团队对齐 150 篇人类摘要与 LLM 摘要的句子，借此测量模型的概念参与模式是否与人类一致，为长文本理解能力提供了新的测量维度。 LLM context lengths have grown, but evidence suggests their ability to integrate information across long-form texts has not kept pace. By comparing human and LLM-authored novel summaries, where compression choices reveal what is narratively important, the authors align sentences from 150 human-written summaries to measure whether models mirror human patterns of conceptual engagement.