Researchers at the University of Maryland, College Park and Google DeepMind say AI-written short fiction can be separated from human fiction by looking past punctuation tics and into the machinery of the story itself.
In a preprint study, the researchers analyzed more than 50,000 AI-generated short stories and found recurring narrative habits across large language models. According to the study, AI stories tend to explain their themes too directly, keep plots on a narrower track, and present cleaner moral arcs than human-written stories. Human fiction in the dataset showed more variation, more complicated time structures, and more ambiguous character choices.
The finding is a useful annoyance for anyone selling generative text as a replacement for writers: the detector is not just counting em dashes or hunting for a suspicious fondness for the word “delve.” It is asking how the story works.
The researchers call the detector StoryScope. Jenna Russell, a University of Maryland researcher and one of the study’s authors, told 404 Media the project was an attempt to move “under the surface” of standard AI text detection and focus on narrative features rather than style alone. Russell is also an intern at Pangram, an AI-detection company.
How StoryScope tested the stories
StoryScope builds on NarraBench, a 2025 benchmark that organized fiction according to narrative traits. The new system examined features including plot development, character description, setting, and the handling of time.
For the test, the researchers selected 10,272 human-written stories and used Gemini 2.5 to turn them back into prompts. They then gave those prompts to Gemini 3 Flash, DeepSeek V3.2, Claude Sonnet 4.6, Kimi K2.5, and GPT 5.4. The prompts and AI outputs have been posted on Hugging Face, according to the study.
The human stories came from Books3, a dataset of about 183,000 books assembled from pirated ebooks. Books3 has been tied to multiple lawsuits and has been used to train an unknown number of large language models. The StoryScope study included more than 10,000 well-known short stories, with authors including Joyce Carol Oates, Stephen King, Louis L’Amour, Charlotte Perkins, and Harlan Ellison.
Russell told 404 Media that the dataset was controversial and said that was why the researchers did not release it publicly. The paper includes a disclosure saying the authors acknowledge copyright concerns around Books3, do not endorse using it for model training or commercial text generation, and limited its use to academic research on authorship, detection, and copyright policy.
What the models gave away
The study reported model-specific habits. Claude produced weaker event escalation, GPT relied more on dream sequences, and Gemini leaned on external descriptions of characters.
Across systems, the researchers found that AI stories made themes explicit more often than human stories. The study said narrators directly stated a story’s theme in 77 percent of AI stories, compared with 52 percent of human stories. AI dialogue was also more likely to become philosophical argument, at 59 percent versus 34 percent, and references to other works were more often vague rather than named.
The paper also found that AI stories used fewer subplots and showed less willingness to rearrange time through flashbacks or jumps. The systems often substituted sensory over-description for simpler emotional statements. Human stories, according to the researchers, tended to involve more characters, more locations, and more specific cultural references.
The preprint also disclosed its own use of AI tools. It said large language models and coding agents, including Claude Code and Codex, helped polish writing and generate some tables and plots. Russell told 404 Media she used AI agents for code implementation and as editing assistants, while manually reviewing their suggested changes.
Russell said many readers and teachers may care less about whether AI touched a draft than whether the human author supplied the understanding or creativity behind it. StoryScope is an attempt to make that distinction less mystical and more measurable, even if the underlying dataset keeps the copyright lawyers awake.
This story draws on original reporting from 404 Media.
