A new study looked at whether AI could be an automated helper in creative tasks, and the results were mixed. AI appeared to help people who weren’t naturally creative write more original stories, but it reduced creativity for the group as a whole. It’s a tradeoff that could become increasingly common as AI tools impact creative endeavors.
The study, published in Science Advances by researchers Anil Doshi and Oliver Hauser from University College London and the University of Exeter, respectively, is necessarily limited by its focus on short stories, but it does seem to confirm the sentiments expressed by many people. In other words, AI can be helpful, but ultimately doesn’t offer anything truly new to creative endeavors.
“Our study represents an early look at a very big question about how large-scale language models and generative AI will impact human activities, including creativity,” Hauser told TechCrunch in an email. “While there is tremendous potential (and undoubtedly enormous hype) for these technologies to have a major impact on media and creativity, it will be important that AI is actually evaluated rigorously, rather than being implemented broadly under the assumption that it will produce positive outcomes.”
In this experiment, hundreds of people wrote very short stories (about eight sentences) on topics that were suitable for a wide audience. One group just wrote, a second group had the opportunity to consult GPT-4 for a single story idea of a few sentences (they could use as many or as few as they wanted), and a third group got up to five such story starts.
After the stories were written, they were evaluated by both the original authors and a second group of people who knew nothing about the AI-generated twist. These people rated the stories for novelty, usefulness (i.e., publishability), and emotional enjoyment.
Low Creativity, High Benefits… High Creativity, No Benefits
Before writing their stories, participants also completed a word production task that serves as a proxy for creativity. Although it is not a concept that can be measured directly, in this case creativity in writing can at least be roughly estimated (no judgment! Not everyone is a natural or skilled writer).
“Capturing the rich and complex nature of creativity on any scale seems fraught with complications,” Hauser writes. “But there is a wealth of research on human creativity, and lively debate about how best to capture the idea of creativity on a scale.”
They said this approach is widely used in academia and has been well validated in other studies.
What the researchers found was that people with low creativity scores got the lowest scores on the story evaluations, which is a testament to this approach. They also did best when they were given the opportunity to use the story ideas they had generated (which most of the participants in the experiment did).
Stories written by people with low creativity scores were rated lower than others in terms of writing quality, enjoyment, and novelty. When given one AI-generated idea, they scored higher on all measures. When given five to choose from, they scored even higher.
For those who struggle with the creative aspects of writing (at least within this context and definition), AI assistants seem to genuinely improve the quality of their work. This will resonate with many people for whom writing is not a natural fit. And a language model that says, “Try this,” is the prompt they need to finish a paragraph or start a new chapter.
But what about those who scored high on the creativity metric? Did their writing reach new heights? Sadly, no. In fact, those participants saw little or no benefit, or even (very close and controversially) were rated worse. Those on the creative side seem to have produced their best work when there was no AI help at all.
There could be a number of reasons why this situation occurs, but the numbers show that in this situation, AI has had no or negative impact on the naturally creative writer.
flattened
But that wasn't what the researchers were worried about.
In addition to the participants’ subjective story ratings, the researchers also performed some of their own analyses. They used OpenAI’s embedding API to evaluate how similar each story was to other stories in that category (i.e., human-only, one AI option, or five AI options).
They found that the approach to generative AI made the resulting stories closer to the average for their category. In other words, they were more similar and less diverse as a group. The overall difference was in the 9-10% range, so the stories weren’t all duplicates of each other. And who knows, this similarity could be a product of less skilled writers perfecting suggested stories and more creative writers coming up with stories from scratch.
Nonetheless, the findings were sufficient to warrant a caution in the conclusions, and I cannot summarize them in full.
These results suggest increased individual creativity, but at the risk of losing collective novelty. An interesting question in general equilibrium is whether AI-enhanced and inspired stories will be able to produce sufficient variation in subsequent output. Specifically, if the publishing (and self-publishing) industry embraces more generative AI-inspired stories, our findings suggest that the stories produced will become less unique and more similar to each other overall. This downward spiral is analogous to a new social dilemma: if individual writers see their generative AI-inspired writings as more creative, they will have an incentive to use generative AI more in the future, but doing so may further reduce the collective novelty of their stories. In short, our findings suggest that despite the enhanced effects of generative AI on individual creativity, there may be caution if generative AI is adopted more widely in creative work.
It reflects fears in the visual arts and web content that AI will lead to more AI, and that if AI is just what it trains, it could eventually fall into a self-perpetuating cycle of boredom. As generative AI begins to permeate all media, it’s this research that serves as a counterweight to claims of infinite creativity or a new era of AI-generated movies and songs.
Hauser and Dorsey acknowledge that their research is just the beginning. The field is very new, and all research, including theirs, is limited.
“There are many avenues that we expect to explore in future research. For example, implementing generative AI ‘in the wild’ will likely look very different from environments we control,” Hauser wrote. “Ideally, our research will help guide how we interact with technology and how we can continue to ensure diversity in creative ideas, whether in writing, art, or music.”
