OpenAI has developed a tool that can catch students cheating by having them write assignments in ChatGPT, but the company is debating whether to actually release the tool, according to the Wall Street Journal.
In a statement to TechCrunch, an OpenAI spokesperson confirmed that the company is researching the text watermarking method described in the journal article, but said it is taking a “deliberate approach” due to “the complexity involved and the potential impact on the broader ecosystem beyond OpenAI.”
“While the text watermarking method we are developing is technically promising, it has significant risks that we are considering as we explore alternatives, including the potential for it to be vulnerable to bypass by bad actors and to disproportionately impact groups such as those who do not speak English,” the spokesperson said.
This is a different approach than most previous attempts to detect AI-generated text, which have largely been ineffective—OpenAI itself discontinued its previous AI text detector last year, citing “low accuracy.”
With text watermarking, OpenAI will focus solely on detecting ChatGPT’s writing, rather than other companies’ models. It will slightly change the way ChatGPT selects words, creating an invisible watermark on the writing that can later be detected by a separate tool.
After the journal article was published, OpenAI also updated its May blog post about its research into AI-generated content detection. The update states that text watermarking has proven to be “highly accurate and effective against localized tampering, such as paraphrasing,” but “less robust against globalized tampering, such as using a translation system, rephrasing with another generative model, or inserting special characters between every word and then asking the model to remove those characters.”
As a result, OpenAI wrote that the method is “trivial for a bad actor to circumvent.” OpenAI’s update also echoed the spokesperson’s point about non-English speakers, writing that text watermarking “could stigmatize the use of AI as a useful writing tool for non-English speakers.”
