Five weeks after Kanav Kariya stepped down as President of Jump Crypto amidst a Commodity Futures Trading Commission investigation, significant developments have unfolded.
Meanwhile, OpenAI is holding back a new “highly accurate” tool designed to detect content generated by ChatGPT. The company is concerned that the tool could be tampered with or that non-English users might avoid generating text with AI models.
In a May blog post, OpenAI mentioned it was working on various methods to identify content created by its products. However, on August 4, the Wall Street Journal reported that plans to release these tools had stalled due to internal debates about their potential impact.
Following this report, OpenAI updated its blog post, stating there is still no timeline for the release of the detection tools. The company acknowledged that one tool is “highly accurate and effective against localized tampering.” However, there are still ways for bad actors to bypass detection, which has led to hesitation in releasing it publicly.
OpenAI also indicated that non-English speakers could be “stigmatized” by the use of AI products due to an exploit related to translating English text to bypass detection. The company expressed concern that watermarking methods might disproportionately affect some groups, including non-native English speakers.
Currently, many products claim to detect AI-generated content, but none have shown high accuracy across general tasks in peer-reviewed research. OpenAI’s system, relying on invisible watermarking and proprietary detection methods, would be the first of its kind for content generated by the company’s models.
OpenAI’s Dilemma Over Releasing AI Text Detection Technology
OpenAI has developed technology to reliably detect AI-generated text, according to sources and documents reported by the Wall Street Journal. However, the company is hesitant to release it due to concerns about its business model and the broader implications.
The AI text detector project has been under discussion internally for about two years and has been ready for use for around a year. Despite being “just a matter of pressing a button,” OpenAI is conflicted between transparency and user demands.
An internal survey revealed that nearly one-third of ChatGPT users would be deterred by a text detector. However, a global survey commissioned by OpenAI showed strong support for AI detection and transparency.
The detector could impact non-native English speakers, potentially affecting the broader ecosystem beyond OpenAI. While “technically promising,” it also carries important risks, according to an OpenAI spokesperson.
The tool assesses how likely it is that a document was written by ChatGPT, with a 99.9 percent effectiveness rate given sufficient text. It uses invisible watermarks detectable by OpenAI technology without affecting text quality. However, its reliability after extensive editing or translation is unclear.
OpenAI can only implement this method for ChatGPT and its models, which could drive users to other language models in the absence of an industry standard. This could harm OpenAI’s business without solving the transparency problem.
Staff discussions have also considered offering the detector to educators or companies helping schools and businesses identify AI-authored work and plagiarism.
The tool stems from work by computer science professor Scott Aaronson during a two-year safety project at OpenAI. Aaronson indicated in late 2022 that OpenAI planned to detail the work soon, but nothing has happened yet.
OpenAI last discussed the watermarking method in early June, deciding to evaluate less controversial alternatives. By fall, the company aims to have an internal plan to influence public opinion and legislation on the issue. Without this plan, OpenAI risks losing credibility as responsible actors.
A working text detector could threaten OpenAI’s business model if people devalue AI-generated text. Studies suggest that in private messages, where the human touch is valued, and even when AI text is higher quality, people still prefer human advice when given a choice.
AI-generated text is becoming more common in search results, and some believe this AI text could have significant commercial value as an omniscient answering machine. As AI writing becomes routine, the focus may shift from creation to content and intent. A reliable detector could prevent harmful uses and help schools prove the use of ChatGPT in essays, potentially slowing AI’s disruption of education and detecting automated disinformation campaigns.
OpenAI Weighs the Release of AI-Detection Tool Amid Concerns
OpenAI has developed a tool that could catch students using ChatGPT to write their assignments, but according to The Wall Street Journal, the company is debating whether to release it.
“The text watermarking method we’re developing is technically promising but has important risks we’re weighing while researching alternatives,” the spokesperson said. These include the tool’s susceptibility to circumvention by bad actors and potential disproportionate impacts on non-English speakers.
This approach differs from previous efforts to detect AI-generated text, which have been largely ineffective. OpenAI itself shut down its previous AI text detector last year due to its low accuracy.
With text watermarking, OpenAI would focus solely on detecting writing from ChatGPT, not other companies’ models. It would achieve this by making small changes to how ChatGPT selects words, creating an invisible watermark detectable by a separate tool.
After the Journal’s story, OpenAI updated a May blog post about its research on detecting AI-generated content. The update indicates that text watermarking has proven “highly accurate and effective against localized tampering, such as paraphrasing,” but less robust against global tampering like using translation systems or rewording with another generative model.
As a result, OpenAI writes that this method is “trivial to circumvent by bad actors.” The update also echoes concerns about non-English speakers, noting that text watermarking could “stigmatize the use of AI as a useful writing tool for non-native English speakers.”
