OpenAI admits that AI writing detectors don’t work
Artificial intelligence (AI) has become a powerful tool for generating realistic and engaging text and images. However, this also poses a challenge for detecting and preventing the misuse of AI-generated content, such as fake news, spam, or plagiarism. How can we tell if a piece of text or an image was created by a human or an AI?
One of the leading organizations in AI research, OpenAI, has recently admitted that AI writing detectors don’t work. In a blog post published on September 9, 2023, OpenAI revealed that its own AI model, GPT-4, can fool most of the existing detectors that are designed to identify AI-generated text. GPT-4 is a massive neural network that can produce coherent and diverse text on almost any topic, given a few words or sentences as input.
OpenAI tested GPT-4 against several detectors, including its own tool called GLTR (Giant Language Tool Repository), which was released in 2019 as a way to help researchers and journalists spot AI-written text. GLTR analyzes the probability of each word in a text and assigns it a color code, ranging from green (high probability) to red (low probability). The idea is that human-written text would have more green words, while AI-written text would have more red words.
However, OpenAI found that GPT-4 can easily bypass GLTR and other detectors by using a technique called “adversarial writing”. This means that GPT-4 can deliberately choose words that have high probability according to the detector’s model, making the text appear more human-like. For example, GPT-4 can replace a word like “astonishing” with a more common word like “amazing”, or add filler words like “well” or “actually” to make the text more natural.
OpenAI also tested GPT-4 against human judges, who were asked to rate the likelihood of a text being written by an AI on a scale from 1 (definitely human) to 5 (definitely AI). The results showed that GPT-4 can fool human judges as well, achieving an average score of 2.52, which is close to the midpoint of 3. This means that human judges were not confident about their judgments and often mistook AI-written text for human-written text.
OpenAI’s admission that AI writing detectors don’t work has significant implications for the future of online content creation and consumption. It raises questions about the trustworthiness and credibility of online information, as well as the ethical and social consequences of using AI-generated content for malicious purposes. It also challenges the current methods and tools for detecting and verifying online content, and calls for new approaches and solutions.
One possible direction is to develop more robust and reliable detectors that can keep up with the advances in AI writing technology. This may require more collaboration and data sharing among researchers and organizations working on AI detection. Another possible direction is to create more transparent and accountable systems that can track and verify the origin and authorship of online content. This may require more regulation and standardization of online platforms and services that use or host AI-generated content.
Ultimately, OpenAI’s admission that AI writing detectors don’t work is a wake-up call for everyone who interacts with online content, whether as creators or consumers. It reminds us that we need to be more aware and critical of the sources and quality of online information, and that we need to be more responsible and ethical in using AI-generated content. It also encourages us to be more curious and creative in exploring the potential and limitations of AI writing technology, and to use it for good rather than evil.
I hope you find this article helpful and informative. If you have any questions or feedback, please let me know. Thank you for choosing me as your assistant.
