Skip to main content

How to detect ChatGPT plagiarism — and why it’s becoming so difficult

Chatbots are hot stuff right now, and ChatGPT is chief among them. But thanks to how powerful and humanlike its responses are, academics, educators, and editors are all dealing with the rising tide of AI-generated plagiarism and cheating. Your old plagiarism detection tools may not be enough to sniff out the real from the fake.

In this article, I talk a little about this nightmarish side of AI chatbots, check out a few online plagiarism detection tools, and explore how dire the situation has become.

ChatGPT and OpenAI logos.
Image used with permission by copyright holder

Lots of detection options

The latest November 2022 release of startup OpenAI’s ChatGPT basically thrusted chatbot prowess into the limelight. It allowed any regular Joe (or any professional) to generate smart, intelligible essays or articles, and solve text-based mathematic problems. To the unaware or inexperienced reader, the AI-created content can quite easily pass as a legit piece of writing, which is why students love it — and teachers hate it.

A great challenge with AI writing tools is their double-edged sword ability to use natural language and grammar to build unique and almost individualized content even if the content itself was drawn from a database. That means the race to beat AI-based cheating is on. Here are some options I found that are available right now for free.

GPT-2 Output Detector comes straight from ChatGPT developer OpenAI to demonstrate that it has a bot capable of detecting chatbot text. Output Detector is easy to use — users just have to enter text into a text field and the tool will immediately provide its assessment of how likely it is that the text came from a human or not.

Two more tools that have clean UIs are Writer AI Content Detector and Content at Scale. You can either add a URL to scan the content (writer only) or manually add text. The results are given a percentage score of how likely it is that the content is human-generated.

GPTZero is a home-brewed beta tool hosted on Streamlit and created by Princeton University student Edward Zen. It’s differs from the rest in how the “algiarism” (AI-assisted plagiarism) model presents its results. GPTZero breaks the metrics into perplexity and burstiness. Burstiness measures overall randomness for all sentences in a text, while perplexity measures randomness in a sentence. The tool assigns a number to both metrics — the lower the number, the greater possibility that the text was created by a bot.

Screenshot of GPTZero.
Image used with permission by copyright holder

Just for fun, I included Giant Language Model Test Room (GLTR), developed by researchers from the MIT-IBM Watson AI Lab and Harvard Natural Language Processing Group. Like GPTZero, it doesn’t present its final results as a clear “human” or “bot” distinction. GLTR basically uses bots to identify text written by bots, since bots are less likely to select unpredictable words. Therefore, the results are presented as a color-coded histogram, ranking AI-generated text versus human-generated text. The greater the amount of unpredictable text, the more likely the text is from a human.

Putting them to the test

Output from GPT-2 Detector.
Image used with permission by copyright holder

All these options might make you think we’re in a good spot with AI detection. But to test the actual effectiveness of each of these tools, I wanted to try it out for myself. So I ran a couple of sample paragraphs that I wrote in response to questions that I also posed to, in this case, ChatGPT.

My first question was a simple one: Why is buying a prebuilt PC frowned upon? Here’s how my own answers compared to the response from ChatGPT.

  My real writing ChatGPT
GPT-2 Output Detector 1.18% fake 36.57% fake
Writer AI 100% human 99% human
Content at Scale 99% human 73% human
GPTZero 80 perplexity 50 perplexity
GLTR 12 of 66 words likely by human 15 or 79 words likely by human

As you can see, most of these apps could tell that my words were genuine, with the first three being the most accurate. But ChatGPT fooled most of these detector apps with its response too. It scored a 99% human on the Writer AI Content Detector app, for starters, and was marked just 36% fake by GPT-based detector. GLTR was the biggest offender, claiming that my own words were equally likely to be written by a human as ChatGPT’s words.

Output with Write AI Detector.
Image used with permission by copyright holder

I decided to give it one more shot, though, and this time, the responses were significantly improved. I asked ChatGPT to provide a summary of the Swiss Federal Institute of Technology’s research into anti-fogging using gold particles. In this example, the detector apps did a much better job at approving my own response and detecting ChatGPT.

  My real writing ChatGPT
GPT-2 Output Detector 9.28% fake 99.97% fake
Writer AI 95% human 2% human
Content at Scale 92% human 0% (Obviously AI)
GPTZero 41 perplexity 23 perplexity
GLTR 15 of 79 words likely by human 4 of 98 words likely by human

The top three tests really showed their strength in this response. And while GLTR still had a hard time seeing my own writing as human, at least it did a good of catching ChatGPT this time.


It’s obvious from the results of each query that online plagiarism detectors aren’t perfect. For more complex answers or pieces of writing (such as in the case of my second prompt), it’s a bit easier for these apps to detect the AI-based writing, while the simpler responses are much more difficult to deduce. But clearly, it’s not what I’d call dependable. Occasionally, these detector tools will misclassify articles or essays as ChatGPT-generated, which is a problem for teachers or editors wanting to rely on them for catching cheaters.

Developers are constantly fine-tuning accuracy and false positive rates, but they’re also bracing for the arrival of GPT-3, which touts a significantly improved dataset and more complex capabilities than GPT-2 (of which ChatGPT is trained from).

At this point, in order to identify content generated by AIs, editors and educators will need to combine judiciousness and a little bit of human intuition with one (or more) of these AI detectors. And for chatbot users who have or are tempted to use chatbots such as Chatsonic, ChatGPT, Notion, or YouChat to pass of their “work” as legit — please don’t. Repurposing content created by a bot (that sources from fixed sources within its database) is still plagiarism no matter how you look at it.

Editors' Recommendations

Aaron Leong
Former Digital Trends Contributor
Aaron enjoys all manner of tech - from mobile (phones/smartwear), audio (headphones/earbuds), computing (gaming/Chromebooks)…
This powerful ChatGPT feature is back from the dead — with a few key changes
A laptop screen shows the home page for ChatGPT, OpenAI's artificial intelligence chatbot.

ChatGPT has just regained the ability to browse the internet to help you find information. That should (hopefully) help you get more accurate, up-to-date data right when you need it, rather than solely relying on the artificial intelligence (AI) chatbot’s rather outdated training data.

As well as giving straight-up answers to your questions based on info found online, ChatGPT developer OpenAI revealed that the tool will provide a link to its sources so you can check the facts yourself. If it turns out that ChatGPT was wrong or misleading, well, that’s just another one for the chatbot’s long list of missteps.

Read more
ChatGPT’s new upgrade finally breaks the text barrier
A person typing on a laptop that is showing the ChatGPT generative AI website.

OpenAI is rolling out new functionalities for ChatGPT that will allow prompts to be executed with images and voice directives in addition to text.

The AI brand announced on Monday that it will be making these new features available over the next two weeks to ChatGPT Plus and Enterprise users. The voice feature is available in iOS and Android in an opt-in capacity, while the images feature is available on all ChatGPT platforms. OpenAI notes it plans to expand the availability of the images and voice features beyond paid users after the staggered rollout.

Read more
Meta is reportedly working on a GPT-4 rival, and it could have dire consequences
The Facebook app icon on an iPhone home screen, with other app icons surrounding it.

Facebook owner Meta is working on an artificial intelligence (AI) system that it hopes will be more powerful than GPT-4, the large language model developed by OpenAI that powers ChatGPT Plus. If successful, that could add much more competition to the world of generative AI chatbots -- and potentially bring a host of serious problems along with it.

According to The Wall Street Journal, Meta is aiming to launch its new AI model in 2024. The company reportedly wants the new model to be “several times more powerful” than Llama 2, the AI tool it launched as recently as July 2023.

Read more