Skip to main content

Here’s why people think GPT-4 might be getting dumber over time

As impressive as GPT-4 was at launch, some onlookers have observed that has lost some of its accuracy and power. These observations have been posted online for months now, including on the OpenAI forums.

These feelings have been out there for a while, but now we may finally have proof. A study conducted in collaboration with Stanford University and UC Berkeley suggests that GPT-4 has not improved its response proficiency but has in fact gotten worse with further updates to the language model.

Recommended Videos

GPT-4 is getting worse over time, not better.

Many people have reported noticing a significant degradation in the quality of the model responses, but so far, it was all anecdotal.

But now we know.

At least one study shows how the June version of GPT-4 is objectively worse than… pic.twitter.com/whhELYY6M4

— Santiago (@svpino) July 19, 2023

The study, called How Is ChatGPT’s Behavior Changing over Time?, tested the capability between GPT-4 and the prior language version GPT-3.5 between March and June. Testing the two model versions with a data set of 500 problems, researchers observed that GPT-4 had a 97.6% accuracy rate in March with 488 correct answers and a 2.4% accuracy rate in June after GPT-4 had gone through some updates. The model produced only 12 correct answers months later.

Another test used by researchers was a chain-of-thought technique, in which they asked GPT-4 Is 17,077 a prime number? A question of reasoning. Not only did GPT-4 incorrectly answer no, it gave no explanation as to how it came to this conclusion, according to researchers.

ChatGPT being asked about a prime number.
Image used with permission by copyright holder

The study comes just six days after an OpenAI executive tried to quell suspicions that GPT-4 was, in fact, getting dumber. The tweet below implies that the degradation in quality of answers is a psychological phenomenon from being a heavy user.

No, we haven't made GPT-4 dumber. Quite the opposite: we make each new version smarter than the previous one.

Current hypothesis: When you use it more heavily, you start noticing issues you didn't see before.

— Peter Welinder (@npew) July 13, 2023

Notably, GPT-4 is currently available for developers or paid members through ChatGPT Plus. To ask the same question to GPT-3.5 through the ChatGPT free research preview as I did, gets you not only the correct answer but also a detailed explanation of the mathematical process.

Additionally, code generation has suffered with developers at LeetCode having seen the performance of GPT-4 on its dataset of 50 easy problems drop from 52% accuracy to 10% accuracy between March and June.

To add fuel to the fire, Twitter commentator, @svpino noted that there are rumors that OpenAI might be using “smaller and specialized GPT-4 models that act similarly to a large model but are less expensive to run.”

This cheaper and faster option might be leading to a drop in the quality of GPT-4 responses at a crucial time when the parent company has many other large organizations depending on its technology for collaboration.

Not everyone thinks the study proves anything, though. Some have made the point that a change in behavior doesn’t equate to a reduction in capability. This is acknowledged in the study itself, stating that “a model that has a capability may or may not display that capability in response to a particular prompt.” In other words, getting the desired result may require different types of prompts from the user.

When GPT-4 was first announced OpenAI detailed its use of Microsoft Azure AI supercomputers to train the language model for six months, claiming that the result was a 40% higher likelihood of generating the “desired information from user prompts.”

ChatGPT, based on the GPT-3.5 LLM, was already known for having its information challenges, such as having limited knowledge of world events after 2021, which could lead it to fill in gaps with incorrect data. However, information regression appears to be a completely new problem never seen before with the service. Users were looking forward to updates to address the accepted issues.

CEO of OpenAI, Sam Altman recently expressed his disappointment in a tweet in the wake of the Federal Trade Commission launching an investigation into whether ChatGPT has violated consumer protection laws.

“We’re transparent about the limitations of our technology, especially when we fall short. And our capped-profits structure means we aren’t incentivized to make unlimited returns,” he tweeted.

Please enable Javascript to view this content

Fionna Agomuoh
Fionna Agomuoh is a Computing Writer at Digital Trends. She covers a range of topics in the computing space, including…
OpenAI might start watermarking ChatGPT images — but only for free users
OpenAI press image

Everyone has been talking about ChatGPT's new image-generation feature lately, and it seems the excitement isn't over yet. As always, people have been poking around inside the company's apps and this time, they've found mentions of a watermark feature for generated images.

Spotted by X user Tibor Blaho, the line of code image_gen_watermark_for_free seems to suggest that the feature would only slap watermarks on images generated by free users -- giving them yet another incentive to upgrade to a paid subscription.

Read more
OpenAI adjusts AI roadmap for better GPT-5
OpenAI press image

OpenAI is reconfiguring its rollout plan for upcoming AI models. The company’s CEO, Sam Altman shared on social media on Friday that it will delay the launch of its GPT-5 large language model (LLM) in favor of some lighter reasoning models to release first.

The brand will now launch new o3 and o4-mini reasoning models in the coming weeks as an alternative to the GPT-5 launch fans were expecting. In this time, OpenAI will be smoothing out some issues in developing the LLM before a final rollout. The company hasn’t detailed a specific timeline, just indicating that GPT-5 should be available in the coming months.

Read more
ChatGPT Plus is free for a limited time: Here’s how to check if you qualify
chatgpt plus promotional offer for students.

ChatGPT didn't just emerge onto the AI scene, it birthed an entire revolution of AI assistants and agents and made them accessible to consumers who were not so friendly with technology. Despite the space now being overcrowded with numerous intelligent chatbots and wrapper apps, ChatGPT is still the most popular of them all. And while you get plenty of features for free now, ChatGPT Plus, its paid tier, gets deeper thinking abilities, priority in times of traffic surge, and quicker access to new models. The downside, however, it is $20 monthly subscription. Thankfully, a select few people can get it for free now.

OpenAI's CEO and co-founder Sam Altman recently announced on X that ChatGPT Plus will be available for free until the end of May. However, the offer is only applicable if you are a college student, and more specifically, studying in a "degree-granting schools in the United States and Canada." The idea basically is to gain popularity among college-goers by helping them cram more before finals in the coming weeks.

Read more