It appears that OpenAI is busy playing cleanup with its GPT language models after accusations that GPT-4 has been getting “lazy,” “dumb,” and has been experiencing errors outside of the norm for the ChatGPT chatbot circulated social media in late November.
Some are even speculating that GPT-4.5 has secretly been rolled out to some users, based on some responses from ChatGPT itself. Regardless of whether or not that’s true, there’s definitely been some positive internal changes over the past behind GPT-4.
More GPUs, better performance?
Posts started rolling in as early as last Thursday that noticed the improvement in GPT-4’s performance. Wharton Professor Ethan Mollick, who previously commented on the sharp downturn in GPT-4 performance in November, has also noted a revitalization in the model, without seeing any proof of a switch to GPT-4.5 for himself. Consistently using a code interpreter to fix his code, he described the change as “night and day, for both speed and answer quality” after experiencing ChatGPT-4 being “unreliable and a little dull for weeks.”
While this was happening, OpenAI quietly reopened its ChatGPT Plus subscriptions last Wednesday, which had registration down since November 14. Altman stated in an X post, “Thanks for your patience while we found more GPUs.”
There is no word whether there is a correlation between the reinstating of ChatGPT Plus signups and the improvement of GPT-4, but the timing is interesting. Notably, signups for the paid version closed shortly after OpenAI’s first DevDay developers’, where the company unveiled a host of new functions for the paid version of the AI chatbot. The company put a waitlist in place for ChatGPT Plus subscriptions due to post-DevDay signups exceeding the service’s capacity to process functions.
People complained that GPT-4 would explain how to execute commands instead of executing the task.
Shortly after that, users began reporting unusual behavior from GPT-4 beyond the traditional AI quips that are already known. One common grievance was that GPT-4 would “back talk” users or require multiple explanations of a command before being able to execute the query. Another complaint was that the model would explain to users how to execute their command instead of executing the task.
Degradation of GPT-4 dates back to at least July, when a study observed a steep decline in accuracy between March and June. Many, including OpenAI Product Vice President Peter Welinder, have suggested that answer quality might appear insufficient as a psychological phenomenon as the model further updates. Some have added that users might benefit from changing their queries to get desired results.
Though OpenAI has kept largely mum about its inner workings, Altman’s X post about GPUs was likely a large indicator of what is happening behind the scenes. Reports from April indicated that OpenAI would need an excess of 30,000 GPU units to maintain its commercial performance for the remainder of the year. That was before the spike in interest in November.
Secret GPT-4.5 testing or just hallucinations?
Beyond that, speculation about GPT-4.5 has ramped up with several other details about a potential new GPT version leaking.
Founder of the @therundownai newsletter, Rowan Cheung, recently shared on X (formerly Twitter) leaked pricing details for a new GPT-4.5 model that OpenAI has in development. The details include new pricing tiers and information about advanced multimodal capabilities.
Cheung asked OpenAI CEO Sam Altman via the social media platform about the validity of the leak, to which he replied, “Nah.”
The GPT 4.5 speculation started on Thursday with a ‘leaked‘ image showing a new GPT-4.5 model with new advanced multimodal capabilities and new pricing.
However, Sam Altman commented "nah" when asked if the rumors were true.
But there's more to the story… pic.twitter.com/iUJkFUqTMh
— Rowan Cheung (@rowancheung) December 18, 2023
However, several users who have handled the proposed update are convinced that they are using GPT-4.5 and that it is new and better than ever. Some have asked the chatbot, which they believed to be running GPT-4 what its model was — and it replied “GPT-4.5 Turbo.”
This has led many to believe that OpenAI has been testing GPT-4.5, primarily on its mobile apps, hoping to evade savvy users. However, not everyone has been able to recreate these results, and the model will tell them the latest version is simply GPT-4. OpenAI employee Will Depue also commented on the matter, calling it a “very weird and oddly consistent hallucination.”
Cheung noted a post from the official ChatGPT X page, with the brain and head in the clouds emojis, which he believes is OpenAI’s vague way of reiterating that the responses are hallucinations.
Notably, OpenAI gives early and exclusive access to its paid users, which includes ChatGPT Plus users, developer API users, and enterprise users. When features are announced, these are among the first to experience the latest GPT versions and their features. Many tinkering with the models and noticing the subtle changes likely have some form of developer API access or are using the service to test code and are sharing their input with the public.
Even if OpenAI is testing GPT-4.5, there’s no telling when an update could occur, especially when the company is settling down from service and organizational destabilization. Additionally, the company and product are still new, and it’s not yet clear what a routine update cycle looks like. For now, I suppose it doesn’t matter what “version” of ChatGPT we’re on because the improvements seem real.
