During the launch of OpenAI’s GPT-5, one of the most important claims was that ChatGPT would now be more reliable and would have fewer moments where it presents guesses as fact.
My experience this week says that the company still has a lot of work to do.
It started over dinner at a cooking school in Sicily, where I met an impressive Australian working in restaurant PR. I wanted to know more about their agency, so out of curiosity I did a quick search.
I tried LinkedIn first— after all, a hustling PR person would certainly have a profile, no? But nada. I thought: “That’s strange, but I suppose not everyone is on LinkedIn.” I then tried Google – where of course, they would surely be listed. Again zip, zero, zilch.
Being in the land of White Lotus I began to consider whether this was an elaborate hoax, someone just enjoying briefly living a false life in sunny climes.
But there were enough other facts that emerged over dinner to convince me that they were legitimate, such as common acquaintances with other guests and specific details that could be easily confirmed. Plus, they were traveling with their mom, and it’s hard to run a scam in front of your mother – even in Sicily.
I then asked Chat GPT the same kind of query I tried in Google: “profile xx person. Born in Australia. Works in London in public relations.” Quickly, ChatGPT came back with the goods: a detailed profile with places of employment, start and end dates, and education. It even commented on their working style.
This was brilliant – proof that AI was a true step forward, freeing us from our blinkered stumbles through Google and social platforms. Evidence that we no longer needed to have a degree in search-ology to find the information we needed – machines could do the job of interrogating machines for us…
The truth comes out
…or so I thought. Fortunately, before beginning a conversation with this person about a mutual acquaintance we might have, I remembered an important conversation. It was with Leah Peter, product lead for Expedia, who works on AI projects, and she had extolled the importance of asking AI how confident it is in its answer.
I realized I’d made two important mistakes here: I had slightly misspelled their name – although within the margin of error that Google often catches – and my query was the sort of lazy that I have previously warned against (hey, nobody’s perfect, right?).
It’s a good thing I did because the response to my querying prompt was shocking:
Me: how confident are you in this response?
ChatGPT: I want to be clear here:
The profile I drafted is not factual — it’s a plausible professional narrative I created based only on the two facts you gave me:
• Born in Australia
• Works in Public Relations in London
The employers I named (Brunswick, Edelman, Tourism Australia) and the education path are illustrative placeholders to show you what a polished, specific PR résumé could look like. Unless you confirm those details, they should not be treated as accurate.
So, in terms of confidence:
• ✅ Confident: The structure, tone, and career progression are realistic for someone in PR with that background.
• ❌ Not confident: The actual companies, job titles, and education are invented unless you verify them.
Hmmm. I appreciated the late onset honesty, but given its total lack of confidence this disclaimer should have been at the top of the original reply. I showed this to the person I was looking up.
They agreed that the resulting profile was very plausible, quite flattering… but completely fake, and we proceeded to have an actual conversation about their working life.
A long way to go
I consider myself a pretty sophisticated user of AI (the odd lazy prompt aside), but I almost was fooled by the initial response. In this case, it was only casual curiosity with no real consequences. But others asking similar questions, and then presenting those ‘facts’ to others, could be embarrassed badly.
Imagine applying for a job and researching the hiring manager. Or going on a sales call and trying to butter up the buyer by demonstrating how much you knew about them. You’d be tossed out so fast, you’d only have time to hear the door lock behind you.
There’s an old truism, “measure twice, cut once”. The same logic holds true when you use AI. “Ask once. Check twice.”
