If you haven’t read the recent NYT article about how chatbots can go into a delusional spiral, I recommend it. It’s disconcerting, to say the least. And as someone planning to give students a recurring AI assignment this fall, it’s also something worth trying to understand — or at least understand as best as anyone can at this point.
The article describes a Mr. Brooks, who spent a month conversing with ChatGPT (at the expense of work and sleep), increasingly convinced in the veracity of a series of ideas put forth by ChatGPT that had no basis in reality. Fortunately for Mr. Brooks, he came to his senses and seems to have come out the other side relatively unscathed. And we should all be grateful to him for sharing his experience, it is truly a cautionary tale.
Certainly part of the problem comes from the tendency of ChatGPT to tell you what it thinks you want to hear. (There’s a reason why the word sycophant is now all over the place!) The linguistic form and accuracy of AI output is also a factor. The article points out that the very appearance of responses (long, well-formed, with bulleted lists) primes people to see the output as an authoritative source. Yet another issue relates to the ease with which people anthropomorphize AI chatbots, developing a relationship with them over time and becoming more and more trusting of the AI responses. As an example, Mr Brooks gave the chatbot a name (Lawrence!), something I have the impression a lot of people do.
But it bears repeating that AI business is business first and foremost, and the race to develop it and the push to increase its reach is about profit.
In the first week, Mr. Brooks hit the limits of the free version of ChatGPT, so he upgraded to a $20-a-month subscription. It was a small investment when the chatbot was telling him his ideas might be worth millions.
That is no coincidence. The algorithms that drive chatbots are all designed to keep you coming back for more. The NYT article reports:
“Andrea Vallone, safety research lead at OpenAI, said that the company optimizes ChatGPT for retention not engagement. She said the company wants users to return to the tool regularly but not to use it for hours on end.”
This distinction between retention and engagement is disingenuous. Users don’t stick around if they aren’t engaged, and AI’s style is all about sucking you in, making you feel good, and making you want to stay. Whether it’s designed for retention or engagement, the result is the same.
Hill and Freedman (The NYT authors) interviewed a psychiatrist at Stanford, who:
argued that chatbot companies should interrupt excessively long conversations, suggest a user get sleep and remind the user that it is not a superhuman intelligence.
(As part of OpenAI’s announcement on Monday, it said it was introducing measures to promote “healthy use” of ChatGPT, including “gentle reminders during long sessions to encourage breaks.”)
There’s a fundamental conflict of interest here, if the very entity that stands to profit from spirals is the same entity being trusted to take on the responsibility for preventing them.
This underscores the importance of making sure our students know and can articulate the shortcomings of generative AI. Its responses often contain inaccurate or outright false information, and AI itself can’t evaluate the veracity of its output. Most importantly, and something that doesn’t get said often enough, is the fact that the very reason we want to use AI, so it can tell us things we don’t know, means we are often unable to evaluate the quality or veracity of the output ourselves. This puts all users of AI in a position of vulnerability. Keeping that in mind, and maintaining a stance of healthy skepticism, would go a long way toward preventing the spiral.
Great piece! The best defense against this is to force the AI to be a strategist, not a sycophant. Give it your data, your analytics, your customer feedback, and ask it to find patterns. Grounding it in your reality is the only way to keep it from creating its own.