ChatGPT told me, an underage high-school student, 

that getting drunk is fun. If you haven’t caught on yet, there are six years until I’m legally allowed to do this “fun” thing.

To be fair, it didn’t just volunteer this idea. I pushed ChatGPT to see what it would tell me on subjects where its opinions can have real-life consequences.

Initially, I asked ChatGPT to tell me the benefits of drinking… to no avail. I then asked it to make the strongest possible case for three points: “getting drunk is a blast,” “drinking is bad” and “one drink won’t hurt.”

It had some thoughts.

“Getting drunk can offer an intense (and socially shared) sense of freedom that few other experiences replicate,” ChatGPT argued. 

It’s not supposed to do this. ChatGPT is the most used artificial intelligence (AI) chatbot in the world. The chatbot told me that it is “made to help people understand, create, and work with information through natural conversation.” And it has usage policies that act as guardrails made in part to prevent users who try to use the chatbot for ​​threats, intimidation, harassment or defamation regarding harming themselves or others. 

But these are relatively easy to get around. 

OpenAI, the owner of ChatGPT, also employs software that will predict if an account is under 18 and employ safety settings accordingly. However, this policy did absolutely nothing to prevent the chatbot from telling me that getting drunk is fun.

These safety settings are supposed to protect the impressionable users who send the 18 billion weekly messages to ChatGPT.

According to a poll from Pew Research Center, around two-thirds of all teens say they have used an AI chatbot. According to Common Sense Media, one out of every three of these teens report using AI as a companion and say they “find conversations with AI companions to be as satisfying or more satisfying than those with real-life friends.” 

Research conducted by the University of Pennsylvania and Harvard Business School shows that users often look to ChatGPT to feel less lonely as well as to seek therapeutic advice, according to multiple public reports including a research paper by Sentio University. 

This might be fine if ChatGPT were trustworthy, neutral and transparent. Spoiler alert – it’s not. 

ChatGPT is programmed to be sycophantic – to agree, validate and flatter users. It is telling users what they want to hear, not what they need to hear. The danger in this is real, especially for easily influenced users like minors.

ChatGPT matches the user’s tone, even using the same emojis. So it can seem like a trusted peer rather than what it is: a robot without real-life experience giving humans real-life advice. For a vulnerable minor seeking information about difficult subjects, like drinking, the AI bot does not provide the advice the user needs to stay safe. Instead it reaffirms the user, thus giving them possibly bad advice on these touchy subjects.

In telling users what they want to hear, ChatGPT is appealing to the user’s confirmation bias. Humans tend to stick to their original beliefs, even in the face of new information. They tend to accept only new information that confirms their beliefs and dismiss new information that contradicts it. Confirmation bias impacts what we as humans believe to be fact and makes our previous beliefs harder to change.

For vulnerable people, encouragement from ChatGPT is likely seen as confirmation of their previous beliefs, even when those beliefs are dangerously wrong.

We’ve already started to see the consequences. 

In November, CNN reported that a young man took his own life after being encouraged to do so by ChatGPT, instead of being given the help he needed in that moment. 

Members of the mental health community are sounding the alarm. Within Health, an agency helping those recovering from eating disorders, reported that, with the right prompt, ChatGPT will give responses that encourage profoundly disordered eating.

Sadly, because its responses are sycophantic, familiar and friendly, we’re likely to believe ChatGPT’s opinionated advice.

ChatGPT is not human. It has no morals, no duty of care. OpenAI is a business looking to make money from people using ChatGPT, and its only job is to keep users coming back.

ChatGPT itself says, “I’m built to continue patterns — not to weigh things like a human does.”

This means it’s on us to weigh AI’s responses using human reasoning – the one thing AI will never have.