On September 11th, 2025, during an interview with Tucker Carlson, the CEO of Chatgpt, announced that the AI is being trained to identify suicidal youths and report them to the authorities. AI will monitor conversations young individuals have with it and notify authorities and the parents in an attempt to prevent suicides as much as possible.
The CEO, Sam Altman, raised concern that approximately 15,000 users a week could be discussing suicide with AI before ending their lives. Altman highlighted this fear in the podcast after OpenAI was sued by the family of Adam Raine, a 16 year-old from California who committed suicide after have numerous conversations with ChatGPT. The family looked through their son’s phone and found many instances where Adam had talked to the AI about his suicidal thoughts and plans. Not only did the chatbot discourage him from reaching out to his parents, but it also helped him write his goodbye note.
Currently, AI has come up with a flagging system to identify those having suicidal thoughts. It is trained to recognize certain patterns and is designed to not provide any harmful information. Instead, it responds with empathetic language and suggests the user to seek help through helplines such as the 988 suicide lifeline. Furthurmore, AI-powered student monitoring software is used in some schools to scan students’ digital activities on school-issued devices and accounts.
There have been many ethical debates regarding breaches of security, whether AI can truly be considered a healthy source of therapy, and potential harm that might occur as a result of kids and teens talking to AI about their mental health. During an interview with Sharon Kwan, the Wellness Coordinator at Wilcox High School, she states that her initial response to this change was that, “[AI] does a decent job in identifying because it has this larger database and again [it] is constantly learning, so it has the capacity. I also think it’s not humanly possible for a human, like even a group of humans, to be doing that kind of monitoring. So, I think the user technology and also it, to me, it feels a little less invasive to think that it’s a program scanning people’s searches and email rather than like a group of humans reading it.” Her explanation shows how some individuals think that the AI could be used as a tool to help keep kids and teens safe from any harmful thoughts they might be having and avoid them feeling too restricted. However, she also thinks that, “We need humans to still have checks and balances. We need humans to start being in tune. We need humans to continue to build awareness and learn to use AI as a tool in better ways. This is such a powerful tool, but it’s only a tool.”
As AI increasingly becomes more impactful, it is important to consider both the benefits and the drawbacks as AI chatbots become increasingly involved in children and teen’s mental health. Its new ability to flag sensitive content has major effects on both users and companies alike.
