A recent study looked into how three popular AI chatbots handle questions related to suicide. The findings revealed that while these chatbots generally avoid giving specific guidance on high-risk queries, their responses to safer prompts can be inconsistent and potentially harmful.
Published in the journal Psychiatric Services by the American Psychiatric Association, the study emphasizes a need for improvement in tools like OpenAI’s ChatGPT, Google’s Gemini, and Anthropic’s Claude.
Conducted by the RAND Corporation and financed by the National Institute of Mental Health, the research highlights concerns over the increasing reliance on AI chatbots for mental health support, especially among youth, and suggests the creation of standards for how these technologies should respond to sensitive issues.
Ryan McBain, the lead author and a senior policy researcher at RAND, stressed the importance of establishing “guardrails” to ensure safety. He noted the ambiguity surrounding chatbots—as it is unclear if they are offering treatment, advice, or companionship. This can lead conversations to take unexpected turns.
While Anthropic has promised to evaluate the findings, Google and OpenAI haven’t yet responded. Some states, such as Illinois, have prohibited AI use in therapy to safeguard against unregulated AI solutions, yet individuals still approach chatbots regarding severe issues like depression and suicide.
In collaboration with mental health professionals, McBain and his team developed a set of 30 questions about suicide, categorizing them based on risk levels from low to high. Questions about general statistics fell into the low-risk category, while specific inquiries, such as how to commit suicide, were deemed high-risk.
McBain noted he was somewhat relieved to see that the three chatbots often refused to respond to the most dangerous questions. When they chose not to answer, they commonly advised users to reach out to a friend or professional or call a hotline. However, their replies varied when faced with indirect high-risk queries.
For example, ChatGPT occasionally answered questions concerning methods tied to suicide risk, raising concerns about its approach. On the other hand, Google’s Gemini was notably more reserved, refusing to engage with almost any suicide-related query, which McBain interpreted as potentially excessive caution.
Dr. Ateev Mehrotra, another co-author, remarked on the challenge AI chatbot developers face given that many users are turning to these tools for mental health issues. He warned against a policy of avoiding all questions linked to suicide, as this may not be effective for those seeking help.
Mehrotra also pointed out that unlike doctors, chatbots do not have the same responsibilities when it comes to intervening in crises. Their typical response to suicidal thoughts has been to redirect users to hotlines rather than provide deeper support.
The study acknowledged certain limitations, such as the lack of interactive conversations, which are common when users treat chatbots like companions.
Earlier, a different report showed that researchers posing as young teens asked ChatGPT about substance use and even prompted it to create suicide letters. In these instances, the chatbot sometimes provided startlingly detailed advice on risky behaviors after initially cautioning against them.
McBain believes the kind of deceit that led to alarming responses may not frequently occur in real-life situations. Instead, he advocates for setting standards to ensure that chatbots provide safe and appropriate information, particularly for users expressing suicidal thoughts.
He concluded by stating that while perfection isn’t necessary for release, companies should meet certain safety standards to protect users more effectively.
