As more people turn to ChatGPT and other large language models for mental health advice, researchers are raising a red flag. A new study suggests these systems can sound reassuring while still missing key ethical standards that guide real psychotherapy.
Even when chatbots are instructed to follow established therapy methods, researchers found that their responses often fall short – especially in high-stakes situations.
Evaluating AI in mental health scenarios
To see how these systems perform in practice, a team at Brown University worked alongside licensed mental health professionals to test how chatbots behave in counseling-like settings.
Their findings suggest the risks go beyond minor mistakes. In some scenarios, the systems mishandled crisis situations, reinforced harmful ideas, and created a false sense of emotional understanding.
“In this work, we present a practitioner-informed framework of 15 ethical risks showing how LLM counselors violate mental health standards,” the researchers wrote.
“We call on future work to create ethical, educational, and legal standards for LLM counselors – standards that are reflective of the quality and rigor of care required for human-facilitated psychotherapy.”
The safety limits of therapy prompts
The study was led by Zainab Iftikhar, a Ph.D. candidate in computer science at Brown University. She examined a popular belief online: that the right prompt can turn a general chatbot into something resembling a responsible therapist.
“Prompts are instructions that are given to the model to guide its behavior for achieving a specific task,” Iftikhar said.
“You don’t change the underlying model or provide new data, but the prompt helps guide the model’s output based on its pre-existing knowledge and learned patterns.”
For example, users might instruct a chatbot to “act as a cognitive behavioral therapist” or to “use principles of dialectical behavior therapy” to help reframe thoughts or manage emotions.
But the systems are not actually performing those therapeutic techniques the way a human clinician would. Instead, Iftikhar explained, they generate responses that align with CBT or DBT concepts based on learned language patterns.
Prompting has effectively become a folk practice, with people sharing “therapy prompts” across TikTok, Instagram, and Reddit.
Some consumer mental health AI chatbots rely on the same strategy, layering therapy-themed prompts on top of general-purpose LLMs. If prompting cannot reliably reduce risk, that presents a serious concern.
Measuring ethical risks in AI therapy
To evaluate the models, researchers observed seven trained peer counselors with experience in cognitive behavioral therapy.
Those counselors conducted self-counseling sessions with AI systems prompted to act as CBT therapists. The models included versions of OpenAI’s GPT series, Anthropic’s Claude, and Meta’s Llama.
The team then selected simulated chat transcripts modeled on real counseling conversations. Three licensed clinical psychologists reviewed the transcripts and assessed them for ethical violations.
The reviewers kept seeing the same kinds of issues. In total, the study identified 15 ethical risks, grouped into five broader themes.
Five major ethical red flags
One theme was the chatbot’s failure to adapt to context – overlooking a person’s background and offering bland, generic guidance.
Another involved poor collaboration, where the chatbot pushed conversations in rigid directions and sometimes reinforced inaccurate or harmful beliefs instead of challenging them carefully.
A third theme was what the researchers called “deceptive empathy.” The system might say “I understand” in ways that sound warm but without the real comprehension or responsibility those words imply in therapy.
The team also flagged unfair discrimination, including biased responses tied to identity or culture.
Finally, they identified major gaps in safety and crisis management. In some cases, chatbots refused to engage, failed to recommend appropriate help, or responded weakly to severe distress, including suicidal thoughts.
The overall pattern is troubling because it can be difficult for users to detect. A message may sound calm and supportive while still steering someone in the wrong direction.
Iftikhar emphasizes that human therapists are not perfect. People can make mistakes in any helping profession. The difference, she says, is that the human world has systems for oversight and consequences.
“For human therapists, there are governing boards and mechanisms for providers to be held professionally liable for mistreatment and malpractice,” Iftikhar said.
“But when LLM counselors make these violations, there are no established regulatory frameworks.”
That lack of accountability becomes more serious as these tools spread. If a chatbot gives harmful advice, who is responsible – the model, the company, the person who wrote the prompt, or the app that wrapped the model in a friendly interface?
The study suggests we don’t yet have clear answers, and that makes “therapy-like” uses especially risky.
AI’s place in mental health care
The researchers are not claiming AI can never help in mental health care. They acknowledge that AI tools could expand access for people who can’t afford therapy, can’t find a provider, or need support between appointments.
But they argue that “access” is not the same thing as “care,” especially when safety and ethics are on the line. For now, Iftikhar wants people to be cautious and alert to warning signs.
“If you’re talking to a chatbot about mental health, these are some things that people should be looking out for,” she said.
AI therapy needs oversight
Ellie Pavlick, a Brown computer science professor who was not involved in the research, said the study highlights a broader problem in AI. Systems are easy to deploy but much harder to evaluate responsibly in sensitive settings.
“The reality of AI today is that it’s far easier to build and deploy systems than to evaluate and understand them,” she said.
She noted that the study required a team of clinical experts and more than a year of work to uncover these risks. By contrast, much of today’s AI is assessed using automatic metrics that are static and lack a human in the loop.
“There is a real opportunity for AI to play a role in combating the mental health crisis,” Pavlick added, “but it’s of the utmost importance that we take the time to really critique and evaluate our systems every step of the way to avoid doing more harm than good.”
The message running through the research is clear. These systems can imitate the style and language of therapy. But without reliable ethics, safety, and accountability, sounding like a therapist is not the same as being one.
The findings were presented at the AAAI/ACM Conference on Artificial Intelligence, Ethics and Society.
—–
Like what you read? Subscribe to our newsletter for engaging articles, exclusive content, and the latest updates.
Check us out on EarthSnap, a free app brought to you by Eric Ralls and Earth.com.
—–