© elenabsl/stock.adobe.com
Despite warnings by experts, millions are increasingly turning to popular AI chatbots such as ChatGPT for therapy-style advice. And recent surveys and polls show those who don’t would consider doing so. But just how ready are these chatbots to become the largest provider of mental health support?
Computer scientists at Brown University in the United States revealed that major AI chatbots routinely break mental health ethics rules, emphasising the necessity for legal standards and oversight. The findings were presented in the ‘Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society’(opens in new window).
Therapy by chatbot
The researchers collaborated with 10 practitioners working with an online mental health support platform over an 18-month period. They observed seven trained peer counsellors while they carried out self-counselling chats with large language models (LLMs) such as OpenAI’s GPT series and Anthropic’s Claude that were prompted to act as cognitive behavioural therapists.
“Prompts are instructions that are given to the model to guide its behavior for achieving a specific task. You don’t change the underlying model or provide new data, but the prompt helps guide the model’s output based on its pre-existing knowledge and learned patterns,” explained author and PhD candidate Zainab Iftikhar, who led the research, in a news release(opens in new window).
She elaborated: “For example, a user might prompt the model with: ‘Act as a cognitive behavioral therapist to help me reframe my thoughts,’ or ‘Use principles of dialectical behavior therapy [DBT] to assist me in understanding and managing my emotions.’ While these models do not actually perform these therapeutic techniques like a human would, they rather use their learned patterns to generate responses that align with the concepts of CBT [cognitive behavioural therapy] or DBT based on the input prompt provided.”
The research team then chose simulated chats based on real human counselling conversations. Three clinically licensed psychologists assessed the chats to determine whether there were any ethics violations. The evaluation revealed 15 ethical risks, from mismanaging crisis situations and reinforcing negative beliefs about themselves and others to exhibiting biased responses.
The (im)perfect therapist
“For human therapists, there are governing boards and mechanisms for providers to be held professionally liable for mistreatment and malpractice,” Iftikhar added. “But when LLM counselors make these violations, there are no established regulatory frameworks.”
Computer science professor Ellie Pavlick stressed the need to thoroughly examine AI systems implemented in mental health situations. “The reality of AI today is that it’s far easier to build and deploy systems than to evaluate and understand them. This paper required a team of clinical experts and a study that lasted for more than a year in order to demonstrate these risks. Most work in AI today is evaluated using automatic metrics which, by design, are static and lack a human in the loop.”
She concluded: “There is a real opportunity for AI to play a role in combating the mental health crisis that our society is facing, but it’s of the utmost importance that we take the time to really critique and evaluate our systems every step of the way to avoid doing more harm than good. This work offers a good example of what that can look like.”