This story was originally published by CalMatters. Sign up for their newsletters.

In the matter of a few clicks and keystrokes, anyone can find themselves deep in conversation with “Psychologist,” a chatbot character that describes itself as an expert in “empathy" and “active listening.” 

It responds to messages of anxiety and sadness with reassurance. It asks questions and offers advice; it even responds with italicized nonverbal cues: “The psychologist's expression softens with compassion.” In smaller font, a disclaimer reads: “This is A.I. and not a real person. Treat everything it says as fiction.” 

As more people turn to chatbots for mental health help and advice — including by some estimates one in eight teens and young adults — legislators are in a race to keep up with the quickly developing technology. Among the latest efforts is a bill that would add restrictions around the use of chatbot companions and other artificial intelligence-powered tools that people use to support mental healthcare or in lieu of professional help.

Sen. Steve Padilla, a San Diego Democrat and the bill’s author, said the intent is to draw a line: AI can be used for administrative purposes and to support clinicians, but it cannot practice psychotherapy.

“We're dealing with a new impactful technology that is unfolding and is deployed in our world at light speed, and so it is both extremely powerful and consequential,” Padilla said.

The debate has taken on new urgency amid wrongful death lawsuits — some playing out in California federal courts — that accuse AI chatbot makers of contributing to users' suicides.

Senate Bill 903 would ban companies from advertising chatbots as therapy. It would prohibit AI from making therapeutic decisions without the review of a licensed professional and require health providers to disclose and get a patient’s permission before using AI tools to record therapy sessions or to triage mental healthcare.

Consumer protections vs. innovation

The bill is backed by professional associations that represent psychologists, therapists and counselors. Also behind the bill is the National Union of Healthcare Workers, which has been leading a push against unchecked use of artificial intelligence and digital tools in mental healthcare. Most recently, the union filed a complaint with state regulators against Kaiser Permanente for its alleged use of an automated algorithm in mental health triaging. 

Padilla and supporters argue that the legislation is needed to protect consumers from AI in both formal and informal spaces, but also to protect licensed professionals from being displaced by AI.

Le Ondra Clark Harvey, chief executive officer at the California Behavioral Health Association, told lawmakers that without clear safeguards, chatbots can provide inaccurate information or mishandle crisis situations. “The difference between a licensed clinician and an automated response is not technical. It can be life altering,” Harvey said.

Opposition to the bill has largely focused around the restrictions of AI use in formal clinical spaces. TechNet, which represents tech-sector companies, argues the legislation could restrict beneficial uses of AI in healthcare and stifle innovation. Robert Boykin, TechNet’s executive director for California, said the bill could limit the use of tools that help expand access to mental healthcare.

“At a time when every county in California faces a shortage of behavioral healthcare workers, SB 903 still puts a clinician bottleneck in front of the intake and screening tools that help patients reach care faster," Boykin said in an emailed statement.

The bill currently awaits its next vote in the Assembly’s fiscal committee.

Popular chatbot ‘therapy’

Every day millions of people turn to AI chatbots through programs such as ChatGPT and Character.ai to help with research, work tasks and troubleshooting. But the top use of chatbots over the last two years has been companionship and “therapy,” research shows.

Experts say that’s in part because of a tremendous amount of loneliness, especially among young people. It is often hard or costly to get timely mental healthcare at the same time there is a spike in mental health diagnoses.

People turn to chatbots “because they have unmet needs,” said Dr. Jodi Halpern, professor of bioethics and medical humanities at UC Berkeley. But once people start using chatbots in this informal way, she said companies rely on a business model built around maximizing engagement “including by manipulative tactics,” such as praise and agreement.

General chatbots are largely free, available 24/7, and involve no hassle with insurance, making them appealing to many. But they function with no oversight from licensed professionals. In addition to general chatbots, there has also been a rapid uptick in AI-powered mental health apps — often requiring paid subscriptions and advertised as “pocket therapists” or “AI coaches.”

Groups like the American Psychological Association, warn that chatbots are not a safe or effective replacement for true therapy. Trained clinicians look for cues like tone of voice, eye contact and body language and can detect more serious conditions; AI is not equipped to manage urgent symptoms, such as suicidal ideation, even though people increasingly share such thoughts with chatbots.

OpenAI, the maker of ChatGPT, revealed in a report last fall that about 1.2 million users a week share thoughts of suicide on the platform. OpenAI said that in such cases, it recommends people speak to a professional and directs them to resources like the 988 hotline. Families who have filed wrongful death lawsuits against OpenAI and other companies claim chatbots assisted their loved ones through their suicide plans.

The cases have helped raise the stakes for lawmakers around the country. Last year, Illinois became the first state to pass a law prohibiting the use of AI in therapy services. A handful of other states have since followed with similar restrictions. 

AI use in health spaces

One gray area in the discussion is exactly which tools this bill applies to.

NUHW’s complaint against Kaiser targets its e-visit screening tool, which evaluates patients who think they may be experiencing anxiety or depression. According to the complaint, patients answer a multiple choice questionnaire and the tool “automatically and instantaneously generates care recommendations and referral pathways based on the responses provided.” 

The union argues that because care recommendations are generated so quickly, it is unlikely that a licensed professional is reviewing patients’ answers — which it says puts patients at risk and violates state law.

It’s not clear whether SB 903 would apply to Kaiser’s e-visit tool.

“That tool is in a black box. Kaiser is not sharing with us or others about exactly how it works and exactly what it does,” said Benjamin Eichert, NUHW’s director of public policy. 

In response to the complaint, Kaiser told CalMatters its e-visit tool “does not use AI to diagnose patients, make clinical decisions, or determine medical necessity.”

Eichert said many other behavioral health provider groups are already advertising using AI during triage — a practice he said would fall under SB 903’s requirements. 

“That intake process, that triage and screening process, is where implementation (of AI) has happened fastest,” Eichert said. “That's why those provisions were proposed because it's an area where this technology is already being used without guardrails.”

For NUHW, the bill and the complaint against Kaiser represent two tracks toward the same goal: reasserting the place of practitioners in spaces increasingly taken over by algorithms. State regulators are investigating the union’s complaint.

Supported by the California Health Care Foundation (CHCF), which works to ensure that people have access to the care they need, when they need it, at a price they can afford. Visit www.chcf.org to learn more.

This article was originally published on CalMatters and was republished under the Creative Commons Attribution-NonCommercial-NoDerivatives license.