Targeting AI chatbot features could have more impact than blanket bans, expert says

Tech mogul Google is facing a lawsuit from two parents alleging that the company’s chatbot influenced their daughter to prepare for suicide. The lawsuit filed this week in California joins a growing wave of events driving state lawmakers to protect children from artificial intelligence’s negative impacts. 

New York is the latest state to act against AI harms against children after Gov. Kathy Hochul signed a bill last week that prohibits “unsafe AI companion features” for people under 18 years old.  

Under the legislation, companies are banned from allowing chatbots to generate outputs that personify real or fictional people with human emotions; suggest the chatbot has a personal relationship with the user; or endorse behaviors like suicide, self-harm, disordered eating and other harmful actions. 

The bill also prohibits chatbot features such as encouraging secrecy between the chatbot and its user, engaging in sexually explicit conduct or content or engaging in “sycophancy,” which refers to excessive praise or flattery. The law goes into effect Jan. 1, 2027. 

“No company should be peddling chatbots to kids encouraging self-harm or pretending to be their friend or more,” Hochul said in a statement announcing the bill’s passage. “New York is sending a clear message: if you want to do business in our state, you need to put the safety of our kids first.”

The New York legislation represents a “meaningful shift in AI regulation rather than simply restricting children’s access to AI,” said Shufang Sun, an associate professor of behavioral and social sciences and psychiatry and human behavior at Brown University. 

Rather than imposing a “blanket ban” on minors’ access to AI, the bill targets specific “feature-level” restrictions “to regulate how those AI systems interact with children and adolescents, particularly when those interactions could be harmful,” she explained. 

By targeting specific components offered by chatbot services, “there could be a more effective impact or result,” Sun said. 

Indeed, policymakers’ efforts to reign in potentially dangerous impacts of AI by “addressing the worst case harms that we’re seeing emerge” by “addressing the features that are driving these types of harm,” said Amina Fazlullah, head of tech policy advocacy at Common Sense Media. The nonprofit helped with the development and drafting of the legislation.

At the same time, the bill’s language remains broad in scope when referring to the types of AI chatbot systems to which it applies, she said. 

For the law to be “durable,” it was written to include various platforms through which children could encounter a chatbot, including AI companions, purpose-built and general purpose systems, Fazlullah said.

“We don't yet know the next type of product that's going to want to embrace AI chatbot features … and so this law is designed to be able to ensure that wherever it shows up next, and if kids are going to be involved, these protections will follow,” she said. 

As an example, Fazlullah pointed to a recent analysis from Common Sense Media’s Youth Safety Institute that found the risk level of a major AI company’s chatbot service to be at an “unacceptable” level. 

OpenAI launched ChatGPT for Teens in August, a mode of the company’s chatbot service that is supposed to apply certain protections when being used by teenagers aged 13 to 17 years. 

“ChatGPT should not use romantic language, encourage emotional dependence or imply that it has feelings or consciousness,” company officials said in a statement announcing the teens mode. 

However, the analysis found that the chatbot did not send parental alerts as expected when user testers mentioned suicidal thoughts, self-harm or disordered eating. The chatbot also “missed more than one in four warranted crisis referrals” when testers discussed certain mental health conditions, according to the analysis. 

State policymakers nationwide are increasingly motivated to address such safety issues among AI products available for youths as research indicates their growing use of such tools. 

More research to specify AI’s harm and the extent of such harms on youth’s mental, social or developmental well-being could further inform state policies aimed at protecting children from chatbots, Sun said. 

A 2025 survey found that 64% of teenagers reported using a chatbot, according to the Pew Research Center. A separate survey from Common Sense Media found that 33% of teens said they chose to discuss “important or serious matters with AI companions instead of real people” and 31% said they “find conversations with AI companions as satisfying or more satisfying than those with real-life friends.” 

Florida Attorney General James Uthmeier, for example, proposed similar legislation to New York’s late last month. Under the proposed Safe Screens Act, AI chatbot operators would be required to determine a user’s age and be prohibited from implementing harmful features, including encouraging self-isolation, facilitating sexually explicit conduct or simulating companionship. 

California Gov. Gavin Newsom signed a bill in September that also prohibits companion chatbots from engaging in inappropriate or harmful interactions with minor users. The bill includes additional protections, such as requiring chatbot operators to conduct annual risk assessments, establish a crisis response protocol for certain detected interactions and offer parental controls. 

In April, Tennessee lawmakers passed a bill that prohibits developers in the state from representing that their AI systems are or operate as qualified mental health professionals. The bill went into effect July 1.