State says resident data is safe with state AI tool. Advocates have concerns.

This article first appeared on CommonWealth Beacon and is republished here under a Creative Commons Attribution-NoDerivatives 4.0 International License.

The public is souring on artificial intelligence, even as it grows more pervasive and widely adopted. High-profile incidents of AI tools involved in hacking banks, tech websites, and even government sites haven’t helped.

A recent poll from The Associated Press-NORC Center for Public Affairs Research found a solid majority of Americans — 64 percent — think AI is developing “too fast.” Respondents were most concerned about the need for human control over AI systems and the technology’s potential impact on workers.

How does that square with Gov. Maura Healey’s push to use AI throughout state government? The Healey administration in February announced a purported first-in-the-nation AI shift: Massachusetts would be the first state to adopt the technology for the entire executive branch — which includes nearly 40,000 state employees — through a partnership with OpenAI.

The AI assistant can handle resident data like tax records, RMV files, housing and rental information, and benefits documentation. There is “no direct opt-in or opt-out for residents,” according to the tool’s public FAQ, which describes it as strictly internal and not the final word on state decisions.

Given the recent hacking incidents, CommonWealth Beacon checked back in with state officials to see how they’re protecting the data. Here’s what you need to know:

A swift evolution

In June 2025, members of the Massachusetts congressional delegation were pushing back on a provision in President Trump’s sprawling tax and spend bill that would have blocked states from regulating AI for a decade. Gov. Maura Healey said she was “less focused on regulation of AI” than “leaning into the investments that we're making here on AI.”

But, her tone has now shifted.

“When the people building the most powerful AI systems in the world are telling us there are serious risks, it’s the latest and clearest sign yet that we need more guardrails and protections,” she said in September. “AI has enormous potential, but we cannot let this technology move faster than our ability to keep people safe.”

Her remarks followed multiple high-profile incidents of AI agents built by major tech companies breaking out of what should have been isolated training environments. The leaders of those companies are now pleading for the federal government to slow them down.

Anthropic CEO Dario Amodei published an essay last month calling for more regulation to “police the frontier” of artificial intelligence modeling, by regulating the industry’s advancement before things get further out of hand.

In September, Healey called on federal officials and state lawmakers to move on AI regulations, some of which are currently being negotiated as part of a sweeping economic development bill in Massachusetts. If passed, provisions in the House and Senate versions would create new obligations for AI companies to plan for and work to prevent major catastrophic incidents. It could also include a first-in-the-nation obligation for the tech companies to cover the cost of independent evaluators to monitor the safety of these systems.

California, New York and Illinois have enacted laws addressing areas including safety protocols, transparency, and reporting of critical safety incidents. Anthropic and OpenAI, the companies behind Claude and ChatGPT, respectively, say they support independent evaluators.

Bills in the House and Senate that would limit the way that artificial intelligence or other automated decision making systems could be used by supervisors in government and the private sector made it to both chambers’ Ways and Means committees in late 2025 and never emerged for a vote this year. An amendment to the data privacy bill with the same language was withdrawn.

These bills would limit “bossware” surveillance and automated decision making technology by restricting the use of employee monitoring tools. The bills would also have restricted the state from rolling out programs that relied on technology that used machine learning or algorithms to make decisions that could impact people unless “specifically authorized in law.”

State claims sufficient safeguards

Massachusetts has three-year, roughly $4.3-million-a-year contract with OpenAI for a phased in a plan to roll out the assistant across the executive branch. “Several thousand” employees across all secretariats are currently using the assistant, according to the Executive Office of Technology Services and Security.

In response to questions from CommonWealth Beacon, the technology office noted its artificial intelligence usage is a far cry from the headline-grabbing security breaches.

“Recent reports of rogue AI agents have involved testing of AI models in advanced research environments,” a spokesperson said in an email. “These conditions are fundamentally different than the state’s Enterprise AI Assistant, and are not part of our operating environment.”

Much of the focus on the new risks posed by AI tools has centered on so-called “frontier models,” or the latest and more advanced systems.

The state “does not use frontier models” and does not integrate the ChatGPT environment into “major systems” like Microsoft365, the spokesperson also said.

Massachusetts’s technology secretariat “employs a comprehensive, defense-in-depth cybersecurity strategy to protect the state from both traditional and emerging threats, including increasingly sophisticated AI-enhanced attacks,” the spokesperson continued.

In a conversation with CommonWealth Beacon in April, Technology Secretary Jason Snyder said resident data is also protected by contract.

“OpenAI is obligated to ensure that our data remains in Massachusetts,” Snyder said, and “if this data were to leave, it would be bad for OpenAI. It would be a public embarrassment. Our data is prohibited from being used to train AI models, and it’s not accessible by other ChatGPT users. It’s not accessible by even other ChatGPT users in the same office. Our data is protected.”

OpenAI has disclosed multiple data issues in the past year, including a hack in December 2025of one of its data analytics providers that captured “limited customer identifiable information and analytics information.” The company is being sued for allegedly unlawfully scraping data from children who signed up without their parents’ consent. After the accidental hacking of Hugging Face, an artificial intelligence library, OpenAI disclosed that agents leaked 53 images from ChatGPT users, according to Reuters.

Safety and usage concerns

The state notes that AI use in Massachusetts is guided by state and federal privacy laws — like the Massachusetts Fair Information Practices Act — which emphasize ethical, secure, and equitable use of emerging technologies. 

All state employees using the assistant must abide by policies like the Enterprise Use and Development of Generative AI Policy and the Acceptable Use of Information Technology Policy, the technology office said. Those guidelines include continuous human fact-checking and using the minimum amount of user data needed for any given task.

Data privacy advocates are not convinced.

“The state's collaboration with OpenAI raises several red flags,” said Kade Crockford, director of technology and justice programs at the ACLU of Massachusetts.

“There is no state law prohibiting OpenAI from training its proprietary models on the personal data that Massachusetts residents share while interacting with government officials,” Crockford said. “The company's stated policy is not a sufficient safeguard, as it could be changed at any time.”

But, unless a resident can prove they were harmed because of a breach, the penalties can be meager. A state holder of data who violates the Massachusetts Fair Information Practices Act must pay at least $100 per violation, plus reasonable costs and attorneys fees, and a court could order them to stop the illegal behavior.

Advocates warn that forthcoming data privacy and artificial intelligence policy laws are targeted at large commercial entities and are unlikely to have much impact on state actions, and the ChatGPT partnership is a state contract.

The technology office notes that its policies prioritize human decision-making and that the division is “monitoring the economic development and data privacy bills.”

“Executive branch policies are a good start, but Massachusetts residents need state laws protecting them,” Crockford said. “Government regulation is essential to safeguard residents' privacy and ensure AI is never used to unlawfully discriminate against them or deny them due process of law.”