Mamdani to kill the NYC AI chatbot that was caught telling businesses to break the law
David Espejo via Getty Images
New York City Mayor Zohran Mamdani says terminating the ‘unusable’ bot will help close a budget gap.
This article is co-reported with THE CITY, a non-profit newsroom that serves the people of New York. Sign up for its newsletter, The Scoop.
In a press conference this week on New York City’s $12 billion budget gap, Mayor Zohran Mamdani zeroed in on the previous administration’s artificial intelligence chatbot as one of “a number of different things we're going to pursue for savings.”
The chatbot, which was released by the Eric Adams administration in fall of 2023, was meant to provide business owners with an accessible way to check city rules and regulations. But as first documented by The Markup and THE CITY, the bot provided answers that, if followed, would lead to illegal behavior by businesses, like taking a cut of employees’ tips.
A spokesperson for the mayor, Dora Pekec, confirmed in a text message that the new administration plans to take down the chatbot. She said a member of the Mamdani transition team had seen reporting on the bot from The Markup and THE CITY and presented it to the mayor as a possible place to save funds.
At the press conference, Mamdani blamed Adams for the budget shortfall, saying he had been handed “a poisoned chalice.” To close the deficit, he said he would raise taxes on the wealthy and corporations and look “under the hood” of the city's budget for potential savings.
When pressed by reporters on what he might cut, he singled out the chatbot.
“The previous administration had an AI chatbot that was functionally unusable,” Mamdani said. “It was costing the administration around half a million dollars. That, in and of itself, is not something that can bridge this kind of a gap, but it's an indication of the ways in which money has been spent while refusing to account for the actual costs of what these programs are.”
The bot, built using Microsoft’s cloud computing platform, was part of an ambitious overhaul of digital services in New York called MyCity. The project was meant to streamline access to government but was criticized for relying on outside contractors.
It wasn’t clear how much it cost to maintain the chatbot. Just building the bot’s foundations reportedly cost nearly $600,000, close to the figure Mamdani provided. Pekec said they didn’t yet have a date for taking down the bot.
A Broken Bot
Testing by The Markup and THE CITY in 2024 showed that, despite promises from the Adams administration, the chatbot would confidently provide incorrect and potentially harmful information to visitors, even on high-stakes topics.
When asked about housing policy, for example, the bot suggested landlords could discriminate against tenants with Section 8 vouchers. Despite being an intended resource for business owners, the bot didn’t know the minimum wage, and told users it was fine to refuse to accept cash for payment despite a city law to the contrary, enacted in 2020.
After The Markup and THE CITY’s initial report was published, readers continued peppering the bot with sometimes farcical questions, which it continued failing to answer properly. The Adams administration defended the bot, saying it would improve over time.
“We’re identifying what the problems are, we’re gonna fix them, and we’re going to have the best chatbot system on the globe,” Adams said at a press conference. “People are going to come and watch what we’re doing in New York City.”
City administrators soon added disclaimers to the bot advising users to “not use its responses as legal or professional advice.” They also improved some of the bot’s answers, but also appeared to limit the kinds of questions the tool was willing to answer.
Today, the bot advises visitors to “ask an NYC government question only” and cautions that “responses may occasionally produce inaccurate or incomplete content.” Visitors must agree to accept the bot’s limitations before using it.
This article was originally published on The Markup and was republished under the Creative Commons Attribution-NonCommercial-NoDerivatives license.