NYC’s AI ban confusion: what’s allowed and what’s banned in city schools

This article was originally published by Chalkbeat.

Days before the school year started, Mayor Zohran Mamdani announced new restrictions on student-facing artificial intelligence tools used in hundreds of New York City public schools.

“We will fully discontinue or disable the AI components of more than 38 previously allowed programs that do not meet our new safety and oversight standards,” the mayor said, though he did not share their names.

After more than a month of requests, Chalkbeat obtained the list of products flagged by the Education Department. It includes widely used reading and math curriculums, student assessment tools, and even a chatbot the Education Department itself developed.

Chalkbeat reached out to the companies behind all 38 products on the list. Several insisted their products don’t use the kind of artificial intelligence prohibited under the city’s policy and said they were unsure why they were flagged.

The Education Department said its restrictions apply to student-facing generative AI features, not necessarily the entire products, and that it is working with companies to determine whether those features can be disabled. A handful of companies said they have already turned off the AI elements of their tools and have been given the green light to operate in city schools.

The companies’ confusion raises questions about how the Education Department vetted the products and implemented sweeping new restrictions just as the school year was getting underway. (The full list of flagged products, along with company responses, appears below.)

Companies Question Why Their Products Were Flagged

Some products were banned outright, including Amira, the controversial AI-powered reading tutor used in more than 200 city schools. Education Department officials said it could not function without AI. The city also discontinued student access to the Education Department’s own chatbot “teaching assistant” that it built in partnership with Microsoft and was deployed in six schools last year.

But other companies were baffled that their products appeared on the list at all.

Wordly, a translation tool that helps educators communicate with students who are learning English, was flagged even though the company says it does not use generative AI, and the city’s policy has a carveout for tools that help English language learners.

“Wordly is working on getting our product off the banned protocol,” a company spokesperson wrote.

Multiple products from HMH — a major player in the city’s new reading curriculum mandate and which employs former New York City schools Chancellor Melissa Aviles-Ramos — were also on the list of products that violate city policy. One of them is Read180, a program designed to help struggling readers. Another HMH product, Arriba la Lectura, is allowed for students learning English but not other students, according to the Education Department. HMH contends that none of its products on the city’s list use student-facing AI.

An Education Department spokesperson said they flagged products with student-facing generative AI but did not explain how they determined which tools met that definition, including in cases where companies disputed that their tools use AI. Officials said they are working with companies on a case-by-case basis to determine if their products can be used without AI and be reinstated.

“We will continue to evaluate the evolving AI landscape and take the steps necessary to ensure implementation aligns with our student protections, individual needs, and educational priorities,” Education Department spokesperson Onika Richards said in a statement. She said parents can contact their schools or local community education councils if they want more information about what tech tools their schools are using.

The list is not exhaustive. Officials confirmed that other products, including the AI-powered tutoring platform Coursemojo and Google’s Gemini chatbot, are also banned. (Education Department officials said Gemini was left out because it was already being deactivated for students, though it is unclear why that excluded it from the list.)

Some critics cautioned against taking the companies’ claims about their AI features at face value and called for a more thorough and transparent vetting process.

“You cannot trust the companies that they’re not using student-facing AI,” said Leonie Haimson, an education technology critic who has raised concerns about whether approved technology tools are adequately protecting student data. “You cannot trust the DOE to independently evaluate and assess these products.”

Other companies acknowledged that their products included AI features prohibited under the city’s new policy and moved quickly to disable them.

All In Learning, a Texas-based company that provides student assessment tools to more than 100 New York City schools, said it scrambled to disable the AI elements of their product as soon as they saw Mamdani’s announcement. The product includes a feature that allows teachers to use AI to help them grade student responses, which is not allowed under the city’s policy.

“We turned on a dime,” said Darren Ward, the company’s chief sales and marketing officer, adding that the AI elements of the tool were not widely used. By the time Education Department officials reached out to the company, Ward said they were already in compliance. Schools were able to keep using the assessment platform without interruption, he noted.

Magma Math, a digital math instruction platform, includes a feature that gives students AI-generated hints when solving problems and wound up on the city’s list of products that violate its AI policy. Company officials said they turned off the AI component on Sept. 4, two days after Mamdani’s announcement. They got the green light from the Education Department to continue operating in city schools on Oct. 5.

“We continue to value student and teacher interactions above all else,” Henrik Appert, the company’s CEO, wrote in a statement.

In addition to turning off AI features, Magma Math reps emphasized that they recently launched another product for districts that are under pressure to reduce screen time: paper versions of its math problems.