How New Jersey’s AI assistant saves the state time and money

As states race to lead the way on adopting artificial intelligence, one state has stood out for its efforts to train its employees, roll out tools and empower executive leaders.

Code for America recognized New Jersey — alongside Pennsylvania and Utah — last month in its Government AI Landscape Assessment as a leading state for AI adoption. The civic tech nonprofit said New Jersey received recognition for its work to appoint a Chief AI Strategist and launch NJ AI Assistant, a generative AI-powered platform for state employees. Code for America also praised the state’s efforts to train its employees on the technology.

And the state looks to be keeping up the momentum. It is developing how-to guides for generative AI and using the technology to make its call centers more efficient.

Route Fifty recently spoke with New Jersey Chief Innovation Officer Dave Cole to learn more about its efforts with AI.

This interview has been edited for brevity and clarity.

Route Fifty: How has the state’s relationship with AI evolved over the past few years?

Dave Cole: We've been working on unemployment insurance modernization since the pandemic, as an example, and in the process of doing that work, on our Office of Innovation team we found one of the biggest challenges is the communications that go out. Emails and largely print mail that go out are written from a perspective of absolute certainty that the legal compliance is right, but not necessarily that the person receiving the mail has any idea of what it means or what to do with it.

We've been working using general human-centered design principles on improving those email and mail templates, and this was around the time when ChatGPT first came out. We found by working with ChatGPT to put prompts in that were basically instructions for how we worked, we could accomplish the same task in a significantly faster time. We were able to scale our first work on one or two templates or document changes that could take a couple of weeks to get done into being able to do it in an hour, and then do it 100-plus times over. On our Office of Innovation team, that was the first real, concrete user-facing project that we worked on, leveraging generative AI.

Soon after, in late 2023, the governor [Phil Murphy] called together senior leadership and put together a task force to prepare the state to take advantage of generative AI, balancing the opportunity with the potential risks and things that we need to know. That led into a multi-pronged approach, particularly on how we leverage generative AI for responsible use, improving the workforce development of both public sector workers as well as the state workforce.

Route Fifty: What went into developing your AI assistant, especially as you did it in-house, rather than buy one off the shelf?

Cole: Like anything, it's a hybrid. We didn't develop our own large language model. By any measure we are using the OpenAI, generally available GPT-4 model that's available through the Microsoft Azure platform. But what we did develop on our own was the actual website and the chat interface.

We did that for a couple reasons. It allows us to make sure that we have full control of what people log into, so we make sure that it's tied into their official state accounts. We make sure that when they log in, they see an alert that says this tool has certain data and security constraints that are built in, so you can feel safe using it. At the same time, you do need to go through a training program, and we have this interactive training program that links them to that. We really want to tightly couple having access to these tools with training, and the net effect of it is — because it's available within the state infrastructure — it presents as an internal state website, it's familiar to folks. Something like 20% of the state workforce has signed in and become active users, and the costs to us are about $1 per user, per month, versus the generally available enterprise plan that's starting at $20 a month. At 14,000 users a month, that could be saving something like three-plus million dollars a year.

Route Fifty: Let's say I'm a state employee. What kind of things can I do when using your AI assistant?

Cole: It's a lot of the traditional things we think large language models are good at. Document summarization is a big one. This can be a report that folks want to pull out key facts to use for further analysis. Or a use case we see a lot is the government gets a lot of feedback, and very often it's hard to process all that feedback and make sense of it. We've had agencies that will take feedback on rule making, on proposed policy changes, on their website, any number of things, upload all of that as an attachment and then pull out key themes from the feedback.

We work with them on how to structure good prompts, and we'll often encourage them to get a prompt, summarize the feedback, but also ask to pull out specific quotes so that you can then go back to the source material and check to make sure those quotes line up. It's just a little bit of a way to check for potential hallucinations. Those use cases are popular. We've seen people starting to look at it to find duplicative regulations or reporting requirements that they have, but it's a general-purpose tool at this point.

Route Fifty: You mentioned the financial impact. But what are some of the big effects that you've seen from AI already, in terms of the day to day?

Cole: We're also asking people, is this helping you? Is it saving you time? Do you generally find this useful? I think we have well over a majority of users telling us that it's positive, but also giving us feedback, too. Initially we allowed for PDF attachments, but folks want to use Word documents and Excel files and things like that, so we've been able to expand the roadmap by being agile and iterative based on the user feedback there. We feel like it's meeting a need. It's helping folks be more productive in their work.

The Department of Labor for unemployment insurance has been a longtime partner. They've encouraged, and I think reached, full coverage of their entire staff going through the training and getting access to the tool. Some agencies have adopted it and encouraged in their usual program management to look for ways to incorporate this into their work.

Route Fifty: What does training look like on this tool? How do you get people up to speed?

Cole: States, like any large organization, have a lot of mandatory training that you go through every year, so we use the same platform for that. There's a series of video modules available on responsible AI. We partner with an organization called InnovateUS, which is a nonprofit that does a lot of skills and upskilling for the public sector workforce. We developed the training and made them available in the state learning system. The governor put out his own video strongly encouraging people to get started with the training, and then work with the agency heads to roll it out. If I've seen the latest numbers on this, 25 different states have adopted the same training program from the InnovateUS consortium in some way.

Route Fifty: Then in terms of ongoing training, is there a way you keep up with the technology and make sure your employees still know what they're doing as it evolves?

Cole: We're planning refreshes periodically. It's on about an annual schedule, though. It's just been one year so far. Right now, we're breaking the training down into two different courses. There's a quick get started course, then there's a little bit more in depth. The point of that is that eventually we want to break out that second level into training that might be more specific to different types of jobs, so it could be a little bit more tailored. We wanted to be practical, and there's some theoretical information, but concrete use cases, things to look out for, and how to mitigate those are the main points.

We are also, beyond just the video training, trying to put out a bunch of resources about prompts that work. We're updating our acceptable use policy, which is the governing document on how to approach AI. Right now, there are about a dozen use cases of specific ways that we encourage people to consider using AI. We're moving that out of the document into interactive websites so we can keep that living. As we learn more use cases from agencies, we will be able to promote that as well.

Route Fifty: As you look to the future, what are you looking forward to with this technology in the next 12 to 24 months? What's exciting you, and what's worrying you as well?

Cole: In my mind, there’s no question it can be a highly disruptive technology. We are excited by ways that it can be used for good within the government context, but also trying to be very careful about making sure that people know about hallucinations, inherent biases, how to make sure you're not sending information that doesn't need to be in the model that could be considered for decisions. It's honestly not too dissimilar, in my mind, from cybersecurity. We always have had a cybersecurity approach, but now we have to get specific around phishing and the types of attacks that people are going to see in their everyday lives, because it's such a prevalent part of existing in the world. We try to take that position with AI as well. People give them the training; [they should] be really eyes wide open about it but also see the opportunity there and then in terms of where that opportunity might go.

We're really interested in some of the more automated use cases some agencies have had around taking the AI assistant, which is a very manual, interactive chat process, and putting it into something that's more of bulk processing. For instance, if you're a department that issues grants and you have to get a lot of documents as part of your uploads, the AI can be pretty good at scanning a document, extracting data and checking to make sure that it looks like it's a legitimate document. People do that already manually.

If we use an AI system to do that, it can shave hours off their time. But if we also bulk process documents in batches, then it can have a transformative impact on the workflow, such that maybe the user, when they submit the document, gets that real time feedback, and they know they just uploaded the wrong document or there's missing information. They don't have to wait for an agent to look at their file a couple weeks later, tell them, and then use a lot of time.  Getting this more embedded into systems, always with a human in the loop in the review process, I think could have a transformative impact on helping people do business with the government much more quickly and hopefully at a lower cost as well.