New Jersey offers guidance for building generative AI tools

As the public sector increasingly dabbles in generative artificial intelligence technology, New Jersey has released a guide to assist officials looking to develop their own gen AI tools. 

The state developed the guide in a bid to “help others be able to use [generative] AI to help their teams not hit the same sort of roadblocks that we've overcome,” said Jessica Lax, senior advisor for responsible AI for the New Jersey Office of Innovation and co-author of the guide. 

Released last week, the guide encourages users to consider if generative AI is an appropriate solution in the first place. 

“Not every government process is appropriate for AI, even if the technology could handle it,” the authors wrote. 

Generative AI is more suited for use cases like administrative tasks, writing user-friendly content, drafting translations and summarizing information, according to the guide. 

As an example, Lax pointed to the state’s generative AI tool that creates drafts of permit instructions with plain language and streamlined information to expedite the business creation process for residents. The product has reduced the time it takes to research and draft permit content by 3.5 hours, which could have taken up to eight hours previously, according to state officials. 

“There’s nearly 2,000 permits in the state that a business might need,” she said. The state’s generative AI-enabled Permit Drafter tool “looks to translate some of these complex permits into really simple plain language.” 

With the tool, officials gather research on a permit — including information like who should obtain it, its requirements, and necessary steps for applicants — to feed into a large language model. The model also considers prompts like how to format the document or relevant style guidelines, Lax said. 

The tool then outputs draft instructions for obtaining a permit that are reviewed and edited by human staff, she explained. The product “makes it easier to do business in New Jersey” and “saves us a lot of time,” Lax added. 

To help confirm if generative AI is efficient for a particular use case, staff can test it with an AI-enabled chat interface. New Jersey staff, for instance, leverage products like the state’s AI assistant, Microsoft Azure or Amazon Bedrock to do so, according to the guide. 

Such tools can output draft explanations of how AI could, for example, complete a certain task. Staff should also compare the outputs with human-made products, Lax explained. 

When comparing AI-generated outputs and human-made content, staff should ask themselves, “Does this look like what the content strategist created?” Lax said. “Sometimes it's yes, sometimes it's no. If the answer is no, [that] indicates either my prompt needs to be refined or maybe AI can't solve this issue.” 

Another critical step to building a successful generative AI tool is ensuring that it is processing data efficiently and correctly, according to the guide. 

“You won’t know if an LLM is reading your data or hallucinating without testing. When LLMs can’t access file data, they won’t tell you; instead, they make up responses without mentioning the problem,” the authors wrote. 

Staff need to do “proactive testing” by, for example, prompting the LLM to recite information found on a specific page of content, Lax said. This means that some data may need to be reprocessed into a format that the tool can properly read. 

The guide also suggests considering if certain processing approaches are more efficient for specific types of data. Optical character recognition, for example, is better for processing scanned, typewritten text images than sources like charts, images or page layouts, according to the guide. 

Staff also need to account for the different ways an LLM connects with an application programming interface, Lax said. Standard APIs, for example, are more useful for simple LLM implementations like “one-off” interactions, Lax said. 

For more complex LLMs, staff should consider leveraging agentic APIs, which enable LLMs to integrate with additional tools like databases or web crawling, according to the guide. 

Lax also highlighted the importance of addressing LLM timeouts, which were “one of the largest things that we had to navigate,” she said. Timeouts occur when a request to an LLM takes too long to process due to factors like a high server load or a complex query. 

As an example, the guide points to the Lambda service from Amazon Web Services. Staff can address timeouts in tools like Lambda through code optimization by reviewing it for inefficiencies, considering implementing asynchronous or parallel processing or improving database queries, according to the guide. 

Lax added that state leaders will continue to explore and experiment with navigating LLM timeouts, which could result in additional insights as they expand their work with the technology. 

More broadly, the guide “is a living document to us. We’re going to keep updating it as we do more work and discover more things,” said Jessica Lax, senior advisor for responsible AI for the New Jersey Office of Innovation. “AI is changing rapidly all the time, and I think a document that keeps that to light is so important.”