Report: How to make government websites AI friendly

Artificial intelligence is altering how constituents interact with their governments and services, as the technology is increasingly woven into benefits administration, public transportation and, now, website navigation, according to a new report. This presents an opportunity for governments to revise their public websites to ensure they remain a relevant source among artificial intelligence-enabled searches. 

“As more residents turn to AI assistants instead of visiting a government website directly, their ability to find the right answer increasingly depends on if that assistant can actually reach — and correctly interpret — government content,” a new report released last week from Georgetown University’s Beeck Center for Social Impact and Innovation reads. 

The report aims to help government staff responsible for technical and content decisions regarding government websites — such as IT offices, chief information officers or web design teams — prepare their content to be accessible for AI tools used to identify and relay such information, according to the report.

Those tools are known as AI crawlers, or automated programs varying in scope and purpose that sweep websites to collect information, which are increasingly requiring agencies to consider how they design website access, structure and content accuracy, Andrew Merluzzi, artificial intelligence and incubation fellow at Georgetown University’s Beeck Center and report author, told Route Fifty. 

Generally, this “is not a super difficult thing for governments to fix, but they have to know they have this problem in the first place in order to go fix it,” he said, adding that “the main goal of this [report] was to try to help governments make the information transparent that they're already trying to make transparent.” 

One step government can take to improve AI crawler access is to assess their current website security configurations. Many websites include settings that block “malicious” bots, which could “inadvertently block a legitimate AI assistant trying to answer a person’s question,” the guidance reads. 

However, agencies must also balance AI crawler access with its potential impact on a website’s hosting capacity, due to increased traffic, according to the report. 

Government can consider a three-tier system to help determine how they approach granting access to certain AI crawlers through their website settings, such as a robots.txt file, Merluzzi said. 

The first tier of AI crawlers refers to tools from “established AI providers,” which the report recommends that a government allow to access its website when their identities can be verified. Such crawlers or crawler setting controls include OpenAI’s GPTbot, Anthropic’s Claude and Google’s Googlebot, among others.

In fact, “you don’t have to care about every possible [large language model] on the planet accessing your website — what you need to care about are the major tools that people are using every single day,” Merluzzi said. 

AI tools under the second tier may be developed from established providers but could have “narrower relevance” for resident use or have documented compliance concerns, according to the report. Before governments design their websites to allow such AI crawlers, staff can monitor and evaluate specific tools on a case-by-case basis. 

For bots that do not match a verified bot signature, which fall under the report’s third tier, government can implement a graduated response system of website access. For instance, staff can configure websites with unique responses to different bots, such as logging a potential bot that appears low risk for further review or automatically blocking traffic from a source highly likely to be a bot, Merluzzi explained. 

The report also points to website structure as a tool for building access for AI crawlers. 

Governments are spending considerable resources and time to improve the usability and interactivity of their websites with features like drop-down menus and clickable dashboards, but plainness and simplicity could be key with AI crawlers, Merluzzi said. 

“The implication here now for AI is just [to] publish a lot of that information as plain text, and that’s all you need to do so that the LLMs can then find a webpage … and read it and relay it back to the user who’s asking a question,” he said. 

In addition to more interactive webpages, government staff can publish critical information — such as deadlines for benefit program applications — as web page text, HTML tables or PDFs with extractable text and data, according to the report. Otherwise, scanned documents or screenshots containing relevant data for users may be overlooked by AI crawlers.

Staff should also retest a website’s interaction with AI crawlers following major website redesigns or backend changes to a content management system, as such alterations “can affect whether AI crawlers and retrieval systems can access important information, even when the visible content appears unchanged,” according to the report. 

The accuracy of information on a website is another critical pillar to AI crawler preparation, Merluzzi said. 

Government should designate a specific person or team, separate from managing the website’s security against AI crawlers, to focus on content accuracy and timeliness, according to the report. 

These individuals can own the process for updating and correcting information across government websites, prioritizing data “consequence” rather than “volume,” the report reads. 

“A rarely-viewed page with a wrong emergency number matters more than a frequently-viewed page with a minor outdated statistic,” according to the report, which also recommends testing if an AI assistant pulls the correct and updated website data when queried. 

Unlike other “technology problems in state and local governments for which the solutions are not easy [because] they’re bureaucratic or likely to take many years,” Merluzzi said, reconfiguring websites to be more searchable by AI crawlers “is one where if you put a few people on this problem for a week or two, you could make really substantial progress without a ton of expertise.”