What states learned from the CrowdStrike outage

If there is a silver lining to this summer’s CrowdStrike outage, it might be to show some states that their disaster recovery plans are up to snuff.

Several state CIOs recently reported that it took them no more than 24 hours to get their operations back up and running after a faulty July 19 update from the cybersecurity company caused major problems on computers running their software. Microsoft’s dreaded “Blue Screen of Death” popped up on screens worldwide, leading to canceled flights, impacting financial institutions, and disrupting border operations at several U.S.-Canada crossings, among other things.

For state and local governments, the incident saw outages or disruptions that affected 911 call centers, Department of Motor Vehicle offices, official websites, election and voter registration databases, transit agencies and more.

At a panel discussion during the National Association of State Chief Information Officers’ annual conference in New Orleans late last month, Oklahoma CIO Joe McIntosh said the state restored 819 servers within 24 hours of the outage and resolved 10,000 endpoints by the end of that week.

“Our response to that outage shows a lot about the maturity of our organization,” he said, “specifically when I think about our security teams, our networks, [and] the way that they came together and basically got emergency services back up and running almost immediately.”

Minnesota CIO Tarek Tomes also said that his state was in a “recovery posture, literally within 24 hours” after the outage. He wryly described it as a “massively significant tabletop exercise,” but said the planning and preparation for such eventualities had paid off.

It wasn’t smooth sailing for everyone. Some states had to quickly stand up temporary websites to continue providing crucial services, while simultaneously mitigating any damage to their networks.

It was particularly tricky in Iowa, which had gone live with a new governmental structure that also entailed several network changes right as the outage hit. As part of a push under Gov. Kim Reynolds, the state was in the midst of consolidating its cabinet agencies from 37 down to 16. Deputy CIO Michelle O’Hollearn had that on her plate when the CrowdStrike update went awry and “completely turned the apple cart upside down.”

For those states that didn’t turn things around in 24 hours, the CrowdStrike incident has become a learning moment. The cybersecurity company itself is even using the event as such.

“CrowdStrike proudly partners with 43 of the 50 U.S. states to help protect them from sophisticated nation-state, hacktivist and criminal threat actors,” a company spokesperson said in an email. “Our focus continues to be on using the lessons learned from this incident to better serve our customers and we are grateful for their support.”

Outside observers say the outage has made governments and businesses of all sizes think even more carefully about their resiliency and disaster recovery plans, especially as the threat of cyberattacks increases.

“That was an enlightening moment to the significant impact that could be there, and that was basically a human error event. That wasn't even a malicious actor out there trying to disrupt our industries,” Shawn Rodriguez, vice president for state and local government and education for software company World Wide Technology, said in an interview at NASCIO. “But it shone a light on how difficult things could be if we ever had that type of event occur at that scale, from malicious actors.”

Being better prepared, though, will cost money, warned Srini Subramanian, a principal at Deloitte & Touche LLP. He added that the Crowdstrike incident or others—whether they be cyberattacks or human error—could be the “singular motivation” to justify funding modernization of tech infrastructure to be more resilient.

“As a society, we have used the medium of the internet for our economy and to connect everything everywhere. Only later once a connection was possible, we started thinking about needing to secure certain information in it,” he said. “Can that be reversed as we start embracing generative AI, and maybe five years from now, when we have quantum technologies come in and completely revolutionize technology? And can we make sure that resilience is an integral part of how the design is done, so that we don't have to retrofit like we are doing now?”

Rodriguez said another silver lining is the awareness the incident raised of how fragile technology can be, and how important it is to prepare for such incidents. A recent NASCIO survey found that CIOs are already taking on an elevated role in state emergency operations in all manner of incidents.

“Every time we unfortunately have a little or a major thing like that happen,” he said, “I think it puts us closer and closer as an industry on how we're collaborating and working together to really holistically engage and execute in cybersecurity.”