Diplomatic patches

The State Department is fighting patchwork with patches. After years during which embassies and other outposts around the globe developed individual, disparate applications'which left department systems vulnerable'State IT officials are establishing centralized software patching and hardening their defenses.CIO James Van Derhoff is leading a bid to end the department's information assurance woes and simultaneously cut costs by deploying new tools, improving training and mandating pilots that centralize security services. The move is at least in part a response to a series of dismal security assessments.'I arrived a year ago,' Van Derhoff said in a recent interview. 'There were clear IT security issues that were pointed out to me immediately when I arrived.'State bears the stigma of an F on its Federal Information Security Management Act report card for 2005. In addition, the department's inspector general had spewed critical reports on State's system security for the preceding five years.A thick stack of scathing IG reports, newly revealed via the Freedom of Information Act, describes years of failure to meet patching requirements, harmonize systems configuration across overseas locations and meet certification and accreditation requirements dating back to the 1980s.GCN received the previously undisclosed IG reports, which state that they are 'sensitive but unclassified,' under the Freedom of Information Act. The reports, dated from 2002 to 2006, were released almost a year after GCN filed the original FOIA request in 2005.The audit reports include many sections that have been 'redacted,' or deleted for security reasons. In some reports, entire pages appear blank because of the redactions, while in others, many paragraphs or individual words have been excised.The IG's analyses and recommendations, and the department's corresponding replies, show that serious security flaws continued for years despite repeated pledges to improve.Among the most prominent areas the auditors singled out for criticism were:Van Derhoff and his team have deployed technology and changed procedures to improve security in these and other areas.'Patch management has improved tremendously over the past year or two,' Van Derhoff said. 'One of the ways that we have addressed that is [we have removed] one of the deficiencies, which was the [Microsoft Systems Management Server] product that we were using. We were using SMS 2.0 [a utility from Microsoft that] pushes operating system patches out. There were a lot of bugs in that.'As a countermeasure, he said, 'We have been moving to SMS 2003 over the past 12 months.' Department officials expect to complete that migration within days, he said.Van Derhoff and his chief information security officer, John Streufert, also use applications from Tenable Network Security Inc. of Columbia, Md., for automated network scanning. The company provides the Passive Vulnerability Scanner for intrusion detection and the Nessus scanner to detect anomalies and minimize network risk.Van Derhoff and other State IT officials rely on a custom enterprise network management tool named iPost that continually displays system health indicators for each of the department's installations.IPost calculates and provides a 'patch score,' for example, that displays in a pentagonal image the percentage completion of five categories of the software repairs, from critical to low. It also monitors the status of the installation's antivirus suite, operating system, server performance and network performance, among other indicators.'If I find a post that is particularly egregious in not putting patches up, I will pick up the phone and call the ambassador,' Van Derhoff said.State IT security operates under parallel laws that divide responsibility for information assurance between the Information Resources Management bureau that Van Derhoff runs and the Diplomatic Security bureau.The sanitized IG reports suggest some of the complexities this legal division creates, as in cases where each bureau responded separately to the auditors' recommendations.Streufert and Don Reid, senior coordinator for security at the DS bureau, work as co-chairman of an Information Security Steering Committee that Van Derhoff created.Van Derhoff came out ahead in his bid to recruit Streufert as CISO last summer. Streufert previously had been acting CIO of the Agency for International Development, which had earned the highest FISMA scores in the federal government for the preceding two years.A group within DS called the Computer Incident Response Team uses automated tools to continually scan State's networks for unauthorized software downloads and other problems, Van Derhoff said.'Every day, five days a week, at five o'clock in the morning, we get a daily report from the CIRT that tells me how many people tried to put unauthorized software on their machines,' Van Derhoff said.The CIRT operation also has the ability to take PCs off of State's OpenNet network when scanning tools detect unauthorized executable code. In those cases, the information system security officer at the affected post carries out an investigation of the incident, Van Derhoff said.State's bid to shore up its patch management technology is linked with a continuing drive to impose uniform configuration management at all the far-flung posts.The inspector general reports document dozens of cases in which posts appealed to IT organizations in Washington to retain custom network configurations, and a culture of resistance to headquarters control gleams through the dry bureaucratic language.'We have a suite of applications for post management,' Van Derhoff said. 'We are trying to get all the posts around the world to standardize on this suite.'Until then, many of the posts will speak their own language.'What happens is, we have very talented foreign national programmers who write code and we wind up with all these homegrown applications,' Van Derhoff said. 'When you are pushing patches out, if you don't have standard configuration out there, the patches that haven't been tested against these local applications will cause them to crash.'Standardizing configuration will be a key step in achieving 100 percent centralized patching, he said.State now is testing its technology for centralized patch management in a pilot at a small Caribbean post. 'We have just completed a pilot [of central patch management] in the last couple of weeks. It was in Curacao, one of the small embassies down in the Netherlands Antilles.'Central patch management is aimed not only at improving security but reducing the workload on ISSOs at posts that can have as few as two foreign service officers. The ISSO job is a part-time gig in those small posts; it's a full-time job at large embassies.'If we can go to using tools, instead of using people, [in some cases] rather than having the ISSO at a post who doesn't have the time to do it all, we can do 90 percent of it and direct him to a specific machine when necessary,' Van Derhoff said.For situations where automated tools alone aren't enough to meet the small posts' IT support needs, Van Derhoff has launched another pilot under which State IT staff based in Fort Lauderdale, Fla., will deploy as needed to fix problems.Meanwhile, the department has responded to the IG's criticism of ISSO readiness by overseeing the training of some 800 foreign-service officers in the required core curriculum, the department said. About 200 of the ISSOs work in this country and 600 overseas, according to department figures.The department seeks to squeeze efficiencies out of its certification and accreditation process by piggybacking on the work done by other agencies. As it stands now, the federal government pays for repeated certification of identical systems by each agency, State IT officials said.The Director of National Intelligence Office CIO, Dale Meyerrose, and the Pentagon CIO, John Grimes, are working to suppress the costs and inefficiencies in that process via their pending certification process reform [GCN.com/739].'One of the things [about the current certification process] is that the companies love it,' Van Derhoff said, because they certify the same systems over and over again.Van Derhoff said that, during certification procedure discussions, 'I said, 'This is crazy, we've got the same Beltway bandits, providing the same services to all the federal agencies.''A better approach to certification, according to Van Derhoff ,is to adopt scanning tools such as those Streufert used at AID. Rather than certifying each system once every three years'and not knowing if a configuration change carried out shortly after certification rendered a system vulnerable for years'new tools allow continual checks.State IT specialists are studying the 1,138 separate controls in the certification process to pinpoint those susceptible to automated scanning, according to State IT security staff.State IT officials echo a refrain heard widely across the federal government: that FISMA grades don't portray agencies' true security postures.'A good deal of what we are doing is not reflected on FISMA,' Van Derhoff said. He noted that the 110-point grading protocol omits consideration of scanning and intrusion detection methods that State uses.But, like a shrewd elementary school principal who finesses the No Child Left Behind tests, Van Derhoff has recruited a staff with a higher grade point average than the State Department.As Van Derhoff said of AID's score on the information assurance report card when Streufert was in charge there, 'The proof is in the pudding.'