White House: Retrieving 'lost' e-mail will cost up to $3 million

By Shruti Dat'GCN StaffDuring the past four years, small but errant code changes made by the White House technical staff prevented an Executive Office system from properly saving the incoming e-mail messages of hundreds of users.The White House likely will have to spend 170 days and $1.8 million to $3 million to reconstruct thousands of e-mail messages that are available only as raw data on server backup tapes, White House counsel Beth Nolan said last month in a letter to the House Government Reform Committee.Committee chairman Dan Burton has suggested that the White House engaged in a cover-up of the error, and he is demanding that it deliver the missing e-mail messages.In a standoff between the White House and Capitol Hill that has become increasingly tense, the Indiana Republican last week called for appointment of a special prosecutor to investigate the administration's response to the code errors.'The big deal is not that a computer technician made a mistake. Mistakes happen,' Burton said at a March 23 hearing on the missing White House e-mail. 'The big deal is how the White House reacted to it.'The records brouhaha arose during the course of the committee's campaign finance investigation. White House officials revealed that they did not have immediate access to all electronic correspondence.In an eight-page letter to Attorney General Janet Reno calling for appointment of a special prosecutor, Burton decried Reno's supervision of the campaign fund-raising investigation. 'Ms. Reno, you cannot use the campaign financing task force, supervised by yourself, to investigate yourself and the Justice Department lawyers who helped keep the e-mails from being produced to Congress, independent counsels and your own campaign financing task force,' Burton wrote.The White House blames two computer code errors for the failure of its records management system to save some incoming e-mail messages. 'Two separate configuration errors occurred, which prevented certain incoming e-mails sent to the Automated Records Management System-managed accounts from being recorded in ARMS for a period of time,' Nolan wrote in her March 17 letter to Burton.The White House Office of Administration has used ARMS, a keyword searchable e-mail archive, since July 1994, Nolan wrote.Several times a minute, ARMS scans White House servers to capture copies of messages. The system creates a special tag for each message to avoid repeatedly copying messages, she said.The White House installed ARMS because correspondence in the administration's Lotus Notes e-mail system resides on servers, not on individual users' PCs, Nolan said.e-mail deleted from PCs is also deleted from the servers, she said. When searches were conducted on PCs in response to subpoenas, a redundant search was not conducted on the servers. 'The only other electronic records of the server consist of tapes made periodically when the server is backed up,' Nolan wrote. 'Backup tapes are thus an inaccurate and incomplete compilation of what is on the system and serve as a last resort only in cases of a catastrophic system failure.'The Executive Office does not search backup tapes because they are not created for archival purposes, are not part of ARMS and are not searchable unless the data is reconstructed and transferred to ARMS, she said.The White House has about 3,400 backup tapes of e-mail created since system administrators found and fixed the errant code, Nolan said.The first error occurred when the White House information systems and technology staff performed maintenance on ARMS in August 1996, she said. During that process, individual user accounts within the White House Office, Office of Administration and Office of Policy Development were moved to a new server, identified in system code as 'Mail2.' But White House computer technicians mistakenly programmed the accounts of 526 users to save e-mail to a server they identified as 'MAIL2.' The case-sensitive ARMS scanning process did not recognize 'MAIL2' and did not capture incoming e-mail for those 526 users, Nolan said. Daniel Barry, a White House records proj-ects computer specialist, first noticed that incoming e-mail did not correspond to outgoing e-mail when he was fulfilling a search for a January 1998 subpoena, Nolan said.In June of that year, Barry and contract employees from Northrop Grumman Corp. who were administering the system discovered a series of messages residing on the server that were coded 'unrecorded' because ARMS had not picked them up, Nolan said.The following November, the White House's information systems and technology staff and Northrop Grumman personnel corrected the glitch and created backup tapes of the data to preserve the unrecorded e-mail, Nolan said.But White House officials, trying to deal with the cost of year 2000 work in the Executive Office, decided not to reconstruct the backup tapes. Northrop Grumman estimated it would cost $600,000 to conduct a feasibility study on reconstructing the files, Nolan said.To complicate matters, computer technicians introduced a new glitch when they were correcting the 'MAIL2' code error, she said. This time, the coding error led the system to stop capturing copies of incoming e-mail to users whose first names began with 'D.' In place of the 'D,' the technicians had entered 'J.' The code mistake affected the e-mail accounts of 200 users, Nolan said. The White House fixed this error last May and again made backup tapes of the server, Nolan said.Most recently, in March, the technical staff determined that ARMS was not fully managing the capture of e-mail on servers in the Office of the Vice President, Nolan said.Before last week, ARMS was not saving all e-mail for user accounts in Vice President Al Gore's office. The White House has made 625 backup tapes of data on servers in the vice president's office.Burton said he has been growing wary of the administration's inaction in producing the missing e-mail. 'From the interviews we've conducted and the correspondence we've received, it looks like they chose to cover it up,' said Burton, adding that presidential records, by law, must be preserved at the National Archives and Records Administration.Testimony from some Northrop Grumman employees working on ARMS suggests that White House officials told them not to disclose the extent of the e-mail problems.Robert Haas, a systems administrator for the company's Lotus Notes group, testified that White House officials threatened him.At the Government Reform Committee hearing, Haas said two White House officials told him 'there would be a jail cell with my name on it' if he disclosed the coding errors. Haas named Laura Crabtree Callahan, former chief of the White House's Desktop Systems Branch, and Mark Lindsay, director of White House management and administration, as the sources of the threat.'Overall, my impression of the meeting was that Ms. Crabtree and Mr. Lindsay were very serious about their warning,' he said.In his testimony, Lindsay denied Haas' accusation. 'Let me give you my word that I never made any such threats,' he told the committee. 'I unequivocally deny that I threatened anyone regarding disclosure of the e-mail situation.'Callahan, now deputy chief information officer at the Labor Department, also testified that she did not threaten ARMS systems administrators.Yiman Salim, a Northrop Grumman software developer, testified that White House officials instructed him not to discuss the issue. 'We were told that the incident was considered sensitive and that we should take it very seriously,' he said. 'I do not remember hearing the word 'jail,' and I never felt threatened.'