The FBI's expanding data center

Upgrading a data center is a big job under any circumstances, but managers at the FBI's Criminal Justice Information Services (CJIS) division have a real challenge ' overhauling a 100,000-square-foot data center while the facility continues to provide around-the-clock data services to police agencies, civil organizations and private companies.

The data center, part of a bucolic campus on almost 1,000 acres of hills and hollows near Clarksburg, W.Va., uses a fleet of mainframes and servers and thousands of miles of cabling. Its upgrade might be likened to replacing the engines of a jumbo jet while in flight, so careful planning and precise execution is essential.

The makeover, known as the realignment project, is designed to accommodate ever-increasing demands for processing and data storage driven partly by unfolding efforts to combat terrorism. Biometric services required by legislation that mandates background checks for teachers, bank employees and other workers are one source of the increased demand.

Thomas Bush, the bureau's assistant director in charge of CJIS, said another of the most pressing new processing demands arises from work to mesh so-called flat fingerprint records held by the Homeland Security Department with rolled-fingerprint data.

'We were on a path of mutually assured destruction,' Bush said of the technology problem DHS and the FBI faced in coordinating the two-finger flat prints from DHS' Automated Biometric Identification System records with the bureau's 10-finger rolled-print records.

'Basically, at the very front end, we focused on the worst of the worst,' violators of both criminal and immigration laws, Bush said. 'So they had the worst of theirs, their most actionable bad guys, and the worst of ours, our most actionable bad guys ' probably 1.3 million bad guys,' Bush said. That 22-month project was launched in 2006 with initial implementation in Boston.

Additional demands for processing power will come from the introduction of new capabilities for the Next Generation Identification (NGI) project, which will expand the bureau's fingerprint database processing capabilities.

The project aims to accommodate future hardware additions needed to support these activities with existing electric-power, chilled water and floor space resources.

During the course of the upgrade, crews will make major changes to cabling installations, the chilled water and air handling components of cooling equipment, and power supply components.

Data center unit program manager Debbie Chapman explained the bureau's realignment plans as she walked briskly around the dozens of high-end computers in the center.

'The data center realignment project involves moving the existing equipment into rows of servers and other equipment that will point at right angles to the current configuration,' Chapman said.

Chapman and other data center project managers described how the new layout will reshape the center's cabling and cooling equipment.

'The new layout calls for modular cells, each with its own cabling and power centers,' according to a data center project manager who asked not to be named. Each cell's power supply provides a redundant support via A and B components that include separate circuit breakers linked to their respective groups of blade servers in a representative row of equipment.

The data cabling components, also located at the ends of the equipment rows, provide neat and efficient links among various storage units and processors. In many cases, the new cabling installations replace earlier counterparts that had been expanded in response to growing requirements. The center uses 3,000 to 4,000 miles of cable.

The new cabling layouts will help data center employees maintain and troubleshoot the equipment, center technologists said. The cabling overhaul extends to cable trays located below the center's three-foot- deep sunken floor. Repositioning those cable trays will also help reduce the power and cooling demands the processors and data storage units impose on the facility's infrastructure.

'Each of the modular rows has a hot side and cool side,' Chapman said as she pointed out the configuration of a unit that stretched about 20 feet across the data center floor. The facility relies on a central chiller facility several hundred yards away from the headquarters building that houses the underground data center.

'The chilled water flows into the data center [via pipes] at a temperature of 52 degrees [Fahrenheit],' a data center technologist explained. Air cooled by heat ex- changers flows under the sunken floor and up through the equipment racks and other computers on the center floor. From there, it exits through the ceiling.

'We plan the air flow patterns beneath the floor and throughout the rest of the facility to assure the proper equipment operation,' one project manager said. 'We use commercial applications for the [air flow calculations] that assure even air distribution and cooling throughout the data center.'

The realignment requires new calculations for those cooling configurations, Chapman and other data center technologists explained. Cooling components balance the heat generated by the center's computers to maintain a temperature of 72 degrees.

The data center demands a large and constant supply of electricity to support its array of IBM mainframes, dozens of server racks, rows of high-end Hewlett- Packard Superdome servers and other equipment. The facility's storage equipment includes several Sun Microsystems Storage Tek SL8500 Modular Library systems.

Under normal circumstances, the CJIS center draws power from the commercial grid. A backup uninterruptible power supply installation can keep essential systems operating for 30 to 45 minutes until diesel generators located in an adjacent power plant can take over.

'The data center's [electric] power consumption is limited by the capacity of the UPS,' according to the FBI project manager who oversees the realignment's electrical requirements.

'The data center realignment also will help us use the [floor space] more efficiently' for added equipment, a data center technologist said during a recent tour of the installation.

Technologists declined to specify the exact parameters of the data center's power and cooling requirements.

The realignment will also reshape systems in the facility's control center, where FBI crews continuously monitor systems in the facility and provide live telephone help-desk support to users in all 50 states and overseas locations in Britain, Australia and other countries.

The data center provides information technology services to five major systems.

• Law Enforcement Online: This nationwide network links the FBI with state, local and tribal police agencies nationwide to help coordinate their counterterrorism, counterintelligence and criminal investigations, among other functions.
  
• National Crime Information Center: The NCIC is a computerized crime data repository that law enforcement agencies use in criminal investigations.
  
• Integrated Automated Fingerprint Identification System: IAFIS is the bureau's fingerprint data storage and processing system now set for expansion via the fledgling NGI project.
  
• National Instant Criminal Background Check System: Launched in 1998, the system provides information that vendors of firearms and explosives use to determine would-be customers' compliance with restrictions imposed by the Brady Handgun Violence Prevention Act of 1993.
  
• Uniform Crime Reporting: The UCR program, launched in 1930, now collates crime statistics from 17,000 police agencies nationwide for the bureau's annual report on crime trends.