Census fields a robust WAN
The Census Bureau's DecennialNet WAN achieved 99.5 percent uptime from its final configuration in March through May, and it probably hit 99.75 percent for June.
The Census Bureau's DecennialNet WAN achieved 99.5 percent uptime from its final configuration in March through May, and it probably hit 99.75 percent for June.'By almost any measure, [uptime] is very good,' said Kevin McGreevy, Census program manager for the network management contractor, Network Designs Inc. of Vienna, Va. 'It's almost unheard-of in a frame relay network.'But reliability was essential to the mission, said Bob Rinaldi, DecennialNet's program manager at Census. The network handled all payroll and other management information traffic for 520 local Census offices, as well as the data collected by those offices about more than 250 million Americans.'If you're payrolling a half-million people, you have to keep your network up between the regional office and the local offices,' Rinaldi said.DecennialNet was built exclusively for this year's census. Officials recently started shutting it down.The all-Cisco network included 1,200 routers and switches from Cisco Systems Inc. of San Jose, Calif. It supported more than 10,000 desktop client systems, more than 700 Dell Computer Corp. servers and another 100 servers from Compaq Computer Corp. and SGI running Novell NetWare 4.11.'We set ourselves a goal of never having a local office down for more than 24 hours,' Rinaldi said. 'I can hardly remember when any office was down that long.'The 256-Kbps frame relay links handled bursts of up to 512 Kbps between the 520 local offices and 12 regional offices. Links between the regional offices and the Census processing center at Suitland, Md., combine frame relay and dedicated digital services.The infrastructure was supplied by AT&T Corp. under an extension to its old FTS 2000 contract.'We opened up the first regional center in 1997,' when AT&T was the only provider available to Census, Rinaldi said. The bureau stuck with the company as the network grew, he said, because 'you don't want to change in midstream.'Network Designs holds a round-the-clock support contract for the Decennial Network Operations Center. The company designed the network management systems and redesigned the network as its needs expanded.The first baseline lockdown for the emerging Novell-Cisco network was in April 1999, when the network ran at 128 Kbps. It was upgraded to 256 Kbps in September 1999 and locked down for a second time. The third and final lockdown, standardizing the operating system on the routers, occurred in March.DecennialNet had more than 530 model 2500 and 2600 Cisco routers, 40 model 3640 routers, 16 model 7204 and 7206 routers, and five model 7205 routers. Cisco Catalyst LAN switches ran the local office LANs.A full-time staff of 12 managed the network from the main operations center at Census headquarters in Suitland during the day. A backup center in Vienna functioned after business hours.Configuration for the routers and switches was handled by CiscoWorks 2000 Resource Manager Essentials. Hewlett-Packard OpenView monitored the network nodes. Network Health software from Concord Communications Inc. of Marlborough, Mass., polled the 22,000 network elements to monitor performance and trends. Network Health was added to the management toolbox because of its customized reports, McGreevy said.'We want our engineers to be free to work on problems rather than look for them,' he said.Despite the record of high availability, problems did pop up. The operations center logged more than 1,500 events per day, responding to an average of five Level 1 problems (site down, cause unknown) that resulted in about an hour of downtime each, and 25 Level 2 problems (site down, probable cause determined) averaging 15 minutes downtime.With more than 500 sites scattered across 50 states and Puerto Rico, McGreevy said, fires, power outages, lightning strikes and backhoes all took their toll.The sites were all temporary offices, Rinaldi said. Census began taking them offline last month.Ninety percent of the network will be down by October when the local offices close, and then the lights will start to go out at the regional offices.Rinaldi said he has not yet begun plans for the 2010 Census network.
Network connects bureau offices around the nation with very little downtime
Power links
Wide area
 | The Census Bureau took three years to construct its massive nationwide network and began taking it down five months after it was completed. |
Power links
Wide area
NEXT STORY: INTERNAUT