Air Force survey seeks and helps solve network problems

| SPECIAL TO GCNWhen the Air Force began surveying network infrastructures at its bases around the world, engineers found some misconfigured servers that were causing lots of problems.'In the past, before fielding new software systems, survey teams have been sent out primarily focusing on the hardware requirements necessary to run the applications,' said Ron Lewis, the team technical leader in the effort and a network engineer for Quality Research Inc. of Huntsville, Ala.'Very little thought [was] given to the impact the automated information systems would have on day-to-day network performance,' Lewis said.It's a huge job that began in October 1999. The first three months were spent writing out the procedures just to do the surveys, said Danny Oliver, chief of Integrated Fielding Support with the Air Force's Headquarters Standard Systems Group. Then two trial surveys were run at bases within driving distance of SSG's Integrated Logistics Systems Program Office at Maxwell Air Force Base-Gunter Annex, Ala.In January 2000 the first team of four network engineers began a tour of European bases. In eight months Lewis amassed more than 33,000 frequent-flyer miles while visiting Air Force bases in England, Spain, Germany, Italy and Turkey. Two more teams were added to help survey U.S. and Pacific Rim bases.The focus is on systems integration. After systems are upgraded, they must be capable of sharing data with other systems without application-specific data bridges, Lewis said.'This means, though, that the 80-20 network rule no longer applies to the systems bandwidth requirements,' Lewis said. The 80-20 rule is that 80 percent of the traffic stays internal to the network and that only 20 percent crosses the gateway. 'The teams' main objective is to determine the base LAN and WAN capabilities and match those against the [logistics information systems] requirements,' he said.To do this, the teams use Network Observer, a protocol analyzer from Network Instruments LLC of Minneapolis. Network Observer can identify each LAN segment's traffic patterns, bandwidth utilization, client-to-client throughput metrics, percentages of broadcast and multicast traffic, and error rates.To understand how a network is used, the teams take something akin to a snapshot of the network at minimum, maximum and average usage, Lewis said.Network Observer is also used for accessing trends, such as identifying the average packet size, the types of traffic, and who a segment's 'top talkers' are, Lewis said.WS Ping Pro from Ipswitch Inc. of Lexington, Mass., sends out ping packets, which bounce off traffic to give a network's Internet Control Message Protocol metrics.On average, the teams survey 30 segments on each base. Before leaving, the team gives the protocol analyzer data, ICMP metrics, and the recorded live data transfer rates to the local network control center.The teams also attempt to forecast the impact an upgrade would have on network load. Segments are tagged red, yellow or green, depending on their ability to support new software. Green means there will be no noticeable impact, yellow that the predicted usage is going to push the network into the saturation zone and slow down performance, and red that the segment could not sustain any additional traffic.Where analysts found current problems, they tried to offer easy solutions.One base had an unusually high network utilization average, Lewis said. Most networks have an average utilization of 4 percent, but this base averaged 80 percent in every building. Networks usually degrade at about 30 percent to 40 percent utilization, he said.The surveyors identified a server that was polling every network client continually, consuming nearly 70 percent of network resources. As soon as the server was reconfigured, network traffic fell to 4 percent, Lewis said.Another base had a browsing function turned on, which meant every 30 seconds the computers were polling the network to see who was out there. This accounted for 50 percent of the traffic on the network. Printers are among the worst offenders when it comes to making meaningless network traffic, Lewis said. Most printers regularly check in with the print server just to say 'I'm still here.' Lewis' team recommended that the base install Hewlett-Packard's Web JetAdmin software to help control and minimize such printer traffic.The teams do not fix problems but make recommendations.There are rules for networking, but the teams found many instances where the rules weren't followed, Lewis said.The survey teams found that on many bases the communications people are responsible for everything from the WAN to the desktop. The workload is overwhelming, and turnaround time is slow, so upgrades are left undone because of lack of time and personnel.