Towns host each other's recovery data
Connecting state and local government leaders
Two Tennessee towns find a COOP solution
Setting up a fully-functional backup site to maintain continuity of operations is an expensive undertaking, especially for a smaller government body with few extra dollars to spend. The city of Maryville, Tenn. has found an intriguing solution: It uses the excess space in the data center of a neighboring town, Alcoa, Tenn., about 3 miles away. And in exchange, Alcoa uses Maryville's site for its own COOP site, according to Terry McCoy, Maryville's director of information technology.
Despite a modest population of 30,000, Maryville has some pretty stringent COOP requirements, McCoy explained, not the least because it acts as an electricity provider for local residents on behalf of the Tennessee Valley Authority. TVA requires the town's billing system to go through audits under the Statement on Auditing Standards 70 protocols. One requirement of SAS 70 is to have documented off-site backup of operations.
In addition to this requirement, the IT department also needed a way to back up 2,200 work stations, mostly from the city's schools, but also municipal office desktop computers and the police force's laptops. The IT department oversees all the databases and file systems for the city. All in all, setting up an off-site data center would cost the city at least about $50,000, McCoy noted.
So McCoy looked for a less expensive work-around, and found an answer in the neighboring town, Alcoa. Alcoa also had the TVA mandate for off-site operations, so the two towns decided to house each other's back-ups. In this way, if Maryville's primary data center went offline for some reason, operations could easily be picked up a few miles away from Alcoa's facilities, and vice-versa.
One advantage of the plan was that much of what the towns needed to make it work was already in place, McCoy said. The towns shared a fiber-optic network, which connected all the office buildings and schools of both cities. McCoy's teams also started using virtualization, which allows them to spin up or move around servers as they are needed. Finally, the town invested in a 24 Terabyte Pillar Axiom 500 Fibre Channel-based Storage Area Network, from Pillar Data Systems to house the data.
The combination of virtualization, high-speed fiber optic network and the SAN brought a new flexibility to the town's operations, McCoy said.
Take the schools, for instance. Each school stored all the materials on one of two servers, each with direct attached storage. Each time a server would go down, the IT staff would have to go set up a new server, and install all the appropriate software, such as the GroupWorks educational program.
While the IT department rebuilt the server, which could take up to two days, the school would be without educational materials. At one point, the primary and backup drive of one of the servers failed simultaneously, and the IT department came perilously close to losing all the data.
Now, all these servers are virtualized and run from a cluster, using Novell Cluster Services. So when a physical machine goes down, the virtual server can be moved to another location with very little downtime. Moving to virtualization also saved in hardware costs. Since newly purchased servers would no longer require hard drives for storage, it could save $7,000 to $8,000 per server, according to the city.
By virtualizing much of the town's operational software it could easily be moved around if need be. This also helped in COOP planning. If something were to happen to the Maryville data center, everything could be run from another location.
Using nearby Alcoa made sense, since they execute many of the same services, such as collecting utility bills and tax bills. "If anything ever happened to one of their buildings, then we could send our cashiers over to their buildings," McCoy said. When Maryville purchased its new 24 Terabyte SAN from Pillar, it sold its legacy 12 terabyte SAN, also from Pillar, to Alcoa, which didn't require as much capacity. The towns are now in the process of mirroring each other's material.
"With the virtual environments, we're trying to make it to the point where our server could be brought up over there and vice versa," he said.