Federal agencies and regional data center operators, including one operated by Amazon Web Services, are still taking stock of the impact of widespread power outages that began Friday night and continue to leave large swaths of greater Washington, D.C., region without electrical power.
Many federal buildings are still without power as a result of unusually violent wind gusts Friday evening, prompting the U.S. Office of Personnel Management to give federal employees in the Washington, D.C. area the option to take unscheduled leave or work remotely. Among agencies taking advantage of the policy is the U.S. Patent and Trademark Office, where 7,000 of the agency’s 10,000 employees now routinely telework at least one day a week, according to Danette R. Campbell, senior advisor, at PTO’s Telework Office.
What is less clear, however, is how widely the power outages may have affected federal IT and communications systems used to support employees.
Northern Virginia and the greater Washington area are home not only to millions of residents, but some of the nation’s largest data center operations. The storm forced data centers in the region to scramble Friday night, including an Amazon Web Services facility, which began seeing “elevated error rates impacting a limited number” of customers,” according to status reports issued on a company blog and subsequently reported by the New York Times.
In response to the incident, an Amazon spokesperson told AOL Government Monday:
“Severe thunderstorms caused us to lose primary and backup generator power to a portion of a single Availability Zone in our US-East Region Friday night. For perspective, in our US-East Region in Virginia, we have in excess of 10 data centers.
“In the thunderstorm on Friday night, several of our data centers had their utility power impacted, but in only one of them did the redundant power not operate correctly (which ended up impacting a single digit percentage of our Amazon EC2 instances in the US-East Region). We began restoring service to most of the impacted customers Friday night and the remainder were restored on Saturday,” said the spokesperson.
Amazon’s was not the only data center that lost power.
“Our data center in Reston lost external power for a relatively short time,” said Unisys Group Vice President for Federal Systems Peter Gallagher. “Generators of course kicked in – so we were never off-line in Reston despite the storm. Our primary data centers in Salt Lake City and Eagan, Minn., were similarly not in the storm zone so our contingency operations for all customers – federal and commercial – were unaffected,” he said.
But as a growing number of organizations have come to rely on Amazon and others for storage, the type of shutdown that occurred over the weekend continues to demonstrate why cloud computing systems are still not a complete answer for organizations requiring dedicated backup options.
“Having systems in the cloud doesn’t mean that those systems are fault tolerant,” said Jason Lewis, chief scientist, Lookingglass Cyber Solutions.
“Amazon has multiple zones that can be setup for failover if a problem in one (zone) occurs. If a customer only has a presence in one location and a weather event takes down that location, then they didn’t plan for failure. I think there is a misconception that everything in the ‘cloud’ will be fault tolerant and redundant. This isn’t the case,” he said.
“The Dulles area has a huge Internet presence,” Lewis said. “I’d expect a lot of affected customers are looking at expanding their backup/fail-over plans. While recent events are rare, planning for them is part of highly available design.
According to Amazon’s blogs however, the power outage served as a reminder that organizations that rely on large scale data and cloud computing centers may face more than a temporary suspension of service in the event of regional power outages. It may also mean that the data itself can come into question, as appeared to be the case when Amazon reported at 10:36 PM PDT that Elastic Block Store storage volumes “may have inconsistent data” and in effect be “Impaired.”
Amazon, which supports a variety of Web based services for consumer, commercial and government customers, provides regular blog posts about the status of its Elastic Cloud Computing (EC2) and other data centers operations.
Those blog posts captured a glimpse of how the storm impacted Amazon’s Northern Virginia server operation as millions of area residents and business suddenly lost power from the storm: continued here.