Apparently a traffic accident is to blame for almost 3 hours of downtime for our corporate websites in the night from Monday to Tuesday. A truck driver lost control over his vehicle due to low blood sugar
and drove into a power transformer on the side of the Dallas-area data center of Rackspace Inc.
in Grapevine, Texas, which hosts our server park.
The datacenter lost electricity shortly after the traffic accident caused damage to the power transformer. Rackspace's generators kicked in but then a few other things went wrong. In this case, two chillers within the data center failed to start back up, and a considerable number of servers had to be taken offline to avoid damage from overheating, including ours and many other popular websites.
The outage was widely noticed in the blogosphere, as Rackspace hosts many popular Web 2.0 sites and blog services, including Laughing Squid, GigaOm and 37 Signals.
The downtime was notable primarily because it is a rare occurrence for Rackspace, which has been one of the most reliable providers in the world over the last six years, and was the industry's best performer in September
, according to Netcraft.
A situation like this happens - regardless of the best disaster plans. The important issue is to handle them properly, which seems to be the case with Rackspace (see their official announcements
We apologize if the outage has caused any inconveniences or difficulties to our customers. We will look into options to avoid situations like these in the future.