hosting

Massive Power Outage at Rackspace’s Dallas Data Center

Summary: The Rackspace data center in Dallas (DFW), where our servers are located, suffered a major power outage tonight lasting approximately two hours. A truck hit one of their transformers, causing damage, which compromised the power of the data center, specifically the cooling system. Our servers were pro-actively shut down in order to prevent overheating while the power was restored. Check out Laughing Squid for the most recent updates on the power outage issue.

Update: 5:55 P.M. PST — Are servers are offline due to a power issue at Rackspace’s Dallas (DFW) data center where Laughing Squid’s servers are located. Their support staff is currently working working on this and we will update this post when we have more information or when the issue has been resolved.

Update: 6:58 P.M. PST — Here’s the latest update from Rackspace:

In a completely unrelated incident to this weekend’s power problems in DFW, a traffic accident caused damage to a power transformer which provides utility power to our DFW data center. Here is the current sequence of events:

* At approximately 6:00 p.m. CST utility power was lost to the DFW data center
* Power automatically switched over to backup generators without disrupting service for any customers
* When generator power was established two chillers within the data center failed to start back up
* Utility power was re-established through a secondary utility source
* As a result of temporary data center temperature increases, we proactively shutdown a number of customer servers to protect them from overheating

At this point, the chillers are back up and running and we are operating on generator power throughout the data center. We have contractors on site to repair the damage and will be in contact with all customers who have been affected by this outage. We apologize for any disruption to your business operations and will work diligently to restore your service.

Update: 7:43 P.M. PST — The power has been restored and our servers are now back online. We are currently running checks on each server to make sure everything is functioning normally.

We sincerely apologize for any problems this downtime has caused. This is the first time we have had power outage like this in the 9 years that we have been in hosting (8 of which at Rackspace). After speaking to our contacts at Rackspace, I know that they will be doing everything in their power to prevent this from happening in the future.

Update: 11:09 P.M. PST — Rackspace has posted an update on their two recent power outages (we were only affected by the 2nd one). Quoting Rackspace:

We cannot promise that hardware won’t break, that software won’t fail or that we will always be perfect. What we can promise is that if something goes wrong we will rise to the occasion, take action, resolve the issue and accept responsibility.

Here’s more coverage of the Rackspace DFW Data Center power outage: