Summary: The Rackspace data center in Dallas (DFW), where our servers are located, suffered a major power outage tonight lasting approximately two hours. A truck hit one of their transformers, causing damage, which compromised the power of the data center, specifically the cooling system. Our servers were pro-actively shut down in order to prevent overheating while the power was restored. Check out Laughing Squid for the most recent updates on the power outage issue.
Update: 5:55 P.M. PST — Are servers are offline due to a power issue at Rackspace’s Dallas (DFW) data center where Laughing Squid’s servers are located. Their support staff is currently working working on this and we will update this post when we have more information or when the issue has been resolved.
Update: 6:58 P.M. PST — Here’s the latest update from Rackspace:
In a completely unrelated incident to this weekend’s power problems in DFW, a traffic accident caused damage to a power transformer which provides utility power to our DFW data center. Here is the current sequence of events:
* At approximately 6:00 p.m. CST utility power was lost to the DFW data center
* Power automatically switched over to backup generators without disrupting service for any customers
* When generator power was established two chillers within the data center failed to start back up
* Utility power was re-established through a secondary utility source
* As a result of temporary data center temperature increases, we proactively shutdown a number of customer servers to protect them from overheating
At this point, the chillers are back up and running and we are operating on generator power throughout the data center. We have contractors on site to repair the damage and will be in contact with all customers who have been affected by this outage. We apologize for any disruption to your business operations and will work diligently to restore your service.
Update: 7:43 P.M. PST — The power has been restored and our servers are now back online. We are currently running checks on each server to make sure everything is functioning normally.
We sincerely apologize for any problems this downtime has caused. This is the first time we have had power outage like this in the 9 years that we have been in hosting (8 of which at Rackspace). After speaking to our contacts at Rackspace, I know that they will be doing everything in their power to prevent this from happening in the future.
Update: 11:09 P.M. PST — Rackspace has posted an update on their two recent power outages (we were only affected by the 2nd one). Quoting Rackspace:
We cannot promise that hardware won’t break, that software won’t fail or that we will always be perfect. What we can promise is that if something goes wrong we will rise to the occasion, take action, resolve the issue and accept responsibility.
Here’s more coverage of the Rackspace DFW Data Center power outage:
- “Rackspace Outage 07” – RKNet
- “Truck driver in Texas kills all the websites you really use” – Valleywag
- “Quick, Plug The Internet Back In: Major Rackspace Outage” – TechCrunch
- “RackSpace Outage Hits Home” – GigaOm
- “Followup: Rackspace outage was third in two days” – Valleywag
- “Failure Happens: An SLA is just a contract & Data Centers are single points of failure too” – O’Reilly Radar
- “Truck Crash Knocks Rackspace Offline” – Data Center Knowledge
- “Truck crash takes Lost Remote offline (seriously)” – Lost Remote
- “RackSpace Data Center Goes Dark After Accident” – Wired (Epicenter)
- “Rackspace: When Bad Drivers Attack!” – Broadband Reports
- “Low blood sugar brought down Rackspace websites” – Valleywag