Just a quick update from the previous post about the Disaster at The Planet’s Houston DC.

• DNS servers have been shipped and setup in an alternate data center.
• Access to ServerCommand.net restored. (If you can repoint your DNS to ‘UP’ servers, do it now)
• ThePlanet’s forum is experiencing a very high load - very slow loading times.
• No official word yet on when the power will be restored.  Vendors and Contractors are currently working to restore power to H1 (Houston datacenter 1)

UPDATE: Damage was worse than anticipated; power will be restored to 6,000 servers on floor 2 within an hour.  Power will be restored to floor one in the evening of June 2.  Thanks for the awesome communication from ThePlanet!

UPDATE 2: Most Servers on the second floor have been resotred to full power.  There are still many servers on the first floor which will be activated later tonight.
If you have a dark server (no power) right now, all that can be done is to change your DNS to an offsite backup server if you have one.  Otherwise, we are waiting for further developments from Texas.

Reports are that the 3 walls surrounding the  transformer room were knocked down by the explosion.  No servers are damaged, but no power is available in the entire datacenter because the backup generators are too dangerous to run because of fire safety concerns.  The Houston firedepartment will not allow them to run the generators until the building can be considered safe.

If you are among the frustrated many, please count to 10 before making a post that you may regret later.  If you are feeling stress from this event, please take a 3 hour break (or longer) to clear your mind so that when services are restored, that you are at your best to take necessary recovery steps.  Sleep deprevation will only make your stress worse.  I didn’t sleep last night, but I decided that I would not function unless I had a solid rest today.  I feel much better for it.

Update:

• Electrical transformer explosion at ThePlanet’s Houston datacenter.
• 3 walls knocked down - but servers are ok
• Backup generators cannot be started due to fire safety concerns
• 9000 servers affected - over 7,500 customers.
• Countless third party services affected by outage

More updated coverage here
When your web host suffers a massive failure.

Today an electrical transformer exploded and caught fire at ThePlanet Internet, my web hosting provider. Not only did they lose incoming power, but their backup generators could not start due to the fire safety issues. I do rely on my web server access for my daily job routines as the company I work for has online collaboration and some essential services hosting on this server. ThePlanet did an amazing job of communicating and fixing the problem in a reasonable manner. (on the weekend no less)

Now this brings me to thinking – very seriously – about critical business risks. What contingency plans do you have if something fails so catastrophically? It’s unrealistic to assume that one can achieve 100% uptime with any web hosting over the long term. When a problem occurs – and it will occur - are you ready? In my case, the company I work for really doesn’t have any disaster plans. In fact, all essential services are managed in house and located on a single server! - Yes, everything depends on one single vulnerable point of failure!

How can you lower your risk of business downtime? Firstly, examine what you and your essential services rely on. How many of those services depend on one or two devices or points of service? Do you check them to ensure that they are running correctly, without bugs and security issues? You can lower your risk by moving some of these functions to the “cloud”. The “cloud” is the term for hosting your services on specialized infrastructure which is dedicated to providing a specialized service. These “cloud” systems are often highly monitored and staffed by experts in their field – They know what they are doing and they do it well. The more business services that you can place into the “cloud”, the more fault tolerant your overall business will be in the event of a failure at any single point.

An Argument of Cloud Computing

The next disaster is on the level of “cloud” computing. Recent bug issues at popular payment processor PayPal have caused widespread problems for many web site owners who rely on it. It should be noted that this error did not affect everyone – rather it only affects those who implemented the “cloud” service of PayPal in a certain lazy/incomplete way. There are certain technical details about this problem which I originally saw and addressed to make my system as fault tolerant as possible. When implemented correctly and for a serious business, PayPal is likely one of the greatest processors. For example, GoDaddy uses PayPal – and Bob Parsons has not complained once about it! Unfortunately some lazy and bratty system admins from various shoddy/chaotic web sites complained loudly about the PayPal bug rather than just fixing it themselves and moving on.

One system admin from StatCounter (currently down) happened to be affected by both the datacenter explosion and the payment processor problem. Rather than going into disaster recovery mode; Statcounter simply complained loudly and let down it’s clients. Not exactly a prime “cloud” service by any means. In fact, I *unfortunately* subscribe to the service and it’s blog which contains numerous posts whining about services from Google adwords, cookies (yes harmless cookies – quite misleading), PayPal, and so on. To this date I have disagreed which each complaint – and found serious flaws in each argument. The StatCounter blog reminds me of the old saying: “for every finger you point, there are three more pointing back at you.” Today’s statcounter flake is the last straw. I’m going to be joining a few of my early adopter friends and upgrading to a better free service like “GoStats”. Started using it for a few hours and I wished I knew about GoStats earlier.

I hope these disaster examlpes were helpful for you.  Please subscribe to my blog via RSS if you liked my post.

(disclosure – I used StatCounter up until today, I use Godaddy, PayPal, Planet, and now GoStats)