Hurricane Ike Aims at Houston; Evacuations Called Options

[Jesse Smith]

http://www.bloomberg.com/apps/news?pid=206...&refer=home

"The total amount of energy is more powerful than Katrina, so we could be seeing a storm surge that could rival Katrina,'' Masters said. The storm is so large "the location doesn't matter much; it is going to inundate a huge part of the Texas coast.''

er, this doesn't look good for our servers in Houston!!

[Kevin Hazard]

Ike is heading our direction, but similar to our preparation for Gustav, we've gone through quite a bit of additional work to ensure we are ready for what appears to be a more direct severe weather impact.

In our Houston data center facilities, we've secured all exterior windows and doors and taken precautions to check the roofs, exterior boundaries and the surrounding areas for loose objects and equipment that could be affected by hurricane-force winds. For reference, the actual computer rooms in our data centers do not have exterior windows, as you would expect.

For data center power, we have confirmed our fuel levels and have brought in an additional 10,000 gallons as backup. Backup generators at both data centers and our Houston headquarters are prepared to shoulder the power needs of the data centers and our downtown offices. We pay premium fees year-round with several providers to guarantee that we are at the front of the line if we need additional fuel. With our current fuel supply, we can continuously power our facilities for up to a week without any required backup.

On the customer support side, we have transferred many of our Houston-based staff members to our Dallas offices to continue service and support availability throughout the weekend should that be required ... these departments include technical support, billing care and network operations. In addition, our Houston data centers will be fully-staffed and have plenty of food and water provisions for employees to remain in place for several days.

Our facilities staff will remain onsite at all of our locations for the duration of the severe weather threat, so we will be able to respond immediately to any hurricane-related events or problems.

[joLS]

Ike is heading our direction, but similar to our preparation for Gustav, we've gone through quite a bit of additional work to ensure we are ready for what appears to be a more direct severe weather impact. ...

What about the data lines to H1 and H2? How apt are they to go down during wide-spread power outages, etc.?

[Kevin Hazard]

Each data center has several 10Gbps connections to Tier 1 bandwidth providers, and each of those providers will be monitoring and responding to the effects of the storm as quickly as possible. In the event that any service-affecting issues arise, we will be updating our forums and blog with the most up-to-date response information.

[DDD]

Kevin, I hope everyone remains safe, but as for as the datacenter goes, I'm less worried about wind than I am about water at this point.

According to the most recent reports I've seen today, the authorities are predicting a possibility of 15-18 feet of flooding and at least eight Houston-area zip codes have been ordered to evacate.

I understand The Planet doesn't like to disclose specific street addresses of its datacenters, but what zip code is the H1 datacenter in and how high above sea level is that area? How far is it from there to the latest bayou or coastline? What is the flooding prediction for the area around H1? Thank you.

[Kevin Hazard]

The Houston data centers are located in North Houston, at least 60 miles from the coast of Galveston Bay on the north side of Harris County. Harris County has not been issued any evacuations -- voluntary or mandatory. I don't have the details about the height above sea level, but Houston is not known for rolling hills or dramatic altitudes, so I would assume both data centers are located at about sea level.

Given the distance from the larger bodies of water and the proven drainage in the area, we do not expect flooding to be a service-affecting issue.

[DDD]

Thanks Kevin, that's potentially very relieving information. Given The Planet's new Houston headquarters seems to be in south Houston and located in an area named "Bayou Place" -- a name that seems to imply a nearby bayou -- I was concerned that a proximity to the Houston datacenters may have been one of the reasons why that location was selected.

[rbgl]

Kevin, I have recently exchanged messages with your CEO, congratulating him for the excellent organization of The Planet, for your exemplary handling of the recent power challenge and for your fair and timely treatment of your clients.

I'm certain that you will agree with me that the next several days will bring one of the most important weather challenges of the decade to the Houston area and to THE PLANET facilities in that area.

This is not the time for 7 or 8 of your customers to be seeking information and "official" (?) hurricane and management updates on a difficult-to-locate BLOG!

JUST AS YOU ARE QUICK TO DISTRIBUTE FREQUENT SALES AND MARKETING E-MAIL MESSAGES TO YOUR CLIENTS, I RECOMMEND THAT YOU IMMEDIATELY CONSIDER A POLICY OF DISTRIBUTING EQUALLY QUICK, FREQUENT, AND TIMELY E-MAIL HURRICANE (AND OTHER DISASTER) UPDATES TO ALL OF YOUR CLIENTS WITH SERVERS IN HOUSTON.

THANK YOU FOR YOUR CONSIDERATION OF THIS REQUEST. IF NECESSARY, IT WOULD ALSO BE INTERESTING FOR YOU (AND YOUR CEO) TO SURVEY YOUR CUSTOMERS ABOUT THE REASONABLENESS AND VALIDITY OF THIS REQUEST.

I RECOMMEND THAT PROACTIVE, NOT REACTIVE, SHOULD BE YOUR COMPANY'S MANTRA IN THIS CUSTOMER COMMUNICATION REGARD!

WITH STRONGEST WISHES THAT YOUR INFRASTRUCTURE, EQUIPMENT AND ALL THE PERSONNEL OF THE PLANET COME THROUGH THIS AND ALL FUTURE STORMS AND EMERGENCIES SAFELY.


RBGL

[Martyn Dale]

Was on the phone to Kevin earlier talking about this. They seem well prepared. As long as oddies done come up, eg 1000 cows being flung at the DC, they will be fine. Those buildings were build to withstand massive amounts of wind. I was told the figure but i forget. It was plenty over 100 though, i remember that much

[Kevin Hazard]

Thank you for the good wishes, RBGL, and I really appreciate your feedback.

A note sent directly to customers is not out of the question, but it is a bit premature at this time. I'd imagine that we will have the best understanding of the impact Ike will have on our facilities tomorrow, so until then, the last thing we want to do is manufacture worry or stress as the storm approaches.

As Martyn mentioned, buildings in the Houston area must meet building codes to withstand winds of at least 120 miles per hour, and the location and provisions of our facilities allow for extended coverage in the event that the weather has a severe impact on our office or our data centers.

I will certainly pass along your request, and we will move forward based upon the predicted impact and severity of the storm as we get more concrete details about it's projected path. Additionally, should any service-affecting issues occur, we will use theplanet.com to direct traffic to resources and we will have additional information available to customers in Orbit and ServerCommand.

[Eric Bush]

A quick update from the Houston Data Centers in regard to our preparations.

Thanks to everyone who is sending well wishes to us from the field. We have been working 'round the clock to check and double check our procedures and preparations. There are a ton of things (too many to list here) that we've been doing to prepare.

I am actually part of a ride-out team that will be locked in one of the data centers (we have teams at both of them). A set of our top technicians have volunteered to be part of the storm ride-out team. We wanted them to make sure their families were safe before coming in. Personally, I'm very glad to be in such a secure and well provided for facility in events like this.

As Kevin has said, we will constantly evaluate the need for updates using all of our information outlets (forums, website, email, etc.). I can't make any promises about frequent updates, but since I'm going to be here for the duration, I'll do what I can to jump into the forums as much as possible.

Regards,

Eric Bush
Data Center Supervisor (Houston)
The Planet

[Martyn Dale]

Were you one of the guys who rode it out a few years back during EV1 time? The staff was great with it back then and im sure also will be now.

Thank you for you and the other teams for volunteering, stay safe

[Eric Bush]

No, I was not around during the EV1 days.. None the less, we continue to be diligent!

Either way, you're welcome! It's our pleasure. I'll pass along your comments to our staff.

Regards,

Eric Bush
Data Center Supervisor (Houston)
The Planet

Were you one of the guys who rode it out a few years back during EV1 time? The staff was great with it back then and im sure also will be now.

Thank you for you and the other teams for volunteering, stay safe

[joLS]

I don't mean to be negative here. Yes, I am sure the staffers are doing a great job, and my thanks goes out to them. But 100% of our income is tied up in h1 and h2 with all but one of our servers there, and frankly I've had enough of these gut twisting episodes every fall. (In many ways we are still recovering from the h1 explosion and fire from last spring, and now this...).

Yes, right now, at this moment it's the fear of the unknown, but that "unknown" just seems to get larger and more fearsome every year. Indeed, it does not take an advanced graduate degree to see that Al Gore and others who write about the potential for rapid climate change are right-on-the-money. Let's face it, this will not get any better, at least not in our lifetime.

Are there any thoughts among ThePlanet management to close h1 and h2, and get the hell out of Houston?

[ajz4221]

If your entire company is based in one city, then I think you need to talk to TP staff and load balance your servers between Dallas and Houston. You also need a disaster recovery plan if your company data is that important to you.
It should be no different than running your own data center. No matter how much money you put towards your infrastructure, “something” might and will go wrong at any time.

Stay safe TP staff. It is great TP management is allowing your families to ride out the storm with you while you work. That will probably be one of the safest places in the city.

[Eric Bush]

Stay safe TP staff. It is great TP management is allowing your families to ride out the storm with you while you work. That will probably be one of the safest places in the city.

Thank you. To further clarify, our ride-out team's families are not staying at our facility. The team was asked to make sure their families were safe before volunteering for the ride out team.

[Kevin Hazard]

I don't mean to be negative here. Yes, I am sure the staffers are doing a great job, and my thanks goes out to them. But 100% of our income is tied up in h1 and h2 with all but one of our servers there, and frankly I've had enough of these gut twisting episodes every fall. (In many ways we are still recovering from the h1 explosion and fire from last spring, and now this...).

Yes, right now, at this moment it's the fear of the unknown, but that "unknown" just seems to get larger and more fearsome every year. Indeed, it does not take an advanced graduate degree to see that Al Gore and others who write about the potential for rapid climate change are right-on-the-money. Let's face it, this will not get any better, at least not in our lifetime.

Are there any thoughts among ThePlanet management to close h1 and h2, and get the hell out of Houston?

At a certain point, any data center anywhere is subject to natural phenomena. If we look at the continental United States: in the northeast, service could be disrupted by a blizzard in the winter; on the west coast, it might be an earthquake; in the midwest, it might be a tornado ... The potential for rapid climate change is present globally, and it materializes in different ways. Because it is not possible to run away from these global phenomena, it is our responsibility to prepare for them.

The last hurricane to make landfall in the Houston area was in 1983, 22 years after the previous hurricane in 1961. While the significance of the storm should not be downplayed, the frequency of the storms shouldn't be sensationalized. Our VP of facilities Jeff Lowenberg has been working in the Houston area for several years and has planned for and responded to severe weather in the past, so we are covering every base to ensure that we are fully prepared to continue service to the greatest extent possible.

[Jesse Smith]

*hint*To the n00bies that havn't figured out how to yet...now would be a good time to learn how to make your own back-ups. *hint*

Any one who doesn't, has no right to complain if their server gets wet!!

[Tomy Durden]

I second the backup idea! Even if your servers aren't in Houston or even with us!

[madsere]

The last hurricane to make landfall in the Houston area was in 1983, 22 years after the previous hurricane in 1961.

I think Kathrine made landfall just a few years ago, certainly not 22 years ago...