H1 Data Center Offline - Status Updates Available Here Options

[Tomy Durden]

The Planet is currently experiencing an outage which is effecting a number of customers' servers. This issue may also be affecting customers' ability to get through to our call center.

We are doing everything we can to remedy this problem as quickly as possible. And will place an updated notice here as soon as there is new information.

[Kevin Hazard]

Today at approximately 5:45 p.m., a transformer in our H1 data center in Houston caught fire, thus requiring us to take down all generators as instructed by the fire department. All servers are down.

We are working with the fire department, with our facilities staff on site, to assess the situation.

To keep you updated, we will send messages every 15 minutes in Orbit and here in our forum.

[Kevin Hazard]

We have determined that no servers in the data center have been damaged. Nonetheless, they are down because power is out. Teams across the board are working to take appropriate action.

We will continue to keep you updated.

[Kevin Hazard]

We have no additional updates at this time. Our team is still evaluating the time required to bring all affected servers online as soon as possible.

[Kevin Hazard]

In our latest assessment, we have determined that networking gear has not been damaged, but we are without power so assessments continue. All disaster recovery systems are in motion, and we have teams already working in the data center.

[Kevin Hazard]

We have no additional updates at this time. Our networking, technology, support, and facilities teams are still working to restore power to all affected customer servers.

[Kevin Hazard]

The ServerCommand customer portal is down, so please contact our customer support team if you have questions. We have begun moving the ServerCommand infrastructure and will keep up updated as it is comes on-line.

[Kevin Hazard]

We have no additional updates at this time. In a post coming shortly, you can expect full details about the incident.

[Kevin Hazard]

Our entire team will be convening at 9:30pm CDT to consolidate a status report. I hope to pass along that information soon thereafter.

[Kevin Hazard]

The senior managers are still meeting about the incident. I will update the thread as soon as I speak with them.

[Kevin Hazard]

From Doug Erwin:
This evening at 4:55pm CDT in our H1 data center, electrical gear shorted, creating an explosion and fire that knocked down three walls surrounding our electrical equipment room. Thankfully, no one was injured. In addition, no customer servers were damaged or lost.

We have just been allowed into the building to physically inspect the damage. Early indications are that the short was in a high-volume wire conduit. We were not allowed to activate our backup generator plan based on instructions from the fire department.

This is a significant outage, impacting approximately 9,000 servers and 7,500 customers. All members of our support team are in, and all vendors who supply us with data center equipment are on site. Our initial assessment, although early, points to being able to have some service restored by mid-afternoon on Sunday. Rest assured we are working around the clock.

We are in the process of communicating with all affected customers. we are planning to post updates every hour via our forum and in our customer portal. Our interactive voice response system is updating customers as well.

There is no impact in any of our other five data centers.

I am sorry that this accident has occurred and I apologize for the impact.

[Kevin Hazard]

Because the management servers were located in H1, services provided by ResellOne, Legacy EV1Severs domain management, and retail SSL have also been affected by the outage. Domain and SSL functionality in Orbit has proven to be unaffected by the outage.

[Kevin Hazard]

As you know, we have vendors onsite at the H1 data center. With their help, we’ve created a list of equipment that will be required, and we’re already dealing with those manufacturers to find the gear. Since it’s Saturday night, we do have a few challenges.

We are prioritizing issues as follows:

1. Getting the network up at H1 is first and foremost. We’re pulling components from our five other data centers – including Dallas – which will be an all-night effort.
2. Getting power back to the data center is key, though it is too early to establish success there.
3. Because ServerCommand is in H1, our legacy EV1 customers are blinded about this incident. We are in the process of moving the ServerCommand servers to other Houston data centers so that we’re able to loop them into communications.
4. We absolutely intend to live up to our SLA agreements, and we will proactively credit accounts once we understand full outage times. Right now, getting customers back online is the most critical.

[Todd Mitchell]

Hello,

Our various departments continue to work very hard to restore service to our H1 data center. We expect to have ServerCommand access restored within the next several hours as well as access to single homed nameservers out of H1 redirected to another facility.

We will continue to provide status updates once an hour. We do not have an Estimated Time to Repair at present.

Todd

[Todd Mitchell]

Ladies/Gents,

Our facilities team continues to work with our vendors at the H1 site to restore service to affected clients. We do not have an Estimated Time to Repair at present. Our staff and management continue to work through the night and we will continue to provide hourly updates.

Todd

[Todd Mitchell]

We continue to work through the early morning hours to restore service. We have vendors on-site working with our facilities group.

We do not have an Estimated Time to Repair at present. Our staff and management continue to work through the night and we will continue to provide hourly updates.

Todd

[Todd Mitchell]

Our UNIX and development team continue to work to restore service to both ServerCommand and EV1 DNS. Based on current information, 4 of the 8 DNS servers are in service and we expect the remaining DNS servers to come online within the next 180 minutes. The same approx. time line holds true for ServerCommand. The server farm has been relocated to another data center and development is currently working on bringing the services back online.

In terms of the facility, we do not have a firm ETR at the moment. Facilities continues to work with our on-site vendors to acquire replacement equipment and get it installed to bring service back online.

We do not have an Estimated Time to Repair at present. Our staff and management continue to work through the night and we will continue to provide hourly updates.

Todd

[Todd Mitchell]

Morning,

We are continuing to work through various issues this morning. We will have additional contractors on-site this morning starting at approx. 7 AM. Some will hand-off from contractors who worked overnight and others will start the recovery/installation of new electrical gear to power the data center.

We are still working through the EV1 DNS and ServerCommand items. We are making progress on both items and expect to have both functional within the next 120 minutes.

In addition to the above, the network engineering group worked overnight to prepare the network for the recovery of H1. We expect the reconvergence of the network to go smooth once H1 comes back online.

We do not have an Estimated Time to Repair at present; we should have a better estimate this morning. Our staff and management continue to work through the night and morning-- we will continue to provide hourly updates.

Todd

[Todd Mitchell]

Hello,

The team here at The Planet continues to work through the various issues that we continue to encounter. We are still making progress on the previous items that I mentioned in my last post. DNS infrastructure has been migrated to another data center and propagation has begun. We are working through some database issues with ServerCommand and fully expect those to be resolved within the next hour.

I’d also like to address the idea of migrating from one data center to another. During the early stages of the H1 data center we opportunistically relocated some customers to another data center. However, due to network and data center (power/cooling) constraints, this option is no longer available and requests for migration cannot be honored. Please rest assured that our teams are working diligently to return service to all affected customers.

At this time we do not have an Estimated Time to Repair at present; we should have a better estimate this morning. Our staff and management continue to work through the night and morning-- we will continue to provide hourly updates.

Todd

[uvashi]

More teams from The Planet are coming along with more contractors from key vendors for electrical and facilities to help get H1 online. At this time, DNS infrastructure continues to propagate. ServerCommand servers are installed, but the teams are making sure all networking is intact and ready. In the meantime, please call our support lines for any issues. Additional support techs are available.

At this time, we do not have an updated Estimated Time to repair. Please continue following this thread for updates.


-----
Urvish Vashi
Director, Product Management
The Planet