EV1Servers Name Servers Status
06/23/06
5:50 pm CDST (-5 UTC)


At ~ 3pm this afternoon we were forced to reboot the nsx.ev1servers.net name servers.

The reload of the zone files took an inordinate amount of time.

Once the reload was complete the DNS response was compounded by repeated retries.

We resolved that issue by removing the servers from the load balancers.

At present the name servers have stabilized.

I apologize for the delay in getting this post out.

I have had my hands full.

I will update you as necessary.

As always I appreciate your patience and your continued support of EV1Servers.

Val

Dumb question.

But shouldn't the name servers be independant and redundant? So if one nameserver goes down the other acts as a backup.

Seems like there is no redundancy and safety if all the nameservers can go down.

So, any updates yet as to the root cause, resolution, or steps taken to minimize future DNS outages?

I'm sorry, but one hurried post (I understand why of course) in the "General Comments/Suggestions" forum doesn't seem like enough communication for an issue of this magnitude, IMHO. Considering how much attention a link/router/forum issue of minimal impact is given in the "Network Status" forum, I would hope this issue is worthy of some more attention and communication to your customers. Heck, even a simple email with a link to this thread would have been better than what I've seen so far.

Thanks much,

andrew

I agree.

Three years ago, I was looking for a new provider for a dedicated server. I came across Rackshack. At the time a transformer blew up and Rackshack was under back up power generation. There was constant updates, pictures of the back up to the back up generators being brought in. Details of the emergency back up plans. I was so impressed at the level of communication I ordered my first server.

Now three years later, the nameservers go down - where was the backup plan, the communication to customers?

I've made changes yesterday to the DNS and ev1.servers.net and ev2.servers.net still don't show the changes. I've put the trouble tickets in. No explanation except for we are still investigating. If changes to DNS have been disabled to 'stabilize' the name servers, where is the communication?

I would recommend to Val - to go back three years ago and take a look at the level of communication that was so impressive that I went ahead and ordered a server despite the fact that Rackshack was under backup generator power at the time and contrast and compare to this level of communication.

Leaving customers in the dark and wondering was not the RackShack way. Don't make it the Ev1Servers way now.

Thanks for listening.

any updates on this?

EV1Servers Name Servers Update
06/26/06
5:00 pm CDST (-5 UTC)

We have identified an unforeseen failure state that resulted in the recent outage of the NS1/NS2 dns arrays. Over the course of the weekend Ev1 staff has implemented a series of solutions that mitigate the specific issue and eliminate the process that interrupted the arrays' redundancy. In additon the improved solution has greatly improved efficiency and throughput of both arrays.

We recognize the critical nature of this service and are commited to providing a stable and redundant solution. Thank your for your patience while we worked to address this issue.

what issue ?

DNS is inherently designed for failover so what was the issue?

hell, i don't even use the EV1 nameservers and this response is riddled with "we don't want to tell you because we're embarrased"

-=dave

Were you ever been a politician in a former lifetime?

/me has visions of a certain day at DC1, an electrician, a pair of channel locks and a A/C power panel, then suddenly everything when dark when the electrician had removed the DC off the batteries to do something (aka took away the redundancy) then shorted the power.