Hey guys, I'm in Orange County, CA and we're experiencing problems connecting with our ThePlanet servers. Here's a tracert:



Any ideas? Anyone else having connection issues today?

Its not just you ... im experiencing some packet loss as well (not nearly as bad as yours though) and submitted a trouble ticket about an hour ago

I cant reach one of my servers right now. Anyone experiencing this today?

Yeah, one of my servers is down too, but they're responding to the problem.

First issue I've had in a year I think.. Last time was a Seagate hard drive dying... I'll take network problems over dead drive issues any day!

Most definitely. A dead hdd will make me sweat, but a network problem will make someone else sweat.

I've been getting alerts about clients servers going down for the past week or more. I can't even reach some of the servers from ThePlanets own network. It seems to be on the old EV1 network segments in Houston.

Same here, several servers I manage / lease from the old EV1 days are unreachable, same hop, and I can't even get to servercommand.net either ..



Would be nice to see an update on the forums about what is happening since we can't get to the TT system .. in the old days we would have had a post on the system and network status space already.

Something is up i lost access to my box on all ip's and can't reach https://servercommand.net/

I can't reach https://www.servercommand.net either. Not sure what is going on. One of my servers is down, but not all of them.

I can't reach https://www.servercommand.net either. Not sure what is going on. One of my servers is down, but not all of them.

http://forums.theplanet.com/index.php?show...c=90183&hl=

It is all of DC1, building is evacuated apparently...

Hi John, nice to see you respond, hope all is well ... long time no talk.

only one of my servers is offline.
=x, i will be waiting, but will be good if I know what happend.

Glad to see I'm not the only one, so it's not my server. Can't even dig servercommand.net from another network. It's like their DNS is totally out.

Cannot access there from europe; Also phone lines don't work.

Ok, now all of my DNS hosted at NS1.ev1servers.net and NS2, NS4 and NS5 are all not responding. Only people that haven't cleared out their cache are able to hit it... what's going on?

Is this the same hack that took down comcast.net earlier this week?

I just pulled a few trace routes.

This is from an optonline.net connection :
Tracing route to 207.44.186.81 over a maximum of 30 hops

1 <1 ms <1 ms <1 ms 223.3.7.12
2 8 ms 8 ms 8 ms 10.x.x.x
3 6 ms 8 ms 10 ms dstswr2-vlan2.rh.brptct.cv.net [67.83.224.162]
4 * * * Request timed out.
5 * * * Request timed out.
6 10 ms 10 ms 13 ms rtr4-tg11-1.in.nycmny83.cv.net [64.15.0.41]
7 32 ms 31 ms 32 ms 64.15.0.98
8 * * * Request timed out.
9 34 ms 35 ms 36 ms so-0-1-0.mpr1.ord2.us.above.net [64.125.30.146]
10 60 ms 54 ms 52 ms so-2-0-0.cr1.dfw2.us.above.net [64.125.30.245]
11 52 ms 54 ms 53 ms xe-1-1-0.er1.dfw2.us.above.net [64.125.26.210]
12 61 ms 54 ms 53 ms ge-2-0-0.mpr2.dfw2.us.above.net [64.125.26.213]
13 55 ms 54 ms 54 ms so-1-1-0.mpr4.iah1.us.above.net [64.125.26.134]
14 57 ms 54 ms 56 ms 289.ge-4-2-1.mpr1.iah1.us.above.net [216.200.251.58]
15 * * * Request timed out.
through
30 * * * Request timed out.

This one is from dnsstuff.com :

1 0 0 * 0.4 ms 75.125.82.131 AS0 IANA-RSVD-0 [Timeout] 255 US
2 0 0 * 0.4 ms [+0ms] 209.85.0.1 AS36420 EVERYONES-INTERNET3 gphou2-209-85-0-1.ev1servers.net. [Router did not respond]
3 0 0 * 0.6 ms [+0ms] 66.98.241.225 AS30315 EVERYONES-INTERNET2 ivhou-66-98-241-225.ev1servers.net. [Router did not respond]

Spoke to Matt in support looks like a network issue no eta yet on a fix though

I am not able to even call in, I get a busy tone.

sure sign of problems. busy signals or straight to voice mail :)

This has been happening for more than a week. I would have thought the problem would be resolved by now.

Same problem here. Server and servercommand inaccessible. I sat waiting for a customer service chat person for a while and gave up when I found this thread. Hope it gets resolved soon...

Aargh!

Same problem from here in the UK, servercommand.net inaccessible and unable to ping any of our servers...

At least it's 00:45am here so we're escaping most customer impact.

Fingers crossed for a quick resolution, hang in there support guys - we know your doing your best!

Colin.

There is a status post (Although its fairly vague) here:
http://forums.theplanet.com/index.php?showtopic=90185

I am also in the UK and wondering if it will be a 12+ hour outage and if I should go to bed and think about it in the morning.

Info from ThePlanet is totally useless on that front

Tom - www.mouselike.org

My server is down too.

Tried to contact the company.. all phones no matter what DC I have called ring busy, or no answer then busy.
Most companies when they have an outage have a recording at the very least, along with updates every couple of hours.

Datacenter Manager of the year announced:
http://searchdatacenter.techtarget.com/new...1314835,00.html

Could someone explain how these changes are preventing the downtime we all are experiencing right now?

Link has a tour of what appears to be the houston DC.

Please post an update to this situation !

This sucks, it's 2.35am here in Europe, have better things to do than to wait some kind of update...

Hard to believe they can't do better than this http://forums.theplanet.com/index.php?showtopic=90185 !

If the DC has gone up in flames they might as well tell us now so we can move on to better things...

All sites on both IPs on my server are unresponsive.

I cannot login to ServerCommand customer portal to open a trouble ticket even.

I can't even get through to their phone # (the 281--- number) as it's permanently busy.

There's going to be HELL to pay with all my paying customers once my email starts working again (of course, hosting my own email on my server also means my email is down so I'm forced to using gmail now).

BUT THE THING THAT PISSES ME OFF THE MOST??: They don't even have any announcements up on their main page (or anywhere) saying there's a problem! So Until I came here and read your msgs I thought that I was the only one!

C'mon The Planet - GET WITH IT. When something serious like this affecting so many people is going on: POST A FRIGGIN MESSAGE ABOUT IT.

I hear yah.

Ok, I called this #: (866)-325-0045 -- and there's a msg saying they're aware of the problem. WOULD BE NICE IF THEY POSTED IT SOMEWHERE!! Like maybe in the portal login web page because that's where everyone will go before anywhere else!!

but of course if they ADMITTED having a problem on a publicly accessible web page it would scare off potential new customers... Sure. Leave your current customers in the dark as to what's going on as long as you can get new customers.


GGGGGGRRRRRRRRRRRR!!!!

Stuff like this never happened when it was just EV1.

well stuff like this will get out, nowdays the Internet moves faster than the speed of light, the test for any company really is how they respond to the issue and how they keep their customers informed. Costs more to get new customers than to keep the ones you got.

You are close to the truth, check the updated post on http://forums.theplanet.com/index.php?showtopic=90185

Thanks for the link drsox.

Ok, I don't want to sound grumpy but I'm in that kinda mood at the moment due to obvious reasons. They say a transformer caught fire then the fire dept. told them to turn off their backup generators... now... correct me if I'm wrong but ISN'T THAT WHAT THE GENERATORS ARE *THERE FOR* ?? IN case you can't get power from the grid !!!???

GRRR!

Sorry everyone for being so pee'd off the last few posts. I'm just not looking forward to answering dozens and dozens of emails to my customers who are [rightfully] complaining about not being able to access their sites, their accounts, lost revenue, asking me for re-imbursements, etc, etc, etc, etc...

I feel for you buddy, I had to find out that our server was down through a customer that contacted be approx. 20 minutes before the transformer supposedly blew .

Well, they're for regular outages/failures. When fire's involved I'm guessing things change.

"Our initial assessment, although early, points to being able to have some service restored by mid-afternoon on Sunday."

SOME service by mid-afternoon Sunday?????????!!!!!!!!!

That is OUTRAGEOUS!! They better be giving out credits or free server upgrades. Wow. I'd be better off with servers from GoD@ddy.

I'm feeling the heat (no pun intended at all) right now as well but getting hot headed about things will just make it worse for all of us.

I will lose my company if my server isn't back up by Monday morning 8:30am. I can't even get to the remote backups! Which I pay money for!

Maybe you need to reviews the SLA between you and your customers.

What was my reaction with i read the word "Some". I think it's crazy that 7,500 customers can be affected for so long with no short-term back-up plan.

Lots of people choose The Planet because they need good quality servers which are usually provided. But when people run businesses off those servers, and have paying customers that depend on your servers, this leads to big problems.

Also, since server command is down the websites I had running on my other 3 servers don't work because DNS is completely dead as well too......

I'm dealing with my customers right now who are stunned their services are down and are demanding re-reimbursement. That's on top of the money I'm losing through ads and everything else.

After all this time I JUST got an email from The Planet letting me know what's going on. I do hope The Planet takes care of the customers who are affected by huge outrageous outage. Free server upgrades would be nice, free service for a month, you know SOMETHING to make sure you retain these customers.

Viewing your sig I couldn't help but laugh. Thanks Mark. I needed it.

Hahaha, yes, it makes me laugh right now too...

What I don't understand is that apparently there is no damage to the network and servers, so what needs to be replaced exactly other than the transformer and why is it taking so long ?

The first reports indicated that the FD had forbidden ThePlanet to use the backup generators, so why aren't these being used now, something is not jiving here...

The explosion damaged alot of the switching equipment and much of the backup switching equipment.

So this is just their H1 center? How many datacenters do they have and where are they located? My server is down and I know that it's located in H1. Got a new server YESTERDAY and that's down too - guess The Planet deployed it in H1 too, *sigh* not a good start of the weekend. Think it's possible to get your servers moved to another of their datacenters after this is fixed? Not too sure about keeping anything in H1 anymore...

Also, annoyed that we didn't get any notices from the monitoring system on ServerCommand. Should get warnings to our email/cellphone if the servers are not responding. Found out about the outage when I woke up today (I am in Europe).

Luckily not anyone of our clients have asked about the outage yet, but then again we have no way of notifying them about the problem either because we have all our support/customer service online. Thank god it's Sunday and warm outside.

ServerCommand systems were housed in H1 so they couldn't send you any warnings :/

Thanks for clarifying, is that switching as in network switches or electrical switches ? Either way that sounds bad because that's going to mean actual replacement of the switches and there must be a ton of these guys ?

I am guessing electrical switches 'cause having network switches nextdoor would probably be a bad idea? Still, this explosion shows even how many backup generators, and ups' you might have for keeping things running there is always a weak link.

Even so I must give Ev1Servers/The Planet credit for keeping things up and running during Hurricane Kathrina, everything worked great and no interuption in service was noticable.

Hope to hear some good news soon.

It was written that name servers (ns1.ev1servers.net, ns2, I guess) were migrated and the network convergence begun. What is the magnitude of the this convergence? Minutes, hours or days?

Well thats only DNS and they don't appear to be working yet.

They have no ETR on the servers in the datacenter.

Yes, most our servers in another data center (Houston 2?, it is a private rack), and they are online, but because of the ns1.ev1servers.net DNS servers went offline they are not accessible to users. This is why I am interested what is the magnitude of the time required for the convergence.

The DNS is up and working now, it looks like it was earlier as well just a little slower then normal.

Unfortunately it doesn't seem to be working from here:
> set timeout=30
> solware.com
Server: ns2.ev1servers.net
Address: 207.218.247.135

DNS request timed out.
timeout was 30 seconds.
DNS request timed out.
timeout was 30 seconds.
*** Request to ns2.ev1servers.net timed-out