I'm getting about 30% packetloss to my colo in DC2. at 4.78.244.14 -> dsr3-1.dllstx3

yup me 2

Okay, it's not just me then. Just spoke with a technician, he said they know about it and are working. *thumbs up*

I'm getting the same. Hope this is resolved soon or I will have a lot of support tickets to answer.

I just don't understand it really.

I've used the same 2 datacenters for over 2 years now.

Both are in Texas and both are very large and do well.

1 has had a TON of issues and the other has been solid as a rock.

:shock:

Wow.. It's insanely lagging on my side... it takes like 5 mins+ for a page that normally takes 2 seconds to load. Does this happen often? I just moved most of my sites over this week.

edit: man even this post took a good two mins before it finally got posted.. :shock:

I'm noticing it just got slightly better. *very* slightly....

Update:

The packet loss is being caused by a very large DDOS attack (6Mpps). We've blocked the packets at our border routers, but the packet load is still high enough to cause at least one of our BGP sessions to flap (causing intermittent routing inconsistencies). We've contacted the provider and asked them to null route the IP that is being attacked. Traffic should return to normal in a few minutes. We apologize for any inconvenience this is causing.

Do you mean 6Gbps? I would seem to think 6Mbps would be miniscule

All better for us over here.

Thanks

6Mpps is correct (Mpps=Million Packets Per Second)

This is probably the largest DDOS I've personally seen (ever). The traffic coming in via Level(3) was high enough to max out the interfaces processing ASIC's.

The Null route is in place and things do indeed appear to be much better.

Everything does seem to be doing much better. Good job!

Oops, read that as bytes. pps is more reasonable, and that is pretty nuts :shock:

Thanks for the update! Good to see that everything is back to normal.

Null routing. Sad it had to come to that, I'd rather see it blocked.

However, I understand that sometimes even the DDoS mitigation isn't enough, and I was also affected by this DDoS (packetloss I mean). I'm glad it got dealt with.

Still, it's too bad it had to come to null routing.

:shock: I've never heard of such an attack..how would one come about getting such heavy bandwidth to launch one of them :shock:

Null routing is an option of last resort. The attack was severe enough to affect the infrastructure and the sources of the packets were spoofed, so filtering was not possible.

Regarding how an attack of this size is launched, it's usually done via large drone and zombie farms.

I was under the impression that virtually all DDoS attacks used spoofed packets, and I was explicitly told by ServerMatrix employees (Possibly yourself, I don't remember) that FloodGuard could handle spoofed packets. Is this not actually true, or is the Cisco Guard solution not as effective as FloodGuard was?

I don't suppose you can tell us who the target was? ;-)

You see folks THIS is why we pay top dollar for hosting. We may have experienced packet loss but not complete infrastructure failure.

In other words, we may have been bruised, but we're still beating the crap out of the offender!

Actually, I don't think we are paying top dollar - there are other, more expensive providers out there. But I do think we are getting a lot for our money, some things that the more expensive providers may not provide. Yes, it was good to see WC on top of this nice and quickly.

It's happening again this morning. I don't know who the target is but it's obviously near the shared load balancers being as those have gone comatose both times (although last night a lot of things were...)

Seems the one this morning is about half the size but continues to go on (check Network Flow graph in Orbit).

well as I saw the graphs, elmost every provider/peer took a dive, even Cogent.

Probably some stupid 12 year old kid with beef.

Aaaaaaaanyway, Guspaz you do realize 6Mpps is one HELL of a lot of data right, and that basically nothing can stop it? :shock:

WCharnock can stop it!

Nevermind, just AT&T having a moment this time...

Yea im getting huge drops from att and it looks like level3 is picking up the slack on the planet's side, this is through our t1 through them...

Same here, we have a couple AT&T T1's here in our office and our vpn drops out, and we are unable to reach theplanet.com even.

I just experianced a total server outage (unable to get to server or to servermatrix/theplanet for that matter).

Seems to be fine now, but for 3-5 mins a short time ago it was out completely.

Z

Yeah I wasn't able to reach my server for an entire 23.5 seconds.
I went bonkers!

I almost went so far as to pack my suitcase, board a plane, parachute into DC4' and solve the problem myself......

then I realized it wasn't that big of a deal.

Moving on....