Help - Search - Members - Calendar
Full Version: Un-explained Bandwidth Usage
The Planet Forums > General > Suggestions/Comments
yittrix
I have noticed a strange trend.

Rackshack's MRTG graphs will see a huge spike in bandwidth usage for about 15 minutes and then it stops however my servers MRTG graph does not show any major activity during this time.

I have checked everything from IMAP usage to FTP usage. I have also checked the firewall logs and there was no activity on the server at this time.

The last spike I have was on August 21, 2003 sometime between 20:00 and 23:00.

http://bandwidth.rackshack.net/RSbw5.cgi?c...7.44.164.247_11

http://rack.yittrix.org/mrtg/localhost_2.html

The spike used about 7.0GB of bandwidth in and out combined according to Rackshack's monitoring however MRTG for the same period only shows 66.82 MB in and 38.19 MB out.

Total usage reported by my MRTG graph is:
1.733 GB down and 0.975 GB Up.

Rackshack's monitoring shows below.


Date In (bytes) Out (bytes) In (Gigs) Out (Gigs) Total (Gigs)
-------------------------------------------------------
8/22/03 4,200,893,535 4,137,684,147 3.91 3.85 7.76
8/21/03 4,166,203,570 4,072,346,650 3.88 3.79 7.67
8/20/03 1,030,686,031 1,837,086,826 0.96 1.71 2.67
8/19/03 1,028,930,812 1,801,357,789 0.96 1.68 2.64


Anyone else seeing the same problem?
Akai25
Yes, same Problem, since rs have change from MRTG to rrtool!

I have written many trouble tickets. Every ticket have the supporter closed with every thing ok, nothing is ok! ;-(
Every computer on "my" rs switch have the same peaks, on the same time, but for rs is this normally...
Now, I am stupid or the supporter.

Look, at my traffik peak at Mon - Tue and yours an on the Weeks..., we have the same traffik! Ohh, magic or a big broken traffik counting system.
Traffik
TechieSurfer
yittrix -

I am aware of a graphing problem that can occur when the switch is unavailable for polling (ie the switch is down or subnet is down.) This can cause an unusual spike for that period of time. I believe that is what happened. Below is a log of our switch monitoring during that time:

20030821 1934 DOWN 207.44.164.247 207.44.164.247 Timed Out
20030821 1939 UP 207.44.164.247 207.44.164.247 missed 5

This seems to exactly correspond to the event, and clearly we could make an adjustment if you were going to be charged for exceeding your bandwidth allotment.


Akai25 -

Your subnet appears to be experiencing a different issue, but the graphing appears to be correct. The aforementioned problem is related to switch/subnet downtime and erroneously reports spikes of both inbound and outbound traffic. Your switch has not suffered recent downtime and the unusual traffic component is all outbound - inbound traffic during this time is absolutely consistent. (Both of these as measured from the switch.) This suggests the traffic is broadcast/multicast activity or port scanning, but unfortunately this activity was not occurring when I investigated, although there was a spike of it around 6:15 am today.

I will continue to try to 'catch' this broadcast/multicast traffic, and it would be nice if you could also try and capture this traffic - possibly based on a trigger.

Sincerely,

Randy Williams, CTO
Akai25
TechieSurfer:

No, not this morning peaks, what I mean. The high peaks was since the last five weeks, the last was Tue morning.

Yes, I am counting the traffic in/out and the broadcast.
On witch E-Mail address, can I send you a copy from the log?

Thanks.
TechieSurfer
Akai25 -

I was also including the 5 mb spike on Tuesday morning. The inbound traffic across all ports is consistant, and therefore a fundamental polling issue can be ruled out.

Include the data in a trouble ticket.

Randy Williams, CTO
BudWay
Same problem here.....
TechieSurfer
Budway -

I see 2 outbound spikes Thursday and Friday morning, but each of them was only about 130 Kbps, and the one on Friday also had a inbound component. I was unable to locate a similar spike on the other graphs from your subnet during those times.

Sincerely,

Randy Williams, CTO
cmafia
Mine spiked so hard at the beginning of the month that I thought I was going to be over 500 gigs. Then, it tapered off! I talked to online techs and the guy couldn't figure out why.
TechieSurfer
cmafia -

Notice that your high traffic has an outbound component that is proportional to the inbound traffic, and this pattern lasted several days. That was legitimate traffic from your server, and is inconsistent with the brief erroneous traffic spikes reported by the thread starter and the traffic I am investigating for Akai25.

Once again the graphing system *does* erroneously report data when the switch or subnet is unavailable (for obvious reasons), and we will adjust the bandwidth billing wherever necessary.

In addition we are investigating a mechanism to detect the anomalies in the raw rrd data when a switch is unavailable. If successful we will be able to automatically adjust the bandwidth totals to account for these anomalies.

Sincerely,

Randy Williams, CTO
cmafia
QUOTE
Originally posted by TechieSurfer
cmafia -  

Notice that your high traffic has an outbound component that is proportional to the inbound traffic, and this pattern lasted several days.  That was legitimate traffic from your server, and is inconsistent with the brief erroneous traffic spikes reported by the thread starter and the traffic I am investigating for Akai25.

Once again the graphing system *does* erroneously report data when the switch or subnet is unavailable (for obvious reasons), and we will adjust the bandwidth billing wherever necessary.  

In addition we are investigating a mechanism to detect the anomalies in the raw rrd data when a switch is unavailable.  If successful we will be able to automatically adjust the bandwidth totals to account for these anomalies.

Sincerely,

Randy Williams, CTO


Thanks Randy.
BudWay
Hehehe....


I was just kind'a i don't have traffic problems, hehe hope not to have till about 1 mo. from now icon_smile.gif


Realy hope there is a Dual Xeon 1 GB DDR RAM RAID 5 3 x 73 GB ( 146 gb ) 1200 gb it cpanel buy than icon_smile.gif

I'm really in love it that server.... icon_smile.gif
axiom
I've had a server for over a year, with an average use of about 10GB total BW a month, until september, when BW monitoring started reporting a use of about 95% of it. Now, the usage for this month (november) shows 99.5%. I don't understand how is this possible. Server has only 7GB of HD space used in total. No sites have been added since august, and there are no big files of any kind (mp3, zips, etc.). PSA usage report shows only 10GB of BW used for the month. What's more strange, In and Out BW appears to be 360GB... how's that possible? if there were people using it for downloads, I believe there would be a difference. The MRTG is located at:

http://bandwidth.ev1servers.net/RSbw4.cgi?...16.127.82.247_5

Does anyone have an idea of what could be causing this?. EV1 Trouble tickets are closed stating everything's OK, but clearly it's not. I checked the server for intruders, but there are none. No site is getting thousands of hits; no site is big nor hosts any kind of big files.

icon_sad.gif I'm getting really depressed over this issue. I cannot believe my server with modest sites is using more BW than big porn sites.
EricP
QUOTE
Originally posted by axiom
I've had a server for over a year, with an average use of about 10GB total BW a month, until september, when BW monitoring started reporting a use of about 95% of it. Now, the usage for this month (november) shows 99.5%. I don't understand how is this possible. Server has only 7GB of HD space used in total. No sites have been added since august, and there are no big files of any kind (mp3, zips, etc.). PSA usage report shows only 10GB of BW used for the month. What's more strange, In and Out BW appears to be 360GB... how's that possible? if there were people using it for downloads, I believe there would be a difference. The MRTG is located at:

http://bandwidth.ev1servers.net/RSbw4.cgi?...16.127.82.247_5

Does anyone have an idea of what could be causing this?. EV1 Trouble tickets are closed stating everything's OK, but clearly it's not. I checked the server for intruders, but there are none. No site is getting thousands of hits; no site is big nor hosts any kind of big files.

icon_sad.gif I'm getting really depressed over this issue. I cannot believe my server with modest sites is using more BW than big porn sites.


Whoa.

Those are some funky looking graphs. I would really ask Tech Support to Please take a deeper look at your server.
TechieSurfer
axiom -

After a great deal of investigation, I figured out what is happening.

Another server is misconfigured and is sending all of their outbound traffic to your server (to your assigned .1 address). Your server must also be configured as a router, and it is forwarding all of that traffic to the correct default gateway. This explains why the inbound and outbound traffic to and from your server appear to be equal.

I have already asked for a ticket to be opened so that the misconfigured server can be corrected, and of course we will not bill you for this bandwidth...

Sincerely,

Randy Williams, CTO
axiom
QUOTE
axiom -  

After a great deal of investigation, I figured out what is happening.

Another server is misconfigured and is sending all of their outbound traffic to your server (to your assigned .1 address). Your server must also be configured as a router, and it is forwarding all of that traffic to the correct default gateway. This explains why the inbound and outbound traffic to and from your server appear to be equal.

I have already asked for a ticket to be opened so that the misconfigured server can be corrected, and of course we will not bill you for this bandwidth...

Sincerely,

Randy Williams, CTO

Thanks, Randy. I will surely have a nice sleep tonight after your reassuring response.
This is a "lo-fi" version of our main content. To view the full version with more information, formatting and images, please click here.
Invision Power Board © 2001-2010 Invision Power Services, Inc.