Hi guys,
I run cacti to monitor my server, and I noticed that around noon today, apache hit 112 busy threads and lasted for about half an hour.
Usually my server has an average of no more than 30 busy threads.
This jump in usage, caused a system load of over 15 for that period and pretty much rendered the server useless.
The server was getting requests of 500 kbytes per hit.
What I need is a way to track down which domain was causing all this traffic/load.
I have over 30 domains, and I'm not sure if the access_logs will show anything. What should I look for in the logs?
I'm running plesk 7.5.1.
I also checked the tippingpoint monitoring in the ev1servers control panel, but didn't find anything in that period of time.

Thanks in advance for any tips/suggestions.

-d

If you have any statistics software installed, you could just parse the reports and determine the culprit there.

If you want a more immediate solution, you could use strace to trace httpd, and grep out which sites are making the most system calls.

example:

for i in $(ps -C httpd --no-headers | awk '{ print $1 }'); do PIDS="$PIDS -p $i"; done
strace -vvFf -o /root/strace.httpd $PIDS

Let this run for about a minute or so during a peak time of the resource usage. After letting this run for a bit, you will need to break out of it ( control + c ). You'll now have a file /root/strace.httpd that contains all the system calls made by httpd.

Since the strace output is undoubtedly going to be lengthy and intimidating, it would be best to grep through it for site usage. You could do this with the following regular expression:

egrep -i '/home/httpd/vhosts/[-_a-z0-9/]+.[a-z0-9]+' /root/strace.httpd | awk '{ print $2 }' FS=""" > /root/httpd.usage

This should make things a bit easier to distinguish, and allow you to easily isolate the culprit.

- Steve

Thanks a bunch for your help. I will try this next time I notice the problem.
Is there a way to track what happened from the previous logs?

Thanks
-d

If you have urchin installed, or are installing it, then yes. Outside of that, it would be very difficult to determine what site caused the load outside the time of the high load average

- Steve

How about enabling server-status in the httpd.conf file

Then all you have to do is go to yourdomain.com/server-status/ and see what every apache child process is doing.

Turn on ExtendedStatus too:

from httpd.conf:


SetHandler server-status
Order deny,allow
Deny from all
Allow from .yourisp.com
Allow from .yourwork.com



and:

ExtendedStatus On

install phpsuexec, might help...

installing mod_dosevasive might help also

regards