[root@srv1 admincp]# uptime
20:29:04 up 15:34, 1 user, load average: 101.35, 60.54, 27.80

My server works fine until all of a sudden the CPU usage sky rockets and then I have to restart the web server...

httpd -k restart

and then it works fine for a bit and then jumps to high usage and eventually crashes.

What do I do? Any help is appreciated.

Greetings:

There are a number of things you or your system administrator could check at the time this occurs (given your monitoring system alerts you):

netstat's to see if you are being DoS or DDoS'd
Apache log files (access and error)
top
etc.

The information provided is too minimal to be more specific.

Thank you.

Here are fresh results from top

As you can see hte server load is extreme and without reason it just jumps so high that my server falls it its knees...

This started happening very recently. I have about 3 sites running from the server with nothing big.

Now what else do I post here for you talented smarties to debug this issue?

I am fairly new to Linux so if you need any info please provide me with the commands. Thanks!

Based upon your Top results, I would look in your Apache access logs to see what is being accessed on the server. You have one httpd process that is eating up 84.8% of the CPU, which to me indicates you have a run away script somewhere.

Hi, where do I check the apache access log file? Is it

/var/logs/httpd/access_log?

Is 1.2 million unique hits too much traffic for a 1.3 Ghz Celeron server?

LOL......that would depend on the time frame i guess.

but 1.2 million unique users at the same time......... i think that would be to much for ANY Celeron server.

Okay, I am really getting annoyed by the server problem. The reason why I asked was because I get about 1.2 million uniques a month so I was thinking the server is toast but luckily I had a friend from work who ran some commands on the box and showed me what was crashing....

From what it seems the httpd server is restarting every second! I dont' know whats causing this but here is some information for you rackies... (rack techies.. lol).

First... the

# ps aux command

Latest top...

I have a lot of this line in my messages log

Jun 2 00:49:19 srv1 Microsoft FrontPage Server Extensions: (OS Error #13 Message: Permission denied) (userid 10003)
Jun 2 00:49:19 srv1 Microsoft FrontPage Server Extensions: http://www.mydomain.com - Error #20003 Message: Cannot create folder "_vti_cnf". (userid 10003)


What does this mean? How to fix it?

Just to educate those who didn't know this error message.... this is a common error message for anybody who runs Gallery with FP extensions. Gallery uses apache user and apache group to create and maintain albums, photos etc. I changed this to the group and user that FP uses and the error message dissapeared from the log but the Gallery stopped working...throwing errors such as permission denied. I renamed it back to apache apache and it works like a charm and the hell with error log ... Didn't know the work around this to keep both MS FP and Galler happy... so I left it as it is

1) 1.2M per month shouldn't cause an issue. Besides, this is not multiple visits you're seeing in top. That is a single httpd instance eating up all of the resources. In top, you'll see a separate httpd entry for each connection.

2) You say you only have 3 sites on the server. Tail the access logs for those individual domains to see which files are being accessed when you see an httpd process running out of control. Unless there's some major misconfiguration I'd be willing to bet that it's a run away script. Those should be located at /home/httpd/vhosts/domain.com/statistics/logs/access_log

3) It'll be easiest to find if you have multiple ssh/command line sessions running at the same time. One watching top and one watching each access_log.

4) When you watch top does the PID number (on the left) change for that one httpd process that is eating up your CPU? Or does it stay the same? If need be you can even issue a kill command to stop the process and see if it reconnects. ie: kill where is the PID of the run away process.

Thanks Squire! I will be getting down to this today. I will have multiple shells open and monitor the activity on three sites. If you see my log from

# ps aux

you will see the multiple httpd sessions that are being started...

As I am going through my log files... who is admin13??? I knwo the path is the path for PSA where it stores the default index.html page why would anybody be accessing the admin13 folder although it never existed! Any thoughts on this?

[Thu Jun 03 04:39:26 2004] [error] [client 213.252.152.15] File does not exist: /home/httpd/vhosts/default/htdocs/~admin13
[Thu Jun 03 04:39:26 2004] [error] [client 213.252.152.15] File does not exist: /home/httpd/vhosts/default/htdocs/~admin13
[Thu Jun 03 04:39:27 2004] [error] [client 213.252.152.15] File does not exist: /home/httpd/vhosts/default/htdocs/~admin13
[Thu Jun 03 04:39:27 2004] [error] [client 213.252.152.15] File does not exist: /home/httpd/vhosts/default/htdocs/~admin13
[Thu Jun 03 04:39:28 2004] [error] [client 213.252.152.15] File does not exist: /home/httpd/vhosts/default/htdocs/~admin13
[Thu Jun 03 05:56:36 2004] [error] [client 202.146.128.127] request failed: error reading the headers
[Thu Jun 03 07:45:58 2004] [error] [client 212.113.164.99] File does not exist: /home/httpd/vhosts/default/htdocs/~admin14
[Thu Jun 03 08:15:34 2004] [error] server reached MaxClients setting, consider raising the MaxClients setting

Now I am getting this in the

==> /var/log/httpd/suexec_log.4 <==

file.

By the way to see all the log files at once this is a best command I found...

# tail /var/log/httpd/*

This command will get the last few lines of all the log files.. you can search for only Jun 03 or anythign else if you wish...

[Thu Jun 03 04:39:26 2004] [error] [client 213.252.152.15] File does not exist: /home/httpd/vhosts/default/htdocs/~admin13
[Thu Jun 03 04:39:27 2004] [error] [client 213.252.152.15] File does not exist: /home/httpd/vhosts/default/htdocs/~admin13
[Thu Jun 03 04:39:27 2004] [error] [client 213.252.152.15] File does not exist: /home/httpd/vhosts/default/htdocs/~admin13
[Thu Jun 03 04:39:28 2004] [error] [client 213.252.152.15] File does not exist: /home/httpd/vhosts/default/htdocs/~admin13

does your machine have HyperThreading? If so, then you should know that HT enabled CPU's are notorious for false reporting of memory usage. There may be nothing wrong at all. My server shows a 99% memory use almost all the time, even when nothing is running.

Just a thought


C

I have a Celeron 1.3hz with 512 Ram.... does that qualify for hyperthreading?

NO.

celery? no way lol