I recently acquired my second server with EV1: a new RHE 3 cPanel 10 server. My new server is unstable. It freezes up every 1-3 days, the only remedy being requesting a reboot trouble ticket. I cannot access the server via HTTP, FTP, or SSH when it is locked up.

I am by no means an expert at security or administration, but have an fair understanding of most things.

I have not made any major changes to the server as far as I know. These are the additional pieces of software I've added to the server:

APF, BFD, LSM, SIM

I had the problem before and after I ran up2date.

SIM has not reported any services that fail or the load getting too high. I am completely unaware of what could be causing me problems.

If anyone has any ideas or suggestions, I would greatly appreciate it. I have searched the forums for several hours looking for what may be causing this.

I am specifically looking for ways to determine what is causing the system to fail: log files to check, checking for memory leaks, etc.

Any feedback is greatly appreciated. If you need any additional information from me, please ask.

Thank you!

do a memory test, perhaps you have some bad ram...

A few things, what is your Main >> Server Setup >> Update Config set at? Sometimes if you have all three options set to AUTOMATIC it can crash the box. What kernel are you running?

It may also be a bad hardrive, if you do not have anything needed like live sites, and the problems continue, and the hard drive is ok they will probably do a restore for you. Sometimes it is a hardware issue thought that they will handle. Just keep in touch with tech support and it will get sorted out.

I ran memtester with 2 loops on 512MB and 768MB of 1GB and did not receive any errors.

I had the following settings in Main >> Server Setup >> Update Config:

cPanel/WHM Updates - Automatic (RELEASE tree)
cPanel Package Updates - Automatic
Security Package Updates - Automatic

For testing, I have now switched that to:

cPanel/WHM Updates - Manual (RELEASE tree)
cPanel Package Updates - Automatic
Security Package Updates - Automatic

I'm running the 2.4.21-27 kernel.

Any more suggestions? I really appreciate the help.

The server crashed again last night...

how big are apache logs at that time of the day? because if the logs are over 2 gigs.. that will probably be a reason for the crash.

I don't know a lot about apache logs, these are the two that I looked at:

/usr/local/apache/logs/access_log
/usr/local/apache/logs/error_log

Both logs are only 2-3MB. I've never viewed these logs before (on any system). In the access log I found several instances of something very similar to the following:

XX.XX.XX.XX - - [04/Mar/2005:22:12:14 -0600] "SEARCH /x90x04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04H
x04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04H
x04Hx04Hx04Hx04Hx04Hx04Hx0
4Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx0
4Hx04Hx04Hx04Hx04Hx04Hx04H
x04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04H
x04Hx04Hx04Hx04Hx04Hx04Hx
04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx
04Hx04Hx04Hx04Hx04Hx04Hx04
Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04
Hx04Hx04Hx04Hx04Hx04Hx04H
x04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04Hx04H ........................ (continues)

Is this some sort of hacking attempt? DoS attack? Not a problem? There were a few different patterns of long strings after the SEARCH command.

In the error_log file, I did not know exactly what to look for. I did find something similar to the following at the time I believe the server crashed. I don't know the exact time since I have not received any sort of monitor warning.

[Sat Mar 5 01:17:50 2005] [warn] pid file /usr/local/apache/logs/httpd.pid overwritten -- Unclean shutdown of previous Apache run?
[Sat Mar 5 01:17:51 2005] [notice] Apache/1.3.33 (Unix) mod_auth_passthrough/1.8 mod_log_bytes/1.2 mod_bwlimited/1.4 PHP/4.3.10 Fro
ntPage/5.0.2.2635 mod_ssl/2.8.22 OpenSSL/0.9.7a configured -- resuming normal operations
[Sat Mar 5 01:17:51 2005] [notice] suEXEC mechanism enabled (wrapper: /usr/local/apache/bin/suexec)
[Sat Mar 5 01:17:51 2005] [notice] Accept mutex: sysvsem (Default: sysvsem)

Also, I did not know what would cause this (I found quite a few occurances of this):

[Fri Mar 4 00:13:12 2005] [error] [client XX.XX.XX.XX] request failed: URI too long

Thank you so much for your help!

This is a a widows NT exploit i do believe harmless to linux

You need to look though your logs to find out the exact time of system down. Chances are, this is at 4am, the same time Cpanel runs though its daily maintenace. If its at 4am then you need to look and find all the jobs running at 4am to determine why the box is going down. Changes are, high load, out of memory conditions are bringing the box down.

Another thing you should do is a full recompile of the box. I had an issue with one box rebooting weekly. After recompiling the entire box using easyapache the reboots stopped. I would probably recompile the entire box first then see if the problem goes away.

I'm sorry, but I don't know what logs to check to determine the time of the crash, let alone what was happening then. Please let me know what logs to check. I've been searching for posts about this, but haven't been able to locate what I think are the right logs.

Logs I've looked at but can't tell if there's information in them:

/usr/local/apache/logs/ -> access_log, error_log
/usr/local/cpanel/logs -> access_log, error_log
/var/logs/messages

Any insight you may have would be helpful.

Also, the server has not crashed in roughly 2 1/2 days.

Thank you!

I would do what aussie said

recompile a new kernel
recompile apache


I had this problem onnce before box would lock up every three days like clock work

it turned out to be it did not like the kernel booted it to the old kernel I had before and it ran just fine


there is a script somewhere on this forum called snapshot I will take an image of your logs top etc . so you can look to see whats hapening at the time of crash .


I cant remember where it is maybe someone can point out where it is

I submitted the server to support, and it turned out the kernel was a multi-processor kernel. I didn't realize it could have been running something like that. I'm assuming, and hoping, that is what was causing the problem.

My server came with the multi-processor kernel installed. That is extremely frustrating and very annoying that I purchased the server improperly setup.

If that doesn't fix the problem then read this:
http://forums.cpanel.net/showthread.php?t=30777

Michael

i had the same problems you are expeienceing on a new cpanel.rhe box ... after i built a new kernel, everything was fine...

i'm having the same problem as you do.

I just ordered the 2nd server on May 2nd with RHE + WHM/cPanel. So, everything is new. The server crashes at least once a day. It's locked up and I couldn't restart via any means. I couldn't login or ssh.

I opened a ticket and the tech change the new server for me and insert my old hard disk that I have been setup AFP and a few websites, but it didn't solve the problem.

Rava, could you tell me how to build a new kernel? Or point me to the right direction.

Thanks.

before you recompile your kernel you need to know little bit about kernel and your hardware, if you dont know please ask somebody to compile for you.

But this post should help you.. if you are planing to compile your kernel yourself

http://www.linuxnetworkcare.org/showthread...php?p=35#post35

hope that helps,

ok the problem has been fixed. the kernel was for multi-processor whereas my server is single processor. the problem came from ev1servers staff that put wrong kernel for the servers.

jay711, I was just going to suggest that. I manage two EV1 servers, and both came with the wrong kernel and were very unstable. Swapping in the correct kernel stabilized them. I can see this happening occassionally, but it floors me that it seems to happen with such regularity.

Well that being said when you get an ev1 server the kernels are usually out of date so you should be upgrading it anyway.

A server with P4 HT uses single or multi processor kernel ?

It depends if the chip supports hyperthreading and if you want it enabled. Normally I just use a single cpu kernel.

I don't have documentation for this, but I've heard from someone close to the RH linux kernel development tree that if HT is enabled in the bios, you should use an SMP kernel, even on a single-proc box.

Correct you can the problem is not all P4's support HT, only some. You have to look at the specific chip to figure out if it has hyperthreading. From all that I have seen and heard hyperthreading does not do that much on a single proc box. Either way you can probably get away with running a SMP kernel without it but occasionally you have stability issues. So in short it is easier to just run a single cpu kernel.