Originally posted by Nethead
In the case of my issue there are no errors recorded.   All logs just abruptly stop.    


I have a script running via cron to record server state every 5 minutes.   Today it took this snapshot no more than 1 minute before a crash.   Examining it, all looked normal with no high loads, no excessive open connections, and no unusual or high number of processes active.


I'm stumped beyond stumped.   I'm running a CPU burn in test now to see if that kills it.   So far its hanging in there.

When EV1 replaced my server they replaced with like hardware and so if it was a hardware conflict then like hardware would cause the same thing to happen which it did.

Originally posted by beams
BTW Nethead - what script are you using to 'snapshot' the server status?

Originally posted by Doobla
While I am sorry for your fortune, I am glad to see that somebody else is having the same problem as I am and that it's not an Ensim thing.

I am having the same problem except that my crashes seem to be more regular than yours happening usually once ever 6-10 days.  I convinced EV1 to replace my server in hopes that the problem was actually a hardware issue, but it was not as within a week I was requesting a reboot again.  Since my clients are frustrated to no end I chose a band-aid approach for now and set cron to reboot the system automatically for me Sunday and Wednesday morning at 2am.

I have tried upgrading kernels, compiling the kernel from source with a minimal driver set (although I am not an expert int hat area by any means) and a host of other software "solutions" to no avail.  I believe that it comes down to a combination of hardware in the server and how the hardware relates to each other, not that I think any hardware is bad per se.  This is speculation at this point but I am frustrated beyond belief with this and don't know where to turn.

Here's hoping that somebody will have a suggestion or two....

Jon

Originally posted by Konrad Frye
Someone suggested it might be a swap issue but I haven't seen any evidence of that.

However as your services are having problem, hence i suggest you to upgrade or reinstall all the services for which you have to upgrade cpanel to latest version. You can do the same from WHM under Cpanel menu and clicking on "Upgrade to Latest Version" link. Do the same and monitor the server for few hours. If the problem still persists, then i suggest you to order a restore in which OS and cpanel will be resinstalled.

The main reason for the server going down was because someone changed the httpd.conf file to set the maxclients variable to 260, this caused apache to spawn processes until it died and took the machine with it.

I think most of us affected have already done most of these things? Other affected please confirm.

In other words, there were no high loads at the time of the crash. In fact there were no more than maybe 2 dozen apache processes running at the time.

Originally posted by beams
This thread seems to be very helpful - it suggests a possible hardware<->redhat problem.

http://forum.ev1servers.net/showthread.php...&threadid=23255

Tom

10/13/03 10:32:12 PM
My server for some time has been locking up at random times requiring a reboot ticket to be put in. It started about a month after I upgraded to Ensim Pro which was several months ago and I believe there was a kernel upgrade that happened around that time. Since then I had my hard drive replaced as the techs found that there were bad sectors on the drive and they thought that might have been the cause of the problems, however it wasnt but a few days later and my system hung again forcing another reboot ticket. Long story short (the full story you would probably read in my trouble tickets) my box was replaced and still I have the problem. After one time of it happening, after the server was replaced, I put cron entries in to reboot the server at 2am on Sunday and Wednesday mornings as a band-aid to the situation as many of my clients were going to leave me. Im still worried about this as no particular load can be observed before the crashes and it obviously wasnt "bad" hardware but I dont have anything out of the ordinary installed on the server. It has been suggested that Redhat Linux sometimes has problems with certain *combinations* of hardware and so I was wondering if you could investigate this and/or advise me on my next steps. Obviously the band-aid approach I have taken will have to be temporary and I have tried everything I could find on the forums as a software solution to no avail.

The previous paragraph was sent in an email to Patrick who never responded to the email. Now over the past two days the server has locked twice for no apparent reason. I have not changed the configuration that was on the hard drive when they gave me a restore except to upgrade Ensim, etc. Nothing out of the ordinary was done and there were no hacks performed on the machine. Going on the idea that it could be a hardware conflict, is it possible to give me a new server with a different hardware config. I dont know if you use different network cards or motherboards or what but I will move my sites to whatever server youd be willing to give me that has a different configuration and then I would give up the old server obviously. Otherwise Ill entertain any suggestions you have on this matter, however I am not the only person that has this problem.

Please refer to these threads for reference to my problem as well as others who have expressed a similar issue.

http://forums.rackshack.net/showthread.php...&threadid=22377
http://forum.rackshack.net/showthread.php?...?threadid=23255  
http://forum.ev1servers.net/showthread.php...?threadid=33908  

Thank you very much in advance for your help.

Jon

10/14/03 9:37:15 AM
Dear Customer,

I was still unable to find any reason for your random crashes. I do have a theory though. I see that you were running apf as a firewall. Is it possible your firewall is misbevin, and it appears that the server is down when its just being filtered?

What I would like to do is to wait until the next outage, and then have DataCenter console into the machine to see if its just the firewall blocking access, and not the machine being down itself.

I ran a memtest & chkrootkit, which came back clean, and the hardware has been totally switched out, so the only thing I can think of is either the firewall misbehaving, or perhaps you just need to reimage the machine and start over.

Ill go ahead and close this ticket. If the machine goes down again, please open a new ticket requesting that DataCenter verify if the firewall is blocking access or is the machine down itself.

-Eve1 Servers Support

10/14/03 5:04:23 PM
I have considered the firewall and eliminated that as a problem. First of all APF wasnt being used when this problem first started. Secondly, there was a period of time recently when I had a kernel installed (trying to see if a kernel configuration change would fix the problem) where I could not get iptables to work so apf wouldnt even run and I still had a coupel of crashes during that time.

As for reimaging the machine, i dont see hwo that wil help since I havent changed the configuration except for security updates and the installation of a few programs that should affect anything (like mrtg).

Also, regarding your comment about the hardware, I do not believe that any hardware is bad but that Redhat is having a problem with the configuration of the hardware. How many different configurations of Celeron boxes do you have? Do you use different motherboards? There is an obvious hardware pattern to this problem as I have only heard of Celeron customers with this problem so my thoughts are (as well as some of theirs) that it is the mix of hardware that makes up the box that redhat doesnt like.

I would entertain any other suggestions that you might have and I would really really appreciate it if you would look at the threads that I mentioned in my original ticket as these people are also having the same problem.  

Thanks in advance for working to resolve this for me. My customers are not happy at all and this seems way out of my control.

10/14/03 5:26:53 PM
Dear Customer,

The problem that seems to appear on your system is an isolated one and as not been seen on many of the other systems. I have gone through the threads and although the issue seems similar (wrt crashes) but the logs do not exactly correlate.

The hardware has been checked and verified, so it cannot be a hardware issue. Also, chkrootkit has been run and system integrity verified.

At this point we have the only option to wait for another outage as suggested earlier and then look at the system to verify whether it is the firewall causing it or some other reason. It is only after this that e can ascertain that the reimage is essential

Please reopen this ticket or put in a new ticket as soon as anotehr outage occurs and I assure you we shall look into it.

Thank You,

Ev1servers Support Team

10/14/03 5:26:53 PM
Dear Customer,

The problem that seems to appear on your system is an isolated one and as not been seen on many of the other systems. I have gone through the threads and although the issue seems similar (wrt crashes) but the logs do not exactly correlate.

Originally posted by beams
Im not so sure your problem is isolated, as I seem to be having the same problem (thankfully not as often).

The problem we face here is that there is no coordinated approach to the problem. On our end, we see quite a few people with similar problems, but we are not working as a team.

On the RS end, they probably have numerous techs doing different tasks and no one person or team is seeing all these problems. So knowledge is being lost or never accumulated.

We really need some kind of 'register' where people with this or similar problems log all the necessary details.

Originally posted by amps
there WAS a post somewhere around here where someone was able to compile a plain kernel with some custom options and fix the issue with the rackshack celery VIA's

I wish you all the best of luck....

Originally posted by Netino
I´m researching this problem since one year ago, and I firmly believe APF have nothing to to with this issue. I do, would be easy to discover: search your log, and see if apf is blocking anyone during the freezes. You will see has nothing logged!

I´m having new directions on research, and the problems exists in all hardware/software and OS combinations. The last discovery is PHP/Apache already caused a similar problem in buffering files in older RedHat 6, I didn´t try to test but apparently the problem was solved. See in <http://www.phpbuilder.com/mail/php-develop...200001/0544.php>.

Regards,
Netino

Originally posted by Doobla
So installing a kernel from source didn't fix your problem netino?  I read that solution in another thread similar to this one.

Originally posted by Doobla
Hey amps,

You fixed yoru probelm with a kernel upgrade right?  And you used a redhat rpm to do that?

What is an smp kernel (vs. some other kernel rpm)????  In other words, what does the smp stand for?

thanks,

Jon

Originally posted by amps
If these guys are in the same hardware issue boat as I was, there will be NO abnormal activity or spike in usage leading up to the crash.  It just suddenly dies out of nowhere... you could be editing a file with Pico with nothing else going on, you could be pushing 100 megs of traffic... there really is no rhyme or reason other than TIME and average load over that time.

Originally posted by Doobla
Although my situation IS a little different than amps in that I only have one drive and no Raid setup going on, I will say that his description is accurate for my situation.  There really is nothing going on and then bam, server goes down.

Originally posted by Nethead
Could some one who is also experiencing these odd crashes post a typical output of the 'vmstat' command on your box?   (...)

Here is my problem box:

CODE

  procs                      memory    swap          io     system         cpu

r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us  sy  id

0  0  0 166988  21888 120952 603956   1  13   107   169  216   149  12   5  84

Thanks!

Originally posted by Netino
Do not use "vmstat" command without the parameter '1'.