The Planet Forums > RH 9.0 and Xeons?

gertiebeth

Dec 28 2003, 02:37 PM

My Xeon (w/ 2 gigs RAM) is almost a year old and, up until about a week ago, was running Red Hat 7.3. Performance was always great. No problems at all. Now that it has been upgraded to 9.0, the load average has been going through the roof! Connections are slow and Apache restarts about every 2 hours or so. Is this a common issue with 9.0 and Xeons? Any help on getting this load average down? I use cPanel and have upgraded the kernel and tried recompiling Apache. No luck.

Thank you for any help!

Erwin

Dec 28 2003, 04:40 PM

QUOTE

Originally posted by gertiebeth
My Xeon (w/ 2 gigs RAM) is almost a year old and, up until about a week ago, was running Red Hat 7.3. Performance was always great. No problems at all. Now that it has been upgraded to 9.0, the load average has been going through the roof! Connections are slow and Apache restarts about every 2 hours or so. Is this a common issue with 9.0 and Xeons? Any help on getting this load average down? I use cPanel and have upgraded the kernel and tried recompiling Apache. No luck.

Thank you for any help!

I have a Dual Xeon with RH9 - been running it for the past 4 months or so. No problems, loads are actually slightly lower.

gertiebeth

Dec 28 2003, 04:44 PM

Thanks Erwin. I'll keep digging a little deeper...

Erwin

Dec 28 2003, 05:52 PM

Optimize your Apache settings. When recompiling Apache and PHP, don't include modules you don't use.

gertiebeth

Dec 29 2003, 12:34 AM

I just can't figure this one out! I put in a TT and RS doesn't have any new suggestions. I have recompiled Apache using only the bare minimum of services, I am using the latest kernel and all other scripts are up-to-date. The only thing different with this server is that I am now using RH 9 instead of 7.3. One thing RS did say is that I should try suEXEC off to reduce the load, but with all this hacking going on, I want to know who owns what. Here is a sample of TOP:

CODE

00:31:15 up 8:02, 1 user, load average: 5.98, 8.46, 7.09

176 processes: 162 sleeping, 7 running, 5 zombie, 2 stopped

CPU states: 79.6% user 15.1% system 0.0% nice 0.0% iowait 5.1% idle

Mem: 2065036k av, 2007496k used, 57540k free, 0k shrd, 92760k buff

1460696k actv, 73332k in_d, 47900k in_c

Swap: 2048276k av, 55420k used, 1992856k free 1710020k cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND

2316 mysql 15 0 24740 9472 1416 S 99.9 0.4 0:16 0 mysqld

24943 root 24 0 1488 1476 228 R 99.9 0.0 1:19 0 httpd

1841 jan 25 0 10584 10M 1776 S 8.3 0.5 0:00 0 php

7 root 15 0 0 0 0 SW 1.1 0.0 1:13 0 kscand/HighMem

1920 al 25 0 2560 2560 1524 R 0.5 0.1 0:00 0 php

6 root 15 0 0 0 0 SW 0.3 0.0 0:20 0 kscand/Normal

1908 bil 21 0 1996 1996 1500 S 0.3 0.0 0:00 0 php

1917 al 21 0 2224 2224 1376 R 0.3 0.1 0:00 0 php

30475 jen 15 0 1088 1088 592 R 0.1 0.0 0:00 0 top

1889 nobody 23 0 0 0 0 Z 0.1 0.0 0:00 0 httpd <defunct>

1904 nobody 24 0 0 0 0 Z 0.1 0.0 0:00 0 httpd <defunct>

1 root 21 0 112 80 56 S 0.0 0.0 0:08 0 init

2 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 keventd

3 root 34 19 0 0 0 SWN 0.0 0.0 0:00 0 ksoftirqd_CPU0

8 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 bdflush

4 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kswapd

5 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kscand/DMA

9 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kupdated

10 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 mdrecoveryd

19 root 15 0 0 0 0 SW 0.0 0.0 0:05 0 kjournald

77 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 khubd

668 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kjournald

1521 root 15 0 192 164 112 S 0.0 0.0 0:00 0 syslogd

1525 root 15 0 176 164 120 S 0.0 0.0 0:00 0 klogd

2122 root 15 0 304 176 80 S 0.0 0.0 0:00 0 sshd

2136 root 15 0 428 356 280 S 0.0 0.0 0:00 0 xinetd

2145 root 15 0 1464 984 524 S 0.0 0.0 0:00 0 antirelayd

2168 mailnull 15 0 504 260 180 S 0.0 0.0 0:00 0 exim

2172 mailnull 25 0 308 4 0 S 0.0 0.0 0:00 0 exim

2178 root 15 0 772 664 484 S 0.0 0.0 0:00 0 antirelayd

Any other ideas? I'm almost at the point of pulling half the sites and putting them on a different box, which is a huge waste of money since this server is only using 20% of its space and 50% of its bandwidth.

Thanks in advance.

Erwin

Dec 29 2003, 02:32 AM

You have a busy site which is database intensive? Both httpd and mysqld are up to 100%...

vma

Dec 29 2003, 03:24 AM

QUOTE

Originally posted by gertiebeth
CODE
00:31:15 up 8:02, 1 user, load average: 5.98, 8.46, 7.09

176 processes: 162 sleeping, 7 running, 5 zombie, 2 stopped

CPU states: 79.6% user 15.1% system 0.0% nice 0.0% iowait 5.1% idle

Mem: 2065036k av, 2007496k used, 57540k free, 0k shrd, 92760k buff

1460696k actv, 73332k in_d, 47900k in_c

Swap: 2048276k av, 55420k used, 1992856k free 1710020k cached

What is your kernel
in the moment I think you don´t use smp-kernel.
rs-server dual-xeon with rh9 don´t have activated smp-kernel by default
edit your bootloader-config and activate smp-kernel.

gertiebeth

Dec 29 2003, 09:23 AM

QUOTE

Originally posted by Erwin
You have a busy site which is database intensive? Both httpd and mysqld are up to 100%...

Yes, I have a very busy site running phpNUKE, but it hasn't gotten any busier than it was when I was using RH 7.3... I specifically chose a Xeon to handle this one particular site, and for almost a year things ran VERY smoothly.

:confused:

gertiebeth

Dec 29 2003, 09:34 AM

QUOTE

Originally posted by vma
What is your kernel
in the moment I think you don´t use smp-kernel.
rs-server dual-xeon with rh9 don´t have activated smp-kernel by default
edit your bootloader-config and activate smp-kernel.

My kernel:
root@cpanel [~]# uname -r
2.4.20-27.9

My lilo.conf:
prompt
timeout=50
default=2.4.20-27.9
boot=/dev/sda
map=/boot/map
install=/boot/boot.b
message=/boot/message
linear

image=/boot/vmlinuz-2.4.20-27.9
label=2.4.20-27.9
initrd=/boot/initrd-2.4.20-27.9.img
read-only
append="root=/dev/sda3"

image=/boot/vmlinuz-2.4.20-24.9
label=2.4.20-24.9
initrd=/boot/initrd-2.4.20-24.9.img
read-only
append="root=LABEL=/"

image=/boot/vmlinuz-2.4.20-24.9smp
label=2.4.20-24.9smp
initrd=/boot/initrd-2.4.20-24.9smp.img
read-only
append="root=LABEL=/"

I don't see the smp-kernel, so I ran up2date and got:
root@cpanel [~]# up2date -u

Fetching package list for channel: redhat-linux-i386-9...
########################################

Fetching Obsoletes list for channel: redhat-linux-i386-9...
########################################

Fetching rpm headers...
########################################

Testing package set / solving RPM inter-dependencies...
########################################
kernel-smp-2.4.20-27.9.i686 ########################## Done.
kernel-source-2.4.20-27.9.i ########################## Done.

/var/spool/up2date reads:

CODE

root@cpanel [/var/spool/up2date]# ls -al

total 69372

drwxr-xr-x 2 root root 4096 Dec 24 22:52 ./

drwxr-xr-x 16 root root 4096 Dec 13 00:47 ../

-rw-r--r-- 1 root root 38659 Dec 24 22:50 httpd-manual-2.0.40-21.9.i386.hdr

-rw-r--r-- 1 root root 835006 Dec 24 22:50 httpd-manual-2.0.40-21.9.i386.rpm

-rw-r--r-- 1 root root 175015 Dec 24 22:50 kernel-2.4.20-27.9.i686.hdr

-rw-r--r-- 1 root root 14004411 Dec 24 22:51 kernel-2.4.20-27.9.i686.rpm

-rw-r--r-- 1 root root 192855 Dec 24 22:50 kernel-smp-2.4.20-27.9.i686.hdr

-rw-r--r-- 1 root root 14572631 Dec 24 22:51 kernel-smp-2.4.20-27.9.i686.rpm

-rw-r--r-- 1 root root 1492979 Dec 24 22:50 kernel-source-2.4.20-27.9.i386.hdr

-rw-r--r-- 1 root root 38943604 Dec 24 22:52 kernel-source-2.4.20-27.9.i386.rpm

-rw-r--r-- 1 root root 453074 Dec 24 22:50 redhat-linux-i386-9.20031223094901

-rw-r--r-- 1 root root 192048 Dec 24 22:50 redhat-linux-i386-9-obsoletes.20031223094901

Given this information, can anyone help me figure out how to activate smp-kernel?

EV1-Curtis

Dec 29 2003, 10:54 AM

At the moment it looks like lilo is configured to load 2.4.20-27.9 by default.

default=2.4.20-27.9

You will need to install the smp kernel for 2.4.20-27.9 and change you will probably need to change that line to read...

default 2.4.20-27.9smp.

It depends on what you end up labeling that kernel in the lilo.conf.

After that you just need to run lilo at the command line.

If there are no error when lilo is run and the * is next to 2.4.20-27.9smp then you should be good to go.

gertiebeth

Dec 29 2003, 11:01 AM

QUOTE

Originally posted by EV1-Curtis
At the moment it looks like lilo is configured to load 2.4.20-27.9 by default.

default=2.4.20-27.9

You will need to install the smp kernel for 2.4.20-27.9 and change you will probably need to change that line to read...

default 2.4.20-27.9smp.

It depends on what you end up labeling that kernel in the lilo.conf.

After that you just need to run lilo at the command line.

If there are no error when lilo is run and the * is next to 2.4.20-27.9smp then you should be good to go.

Thanks Curtis! I'm off to give it a try... *fingers crossed*

gertiebeth

Dec 29 2003, 11:25 AM

Kernel booted OK

2.4.20-27.9smp

and will monitor TOP for a bit. Hopefully load averages will go down...

swilliams

Dec 29 2003, 02:39 PM

QUOTE

Originally posted by gertiebeth

CODE
00:31:15 up 8:02, 1 user, load average: 5.98, 8.46, 7.09

176 processes: 162 sleeping, 7 running, 5 zombie, 2 stopped

CPU states: 79.6% user 15.1% system 0.0% nice 0.0% iowait 5.1% idle

Mem: 2065036k av, 2007496k used, 57540k free, 0k shrd, 92760k buff

1460696k actv, 73332k in_d, 47900k in_c

Swap: 2048276k av, 55420k used, 1992856k free 1710020k cached

If you have a dual xeon, shouldn't it be showing 4 processors in top?

gertiebeth

Dec 29 2003, 03:26 PM

QUOTE

Originally posted by swilliams
If you have a dual xeon, shouldn't it be showing 4 processors in top?

Hummm... I wonder why all 4 processors didn't show up in that shot?

Here is a sample with the new kernel, now running for 4 hours:

CODE

15:27:32 up 4:02, 1 user, load average: 2.53, 3.34, 3.13

137 processes: 134 sleeping, 1 running, 2 zombie, 0 stopped

CPU0 states: 30.4% user 6.1% system 0.0% nice 0.0% iowait 62.3% idle

CPU1 states: 22.0% user 8.1% system 0.0% nice 0.0% iowait 69.2% idle

CPU2 states: 33.2% user 9.4% system 0.0% nice 0.0% iowait 56.2% idle

CPU3 states: 34.0% user 10.2% system 0.0% nice 0.0% iowait 55.1% idle

Mem: 2064404k av, 2038556k used, 25848k free, 0k shrd, 88808k buff

1559812k actv, 116k in_d, 44892k in_c

Swap: 2048276k av, 46200k used, 2002076k free 1744920k cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND

2226 root 21 0 1396 348 248 S 99.9 0.0 2:04 1 httpd

2309 mysql 15 0 24008 14M 1252 S 99.9 0.7 0:21 3 mysqld

10441 eppy 21 0 4608 4608 1788 S 6.9 0.2 0:00 2 php

1 root 20 0 112 84 56 S 0.3 0.0 0:09 3 init

9011 jen 15 0 1304 1304 872 R 0.3 0.0 0:00 3 top

10450 nobody 23 0 0 0 0 Z 0.1 0.0 0:00 1 httpd <defunct>

2 root RT 0 0 0 0 SW 0.0 0.0 0:00 0 migration/0

3 root RT 0 0 0 0 SW 0.0 0.0 0:00 1 migration/1

4 root RT 0 0 0 0 SW 0.0 0.0 0:00 2 migration/2

5 root RT 0 0 0 0 SW 0.0 0.0 0:00 3 migration/3

6 root 15 0 0 0 0 SW 0.0 0.0 0:00 1 keventd

7 root 34 19 0 0 0 SWN 0.0 0.0 0:00 0 ksoftirqd_CPU0

8 root 34 19 0 0 0 SWN 0.0 0.0 0:00 1 ksoftirqd_CPU1

9 root 34 19 0 0 0 SWN 0.0 0.0 0:00 2 ksoftirqd_CPU2

10 root 34 19 0 0 0 SWN 0.0 0.0 0:00 3 ksoftirqd_CPU3

15 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 bdflush

11 root 15 0 0 0 0 SW 0.0 0.0 0:09 1 kswapd

12 root 15 0 0 0 0 SW 0.0 0.0 0:00 3 kscand/DMA

13 root 15 0 0 0 0 SW 0.0 0.0 0:26 1 kscand/Normal

14 root 16 0 0 0 0 SW 0.0 0.0 0:55 3 kscand/HighMem

16 root 15 0 0 0 0 SW 0.0 0.0 0:02 2 kupdated

17 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 mdrecoveryd

26 root 15 0 0 0 0 SW 0.0 0.0 0:08 0 kjournald

84 root 25 0 0 0 0 SW 0.0 0.0 0:00 3 khubd

675 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kjournald

1531 root 15 0 240 212 156 S 0.0 0.0 0:00 0 syslogd

1535 root 15 0 176 164 120 S 0.0 0.0 0:00 2 klogd

Still not great, but Apache hasn't shut itself down all afternoon and there were 600 people connected to my NUKE site when this shot was taken.

Thanks for the help guys! Just one more example of how this forum ROCKS!

char

Dec 29 2003, 05:07 PM

You should also add

append="noht"

to your /etc/lilo.conf under your new smp configuration to disable hyperthreading.

Linux 2.4 doesn't play well with it, and it can cause I/O bottlenecks (search it on this forum - there's a complete HOW-TO for it that someone nicely wrote up).

gertiebeth

Dec 29 2003, 06:13 PM

QUOTE

Originally posted by char
You should also add

append="noht"

to your /etc/lilo.conf under your new smp configuration to disable hyperthreading.

Linux 2.4 doesn't play well with it, and it can cause I/O bottlenecks (search it on this forum - there's a complete HOW-TO for it that someone nicely wrote up).

Found it! Thanks char! http://forum.ev1servers.net/showthread.php...+hyperthreading

rusdicyber

Jan 6 2004, 08:30 PM

Uhmm, what is the difference between normal Kernel and SMP Kernel? :o

alfatech

Jan 6 2004, 09:57 PM

SMP = symetric multi processing.... basically it uses only 1 processor instead of both... defeats the purpose of having a dual xeon

rusdicyber

Jan 6 2004, 11:26 PM

So, if my box have 1 CPU only.It was impossible to use SMP Kernel??

alfatech

Jan 7 2004, 07:16 AM

you can use smp kernel but it won't do you any good.

rusdicyber

Jan 7 2004, 07:15 PM

Thanks For the info

amusive.com

Jan 8 2004, 01:17 AM

QUOTE

Originally posted by alfatech
SMP = symetric multi processing.... basically it uses only 1 processor instead of both... defeats the purpose of having a dual xeon

How does enabling SMP use only one processor? SMP is the only linux kernel capable of distributing load across multiple processors. The "opposite" of SMP is AMP where processors are not evenly distributed; one would be dedicated to IO, video, or whatnot.

Here's a FAQ if you're confused

http://2cpu.com/FAQ/2cpusmpfaq.html

alfatech

Jan 8 2004, 07:24 AM

hah I just read my post, yeah that obviously didn't make sense.... what I was responding to was "whats the difference between normal and smp"

DigitalT

Jan 9 2004, 04:53 PM

on busy boxes hyperthreading can be a big bottleneck.
provide unwanted lag and stuff like that.
You should just see two cpu's both saying that it are Xeons double check:

CODE

cat /proc/cpuinfo

.

Should tell you that you have xeons

newuser

Jan 9 2004, 08:53 PM

jfyi - I tried 2.6 with HT and just like 2.4, HT lowers performance by quite a bit.

HT is now disabled again.