Help - Search - Members - Calendar
Full Version: EMERGENCY HELP: server is slow, and load is almost perfect..
The Planet Forums > System Administration > Web Hosting
ns1
Today server is slow all day, sometimes it doesn't opet pages at all.
I managed to go in WHM and checked server load and it's almost perfect: 0.24 (4 cpus), Memory Used: 31.3 %.

I tried to restart DNS and apache, and after that i rebooted entire server. Still no help.

Server configuration is:
Xeon 3210, 2Gb RAM - 100 websites (more or less) of which only few are highly active.
cPanel 11.24.4-S36281 - WHM 11.24.2 - X 3.9
REDHAT Enterprise 5.2 i686 standard on server2

Server was working perfectly for 7 months, that is since we bought it.
Tomy Durden
Are DNS lookups slow from the server? Are you using MySQL or a 3rd party analytics?

Is it a certain page or is it all pages?
Tomy Durden
I pulled up a site you mentioned in a previous post and yeah, definately something going on. Tracing to your IP, I'm showing a 39% packet loss, but the switch is showing 0%.

When you run iostat, is the iowait pretty high?
ns1
all pages, best to test it on:

www.fantasy-hr.com
http://eydis.fantasy-hr.com
www.optike.hr

DNS lookup from dnsreport.com:
QUOTE
DNS Lookup: fantasy-hr.com A record

Searching for fantasy-hr.com A record at k.root-servers.net [193.0.14.129]: Got referral to b.gtld-servers.net. (zone: com.) [took 69 ms]

Searching for fantasy-hr.com A record at b.gtld-servers.net. [192.33.14.30]: Got referral to ns3.server-ns1.com. (zone: fantasy-hr.com.) [took 312 ms]

Searching for fantasy-hr.com A record at ns3.server-ns1.com. [216.40.234.218]: Timed out. Trying again.

Searching for fantasy-hr.com A record at ns3.server-ns1.com. [216.40.234.218]: Timed out. Trying again.

Searching for fantasy-hr.com A record at ns3.server-ns1.com. [216.40.234.218]: Timed out. Trying again.

Searching for fantasy-hr.com A record at ns4.server-ns1.com. [216.40.234.219]: Timed out. Trying again.

Searching for fantasy-hr.com A record at ns3.server-ns1.com. [216.40.234.218]: Timed out. Trying again.

Searching for fantasy-hr.com A record at ns3.server-ns1.com. [216.40.234.218]: Timed out. Trying again.



There is no need to refresh the page -- to see the DNS traversal, to make sure that all DNS servers are reporting
the same results, you can Click Here.

Note that these results are obtained in real-time, meaning that these are not cached results.
These results are what DNS resolvers all over the world will see right now (unless they have cached information).


I am not sure what do you mean under "analytics" but server is MySql for database.
ns1
I am having trouble accessing it via SSH, since it's slow...
ns1
hmm, I don't like it... it seems that now everything is ok...
ns1
CODE
root@server2 [~]# iostat
Linux 2.6.18-92.1.22.el5 (server2.server-ns1.com)       07/17/2009

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.92    0.19    0.50    3.64    0.00   94.76

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              52.27       960.93       312.83    1864804     607093
ns1
but that is now, when all is ok...
Tomy Durden
QUOTE (ns1 @ Jul 17 2009, 01:49 PM) *
hmm, I don't like it... it seems that now everything is ok...

Noticed that, I didn't do anything yet. I'll get with the NOC to see if something was going on.
ns1
since i am from Europe (Croatia), as is 99% of my clients, could it be something in network between us and US?
ns1
just a question:
if iowait is high (how much is high?), what does it mean?
Tomy Durden
QUOTE (ns1 @ Jul 17 2009, 01:54 PM) *
since i am from Europe (Croatia), as is 99% of my clients, could it be something in network between us and US?

Well, looking at the other customers on the same switch as you, it doesn't seem that anyone else was affected. The NOC just informed me that there weren't any DDoS's that would've affected you.

I'm in Dallas and I saw the latency as well which, along with no other indications from nearby customers, makes me believe that it was something specific to the server.

There may be some evidence in /var/log/messages as to what caused it. It could be something like someone flooding your server's open ports, slow resolving NS slowing down MySQL, or even a hard drive going bad.

I'd suggest having one of our techs take a look for signs of hard drive failure first and then looking for other problems.
ns1
ok, i'll do that. thank you.
Tomy Durden
QUOTE (ns1 @ Jul 17 2009, 02:02 PM) *
just a question:
if iowait is high (how much is high?), what does it mean?

That's a tricky question. While IOWAIT% doesn't directly apply to hard drives, I find that it's somewhat indicative of a drive failure. IOWAIT% is how much of the time the CPU is waiting on a I/O transaction to complete, but is otherwise idle.

I get paranoid if I see a sustained iowait of over 20-50% but, I've seen cases where that's perfectly fine.

Little more useful information which better explains it than I do:
http://ps-2.kev009.com:8081/AIXtip/iowait.htm
ns1
i have checked "messages" and found this:

CODE
Jul 17 19:46:17 server2 pure-ftpd: (?@127.0.0.1) [INFO] New connection from 127.0.0.1
Jul 17 19:46:28 server2 pure-ftpd: (?@127.0.0.1) [INFO] __cpanel__service__auth__ftpd__fcaIWW3iXlZxY5mEMfFmNsTQRgAqGcb0W_C7VOhaghPdByndh
5KxD8jWanzAEh4a is now logged in
Jul 17 19:46:28 server2 pure-ftpd: (__cpanel__service__auth__ftpd__fcaIWW3iXlZxY5mEMfFmNsTQRgAqGcb0W_C7VOhaghPdB
yndh5KxD8jWanzAEh4a@127.0.0.1) [INFO] Logout.
Jul 17 19:49:48 server2 named[21601]: unexpected RCODE (SERVFAIL) resolving '201.22.49.70.dynamic.adsl.gvt.net.br/A/IN': 200.175.89.133#53
Jul 17 19:51:18 server2 pure-ftpd: (?@127.0.0.1) [INFO] New connection from 127.0.0.1
Jul 17 19:51:29 server2 pure-ftpd: (?@127.0.0.1) [INFO] __cpanel__service__auth__ftpd__s1jS8i61Rw4_e1dH4JES5ntNZxq6iPiaHhAgGtyp_kX53sJ1M
UV2E1cf89iSmyM5 is now logged in
Jul 17 19:51:30 server2 pure-ftpd: (__cpanel__service__auth__ftpd__s1jS8i61Rw4_e1dH4JES5ntNZxq6iPiaHhAgGtyp_kX53
sJ1MUV2E1cf89iSmyM5@127.0.0.1) [INFO] Logout.
Jul 17 19:52:56 server2 named[21601]: lame server resolving 'mail.rchawaii.com' (in 'rchawaii.com'?): 168.215.210.50#53
Jul 17 19:53:42 server2 named[21601]: shutting down: flushing changes
Jul 17 19:53:42 server2 named[21601]: stopping command channel on 127.0.0.1#953
Jul 17 19:53:42 server2 named[21601]: no longer listening on 127.0.0.1#53
Jul 17 19:53:42 server2 named[21601]: no longer listening on 216.40.234.218#53
Jul 17 19:53:42 server2 named[21601]: no longer listening on 216.40.234.219#53
Jul 17 19:53:42 server2 named[21601]: exiting
Jul 17 19:53:44 server2 named[20840]: starting BIND 9.3.4-P1 -u named
Jul 17 19:53:44 server2 named[20840]: found 4 CPUs, using 4 worker threads
Jul 17 19:53:44 server2 named[20840]: loading configuration from '/etc/named.conf'
Jul 17 19:53:44 server2 named[20840]: listening on IPv4 interface lo, 127.0.0.1#53
Jul 17 19:53:44 server2 named[20840]: listening on IPv4 interface eth0, 216.40.234.218#53
Jul 17 19:53:44 server2 named[20840]: listening on IPv4 interface eth0:1, 216.40.234.219#53
Jul 17 19:53:44 server2 named[20840]: command channel listening on 127.0.0.1#953
Jul 17 19:53:44 server2 named[20840]: zone 0.in-addr.arpa/IN/localhost_resolver: loaded serial 42
Jul 17 19:53:44 server2 named[20840]: zone 0.0.127.in-addr.arpa/IN/localhost_resolver: loaded serial 1997022700
Jul 17 19:53:44 server2 named[20840]: zone 255.in-addr.arpa/IN/localhost_resolver: loaded serial 42
Jul 17 19:53:44 server2 named[20840]: zone 0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa/IN/localhost_resolver: loaded serial 1997022700
Jul 17 19:53:44 server2 named[20840]: zone localdomain/IN/localhost_resolver: loaded serial 42
Jul 17 19:53:44 server2 named[20840]: zone localhost/IN/localhost_resolver: loaded serial 42
Jul 17 19:53:44 server2 named[20840]: zone croatia-escort.biz/IN/internal: loaded serial 2009011201
Jul 17 19:53:44 server2 named[20840]: zone apartman-rak.com/IN/internal: loaded serial 2009011201
Jul 17 19:53:44 server2 named[20840]: zone apartmani-grbavac.com/IN/internal: loaded serial 2009011201
Jul 17 19:53:44 server2 named[20840]: zone apartmanimarkov.com/IN/internal: loaded serial 2009010901
Jul 17 19:53:44 server2 named[20840]: zone art-um.com/IN/internal: loaded serial 2009011201
Jul 17 19:53:44 server2 named[20840]: zone assloot.com/IN/internal: loaded serial 2009011201
Jul 17 19:53:44 server2 named[20840]: zone bamboocha-bar.com/IN/internal: loaded serial 2009011201
Jul 17 19:53:44 server2 named[20840]: zone best-split.com/IN/internal: loaded serial 2009062301
Jul 17 19:53:44 server2 named[20840]: zone city-ryu.com/IN/internal: loaded serial 2009011203
Jul 17 19:53:44 server2 named[20840]: zone clear-pores.com/IN/internal: loaded serial 2009011202
Jul 17 19:53:44 server2 named[20840]: zone cro-agent.com/IN/internal: loaded serial 2009011201
Jul 17 19:53:44 server2 named[20840]: zone croatiaphotomodels.com/IN/internal: loaded serial 2009011201
Jul 17 19:53:44 server2 named[20840]: zone dangerousgroove.com/IN/internal: loaded serial 2009011201
Jul 17 19:53:44 server2 named[20840]: zone dirtypunkgirls.com/IN/internal: loaded serial 2009011201
Jul 17 19:53:44 server2 named[20840]: zone duhitijelo.com/IN/internal: loaded serial 2009011201
Jul 17 19:53:44 server2 named[20840]: zone erbe-design.com/IN/internal: loaded serial 2009042002
Jul 17 19:53:44 server2 named[20840]: zone horvat.eu.com/IN/internal: loaded serial 2009010901
Jul 17 19:53:44 server2 named[20840]: zone fantasy-hr.com/IN/internal: loaded serial 2008122102
Jul 17 19:53:44 server2 named[20840]: zone goblins-store.com/IN/internal: loaded serial 2009011201
Jul 17 19:53:44 server2 named[20840]: zone gotovodice.com/IN/internal: loaded serial 2009011201
Jul 17 19:53:44 server2 named[20840]: zone grabusic.com/IN/internal: loaded serial 2009011201
Jul 17 19:53:44 server2 named[20840]: zone gustcroatia.com/IN/internal: loaded serial 2009060802
Jul 17 19:53:44 server2 named[20840]: zone gutschy.com/IN/internal: loaded serial 2009042001
Jul 17 19:53:44 server2 named[20840]: zone her-solution.com/IN/internal: loaded serial 2009010902
Jul 17 19:53:44 server2 named[20840]: zone imatepriliku.com/IN/internal: loaded serial 2009011902
Jul 17 19:53:44 server2 named[20840]: zone irialeona.com/IN/internal: loaded serial 2009011202
Jul 17 19:53:44 server2 named[20840]: zone jasnica.com/IN/internal: loaded serial 2009011102
Jul 17 19:53:44 server2 named[20840]: zone ladyboyinfo.com/IN/internal: loaded serial 2009011201
Jul 17 19:53:44 server2 named[20840]: zone male-supplement.com/IN/internal: loaded serial 2009011202
Jul 17 19:53:44 server2 named[20840]: zone mceran-art.com/IN/internal: loaded serial 2008122202
Jul 17 19:53:44 server2 named[20840]: zone milfbase.com/IN/internal: loaded serial 2009011201
Jul 17 19:53:44 server2 named[20840]: zone montazaklima.com/IN/internal: loaded serial 2009060301
Jul 17 19:53:44 server2 named[20840]: zone narnia-nekretnine.com/IN/internal: loaded serial 2009011201


this with "zone" repeats throughout the day as far as i can see...
Tomy Durden
QUOTE (ns1 @ Jul 17 2009, 02:45 PM) *
i have checked "messages" and found this:

CODE
Jul 17 19:46:17 server2 pure-ftpd: (?@127.0.0.1) [INFO] New connection from 127.0.0.1
Jul 17 19:46:28 server2 pure-ftpd: (?@127.0.0.1) [INFO] __cpanel__service__auth__ftpd__fcaIWW3iXlZxY5mEMfFmNsTQRgAqGcb0W_C7VOhaghPdByndh
5KxD8jWanzAEh4a is now logged in
Jul 17 19:46:28 server2 pure-ftpd: (__cpanel__service__auth__ftpd__fcaIWW3iXlZxY5mEMfFmNsTQRgAqGcb0W_C7VOhaghPdB
yndh5KxD8jWanzAEh4a@127.0.0.1) [INFO] Logout.
Jul 17 19:49:48 server2 named[21601]: unexpected RCODE (SERVFAIL) resolving '201.22.49.70.dynamic.adsl.gvt.net.br/A/IN': 200.175.89.133#53
Jul 17 19:51:18 server2 pure-ftpd: (?@127.0.0.1) [INFO] New connection from 127.0.0.1
Jul 17 19:51:29 server2 pure-ftpd: (?@127.0.0.1) [INFO] __cpanel__service__auth__ftpd__s1jS8i61Rw4_e1dH4JES5ntNZxq6iPiaHhAgGtyp_kX53sJ1M
UV2E1cf89iSmyM5 is now logged in
Jul 17 19:51:30 server2 pure-ftpd: (__cpanel__service__auth__ftpd__s1jS8i61Rw4_e1dH4JES5ntNZxq6iPiaHhAgGtyp_kX53
sJ1MUV2E1cf89iSmyM5@127.0.0.1) [INFO] Logout.
Jul 17 19:52:56 server2 named[21601]: lame server resolving 'mail.rchawaii.com' (in 'rchawaii.com'?): 168.215.210.50#53
Jul 17 19:53:42 server2 named[21601]: shutting down: flushing changes
Jul 17 19:53:42 server2 named[21601]: stopping command channel on 127.0.0.1#953
Jul 17 19:53:42 server2 named[21601]: no longer listening on 127.0.0.1#53
Jul 17 19:53:42 server2 named[21601]: no longer listening on 216.40.234.218#53
Jul 17 19:53:42 server2 named[21601]: no longer listening on 216.40.234.219#53
Jul 17 19:53:42 server2 named[21601]: exiting
Jul 17 19:53:44 server2 named[20840]: starting BIND 9.3.4-P1 -u named
Jul 17 19:53:44 server2 named[20840]: found 4 CPUs, using 4 worker threads
Jul 17 19:53:44 server2 named[20840]: loading configuration from '/etc/named.conf'
Jul 17 19:53:44 server2 named[20840]: listening on IPv4 interface lo, 127.0.0.1#53
Jul 17 19:53:44 server2 named[20840]: listening on IPv4 interface eth0, 216.40.234.218#53
Jul 17 19:53:44 server2 named[20840]: listening on IPv4 interface eth0:1, 216.40.234.219#53
Jul 17 19:53:44 server2 named[20840]: command channel listening on 127.0.0.1#953
Jul 17 19:53:44 server2 named[20840]: zone 0.in-addr.arpa/IN/localhost_resolver: loaded serial 42
Jul 17 19:53:44 server2 named[20840]: zone 0.0.127.in-addr.arpa/IN/localhost_resolver: loaded serial 1997022700
Jul 17 19:53:44 server2 named[20840]: zone 255.in-addr.arpa/IN/localhost_resolver: loaded serial 42
Jul 17 19:53:44 server2 named[20840]: zone 0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa/IN/localhost_resolver: loaded serial 1997022700
Jul 17 19:53:44 server2 named[20840]: zone localdomain/IN/localhost_resolver: loaded serial 42
Jul 17 19:53:44 server2 named[20840]: zone localhost/IN/localhost_resolver: loaded serial 42
Jul 17 19:53:44 server2 named[20840]: zone croatia-escort.biz/IN/internal: loaded serial 2009011201
Jul 17 19:53:44 server2 named[20840]: zone apartman-rak.com/IN/internal: loaded serial 2009011201
Jul 17 19:53:44 server2 named[20840]: zone apartmani-grbavac.com/IN/internal: loaded serial 2009011201
Jul 17 19:53:44 server2 named[20840]: zone apartmanimarkov.com/IN/internal: loaded serial 2009010901
Jul 17 19:53:44 server2 named[20840]: zone art-um.com/IN/internal: loaded serial 2009011201
Jul 17 19:53:44 server2 named[20840]: zone assloot.com/IN/internal: loaded serial 2009011201
Jul 17 19:53:44 server2 named[20840]: zone bamboocha-bar.com/IN/internal: loaded serial 2009011201
Jul 17 19:53:44 server2 named[20840]: zone best-split.com/IN/internal: loaded serial 2009062301
Jul 17 19:53:44 server2 named[20840]: zone city-ryu.com/IN/internal: loaded serial 2009011203
Jul 17 19:53:44 server2 named[20840]: zone clear-pores.com/IN/internal: loaded serial 2009011202
Jul 17 19:53:44 server2 named[20840]: zone cro-agent.com/IN/internal: loaded serial 2009011201
Jul 17 19:53:44 server2 named[20840]: zone croatiaphotomodels.com/IN/internal: loaded serial 2009011201
Jul 17 19:53:44 server2 named[20840]: zone dangerousgroove.com/IN/internal: loaded serial 2009011201
Jul 17 19:53:44 server2 named[20840]: zone dirtypunkgirls.com/IN/internal: loaded serial 2009011201
Jul 17 19:53:44 server2 named[20840]: zone duhitijelo.com/IN/internal: loaded serial 2009011201
Jul 17 19:53:44 server2 named[20840]: zone erbe-design.com/IN/internal: loaded serial 2009042002
Jul 17 19:53:44 server2 named[20840]: zone horvat.eu.com/IN/internal: loaded serial 2009010901
Jul 17 19:53:44 server2 named[20840]: zone fantasy-hr.com/IN/internal: loaded serial 2008122102
Jul 17 19:53:44 server2 named[20840]: zone goblins-store.com/IN/internal: loaded serial 2009011201
Jul 17 19:53:44 server2 named[20840]: zone gotovodice.com/IN/internal: loaded serial 2009011201
Jul 17 19:53:44 server2 named[20840]: zone grabusic.com/IN/internal: loaded serial 2009011201
Jul 17 19:53:44 server2 named[20840]: zone gustcroatia.com/IN/internal: loaded serial 2009060802
Jul 17 19:53:44 server2 named[20840]: zone gutschy.com/IN/internal: loaded serial 2009042001
Jul 17 19:53:44 server2 named[20840]: zone her-solution.com/IN/internal: loaded serial 2009010902
Jul 17 19:53:44 server2 named[20840]: zone imatepriliku.com/IN/internal: loaded serial 2009011902
Jul 17 19:53:44 server2 named[20840]: zone irialeona.com/IN/internal: loaded serial 2009011202
Jul 17 19:53:44 server2 named[20840]: zone jasnica.com/IN/internal: loaded serial 2009011102
Jul 17 19:53:44 server2 named[20840]: zone ladyboyinfo.com/IN/internal: loaded serial 2009011201
Jul 17 19:53:44 server2 named[20840]: zone male-supplement.com/IN/internal: loaded serial 2009011202
Jul 17 19:53:44 server2 named[20840]: zone mceran-art.com/IN/internal: loaded serial 2008122202
Jul 17 19:53:44 server2 named[20840]: zone milfbase.com/IN/internal: loaded serial 2009011201
Jul 17 19:53:44 server2 named[20840]: zone montazaklima.com/IN/internal: loaded serial 2009060301
Jul 17 19:53:44 server2 named[20840]: zone narnia-nekretnine.com/IN/internal: loaded serial 2009011201


this with "zone" repeats throughout the day as far as i can see...



Is the time between the named restarts fairly consistent or does it seem random?

Also, what's in /etc/resolv.conf?

If your server is providing the recursive name server for itself, then every time named reloads, MySQL will get real slow due to reverse lookups timing out. Unless you specifically need MySQL to do reverse lookups, you can disable it by using the "--skip-name-resolve" switch when it's started. The alternative would be to use our recursive name server farm.

Take a look at http://dev.mysql.com/doc/refman/5.0/en/dns.html for more info on that.
joec@home
REDHAT Enterprise 5.2

That plus WHM cPanel caught my eye, it should be running 5.3. There is going to be a problem with the update settings some where. Several things to check:

- Does yum work?

yum list

If this fails send in a ticket to have it re-registered.

- Are there WHM update setting errors?

cat /etc/cpupdate.conf |wc -l

This should return 11, otherwise check the WHM update settings by clicking on the version number in the upper right hand side. Make certain everything has a check by it. If anything was missing then do a forced update.

If it does not change to 5.3, send in a ticket!
This is a "lo-fi" version of our main content. To view the full version with more information, formatting and images, please click here.
Invision Power Board © 2001-2009 Invision Power Services, Inc.