The Planet Forums > Help tweaking server (High Loads and memory usage)

1EightT

Aug 16 2005, 04:34 PM

Hi guys, i've been a racksha....errrr ev1 customer for 3 or 4 years now. I've got a dual xeon right now with 2GB of ram and a pair of SCSI drives. few days ago I started retweaking some settings and the server load has gone through the roof. It's fine for a few hours after the changes, then it starts swapping to disk and the load goes up to 20 or so. It used to hover around 1-2 at peak.

I've added eaccelerator for the php caching. Below are my stats output:

CODE

Tue Aug 16 16:30:23 MDT 2005

16:30:24 up 1 day, 1:03, 1 user, load average: 7.22, 7.19, 8.25

154 processes: 146 sleeping, 8 running, 0 zombie, 0 stopped

total 90.7% 0.0% 5.8% 0.0% 0.7% 0.0% 2.6%

cpu00 86.4% 0.0% 8.7% 0.0% 2.9% 0.0% 1.9%

cpu01 94.2% 0.0% 0.0% 0.0% 0.0% 0.0% 5.7%

cpu02 90.2% 0.0% 8.7% 0.0% 0.0% 0.0% 0.9%

cpu03 92.2% 0.0% 5.8% 0.0% 0.0% 0.0% 1.9%

Mem: 2061576k av, 1981444k used, 80132k free, 0k shrd, 30576k buff

1651540k actv, 219112k in_d, 5208k in_c

Swap: 2048276k av, 191748k used, 1856528k free 1523228k cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND

24594 nobody 18 0 11292 11M 4840 R 24.2 0.5 0:11 1 httpd

22662 nobody 20 0 1068 1068 776 R 0.4 0.0 0:00 2 top

Http processes currently running = 47

Mysql processes currently running = 25

Netstat information summary

1    0 TIME_WAIT

1    1 TIME_WAIT

1    17 TIME_WAIT

1    18 TIME_WAIT

1    21 TIME_WAIT

1    9 ESTABLISHED

1    93 ESTABLISHED

2 FIN_WAIT1

2    2 TIME_WAIT

2    3 TIME_WAIT

2    4 TIME_WAIT

2    6 TIME_WAIT

3 FIN_WAIT2

5 SYN_RECV

11 ESTABLISHED

29 LISTEN

2774 TIME_WAIT

+----------------------------+------------+

| Variable_name | Value |

+----------------------------+------------+

| Aborted_clients | 72 |

| Aborted_connects | 60 |

| Binlog_cache_disk_use | 0 |

| Binlog_cache_use | 0 |

| Bytes_received | 128695626 |

| Bytes_sent | 1511054245 |

| Com_admin_commands | 3 |

| Com_alter_db | 0 |

| Com_alter_table | 0 |

| Com_analyze | 0 |

| Com_backup_table | 0 |

| Com_begin | 0 |

| Com_change_db | 37565 |

| Com_change_master | 0 |

| Com_check | 0 |

| Com_checksum | 0 |

| Com_commit | 0 |

| Com_create_db | 0 |

| Com_create_function | 0 |

| Com_create_index | 0 |

| Com_create_table | 0 |

| Com_dealloc_sql | 0 |

| Com_delete | 4681 |

| Com_delete_multi | 0 |

| Com_do | 0 |

| Com_drop_db | 0 |

| Com_drop_function | 0 |

| Com_drop_index | 0 |

| Com_drop_table | 0 |

| Com_drop_user | 0 |

| Com_execute_sql | 0 |

| Com_flush | 0 |

| Com_grant | 0 |

| Com_ha_close | 0 |

| Com_ha_open | 0 |

| Com_ha_read | 0 |

| Com_help | 0 |

| Com_insert | 13132 |

| Com_insert_select | 13 |

| Com_kill | 0 |

| Com_load | 0 |

| Com_load_master_data | 0 |

| Com_load_master_table | 0 |

| Com_lock_tables | 114 |

| Com_optimize | 6 |

| Com_preload_keys | 0 |

| Com_prepare_sql | 0 |

| Com_purge | 0 |

| Com_purge_before_date | 0 |

| Com_rename_table | 0 |

| Com_repair | 0 |

| Com_replace | 3215 |

| Com_replace_select | 0 |

| Com_reset | 0 |

| Com_restore_table | 0 |

| Com_revoke | 0 |

| Com_revoke_all | 0 |

| Com_rollback | 0 |

| Com_savepoint | 0 |

| Com_select | 135521 |

| Com_set_option | 264 |

| Com_show_binlog_events | 0 |

| Com_show_binlogs | 3 |

| Com_show_charsets | 55 |

| Com_show_collations | 55 |

| Com_show_column_types | 0 |

| Com_show_create_db | 4 |

| Com_show_create_table | 4 |

| Com_show_databases | 26 |

| Com_show_errors | 0 |

| Com_show_fields | 25 |

| Com_show_grants | 3 |

| Com_show_innodb_status | 0 |

| Com_show_keys | 26 |

| Com_show_logs | 0 |

| Com_show_master_status | 0 |

| Com_show_new_master | 0 |

| Com_show_open_tables | 0 |

| Com_show_privileges | 0 |

| Com_show_processlist | 75 |

| Com_show_slave_hosts | 0 |

| Com_show_slave_status | 0 |

| Com_show_status | 2 |

| Com_show_storage_engines | 0 |

| Com_show_tables | 1046 |

| Com_show_variables | 115 |

| Com_show_warnings | 0 |

| Com_slave_start | 0 |

| Com_slave_stop | 0 |

| Com_truncate | 3 |

| Com_unlock_tables | 114 |

| Com_update | 40112 |

| Com_update_multi | 59 |

| Connections | 25313 |

| Created_tmp_disk_tables | 2469 |

| Created_tmp_files | 198 |

| Created_tmp_tables | 7984 |

| Delayed_errors | 0 |

| Delayed_insert_threads | 0 |

| Delayed_writes | 0 |

| Flush_commands | 1 |

| Handler_commit | 0 |

| Handler_delete | 3285 |

| Handler_discover | 0 |

| Handler_read_first | 18168 |

| Handler_read_key | 12741770 |

| Handler_read_next | 17584393 |

| Handler_read_prev | 53539 |

| Handler_read_rnd | 1020838 |

| Handler_read_rnd_next | 66502639 |

| Handler_rollback | 0 |

| Handler_update | 1322507 |

| Handler_write | 5504562 |

| Key_blocks_not_flushed | 0 |

| Key_blocks_unused | 1 |

| Key_blocks_used | 14497 |

| Key_read_requests | 33429664 |

| Key_reads | 85390 |

| Key_write_requests | 733999 |

| Key_writes | 34016 |

| Max_used_connections | 19 |

| Not_flushed_delayed_rows | 0 |

| Open_files | 1526 |

| Open_streams | 0 |

| Open_tables | 858 | 84% of table_cache in use

| Opened_tables | 969 |

| Qcache_free_blocks | 2167 |

| Qcache_free_memory | 6752168 |

| Qcache_hits | 213665 |

| Qcache_inserts | 131399 |

| Qcache_lowmem_prunes | 34230 |

| Qcache_not_cached | 4133 |

| Qcache_queries_in_cache | 5105 |

| Qcache_total_blocks | 14345 |

| Questions | 475131 |

| Rpl_status | NULL |

| Select_full_join | 26 |

| Select_full_range_join | 1 |

| Select_range | 16918 |

| Select_range_check | 2 |

| Select_scan | 27140 |

| Slave_open_temp_tables | 0 |

| Slave_running | OFF |

| Slave_retried_transactions | 0 |

| Slow_launch_threads | 0 |

| Slow_queries | 87 | (execution time > 10 secs)

| Sort_merge_passes | 99 |

| Sort_range | 17558 |

| Sort_rows | 10909857 |

| Sort_scan | 11701 |

| Table_locks_immediate | 305276 |

| Table_locks_waited | 135 |

| Threads_cached | 14 |

| Threads_connected | 5 |

| Threads_created | 19 |

| Threads_running | 2 |

| Uptime | 22415 | 6 hrs 13 mins 35 secs

+----------------------------+------------+

Key Reads/Key Read Requests = 0.002554 (Cache hit = 99.997446%)

Key Writes/Key Write Requests = 0.046343

Connections/second = 1.129 (/hour = 4065.438)

KB received/second = 5.607 (/hour = 20184.894)

KB sent/second = 65.833 (/hour = 236997.404)

Temporary Tables Created/second = 0.356 (/hour = 1282.284)

Opened Tables/second = 0.043 (/hour = 155.628)

Slow Queries/second = 0.004 (/hour = 13.973)

% of slow queries = 0.018%

Queries/second = 21.197 (/hour = 76309.239)

MySQL Query Cache hits = 213665/349197(61%)

1EightT

Aug 16 2005, 04:36 PM

Here is the relavent portion of my httpd.conf (i've tried keepalives on and off and didn't notice a load difference)

CODE

Timeout 60

KeepAlive Off

MaxKeepAliveRequests 1000

KeepAliveTimeout 15

MinSpareServers 32

MaxSpareServers 96

StartServers 32

MaxClients 256

MaxRequestsPerChild 0

Here is my my.cnf (i've tried about 10 variations in the last 4 days.

CODE

[mysqld]

skip-locking

skip-innodb

query_cache_limit=1M

query_cache_size=32M

query_cache_type=1

max_connections=500

interactive_timeout=100

wait_timeout=100

connect_timeout=10

thread_cache_size=128

key_buffer=16M

join_buffer=1M

max_allowed_packet=16M

table_cache=1024

record_buffer=1M

sort_buffer_size=2M

read_buffer_size=2M

max_connect_errors=10

thread_concurrency=4

myisam_sort_buffer_size=64M

log-bin

server-id=1

[mysql.server]

user=mysql

basedir=/var/lib

[safe_mysqld]

err-log=/var/log/mysqld.log

pid-file=/var/lib/mysql/mysql.pid <-- Not necessary

open_files_limit=8192

[mysqldump]

quick

max_allowed_packet=16M

[mysql]

no-auto-rehash

[isamchk]

key_buffer=64M

sort_buffer=64M

read_buffer=16M

write_buffer=16M

[myisamchk]

key_buffer=64M

sort_buffer=64M

read_buffer=16M

write_buffer=16M

[mysqlhotcopy]

interactive-timeout

Please help. This is driving me crazy.

1EightT

Aug 17 2005, 09:50 AM

Some guys on another forum suggested the following for my httpd

Timeout 45
KeepAlive On // On works VERY good for me, but many people say off for lower loads, try On for a while, my load would SHOOT up with it off, now it's really stable and lower than ever.
MaxKeepAliveRequests 150
KeepAliveTimeout 5
MinSpareServers 30
MaxSpareServers 60
StartServers 15
MaxClients 1024
MaxRequestsPerChild 75

After 6 hours my server load is down to .35

chess

Oct 26 2005, 09:12 PM

QUOTE (1EightT)

Some guys on another forum suggested the following for my httpd

Timeout 45
KeepAlive On // On works VERY good for me, but many people say off for lower loads, try On for a while, my load would SHOOT up with it off, now it's really stable and lower than ever.
MaxKeepAliveRequests 150
KeepAliveTimeout 5
MinSpareServers 30
MaxSpareServers 60
StartServers 15
MaxClients 1024
MaxRequestsPerChild 75

After 6 hours my server load is down to .35

ditto works well for me too

eth00

Oct 26 2005, 11:45 PM

KeepAlive should be ON unless you are running an image or file server. Unless you are 100% sure of what you are doing leave it on

Take a look at the apache docs if you are curious why.

papayiya

Oct 27 2005, 10:06 AM

Hi,

This is a stupid question cause I use windows at home -- but were do I find the http.conf file on my server?

Thank you,
papayiya

papayiya

Oct 27 2005, 10:09 AM

sorry found it

Dave#

Oct 30 2005, 01:26 PM

QUOTE (eth00)

KeepAlive should be ON unless you are running an image or file server. Unless you are 100% sure of what you are doing leave it on

Take a look at the apache docs if you are curious why.

This isn't true.

KeepAlive On = less CPU but more Ram
KeepAlive Off = more CPU less RAM

For busy webservers ie those serving more than 500 similtanous clients then KeepAlive is unlikely to work well as you are going to run out of real memory very quickly

eth00

Oct 30 2005, 01:30 PM

QUOTE (Dave#)

This isn't true.

KeepAlive On = less CPU but more Ram
KeepAlive Off = more CPU less RAM

For busy webservers ie those serving more than 500 similtanous clients then KeepAlive is unlikely to work well as you are going to run out of real memory very quickly

True BUT with a DDOS attack keepalive can help. The problem with a DOS attack is that each bot that connects will try and stay connected for the max time allowed by the server. If keepalive is turned off then the connection is going to end pretty quicky. Under normal circumstances keepalive should be on but for a DDOS I have found turning it off as well as tweaking the timeouts can help a lot.

Dave#

Oct 30 2005, 02:03 PM

Why are you talking about DOS attacks, the OP doesn't mention them at all. he does however show that he has swapped out 2Gb of Ram and you are suggesting retaining KeepAlive, madness imvho

eth00

Oct 30 2005, 02:09 PM

QUOTE (Dave#)

Why are you talking about DOS attacks, the OP doesn't mention them at all. he does however show that he has swapped out 2Gb of Ram and you are suggesting retaining KeepAlive, madness imvho

I usually lower the timeout rates in apache on a busy server. With a low timeout rate you get the best of both worlds. I have tried keepalive off and even with a busy server it has, from what I have seen, not been as effective as changing the timeouts. Each situation is different, it is easy enough to simply test it out for a few hours and see what works best for the server. It is best to change the values and see what works best for you.