AHA!

I'm starting to realize that the issues that I'm having on my server are all tied to HTTP POSTs.

In short my server is very active, up to 167 MySql queries per second, but loads are typically around 1. Me and my users are having no problems with GET requests, but occasionally POSTs are failing.

I'm getting reports from my users that complain about problems submitting forum posts (408/409/white page), submitting scores, and even PayPal is having a terrible time doing IPN POST submits back to my server.

Any ideas?

I think I may have figured this out, and I'll post some ideas for reference in case others run into a similar issue.

What's special about POSTs? They need to insert/update data into the database.

What's special about INSERTs/UPDATEs? They need to lock a table?

Why is that so hard? My system has about 160,000 tables, and the default MySQL table_cache is 1024.

After bumping my table_cache to 10000, I'm getting less Table_locks_waited, and apparently more throughput. My queries per second has doubled!

It hasn't been long enough for me to know for sure I've fixed my problem, but it's looking better.

> My system has about 160,000 tables

Hello!

That is a lot of tables

Glad you could find the problem