I'm not sure if this is something really obvious, but I thought I'd share it on the off-chance that others are having problems getting their sites back online after all the problems that TP are having with the H1 data center.

I realised that we were having problems when I tried to upload some new code to our site this morning and started investigating the issue. We have three dedicated servers and all of them, thankfully, were by that point alive and kicking.

What I did notice however is that the web servers couldn't talk to the dedicated DB server, running MySQL 5. The login process to MySQL was just hanging until MySQL ran out of available connections. Not ideal.

It looks like the default nameservers for the old EV1 servers (which includes ours) are either down or limping along and this was affecting our MySQL DB. By updating the nameservers in the /etc/resolv.conf, the problem was solved. No idea, if this is correct, but I've used:

nameserver 207.44.128.228
nameserver 70.86.61.133

which are two of the IPs listed for ns1.theplanet.com (figuring that these wouldn't be affected by the outage). We don't use a control panel, so not sure what the process is for updating cPanel and the like.

Spent quite a bit of time figuring this out, as well as updating MySQL's standard config file to make it a little more robust if this happens again. I'd recommend adding:

wait_timeout=120
interactive_timeout=3600
skip-name-resolve

to /etc/my.cnf as well. If you do use the last option, you'll need to make sure all the entries in the privileges table use IP addresses instead of domain names. Also, on our MySQL server (5.0.22) the server wouldn't start with this option hence having to sort out working nameservers.

Anyway, i'm blathering on, but thought I'd post this in case anyone else is having similar hassles. Good luck to everyone affected with the outage and the no-doubt sleep-deprived souls working to get the data center up and running again.

Toodle Pip

Sam
http://www.chinwag.com
http://www.toodlepip.co.uk