Interesting Idea? Please advise. Options

[Guest_rickkumar_*]

Hello everyone,

Thanks in advance for all your comments.

I am trying to get some feedback on an interesting idea suggested to me on another board. This is in regard to our soon to be launched dating network that will use PHP/MySQL/Linux (similar sites match.com and AmericanSingles.com etc.)

IDEA:
---------------------------------------------------------------------------
If we get a dual Hard disk RAID server at RS & another dual hard disk RAID server at another host AND then link them via ZoneEdit.com's DNS servers (i think it is free) using their (load balancing and failover) feature.

We can make one server primary and the other secondary back-up server.

i.e.

NS1.RS server.com ----> Main server
NS2.Other host server.com ----> secondary back-up server.

This way if one of the hard disks crash on main server, it can still be running using the other disk because of RAID feature.

If the entire main server goes down, the back-up server can automatically pick up the load and sites will still be online while main server is being restored.

We can secure the data in 3 simultaneous ways here: RAID in main, RAID in seconday back-up server, and real-time back-up from 'main' to 'seconday' back-up server.

Basically what this entire idea can do is keep the sites online all the time and also provide data sedurity at the same time.
Questions:

1. Is it actually possible?

2. Can we do the real-time back-up from 'main' to 'secondary' server?

3. How can this be made possible realistically if possible?

Please feel free to suggest any option you may feel better fit our database oriented project needs as explained above.

Thanks for your comments.

Regards,

Kumar

[rackAID]

We have implemented similar solutions for clients but unfortunately, it is not as simple as your current scheme and cannot be done easily at RS.


Load Balancing vs. Failover
The round-robin DNS scheme you propose works well for load balancing but does not work well for failover. Many large ISP's cache DNS. If they cache the result from NS1 and it goes down, then the ISP's clients will not be able to reach your site as their DNS still points to the main server. True failover requires IP takeover or higher level routing solutions such that traffic destined for the main server's IP is re-routed to the secondary server. RS does not provide such a routing service.

Data Mirroring.
The other problem you face is how to handle the MySQL database transactions. Presumably your site allows input of data from your visitors. If you use round-robin DNS, then some visitors will be at server 1 while others will be on server 2. How do you intend to reconcile the differences between these servers? MySQL 3 does not support two-way replication very well.


Suggestion:
Though not ideal, the following can keep your downtime to a min in case of a catastrophic failure at the main server. You can setup your DNS to point to main server. You setup the main server to mirror data and replicate the database to server 2.


In the event of a failure, you can change your DNS to point to server 2. Though there will be some delays and downtime, once DNS refreshes people will be going to the new server.

Once the main server is up, you can also throw up quick redirect page until you have the server restored fully. Make sure your HTML is portable in that URLs are not hard-coded. This allows you to re-direct clients to http://www2.domain.com/ from the downed server until you get it back online.

With such a scenario your maximum downtime would be the shorter of getting the redirect page put up on the main server (2 hours if you got a priority restore) or the time it takes for a persons DNS to refresh.

[Guest_rickkumar_*]

Thanks for your post. I guess it is not as simple as we thought after someone suggested it to us at WHT.

Having said that, what if any, other option can you suggest which gives us what we want for our database intensive website as per my original message i.e. secure back-up internal in the server as well outside the server, good system that can efficiently handle member searches, provides maximum uptime, and still meets our budget of about $350 to $400 a month.

Thanks a lot for all the help.

Kumar

[lpirlet.net]

In all honesty, if you keep your server well maintained, you will rarely experience an unforeseen outage (software wise). For a hardware or data center outage, that's what your protecting against with an additional server.

It is a good idea to do hucks suggestion near the end. If RS took down your main server due to abuse/TOS violation, you still have a backup.

If my main server goes down, I would rather display a message to the visitor that "Our server is experiencing technical difficulties, please check back later." over "The page could not be displayed."

HUCK: I would appreciate your feedback on the following.... I tried this with a customer, but it broke FrontPage (since it doesn't get redirected like a http visitor), so we never were able to fully test it out.

If you provide two servers make one NS1 and the other NS2, that creates round-robin DNS. I believe that ISP's cache all your DNS records, not just NS1. So if one server goes down, the ISP cache will still get you to the next, working server. Like huck said, round-robin works more for load balancing, but since most of your pages are going to be php, you could do the following before displaying content (pseudo code):

if( grab 1x1 image from 111.222.222.111 ) {
//Server is up, send traffic there
header("Location: 111.222.222.111/this/page.php");
exit();
} else {
//Server is down, display sorry message and cached page.
echo "Sorry, our main server is down. Here is a cached page which was last cached at xx/xx/xx x:xx xm. Please check back later for updated content";
}
?>

That way everyone will go to your main server while it is up, and should it fail, they will get your non-current server. When you setup the database on the second server, just don't allow people to add/edit/delete database records, only view.

I don't know if that will work... but customers and visitors alike like it better when they know your experiencing an outage rather then figuring it out themselves.

[rackAID]

Even if an ISP caches both IPs, some do not make multiple requests, there is still a problem of my DNS client only doing 1 query and getting a response. If that response is the server that is down, then I will get a server unavailable page in IE.

The issue is that the DNS lookup is still valid. Depending on your DNS resolver and your ISP's cache, you may not ever get the other IP.

One of our larger clients has a failover system. It is controlled in two parts. The first is a load balancing/failover hardware box that distributes load between two servers. The system acts as a NAT box for the two servers behind it. So if one fails the failover box switches to just using one server.

You can achieve great uptime with just 1 server -- however it will require updates and maintence. You will need to run the costs of setting up a maintaining a redundant system vs. the cost of downtime.

[lpirlet.net]

Originally posted by huck
You will need to run the costs of setting up a maintaining a redundant system vs. the cost of downtime.

Right... which for me I have very *little* downtime.... BUT, I am not Match.com or Yahoo.com. I would be willing to bet that those guys have a hefty bill inorder to keep their uptime with their millions/billions of visitors per day.

Thanks for the info!

[Guest_rickkumar_*]

Thanks for your help and comments. In fact very practical comments.

I guess given the fact that we are not a big match.com or AmericanSingles.com but just a small start up. Even though we are aspiring to be among the best dating sites in the future but we really have not extra money and no venture capital. As we start making some profit in may be 6 to 12 months time frame, we can then move to a more decent hosting solution. Our hosting budget is about $300-$400 max per month.

What I understand here is that:

1.We should get a good server that can efficiently handle the searches in the MySQL database. This can be a Dual Xeon with at-least 2 Hard Disks probably 80 GB or more each.

2. Use one of these disks as an internal back-up in the server. (Can it be done ?).
If it is not possible, then may be just have a RAID server.

2A. If this server ever goes down, just display the message "Dear Visitors, Our network is experiencing some technical difficulties. We are working to resolve these issues as soon as possible. Please check back later. Thanks."

3. In addition have an external back-up such as what RS provides. The only issue I see with RS back-up system is that if I do daily back-ups it will eat up our allowed server bandwidth limit quite fast as RS counts it against your server bandwidth. May be you can suggest some other option here?

4. Its better to have server either from a managed provider or hire some inexpensive server management service such as BobCares.com or like. I guesss I can afford to buy a blcok of 10 trouble resolution tickets for very cheap price and then use them as needed.

Once again, thank you very much for the help.

Appreciate it very much.

Regards,

Kumar

[lpirlet.net]

Originally posted by rickkumar
Thanks for your help and comments. In fact very practical comments.

I guess given the fact that we are not a big match.com or AmericanSingles.com but just a small start up. Even though we are aspiring to be among the best dating sites in the future but we really have not extra money and no venture capital. As we start making some profit in may be 6 to 12 months time frame, we can then move to a more decent hosting solution. Our hosting budget is about $300-$400 max per month.

What I understand here is that:

1.We should get a good server that can efficiently handle the searches in the MySQL database. This can be a Dual Xeon with at-least 2 Hard Disks probably 80 GB or more each.

2. Use one of these disks as an internal back-up in the server. (Can it be done ?).  
If it is not possible, then may be just have a RAID server.

Internal backup is good should you need to do a restore, or you delete something you shouldn't have.  I would suggest that if you have another server that you make monthly backups of the site, and do daily database dumps.  Your database is the most important.  As you grow, start doing it 2, then 3, the 4 times a day.  If you don't have another server, you can easily ask someone for space, or rent space for doing backups.  A simple cron script would allow you to make nightly dumping.

2A. If this server ever goes down, just display the message "Dear Visitors, Our network is experiencing some technical difficulties. We are working to resolve these issues as soon as possible. Please check back later. Thanks."

Well, that's what I asked Huck about.  Technically, yes it would work that way... but it is visitor dependent (depends on their ISP).  So some visitors would see the message, some would see the normal "The page couldn't be displayed."  The thing is you have to determin, is it worth the price of another server to display a message only half (give or take) the visitors can see?  Maybe only have one server and rent backup space, then 6-12 months down the road move to a DC that offers a redundent server solution.

3. In addition have an external back-up such as what RS provides. The only issue I see with RS back-up system is that if I do daily back-ups it will eat up our allowed server bandwidth limit quite fast as RS counts it against your server bandwidth. May be you can suggest some other option here?

Again, that would work for restore, but you would be better off in another datacenter.  You don't have to own the server, you can rent space.

4. Its better to have server either from a managed provider or hire some inexpensive server management service such as  BobCares.com or like. I guesss I can afford to buy a blcok of 10 trouble resolution tickets for very cheap price and then use them as needed.

If you don't know how to manage a server, I would suggest that you get it managed.  Bobcares is a service that can help/replace your support department.  However you still have to manage the server.  I would suggest instead service from CheetaWeb.com or R-Fx.net.  They are popular people on these boards, and know what they are doing.

I guess I would start with a server and somewhere to store daily backups of the database. As you grow, look for other options at another Datacenter. It would be cool if I could recommend EV1, but there is no way you can do what your talking about here.

Hope things go well.

[Guest_rickkumar_*]

Jeremy: Thanks very much for that very detailed and helpful answer. In fact there were few things I was not even aware of before.

I have actually talked with my developer who is going to install and maintain my sites on these servers. My developer agrees with your idea.

First of all we will not display the 'server not available' message at this time, may be later when we make some profit.

Yes, as you said we will get a 'managed' nice (probably Dual Xeon) server with dual hard disks of about 80 GB each to start with. We will ask some managed DC to set it up in case they won't have RAID set up already available.

I have seen around and we can get this kind of server in about 250 to 300 dollars.

And I also agree that its better for us to rent space for daily back-ups. In future we can then start few times daily or may be even real time back-ups as our finances permit. We will try to see if we can actually find a DC that provide 'affordable' managed back-ups.
If not, then like you said we will do it using the cron script to do the daily dumping.

Here we need to look around who provides affordable and easy to use back-up space.

My developer has access to and frequently uses very affordable contract network engineers which can help as and when needed on a very short notice. If really needed we will use CheetaWeb as well --> they seem to be pretty good in cost and very thorough in services.

Again Sir thanks a lot for the time that you took to write your detailed comments.

Regards,

Kumar