I recently got set up with a cluster (two 2.4ghz p4), so I though I would start a thread with a few observations and questions. I talked to EV1 today, and they are working to get more detailed docs up about the service soon (hopefully this can help get it rolling ).

Along the way, I've learned some interesting info that might help others looking at this option.


What You Get

A set of 4 IP addresses for each server, and a shared set of 4 IP addresses that support load balancing and failover on ports 21 (FTP), 80 (http), and 443 (HTTPS, aka SSL). The load balancer will automatically direct incoming traffic between the two servers, and do all the network translation correctly (so your web logs don't get messed up). If one of the servers goes down, all requests go to the remaining server.

If you are serving static HTML, files or images, load balnacing and failover is easy. Just keep synchronized copies of each site on both servers.

If you have scripted applications using PHP or Perl that use a database, or allow users to upload files to the server, things will get more complicated. There are a couple of solutions:

Load Balancing with MySQL (but no fail-over)

Install all your databases on server A. Configure applications on server A to use the local machine. Configure server B with the same applications, but tell them to use the database on server A (instead of locally). You should use the private network for this purpose.

Pros: For CPU intensive applications, you will have two servers to cut through the code, and configuration is a snap. More powerful, responsive, and reliable than a single server.

Cons: If the server running the database goes down, then the application will probably fail. In this case you should have defined a default 404 page so those database errors won't look embarrasing.

Load Balancing with MySQL (and fail-over)

You can configure each database on each server in replication mode -- one server is the 'master' and the other is the 'slave'. The two servers replicate over the local private network. Your applications will only be able to write to the master server, but can read from either server. Most off-the-shelf applications will not work this way, you will just have to tell them to use the master server.

You could also have a failover monitoring script that can change the master database on the fly. For more information about all this madness, you might want to read about MySQL Replication, or just find a smart db tech to help you set it up right.

Pros: Truly redundant and fail-safe for database applications. When was the last time you got such high-level IT infrastructure at this price?

Cons: Tricky to set up, applications may need to be customized.

Load Balancing Filesystems (with fail-over)

If you need to keep two servers in a mirrored set-up, you might consider using rsync (which is built into RedHat) or Unison. Since you have a private network, you can keep things syncing often, either through a cron job, or just as a script the repeats forever, sleeping a bit between runs. You could also cross-mount drives between servers (over the private network), so you could share data that way.

Pros: Easy to set up and effective sharing of resources

Cons: Rsync is really just master/slave, you will need something like Unison to do two-way file replication and keep things in sync.

Conclusions

Even if you choose not to use load-balancing and fail-over features, you can benefit from having two server connected by a private network. It makes backups easy, and is a perfect solution for developers that need a production / testing environment that are identical.

EV1 has indicated that they can have some configuration changes they can make that will fix some of the more common sharing issues (like a potential PHP shopping-cart problem: php sessions are written to disk, but the user jumps to the other server while in HTTPS secure mode, and gets logged out).

Unanswered Questions

I'm still working though the details and options, and running into questions as I go along. The possiblities are really endless.

For starters, I'd like to see some of these issues addressed in the documentation and product information areas --

- The second-level server package says '1400 GB Monthly Transfer per server', but after signing up, I see 700GB/month as the limit for each server.

- There are two 80GB hard drives in each server (for the 2nd and 3rd level packages). Would it be possible to run these drives in mirrored RAID-0 configuration? This could be done with software, but it would most certainly require use of the KVM switch rental to manage the reboots and reinstalls. And what about the RedHat enterpise CD ?

- The remote console service over SSH is cool, but what good is it for? I tried rebooting my server while I was connected to the console serial port, and although I stayed connected, I could not really do anything. I guess if you need to get into single user mode, this would be very helpful. EV1 mentions system monitoring, so I guess you could have stats of some kind dumped to a com port and collected remotely?

- How does the RedHat's update service work and for how long? Will I have to pay $349 per server/per year for access to the RedHat services past the first year? What (if any) RedHat support or access is included with Enterprise ES?

- How well do the control panel apps (like Ensim and Plesk) really deal with these types of configurations? What other types of tools come with RedHat Enterprise to help with managing the cluster?

- How do the Urchin licenses work with clusters? One per server?

Overall, this is a unique service that I'm looking forward to working with. I've dealt with dual-server setups before (usually co-location custom configs), but never as dedicated rentals, and never at this price! You can leave so much of the infrastructure and network issues for somebody else, it's a big value add.

Thanks, EV1!

Thanks for sharing your experience, somebox.

I’m probably going to setup a cluster in the near future so your info will be helpful in that regard.

Derek

Just a warning with MySQL replication, slaves can get behind. Thus you should not just blindly trust that your slave has the same data as the master.

I'd really like to see this deployed so customers can balance between more than two servers.

Another question, how does the load balancer determine which server to send the request to? Random, weighted, least connections, etc?

daveman692,



The equation for the load balancer is a function of active connections instead of the actual load of the server......... so it is more like active connections balancing.

The standard algorithym is 'least connections' but most loadbalancers allow 'round robin' and several other ways of distributing traffic

Headsurfer - are there are any plans for true clustering - multi-zoned array based storage over a switched fibre network; redundant private network connections for inter-node heartbeat monitoring and failover software supporting an active/passive cluster model with failover of file systems as well as applications. The solutions being passed off as "clustering" don't really meet the common definition of the term, although there are few situations in which they offer benefit. What they don't offer is the ability to identify that a system has failed, move it's storage to another system, and restart the application on the other system automatically - something all the major clustering systems offer as core functionality.

I'm not sure the cost of a true clustering solution (as opposed to duplicate systems run in a load balancing mode, which only vaguely resembles a true cluster) would make sense for my modest applications, but it would be a nice option to consider.

Hello,

Is there anybody having experience of setting up database replication inbetween two servers using Unison ?

As i am planning to do it so any problems while setting up or post setup problems that any one would like share with me

Thanks
Yashodeep

I have seen a patch for realtime filesystem mirroring from one machine to another, as soon as the file is written its replicated directly no use for rsync or other crontabbed solution.

Regardning MySQL, Why not try out 4.1 Cluster ?

if Ev1 gave me a pair of the cheapest i could spend
a month getting it to work and ill at the same time see
if its fit H-sphere too and give the other users some howto
to setup a solution, i bet many thinks cluster is nice but
stops when they are thinking about how to setup it.

I missed this one the first time around. I'll get randy to clarify a few questions but ...

1. That is an error on the bandwidth. Each server is 1400 or more gb bandwidth
2. drives could have hardware raid 0 or 1. We are soon to launch a few options.
3. up2date is valid as long as you have your service with us
4. Cps are not really designed with this in mind. The application that we typically envisioned was clustering for http and ftp content.

The custom applications that a few have mentioned could be handled with a dedicated shelf or rack along with load balancing hardware dedicated to you and under your exclusive control.

I am very please to now have one of these new cluster setups!

however can anyone point me to a primer or provide a quick how-to
on what needs to be done with the "load balancing" IPs, in order to utilize the
load balancing feature of this setup.

While setup for a normal server is simple, I can't seem to understand
how to use the virtual IPs and how to setup the dns for my domains with these
IPs.

Also I never received my welcome email when purchasing this setup, so
I don't know if there was more relevant info or links there to help me out.
Can anyone provide some much needed enlightenment?

Much Much Thanks in advance.

rottie -

The welcome email explains how the load balancing IPs are configured.

I have sent a request to customercare [at] ev1servers.net. They should re-send your welcome email...

Randy Williams, CTO

Load Balancing information has been entered into the related trouble ticket and also emailed to your admin contact email address.

How are sessions handled, especially with respect to SSL shopping cart type applicatons. What hardware is being used to do the load balancing?

Just be wary when dealing with systems requiring authentication and sessions. Unless the load balancer or the application assures that sessions are passed through to the same server, you will have many issues.

I think as long as you have an active connection (not idle for more than 60 minutes), the load balancer will assign traffic to the same physical server.

http://www.ev1servers.net/english/services...ers_details.asp

This is in the notes at the bottom:

A client will always be mapped to the same physical server while communicating while using the same protocol. This will remain true until either the service fails, or the client has not communicated with the load balancer on that protocol for 1 hour.

Would not trust my ecommerce operations to an "I think"

Be sure to check as many low end load balancers do not maintain state regardeless of how long the connection last. Also consider that your browser inititates many connections as once. For a single client, some requests may go to one system and additional requests to another. What if a person answers the phone for 5 minutes and then tries to checkout? Things to get specs on before migrating your system to a cluster.


[additional example]
Say I am shopping through HTTP and then go to HTTP, how will the request be handled?

Question?
What load balancing hardware is being used? I am sure the supplier has specs and answers to many of these questions.

Huck, "I think" you know what I mean .... I think...

As WebSales_Aaron has stated (the offical response), an active connection is not more than 60 minutes idle. So for a serious buyer, he/she shouldn't take more than 60 minutes to check out, in addition, your shopping cart session/cookies should be set to timeout less than 60 minutes to avoid confusion.

huck -

It is very straight forward:

1) client requests on the same protocol are sent to the same server unless the protocol becomes unavailable on that server, or the client does not communicate for 1 hour on that protocol.

2) server affinity is available on request between http and https, and therefore the client will be directed to the same server across protocols. This is disabled by default, because both services need to be active on the physical server for either service to be eligible for traffic.

Randy Williams, CTO

There is no reason to assume that even during the same session a persons IP will not change.

My advice to potential adopters is to make sure your web app is cable of tracking sessions through cookies or other such tokens and not through server-specific solutions, such as PHP sessions. Many people do not consider these problems when moving to a clustered solution.

Just questions to raise before considering this solution.


This is not a reason to not use the clusters -- but just advice to research your application first.


From some nortel docs, which more clearly states the problem.
:
Client IP addresses are becoming more virtual than real. With the growing popularity of wireless access devices and the use of DHCP servers and proxy firewalls, IP users are no longer uniquely identifiable by IP address. Even sequential user requests within a session might carry different source IP addresses, thanks to widespread use of proxies at Internet access points and within enterprise networks.

If you can't uniquely identify users by IP address, how can you differentiate services based on user categories? How can you make sure multiple requests from a user are processed by the same server? This requirement, known as 'persistent connections,' maximizes server efficiency and ensures users aren't dropped when they're filling an on-line shopping cart or completing a multi-page form.

Many Web sites address this issue by inserting an electronic "cookie" with a unique user identifier into the user's browser. This cookie is transmitted back with the user's next request. The Layer 4-7 switch must be able to recognize this cookie to provide any user-aware traffic management services.

(emphasis mine)

And TechieSurfer says...

TechieSurfer,



Under what condition(s) will trigger the load balance to think the service from a physical server is no longer available and route traffic to its pair? Is it a simple timeout period or something more low level? If it is specified by a timeout period, is it user configurable?

Thank you.

From: http://www.ev1servers.net/english/services...ers_details.asp

The load balancer performs layer 4 checks (opens a TCP connection to the service port) to determine if each service on both servers is "up". These checks are done every 10 seconds, and the service will be marked "down" if it fails 4 consecutive tests. The next time the server responds on the service port, it will be again be marked as available for requests.

Cookie switching, although possible, is not available on the pre-configured solution ($10 extra / server). This would be an option for a custom private solution, as would http 'cookie insertion.' (Cookie insertion eliminates the need for application layer support.)

Most of the time IPs do not change for a single end user during a single session. More importantly cookie switching does not work for HTTPS without moving the SSL burden to the load balance hardware...

Randy Williams, CTO

I have now received my welcome email thanks for that TechieSurfer and WebSales_Aaron

However my virtual IP's don't resolve, just hangs...
was there something I needed to set up?

I did post a trouble ticket but that was over two hours ago,
I know its a long weekend is upon us , but unfortunately poor slugs
like me have to work still

Anyway there is also the two DNS IP's that were provided,
at the risk of sounding extremely dense... what do I do with
those DNS IP's ?

Does setting the servers as Master/Slave need to be done first?
Can any helpful soul please provide some how-to or a point me
in the right direction?
I will certainly provide help to all future cluster newbies like myself
in any way I can once I get this all figured out.
Thanks to all in advance.

rottie -

My understanding is that a mismatched netmask on the server (possibly cpanel related) caused this issue. The netmasks were corrected, and I now show your virtual IP numbers as 'up'.


These are simply the suggested EV1 maintained DNS servers for our customers to use as resolvers.


Typically in a load balance/automatic failover situation you setup the exact same web site(s) on both servers. (The manner by which you synchronize the web content between servers is entirely your choice.) Once synchronized you then point the DNS address of the site(s) towards the virtual IP, and the requests to the virtual IP numbers are transparrently balanced between the corresponding real servers that are 'up'.

Randy Williams, CTO

Thanks very much for the help TechieSurfer.

Unfortunately, now only the main server IPs and its associated virtual IP resolve.

All other physical and virtual IPs do NOT resolve.

Any more thoughts?

rottie -

I am told that this problem was resolved. The IP numbers were unbound from the interface.

Randy Williams, CTO

Can anyone explain how the Virtual IPs are to be used?

I know that each virtual IP is associated with a pair of physical IPs,
one on each clustered server, and it does work wonderfully.

However, can the virtual IPs be used to attach a nameserver?
So that I can assign these to multiple domains?
If so, where/how can this be configured?

Or are these virtual IPs ONLY associated with the specific set of physical IPs?

Once I understand this, all will be good
Thanks for all the help.

rottie -

> Or are these virtual IPs ONLY associated with the specific set of physical IPs?

This is a true statement, and if I understand your question you can host large numbers of domains on any of the virtual IP numbers using name-based hosting.

Randy Williams, CTO

Thanks TechieSurfer, but...

I wanted to know how to add a custom nameserver to these IP addresses.
With IPs on my servers I can create nameservers on those IPs,
but with the Virtual IPs where can I create nameservers for those?
So that I can use the nameservers for any domain I want to
use on the cluster servers with load balancing.

I think the load balancer only is configured to for traffic on http,https,ftp and smtp. For DNS you would need to get that custom configured, I'm guessing.

Since DNS is already a load-balancing system (primary, secondary, etc.) you might think about adding more secondary servers to take advantage of this, using EV1's dns service, or by using a outsourced dns provider like secondary.com.

I use zoneedit.com for my dns, it's usually not a good idea to have your dns on the same network as your host. If the network goes down, you may want to re-route. They also have dns load balancing/failover that does round-robin dns and checks your servers to see if something goes down. This allows you to do failover using several machines.

So the ultimate solution is multiple clusters and dns load balancing between them

A lot of people don't realize that all dns supports round robin load balancing. Not exactly scientific, but for heavy traffic web sites you end up with the aproximate same loads.

I use round robin dns balancing for sites that pull more than 1400 GB month, great cheap and easy solution and each server can be thousands of miles apart and scales up to N nodes too.

Can Round Robin DNS handle sessions properly? For a static site, indeed RRDNS is a quick and simple solution.

Yes, round robin DNS can be useful for very, very busy sites, even with the overhead of sessions. Just store session data in a DB, I do this with one large site and a java JVM (resin).

Often though for sites we just bypass the issue - sometimes 90% of the traffic is static and can be distributed across RRDNS, while the 10% that uses HTTPS or needs sessions can be constrained to one node.

Ok the dea with Round Robin DNS is when a client goes to resolve a IP it could get a different IP each time. As far as state, a normal web browser will only resolve a name one time and keep a record of that IP for as long as the browser is open (some fulsh them now and then, but not to often) So a client going to a web page will go to the same server using the same session untill he closes the browser, at that point the client would have to start a new session when he went back a site after closing the browser.

We have a lot of sites with very short TTL - 20 minutes - in the dns and *many* browsers seem to get it right - i.e., 20 minutes later they pick up a new server if we've updated dns, even if they left the browser open.

However, some brain dead isps and computers cache the name for days anyway and hit servers we have turned off.

Bottom line - I think when you use round robin you have to expect a user session to hit multiple nodes during a session.

I do know that Windows XP keeps a resolver cache, and each positive responses is keept in WinXP's resolver cache for 1 day. However I do not know if the DNS TTL affects the TTL of the resolver cache. It could be that because you have your DNS TTL set to 20 min WinXP (or other OS's that support a resolver cache) will set its local TTL in the resolver cache to the same.

Well, I've had my cluster setup in production mode for about 4 days.

Everything is fine with mirroring and database replication, etc.

BUT THERE APPEARS TO BE NO LOAD BALANCING OCCURING WITH THE VIRTUAL IP.

Now I plan to take a more scientific approach, but I can only be sure of this because of the actual load on the servers AND the fact that the slave server's "slow mysql query log" has only one entry in it (the one from a search i did on the slave server site today).
And because our site is search heavy i KNOW that this log should be filled if people were being directed to this slave server!

Can anyone explain why this is not working? I can only imagine it is an EV1 problem with the load balancing switch.

Thanks.

PS> let me say that the high availability part is fine, that is I have tested the failsafe of bringing one of the servers down and it did redirect to the other live server.

rottie -

I just looked at your ticket. The server's subnet masks on the interfaces are mismatched. They should be all be /23, but three are configured for a /24.

Randy Williams, CTO

Thanks for the help. Just for the record though...

I never touched the IP settings, these were the default IP's and I would've assumed they would be correctly configured... so essentially I'm being asked to fix something that I never broke, or wouldn't know about. It may be possible that the reimage to cpanel/whm
broke this.
So I'm bringing this up so that others may know why their setups aren't load balancing as expected.

Oh and btw, anyone know if there's a place in WHM where I can change the netmask of existing IP's? Thanks.

Well, I have corrected the IP's netmask using 'ifconfig'

Again, I must say that I was asked to fix something which was given to me configured incorrectly.

So now that its been fixed as suggested. THERE IS STILL NO TRAFFIC BEING SENT TO THIS SERVER. I have submitted yet another trouble ticket.

Another thing I noticed was that for some reason (again not at my own doing) there was an IP configured on eth0:4 which doesn't seem like it should be there but correctly is configured on eth1. Not sure if this is even causing a problem but it should be mentioned.

Is it possible that someone at Tech support could finally put this issue to rest
It would be greatly appreciated.

rottie -

The root of your problem is that the load balancer cannot verifiy the health of the .77 IP, and therefore thinks the .77 IP is 'down'. Note it cannot even ping that IP:

#ping 67.15.xx.77
No reply from remote host.

But can ping the other IP without issue:

#ping 67.15.xx.81
Reply from 67.15.xx.81 : bytes=16 time<1ms TTL=64

This was working at one time as there are logged connections for HTTP on the .77 IP:

Overall_Connections: 82
Packets_Received: 556
Packets_Transmitted: 680

Randy Williams, CTO

techiesurfer,
those http requests were made by me directly,
and I don't know why you say you can't ping the IP, I can reach all day/nite from regular http!
In either case you are not resolving my issue, I already know something is wrong why is it so hard to you guys to fix this?
Especially since you guys haven't configured it properly!?

I don't understand??????? You want me to fix something which isn't my problem????

Please explain.

FYI, i can ping it just fine from the other cluster server and any other place!!!

rottie -

The load balancer (.255) cannot ping the .77 IP. This is a server issue, and I understand that the problem may have existed when you received the server.

The statistics are from the load balancer, not directly to the IP. They show that .77 was 'up' at one time and routed HTTP requests. If I read the monitor properly, it stopped responding to requests 55 hours ago.

It has been hypothesized that CPANEL (not sure if that is what you have) may be causing the problems with the server netmasks. As a precaution, we will change the IP of the load balancer's health checks next week to prevent accidental issues like this.

Randy Williams, CTO

Thanks for your attention to this TechieSurfer.

the 55 hour time frame doesn't make sense on my end, because I only changed the netmask as you suggested about 6 hours ago (with ifconfig).

When I ordered this setup I did request that cpanel be put on it. And I do have cpanel now.

I do know that there was a problem that I opened a ticket for about two days ago because WHM's rpm installer feature wouldn't allow it to connect to rpm updates and although I haven't confirmed it... a support tech did say that this issue had been fixed, not sure if this messed up the load balancing but the timeline is close to the 55 hour mark.

Anyway please let me know when things are working as expected. Thanks again for
the support.

rottie -

According to the ticket, the server issue was resolved last night. I can now ping the address from the load balancer:

#ping 67.15.xx.77
Reply from 67.15.xx.77 : bytes=16 time<1ms TTL=64

And that server has been taking connections through the load balancer:

Overall_Connections: 12215
Packets_Received: 765860
Packets_Transmitted: 567786

Randy Williams, CTO

Thanks to you TechieSurfer and your support staff.

I do notice the httpd traffic now and it does appear that load balancing is working as expected.

EV1 is #1!

Any idea how Google indexing responds to the virtual I.P.? I would imagine that it treats it like normal, but looking for a confirmation from someone who has experience.

Also, when checking URCHIN reports, how does one consolidate the traffic, other than manually?

As long as you have the two servers set up the same Google will not know that they are running behind the loadbalancer, so will think that it is one server running on the IP address of the server. So, in other words, it has no effect.
I use awstats and a script called logresolvemerge to merge the two different log files and then feed this to awstats. I imagine you can do a similar thing with urchin (although I think it might support native cluster stats).

Alex

can more servers be added to the original cluster and virtual i.p.?