Re: Server Redundancy
Gerald wrote: We all hedged bets that Cisco was going to absorb the CSS and just make it a software feature on the Catalyst switches. I haven't heard of that actually happening yet though. No, but there is some interesting new functionality in the latest revs of IOS which look awefully borrowed from the CSS. Haven't had time to dive in yet, though. -Jack
Re: Server Redundancy
If you go out and spend a few thousand you can also get Allied Telesyn L2-L4 products that now support Load Balancing. Actually the rapier 24i is about $2000 Canadian. (I'd have to check the VAR pricing) Jason On 6 Aug 2003 at 22:59, Paul Vixie wrote: Using outboard appliances for server load balancing is unnecessary, and it adds more powered boxes (thus decreasing theoretical reliability). If your upstream router can speak OSPF and is made by either Cisco or Juniper then it will implement ECMP (equal cost multipath). If you put your service address on lo0 as an alias, and you run Zebra or GateD on the service hosts which possess that alias address, then each such host will appear to be a router toward the service address as a stub host and your upstream routers will dtrt wrt flow hashing for udp or tcp traffic (that is, the udp/tcp port number will figure into the hash function, so you won't multipath your tcp sessions.) This is how f-root has worked for years. Look ma, no appliances. -- Paul Vixie
Re: Server Redundancy
On Wed, Aug 06, 2003 at 01:50:33PM -0400, Jason Dixon wrote: I second this suggestion. I worked briefly at F5 Networks in 2001 and was responsible for supporting Big-IP and 3DNS. Both are very nice products, but NOT cheap. I've used them all fairly heavily, except the Foundry gear. Alteon's my personal fave. Biggest problem with the F5: hard drive. In my book, that means you instantly need two, doubling the price. For price concerns, tho, just check ebay. $13k AD3s for $2500...don't say nothing good came from the dotcom crash. John
Re: Server Redundancy
Austad, Jay wrote: We all hedged bets that Cisco was going to absorb the CSS and just make it a software feature on the Catalyst switches. I haven't heard of that actually happening yet though. If they did that, how would they sell the CSS hardware? :) I would think that the closest you are going to get to that is the CSS blade for the Cat 6500's. Although, wasn't there a version of code for the 6500's that had some local director features in it awhile back? Or did you actually need a local director blade? -jay Cat6500's in native mode support IOS sever load balancing, which is like a not quite as intelligent version of the CSS, but does use the PFC's hadware accelartion. (Although for this specific application of the original poster, to support servers on different IP subnets requires the SLB function to NAT the client IP address as well as the server IP, to ensure return traffic flows back through the SLB. In this mode, it cannot use the PFC hardware switching.)
RE: Server Redundancy
On Wed, 6 Aug 2003, Austad, Jay wrote: If they did that, how would they sell the CSS hardware? :) That was our concern. Cisco already had hardware to do as good or better than what ArrowPoint was doing. They would suck in the intellectual property, discontinue the CSS line, and roll out a software update to the Catalyst that would do all of the same things the ArrowPoints would. Our 1100's SPOF was the single IDE drive that powered the whole thing. Their answer to that observation was: buy 2 1100's. (...which we did.) G
Re: Server Redundancy
[EMAIL PROTECTED] (Jason Robertson) writes: If you go out and spend a few thousand you can also get Allied Telesyn L2-L4 products that now support Load Balancing. Actually the rapier 24i is about $2000 Canadian. (I'd have to check the VAR pricing) how much would i have to pay to not have that extra powered box between my data and my customers? oh, i forgot, it's zero, isn't it? re: Using outboard appliances for server load balancing is unnecessary, and it adds more powered boxes (thus decreasing theoretical reliability). If your upstream router can speak OSPF and is made by either Cisco or Juniper then it will implement ECMP (equal cost multipath). If you put your service address on lo0 as an alias, and you run Zebra or GateD on the service hosts which possess that alias address, then each such host will appear to be a router toward the service address as a stub host and your upstream routers will dtrt wrt flow hashing for udp or tcp traffic (that is, the udp/tcp port number will figure into the hash function, so you won't multipath your tcp sessions.) This is how f-root has worked for years. Look ma, no appliances. -- Paul Vixie
Re: Server Redundancy
On Thursday, 7 August 2003, at 07:28AM, Rob Pickering wrote: Then you've just got your BGP convergence time and unequal load balancing effects to worry about. Whilst I'm not knocking Paul's solution in an application like running a root NS for which it is perfect, I'm not so sure it's necessarily best for every kind of service load balancing. We're using the technique Paul used in local clusters with OSPF; the convergence time in an OSPF area which contains only a small number of server and a couple of routers in a single area is pretty small. There's no BGP convergence issue in this application (there's no BGP within the server cluster). We're using another anycast technique in the wide area, using BGP to advertise covering supernets for services which are offered autonomously in multiple locations. BGP is involved in this one, but we are mitigating the potential for flap damage or transient convergence loops by offering service from remote nodes to a local community only, and not the whole Internet (i.e. the service supernet is offered as a peering route, with restricted propagation, and not for global transit). The general approach we're taking with the wide-area, global service distribution technique is described here: http://www.isc.org/tn/isc-tn-2003-1.html http://www.isc.org/tn/isc-tn-2003-1.txt I've used both the route hack based and commercial NAT load balancers, and they both have their place. It's not really that much of a hack; it's just anycast over an IGP coupled with routers which can populate the FIB with multiple equal-cost routes with different next-hops, with some manner of flow hash to keep traffic from a s single session pointing at the same server. If you are running complex web services (think expensive per server sw licences etc) then the investment in a pair of redundant load balancers for the front end to give more consistent performance under load as well as resilience can look very sane indeed. I've deployed services behind foundry layer-4/layer-7/content/SLB/buzzword-du-jour switches before, and they worked very well; from the brief time I spent with them, they seemed well-designed and feature rich. However, the foundries still suffered from the (near) single point of failure problem. It only takes one person to mess up the switch config whilst modifying a service or adding a new one, or a firmware upgrade that goes bad, and you lose all your services at once. As Paul mentioned, the advantage of using local-scope anycast with an IGP to build a cluster is that there are no additional components, and hence no additional points of failure. Joe
Re: Server Redundancy
On Wed, 6 Aug 2003, Gerald wrote: vrrp on FreeBSD is supposed to be a free solution to allow machines to watch each other and take over IP addressing if connectivity is lost. Depending on how remote your IP blocks are and how much control you have over the routing equipment in between, your only choice may be a commercial solution. Two things to keep in mind: VRRP is not a load balancing solution, it is a failover solution and (AFAIK) VRRP only operates within-network. allan -- Allan Liska [EMAIL PROTECTED] http://www.allan.org
Re: Server Redundancy
In the immortal words of [EMAIL PROTECTED] ([EMAIL PROTECTED]): On Wed, 6 Aug 2003, Austad, Jay wrote: As a side note, I've used Cisco's CSS, F5's stuff, Alteon, and Foundry. Out of all of them that I've used, the Foundry had the least problems and had a nicely structured config. Foundry seems to be fine for www traffic, but has serious issues with handling long FTP sessions. FTP works while you're in your stickiness period (up to 2 hours on the non-XL serveriron), but after that it will forget which FTP server has the control session and send your next data session to another server which won't recognise it. Last time I spoke to Foundry, this was still considered a feature. Do other vendors handle this properly? I recall that Resonate Central Dispatch handled this well the last time I looked, but the last time I looked was about 3 years ago now, so take that for what it's worth. (www.resonate.com) -n [EMAIL PROTECTED] My goal is real simple: to write better than anyone who can write faster than me, and faster than anyone who can write better than me. (--J.M. Straczynski) http://blank.org/memory/
RE: Server Redundancy
I've used them all fairly heavily, except the Foundry gear. Alteon's my personal fave. Biggest problem with the F5: hard drive. In my book, that means you instantly need two, doubling the price. Same thing with the Cisco CSS. Even without a hard drive, you should have 2 of them anyway. How do you plan to do software upgrades or other certain types of maintenance without an outage if you don't have a second one? I've seen them flake out too, either by not passing traffic or by load balancing getting messed up somehow. In most of these situations, a failover to the standby unit fixed the problem (this was with F5 gear). -jay
Re: Server Redundancy
On Wed, 2003-08-06 at 13:39, Allan Liska wrote: On 6 Aug 2003, Jason Greenberg wrote: Can I have some suggestions on how to load balance servers that are on seperate IP blocks? Is there any way to perform translation at this level? Exclude DNS based balancing please... Take a look at Nortel's Alteon product line, Cisco's CSS product line, or F5's BigIP Product Line. All of which have Global Server Load Balancing capability. The GSLB can be done a number of different ways on these boxes including stupid DNS tricks (not your typical round robin stuff, but still DNS) and using a BGP configuration. I second this suggestion. I worked briefly at F5 Networks in 2001 and was responsible for supporting Big-IP and 3DNS. Both are very nice products, but NOT cheap. -J.
Re: Server Redundancy
--On 07 August 2003 08:29 +0100 Simon Lockhart [EMAIL PROTECTED] wrote: The gated solution sounds interesting, but doesn't automatically have the feedback loop of stopping advertising itself when apache stops responding, but the box is still up (which is a fairly common occurrence in our Apache2 testing). It seems like a fairly trivial hack to put together a script which polls HTTP requests to port 80 and drops the loopback service address if it is consistently failing. Then you've just got your BGP convergence time and unequal load balancing effects to worry about. Whilst I'm not knocking Paul's solution in an application like running a root NS for which it is perfect, I'm not so sure it's necessarily best for every kind of service load balancing. I've used both the route hack based and commercial NAT load balancers, and they both have their place. Commercial NAT based load balancers are able to do things like distribute requests according to actual measured server response characteristics. This is great if you have clusters of servers with different specs but want to extract the best performance under peak load from the whole cluster. It also helps if you are running complex services where individual servers can develop a pathological slow but not failing response for some reason. They are also able to do the kind of service polling as above and react quicker to a down server than one which relies on routing protocols. Neither of the above are much advantage if you are running a cluster of BIND servers who's performance is equal and deterministic and where dropping a proportion of requests for a second or two if a server ever dies is no big deal. If you are running complex web services (think expensive per server sw licences etc) then the investment in a pair of redundant load balancers for the front end to give more consistent performance under load as well as resilience can look very sane indeed. -- Rob.
Re: Server Redundancy
On Thu Aug 07, 2003 at 12:14:43AM -0700, Bill Woodcock wrote: On 7 Aug 2003, Paul Vixie wrote: If you go out and spend a few thousand you can also get Allied Telesyn L2-L4 products that now support Load Balancing. Actually the rapier 24i is about $2000 Canadian. (I'd have to check the VAR pricing) how much would i have to pay to not have that extra powered box between my data and my customers? oh, i forgot, it's zero, isn't it? Yup, ah've allus been a mite suspicious of products fo' which the competitive upgrade is a patch-cord. Likewise. I have a bit of a dislike of putting a single port 80 terminating box in front of the 10's of servers I've just put into the webfarm. I've built all this redundancy into the server side of things, and then I have to funnel all the port 80 traffic through a single box (well, 2 for redundancy). We currently use DNS load-balancing for both global and local loadbalancing, and it works well, apart from not being able to immediately drop a box out of load-balance. The gated solution sounds interesting, but doesn't automatically have the feedback loop of stopping advertising itself when apache stops responding, but the box is still up (which is a fairly common occurence in our Apache2 testing). Simon -- Simon Lockhart | Tel: +44 (0)1628 407720 (x37720) | Si fractum Technology Manager | Fax: +44 (0)1628 407701 (x37701) | non sit, noli BBC Internet Services | Email: [EMAIL PROTECTED]| id reficere BBC Technology, Maiden House, Vanwall Road, Maidenhead. SL6 4UB. UK
RE: Server Redundancy
We all hedged bets that Cisco was going to absorb the CSS and just make it a software feature on the Catalyst switches. I haven't heard of that actually happening yet though. If they did that, how would they sell the CSS hardware? :) I would think that the closest you are going to get to that is the CSS blade for the Cat 6500's. Although, wasn't there a version of code for the 6500's that had some local director features in it awhile back? Or did you actually need a local director blade? -jay
Re: Server Redundancy
On Wed, 6 Aug 2003, Jason Greenberg wrote: Can I have some suggestions on how to load balance servers that are on seperate IP blocks? Is there any way to perform translation at this level? Exclude DNS based balancing please... vrrp on FreeBSD is supposed to be a free solution to allow machines to watch each other and take over IP addressing if connectivity is lost. Depending on how remote your IP blocks are and how much control you have over the routing equipment in between, your only choice may be a commercial solution. http://www.bsdshell.net/hut_vrrpimpl.html I've not used it, and the documentation is currently in French. The HUT project also has FreeBSD load balancing software for free that is supposed to function like F5/Alteon/Cisco LB. I've maintained the Cisco CS 1100 (when it was Arrowpoint) in production. You could VLAN remote machines into what you want to do on that. I think that equipment has changed quite a bit though since Cisco bought them and my experience is over a year old. G
Re: Server Redundancy
On 7 Aug 2003, Paul Vixie wrote: If you go out and spend a few thousand you can also get Allied Telesyn L2-L4 products that now support Load Balancing. Actually the rapier 24i is about $2000 Canadian. (I'd have to check the VAR pricing) how much would i have to pay to not have that extra powered box between my data and my customers? oh, i forgot, it's zero, isn't it? Yup, ah've allus been a mite suspicious of products fo' which the competitive upgrade is a patch-cord. -Bill
Re: Server Redundancy
On Wed, 6 Aug 2003, Gerald wrote: On Wed, 6 Aug 2003, Jason Greenberg wrote: Can I have some suggestions on how to load balance servers that are on seperate IP blocks? Is there any way to perform translation at this level? Exclude DNS based balancing please... vrrp on FreeBSD is supposed to be a free solution to allow machines to watch each other and take over IP addressing if connectivity is lost. Depending on how remote your IP blocks are and how much control you have over the routing equipment in between, your only choice may be a commercial solution. Don't forget pen, which runs on FreeBSD (and even NT according to the author). http://siag.nu/pen/ It's not for the enterprise, but does provide simple load-balancing for people who can't afford a proper switch. Andy --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 ---
RE: Server Redundancy
We've been using the Linux Virtual Server project (which a previous poster mentioned) to do load balancing (locally) on web based apps, pop3, smtp and now iptable firewalls. It scales well, has multiple lb algorithms (wlc, rr, lc, wrr, etc.) and even multicasts out the connection info if you want so you can set up a bank of redundant lb's. It can work w/ NAT or ip tunneling or direct routing so it might be able to do global load balancing depending on the setup. We have a 1+ user mail farm behind one and haven't seen a single hiccup... http://www.linuxvirtualserver.org/ -- Don Mills
Re: Server Redundancy
On 6 Aug 2003, Jason Greenberg wrote: Can I have some suggestions on how to load balance servers that are on seperate IP blocks? Is there any way to perform translation at this level? Exclude DNS based balancing please... Take a look at Nortel's Alteon product line, Cisco's CSS product line, or F5's BigIP Product Line. All of which have Global Server Load Balancing capability. The GSLB can be done a number of different ways on these boxes including stupid DNS tricks (not your typical round robin stuff, but still DNS) and using a BGP configuration. Hope this helps! allan -- Allan Liska [EMAIL PROTECTED] http://www.allan.org
Re: Server Redundancy
The gated solution sounds interesting, but doesn't automatically have the feedback loop of stopping advertising itself when apache stops responding, but the box is still up (which is a fairly common occurence in our Apache2 testing). Most folks tie Big Brother or Netsaint or just an expect script into the loop, and withdraw the advertisement and sound an alarm when a service is offline. -Bill
Re: Server Redundancy
Using outboard appliances for server load balancing is unnecessary, and it adds more powered boxes (thus decreasing theoretical reliability). If your upstream router can speak OSPF and is made by either Cisco or Juniper then it will implement ECMP (equal cost multipath). If you put your service address on lo0 as an alias, and you run Zebra or GateD on the service hosts which possess that alias address, then each such host will appear to be a router toward the service address as a stub host and your upstream routers will dtrt wrt flow hashing for udp or tcp traffic (that is, the udp/tcp port number will figure into the hash function, so you won't multipath your tcp sessions.) This is how f-root has worked for years. Look ma, no appliances. -- Paul Vixie
RE: Server Redundancy
The feature you are referring to is IOS Server Load Balancing, it's a limited subset of CSS features but fairly useful on the 6500 or a fast 7200. The Content Services Module (CSS blade) is very powerful and expensive, but if you need to balance multiple gigabits of traffic is ideal. Simon -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of Austad, Jay Sent: Wednesday, August 06, 2003 11:51 AM To: 'Gerald'; Austad, Jay Cc: Jason Greenberg; [EMAIL PROTECTED] Subject: RE: Server Redundancy We all hedged bets that Cisco was going to absorb the CSS and just make it a software feature on the Catalyst switches. I haven't heard of that actually happening yet though. If they did that, how would they sell the CSS hardware? :) I would think that the closest you are going to get to that is the CSS blade for the Cat 6500's. Although, wasn't there a version of code for the 6500's that had some local director features in it awhile back? Or did you actually need a local director blade? -jay
RE: Server Redundancy
If the servers are in two separate locations, like two datacenters on either side of the country, you are stuck with DNS-based load balancing. Like others have mentioned, Cisco, F5 and others have products which will handle this for you and take into account some other factors when directing traffic. DNS load balancing works quite well, I've used the F5 BigIP and 3dns extensively, and the Foundry ServerIron (which is fairly cheap). A little more detail into what you are trying to do would help. The most common setup with this is to have multiple datacenters, and each datacenter has a cluster of identical servers behind something like a BigIP. The traffic is load balanced at that level, but your Global load balancer which hands out DNS communicates with the local guy to figure out what the current traffic ratio is and modifys its dns replys accordingly. There used to be a free one for linux called Eddie, which looked quite robust. I think it was eddieware.org or eddieware.com. There is also the linux virtual server project, but I don't believe it has support for Global load balancing, only local. As a side note, I've used Cisco's CSS, F5's stuff, Alteon, and Foundry. Out of all of them that I've used, the Foundry had the least problems and had a nicely structured config. I would recommend the CSS, but it seems to have quite a few bugs in the code that still need to be worked out, but the support for SSL acceleration is nice. F5... I used to really like F5. In fact, I was one of their beta sites back in 1999 and 2000. After some problems with code that broke things, we discontinued the beta program with them. Shortly after, their new releases were getting worse and worse, their support seemed unwilling to help (for almost $100k a year in support, you'd think they would care), so I switched to Foundry. An insider over at F5 told me that most of the people who had written the original code back in 1999/2000 were all gone, and most of the problems were a result of the new people not yet wrapping their heads around the code. This was about 2 years ago, so it's possible they've figured out how everything is put together and it's better now. For awhile though, it was quite bad. Feature-wise, F5 has more features than any of the other ones, Cisco CSS comes in a somewhat distant second place. For most people, any of the above will suffice and most of the features available in F5 and Cisco are just nice-to-have's and not a requirement. -jay -Original Message- From: Gerald [mailto:[EMAIL PROTECTED] Sent: Wednesday, August 06, 2003 1:12 PM To: Jason Greenberg Cc: [EMAIL PROTECTED] Subject: Re: Server Redundancy On Wed, 6 Aug 2003, Jason Greenberg wrote: Can I have some suggestions on how to load balance servers that are on seperate IP blocks? Is there any way to perform translation at this level? Exclude DNS based balancing please... vrrp on FreeBSD is supposed to be a free solution to allow machines to watch each other and take over IP addressing if connectivity is lost. Depending on how remote your IP blocks are and how much control you have over the routing equipment in between, your only choice may be a commercial solution. http://www.bsdshell.net/hut_vrrpimpl.html I've not used it, and the documentation is currently in French. The HUT project also has FreeBSD load balancing software for free that is supposed to function like F5/Alteon/Cisco LB. I've maintained the Cisco CS 1100 (when it was Arrowpoint) in production. You could VLAN remote machines into what you want to do on that. I think that equipment has changed quite a bit though since Cisco bought them and my experience is over a year old. G
