Re: Two HAProxy instances with a shared IP
On Mon, 8 Apr 2013 14:36:52 +0100 (BST) in <564091370.1183442.1365428212786.javamail.r...@innovot.com>, Phil Daws Phil Daws wrote: > Hello, Hello, > am making my first foray into setting up a test lab to play around with > HAProxy. Ideally I am hoping to build an environment which consists of two > HAProxy nodes that share an IP address, and each node then offloads HTTP > connections to two backend web servers. Basically build a meshed architecture > for no single point of failure. A mesh for a shared IP ? How ? Encapsulating the IP datagram ? >It looks like the best route would be to use Wackamole and Spread for the >shared IP address. or keepalived or carp. > Am building on CentOS 6.4 so would be grateful for your thoughts on this > setup or whether there is a more appropriate one. If all works well then > hopefully can start to look at LB/HA for other services. The wackamole solution seem only valuable if you have more than one shared IP. ++. -- Jérôme Benoit aka fraggle La Météo du Net - http://grenouille.com OpenPGP Key ID : 9FE9161D Key fingerprint : 9CA4 0249 AF57 A35B 34B3 AC15 FAA0 CB50 9FE9 161D signature.asc Description: PGP signature
Re: Active/active HAProxy
On Thu, 21 Mar 2013 11:00:37 +0100 in , Lukas Tribus Lukas Tribus wrote: > > > It's a point in time dump and restore of the in flight packets. > > Can't dump the details and in flight content of a TCP session if > the host is already dead. You're right. Its primary goal is system update without service interruption. > So either this will work only for manual switchovers (but not for > sudden hardware/software failure; also at this point TCP connection > repair would probably be a solution) or you sync everything in realtime > with the other proxy, but to do that, you will need a huge uplink > between them. You're right in the sense that the requirement for layer 4 LB two nodes synchronization is not just synchronization of some IP headers like in the firewalling case, so the synchronization flow need to transport the whole packets contents. It's tricky to implement : you need to get only the packets from the sockets used inside HAProxy and I'm not familiar with the code in HAProxy : are SOCKET_RAW used and are packets written to a file descriptor ? > > Also, suppose you can implement this through netfilter/pfsync (I've my > doubts about that) and by patching haproxy, how do you avoid that the > "standby" TCP session on proxy 2 interferes with the TCP session prior > to a switchover/failover? I guess you would need additional > kernel hacks. After some sleep and thoughts, the API for netfilter/pfsync will just be an overcomplicated implementation of a packet mirroring system. > > In the end, you will end up spending so much cpu and memory > for standby tcp session and the syncing, that the solution will be > as performant as an active/standby solution and it will increase > the complexity in your load-balancer. Agree for the code complexity and memory usage, not agree for the CPU usage. If HAProxy code already make an extensive usage of raw sockets and write packets to a fd, you just pipeline the fd to the socket especially created for syncing purpose and let the current code handle the fd as it's used to (but I do not know the HAProxy internals). The network load is really a pb then to achieve that. The syncing protocol is then a real headache to minimise race condition and/or out-of-sync case and maintain only the non closed connexion in memory of the backup HAProxy. > > > prefer an proper solution to avoid the renegotiation on > > the client side > > Its a huge and complex task to do, which I didn't see anyone > doing before. If you or your client has the resources to > implement this, please go ahead and tell us how exactly you > did it. Well, I'm going to put in balance the use cases of such a feature and the overall code complexity and of course if the HAProxy community want such a feature given the code complexity added. The first step will be to read the HAProxy code entirely :) The second to deeply think about an effective packets mirroring protocol :) ... > > But for a "would prefer a stateful solution to avoid a TCP RST > + a new TCP handshake when a proxy dies" (so to speak; if iptables > is configured accordingly), I would certainly not do it. > > The benefit of it simply doesn't justify the effort, imho. You're probably right. Time to sleep. Regards, -- Jérôme Benoit aka fraggle La Météo du Net - http://grenouille.com OpenPGP Key ID : 9FE9161D Key fingerprint : 9CA4 0249 AF57 A35B 34B3 AC15 FAA0 CB50 9FE9 161D signature.asc Description: PGP signature
Re: Active/active HAProxy
On Tue, 19 Mar 2013 20:03:14 +0100 in , Lukas Tribus Lukas Tribus wrote: > > > conntrackd permit to also share TCP states between boxes that will > > also run iptables > > With conntrackd-syncing you just allow the packet to pass the iptables > barrier; but the session will still be dropped by the OS because the > TCP stack doesn't know the socket, and so does not the application. Right. > To do this you would need something like TCP connection repair [1], but > that requires support in both the kernel and userspace. While this crazy > feature seems to have made it into the 3.5 kernel, I'm not aware that > this is supported in haproxy. I can think of an alternative implementation using ... netfilter kernel API. Probably not a good idea since the implementation will require netfilter rules to be configured by HAProxy and is not portable. > > In fact, while rethinking, I'm not sure TCP connection repair can be used > for failover anyway, its just a technology to move the TCP session from one > host to another gracefully, but it requires both hosts to be alive afaik - > so it doesn't make sense for failover. It's not its primary goal ... but it can't be used in active/active, only active/passive setup with a bit of hack that, personally, I'll put in the software that ensure the HA like keepalived. It's a point in time dump and restore of the in flight packets. > > > > HAProxy can share as of 1.5 its connection table, which is really a > > appreciated feature :) > > HAProxy can share stick-tables [2], but that doesn't mean you can > implement stateful failover. Which will at least ensure that session affinity with backends is kept between several HAProxy instance which is a good step for active/active HAProxy setup. > If you want to do this with haproxy, you will probably need to drop the > idea of stateful failover, imho, no user-space software can accomplish > this. Well, in a non portable fashion, it's possible if you reuse the kernel API that pfsync or conntrackd use to share the connection table between two boxes. I'm pretty sure one can patch HAProxy code to share more than the stickiness criterions of a frontend or a backend. And to implement this, you do not need the fancy TCP_QUEUE_REPAIR flag, only for example a forked process over an event driven API to guarantee syncing and of course some time to implement that cause that not exactly what I call a trivial piece of code :) > On the other side, if maintaining the TCP sessions when a failover occurs > is a requirement for you, you should stick to LVS + conntrack syncing, > as thats probably possible. Of course you can work only up to > Layer 4 and won't see the application Layer on your load-balancer. It's not a strong requirement for all the applications served only for a few that really prefer an proper solution to avoid the renegotiation on the client side is case of a crash and a restart from scratch. LVS only will not fit the functional requirements of the backend application, I really need layer 4 LB criterion. Regards, -- Jérôme Benoit aka fraggle La Météo du Net - http://grenouille.com OpenPGP Key ID : 9FE9161D Key fingerprint : 9CA4 0249 AF57 A35B 34B3 AC15 FAA0 CB50 9FE9 161D signature.asc Description: PGP signature
Re: Active/active HAProxy
On Tue, 19 Mar 2013 07:04:02 +0100 in , Baptiste Baptiste wrote: Hello, > conntrack is a bad idea with haproxy ;) Could you elaborate ? Does HAProxy already fill the connection table of the underlying OS so conntrackd is just not required ? The connection tracking at the OS firewall level is not recommended for high traffic in order to ensure that the LB tiers running firewall and LB with HA will not drop a TCP session that get to finish on an other LB by the firewall (should never happen in theory, in practice, that might). I can yes disable the conntrack on the firewall on some VIP and let HAProxy handle gracefully the should never happen case that happen :p Thks. -- Jérôme Benoit aka fraggle La Météo du Net - http://grenouille.com OpenPGP Key ID : 9FE9161D Key fingerprint : 9CA4 0249 AF57 A35B 34B3 AC15 FAA0 CB50 9FE9 161D signature.asc Description: PGP signature
Re: Active/active HAProxy
On Tue, 19 Mar 2013 12:06:47 -0400 in <2cf188ce-49a7-43cc-aa21-81cdc10cd...@davidcoulson.net>, David Coulson David Coulson wrote: > > > On Mar 19, 2013, at 9:52 AM, Jérôme Benoit wrote: > > > > cheap hosting with no control on their backbone and network load on one > > box reach the max. > > So what happens when you lose a system? If you are doing active/active and > either/both systems are above 50% utilized, you're going to have an issue > when a failure occurs. I'll do back to a degraded state ... (that will hopefully not last too much), that really all I can do without access to the backbone equipment setup, wiring, etc. ... > You know you can run LVS on the same boxes as HAProxy? That wouldn't require > a topology change, at least from a network perspective. Hum, it will depend on the LVS setup since my main concern is network load repartition, that can do the trick with one VIP and LVS setup with DR from the HAProxies (it's just a replacement of RR DNS). The session initialization sequence (well the first packet) will still be handled just be one box. Sound better than RR DNS from a failover point of view. -- Jérôme Benoit aka fraggle La Météo du Net - http://grenouille.com OpenPGP Key ID : 9FE9161D Key fingerprint : 9CA4 0249 AF57 A35B 34B3 AC15 FAA0 CB50 9FE9 161D signature.asc Description: PGP signature
Re: Active/active HAProxy
On Tue, 19 Mar 2013 07:04:02 +0100 in , Baptiste Baptiste wrote: > Hi Jerome, > > Do you have any good reason to setup an active/active "cluster"? cheap hosting with no control on their backbone and network load on one box reach the max. > crossed VIPs hosted by VRRP is recommended for "simple" active/active > setup then as you mentioned, playing with DNS RR. > conntrack is a bad idea with haproxy ;) OK > > If you expect a massive traffic, it's better to use a first layer of > Layer 4 LoadBalancers using LVS or some routing protocols (or Cisco > ECMP). I can't change the network topology or network equipment setup. -- Jérôme Benoit aka fraggle La Météo du Net - http://grenouille.com OpenPGP Key ID : 9FE9161D Key fingerprint : 9CA4 0249 AF57 A35B 34B3 AC15 FAA0 CB50 9FE9 161D signature.asc Description: PGP signature
