On 15.02.2015, Barry Haycock wrote: > I am looking to build for a client an Active/Active load balancer supporting > ~5000 machines
Your question isn't something I'd expect any FAQ for, it is more of an advanced configuration, and does raise multiple concerns. Active/Active: is there a strong reason for doing so? Why I'm asking: active/active may result in oversubscription, and one needs to very carefully keep that in mind, along with various other parameters. Otherwise, you may be running with two loadbalancers at 45% of capacity each and discover just in the second where one of both balancers fails, the combined capacity of 90% exceeds the typical 70%-threshhold where things start stalling. Regarding 5000 machines: unless you're looking into tunneling (which does have its own merits), IPVS/LVS requires you to have all of your loadbalancers and real servers on the same layer 2 network(s). A single broken or misbehaving network card can flood the same L2 network. Such floods may be hard to track down, but will surely result in downtime for your whole server farm - that's just one of the many aspects why about every network engineer tries to stay well below thousands of servers per layer-2-domain. One can attempt to work around this topic by attaching multiple VLANs to the same loadbalancer, and distribute real servers onto those different VLANs, but this doesn't really decrease complexity. Maybe your network design requires some more thoughts, maybe I'm just worrying about some missing information. IPVS/LVS "out of the box" (default kernel configuration) uses a connection table size CONFIG_IP_VS_TAB_BITS=12, which equals 4096 records; with 5000 machines, that's most likely something you'd like to tune (the help text for CONFIG_IP_VS_TAB_BITS tells how). However, I wouldn't want to have more than e.g. 50 machines behind a single loadbalancer pair: typically, involved traffic and bandwith can saturate enough capacity. Technically, I'd expect IPVS to handle hundreds, possibly thousands of real servers. I just don't come up with a box being able to handle the resulting possible traffic. > Previously we have looked at implementing this solution using > pacemaker/corosync/haproxy on RHEL 6.4 but that solution does not support > syslog/514/UDP. UDP-based protocols do assume the traffic may be dropped silently or the application will ensure retransmit. syslog doesn't care about re-transmits, and so does choose to have its traffic dropped. If you're logging traffic remotely for auditing, intrusion detection or anything else which case about the logged information: you probably will want to avoid UDP-based remote syslog at all. About any syslog implementation (rsyslog, syslog-ng, ...) updated during the past decade does implement syslog on arbitary tcp ports, which may save you from this hazzle. rsyslog also does offer its own protocol (RELP) for reliable transmission. > Looking at LVS there is support for UDP as well as maintaining the TCP state > which is the other major requirement. > > My question is, since we are only load balancing UDP/TCP rsyslog & HTTPS > (Posting to a single URL), would there be a requirement to install keepalived > ontop of LVS? From my view it appears to add another layer of complexity for > no gain that I can see. keepalived does configure IPVS, the same kernel component used by LVS; so it's not another layer of complexity, it's exactly the same layer, but you may just using different tools, depending on what you're expecting :) > I have been searching for a configuration for an active/active LVS solution > and while there are people stating that they are using it I haven't been able > to find any information on the configuration of a Active/Active load balancer. There are multiple approaches regarding "active/active". keepalived's high availability concept relies on VRRP, which at a first look is a pure active/passive-technology (there's just one active master, and everyone else is passive backup). However, you could turn this into active-active: - your service uses multiple IP addresses. load balancer A does handle IP X, load balancer B does handle IP Y, if any of them fails, the other one also handles the failed IP. This behaviour should be achievable by using multiple vrrp syncgroups with different states per balancer (syncgroup C is master on node A and backup on node B, syncgroup D is backup on node A and master on node B). Of course, your DNS record needs to show both IPs. service.example.tld. IN A X service.example.tld. IN A Y Downside: your traffic may not be very evenly split. Over time and with many different clients, it is somehow an even split (50:50), but it may als sometimes become 45:55 or even 40:60. It does also depend on client behaviour, and so that's not exactly what you may be asking for. - you're using some other way than ARP to attract incoming network traffic, e.g. by using specific routing protocols (like BGP, OSPF, ...). Such routing protocols can be used to perform equal-cost multipathing on the network layer: your loadbalancers are announcing the balanced IPs to your network router. Your network router in turn does check the source IP address of every IP address, and distributes e.g. the "odd" IP addresses to one loadbalancer node, the "even" ones to a different loadbalancer node. So from some perspective, this will introduce another layer of "loadbalancing". Depending on used equipment and configuration, you could have e.g. 16 loadbalancer nodes announcing the same IP address to your network, and all of them were active at the same time. http://sysadvent.blogspot.com/2014/12/day-11-turning-off-pacemaker-load.html does have a nice introduction into this topic. Anders -- 1&1 Internet AG Expert Systems Architect (IT Operations) Brauerstrasse 50 v://49.721.91374.0 D-76135 Karlsruhe f://49.721.91374.225 Amtsgericht Montabaur HRB 6484 Vorstand: Frank Einhellinger, Robert Hoffmann, Markus Huhn, Hans-Henning Kettler, Uwe Lamnek Aufsichtsratsvorsitzender: Michael Scheeren _______________________________________________ Please read the documentation before posting - it's available at: http://www.linuxvirtualserver.org/ LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org Send requests to lvs-users-requ...@linuxvirtualserver.org or go to http://lists.graemef.net/mailman/listinfo/lvs-users