Hi, One of the more obvious issues with direct routing and tunneling mode in general looks like this:
-the balanced service is running at a VIP address. -keepalived/ldirectord does perform some availability check on the realserver's management IP address, e.g. perform some HTTP request. -The realserver's management IP address is different than the VIP address, and it doesn't really tell that the VIP address is actually configured or the service is running on the VIP address. As a result, the balancer might be balancing load to a misconfigured realserver. At first sight, there are the following workarounds for this: -accept the situation as it is. E.g. binding the webserver to 0.0.0.0, connecting to one IP interface, assume it'll work anyway on any other interface as well. -switch to masquerading/NAT. This requires your loadbalancer to be your gateway, and your loadbalancer in turn would have to transport a lot more traffic than in DR/TUN mode, which impacts scalability. It also replaces the original destination IP address, and so a realserver probably won't know the original destination IP address. This is an issue if a realserver is serving multiple VIPs. -have a port forwarder running on the realserver, forwarding traffic for a specific port at the realserver's management IP address to the service port at the VIP. The balancer can connect to that port forward and check the "actual" service by using that port forward as a proxy. Risk: if the IP address is not configured on the realserver, the port forward may be routed to the actual VIP back to the loadbalancer, and so probably contact a working service at some other realserver. The check will succeed, and so the (misconfigured) realserver put into rotation - resulting in errors. -check locally on every realserver if the service is configured correctly, and whenever the balancer is doing its service checks at the management IP address, reply with the local result. This may be as simple as adding some check_http script to (x)inetd and asking e.g. keepalived to check for "OK" output on that tcp port. Same risk as in the port forwarding scenario. -manually craft packets on the loadbalancer to check the availability of the IP address or TCP service at the given realserver. The later one can be implemented by combining iproute2 with the "nping" utility from nmap: $ nping --arp --count 1 192.0.2.10 ... to get the current MAC address for realserver 192.0.2.10, e.g. 01:23:45:67:89:ab $ ip route get 192.0.2.10 gives the source IP address used for contacting the realserver, according to the balancer's local routing table for unspecified connections, e.g. 192.0.2.10 dev eth1 src 192.0.2.2 ... so an application requesting a connection to 192.0.2.10 without a specific source IP address would result in a connection from 192.0.2.2 to 192.0.2.10. Assuming a VIP address of 192.0.2.20, we can combine the crafted packet like this: $ nping --dest-mac 01:23:45:67:89:ab --ttl 1 --source-ip 192.0.2.2 --dest-ip 192.0.2.20 --count 1 --interface eth1 --icmp If the resulting nping output gives a line like this: RCVD .* ICMP [192.0.2.20 > 192.0.2.2 Echo reply .* ... we've successfully received an icmp reply from the balanced IP address and so know the balanced IP address is actually configured on this realserver. One may also use "nping --tcp" to craft a TCP SYN packet for the balanced IP address and check for the Syn/Ack-packet - which essentially tells that the realserver does not only have the IP address configured, but also does perform the next step of a full 3way-handshake: there's some application trying to accept incoming connections. I'm fully aware that this is not a fully working test, but it's worth an additional check to prevent certain kinds of misconfigurations. Another option for crafting a full test session is to create seperate network namespaces for every realserver, adding distinct routing tables within every single network namespace for the VIP to be routed via the management IP of a specific realserver, and perform every realserver's check within that specifific namespace. This involves setting up multiple veth-interfaces and bridging them with acutal interfaces, setting up multiple routing tables, ... I'm omitting an example :-) At best, I'd like to have two checks in keepalived: -verify every real server to respond at its VIP address (crafted icmp packet) -check some service at the real server's management IP address to confirm the application to be okay. Did anyone on this list come across those issues and did probably develop a solution for this? Best, Anders -- 1&1 Internet AG Expert Systems Architect (IT Operations) Brauerstrasse 50 v://49.721.91374.0 D-76135 Karlsruhe f://49.721.91374.225 Amtsgericht Montabaur HRB 6484 Vorstand: Ralph Dommermuth, Frank Einhellinger, Robert Hoffmann, Andreas Hofmann, Markus Huhn, Hans-Henning Kettler, Uwe Lamnek, Jan Oetjen, Christian Würst Aufsichtsratsvorsitzender: Michael Scheeren _______________________________________________ Please read the documentation before posting - it's available at: http://www.linuxvirtualserver.org/ LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org Send requests to lvs-users-requ...@linuxvirtualserver.org or go to http://lists.graemef.net/mailman/listinfo/lvs-users