Certainly and that makes sense, I will consolidate what I've emailed before with the additional information here.
# PC info: Linux 3.12.5 for real servers 1 and 2, and Linux 3.9.10 for the client box. There are 3 boxes total, client box, director/RIP1( real server 1) and RIP2 (real server 2): - client box: inet 172.17.0.2/16 brd 172.17.255.255 scope global eth1 #CIP - director which is the same as real server 1 (RIP1). The client is on a separate box. inet 172.17.0.16/16 brd 172.17.255.255 scope global bond0 #RIP1 inet 172.17.0.24/16 brd 172.17.255.255 scope global secondary bond0:2 #VIP - real server 2 (RIP2) inet 172.17.0.24/32 scope global lo:0 #VIP on loopback inet 172.17.0.17/16 brd 172.17.255.255 scope global bond0 #RIP2 # ipvs setup on real server 1 (RIP1) only ipvsadm -C ipvsadm -A -f 100 -s rr ipvsadm -a -f 100 -r 172.17.0.16 -w 100 ipvsadm -a -f 100 -r 172.17.0.17 -w 100 # iptable rules (these rules are set for both real server 1 and real server 2) iptables -t mangle -A PREROUTING -d 172.17.0.24/32 ! -i lo -p tcp -m tcp --dport 80 -j MARK --set-xmark 0x64/0xffffffff iptables -t nat -A PREROUTING -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 50000 iptables -t nat -A OUTPUT -o lo -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 50000 The test I'm conducting is an http get from the client box connecting to the VIP: - Issue the following command on the client box: curl -v 'http://172.17.0.24' On both real servers there is an nginx webserver listening on port 50000 I also turned on debugging and ran the curl command with port mapping using level 12 debug (this is output when the issue occurs of no load balancing). Debug output on real server 1 after executing the curl command the first time: Jan 24 23:05:44 pc01 kernel: IPVS: RR: server 172.17.0.17:0 activeconns 0 refcnt 1 weight 100 Jan 24 23:05:44 pc01 kernel: IPVS: Bind-dest TCP c:172.17.0.2:37455 v: 172.17.0.16:50130 d:172.17.0.17:50130 fwd:R s:65276 conn->flags:183 conn->refcnt:1 dest->refcnt:2 Jan 24 23:05:44 pc01 kernel: IPVS: Schedule fwd:R c:172.17.0.2:37455 v: 172.17.0.16:50130 d:172.17.0.17:50130 conn->flags:101C3 conn->refcnt:2 Jan 24 23:05:44 pc01 kernel: IPVS: TCP input [S...] 172.17.0.17:50130-> 172.17.0.2:37455 state: NONE->SYN_RECV conn->refcnt:2 Jan 24 23:05:44 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit, net/netfilter/ipvs/ip_vs_xmit.c line 1009 Jan 24 23:05:44 pc01 kernel: IPVS: Enter: ip_vs_out, net/netfilter/ipvs/ip_vs_core.c line 1116 Jan 24 23:05:44 pc01 kernel: IPVS: Leave: ip_vs_dr_xmit, net/netfilter/ipvs/ip_vs_xmit.c line 1031 Jan 24 23:05:44 pc01 kernel: IPVS: Enter: ip_vs_out, net/netfilter/ipvs/ip_vs_core.c line 1116 Jan 24 23:05:44 pc01 kernel: IPVS: Enter: ip_vs_out, net/netfilter/ipvs/ip_vs_core.c line 1116 Jan 24 23:05:44 pc01 kernel: IPVS: lookup/out TCP 172.17.0.16:50130-> 172.17.0.2:37455 not hit Jan 24 23:05:44 pc01 kernel: IPVS: lookup/in TCP 172.17.0.16:50130-> 172.17.0.2:37455 not hit Jan 24 23:05:44 pc01 kernel: IPVS: lookup service: fwm 0 TCP 172.17.0.2:37455 not hit Jan 24 23:05:44 pc01 kernel: IPVS: Enter: ip_vs_out, net/netfilter/ipvs/ip_vs_core.c line 1116 Jan 24 23:05:44 pc01 kernel: IPVS: lookup/out TCP 172.17.0.2:37455-> 172.17.0.16:50130 not hit Jan 24 23:05:44 pc01 kernel: IPVS: lookup/in TCP 172.17.0.2:37455-> 172.17.0.16:50130 hit Jan 24 23:05:44 pc01 kernel: IPVS: TCP input [..A.] 172.17.0.17:50130-> 172.17.0.2:37455 state: SYN_RECV->ESTABLISHED conn->refcnt:2 Jan 24 23:05:44 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit, net/netfilter/ipvs/ip_vs_xmit.c line 1009 Jan 24 23:05:44 pc01 kernel: IPVS: Enter: ip_vs_out, net/netfilter/ipvs/ip_vs_core.c line 1116 Jan 24 23:05:44 pc01 kernel: IPVS: Leave: ip_vs_dr_xmit, net/netfilter/ipvs/ip_vs_xmit.c line 1031 Jan 24 23:05:44 pc01 kernel: IPVS: Enter: ip_vs_out, net/netfilter/ipvs/ip_vs_core.c line 1116 Jan 24 23:05:44 pc01 kernel: IPVS: Enter: ip_vs_out, net/netfilter/ipvs/ip_vs_core.c line 1116 Jan 24 23:05:44 pc01 kernel: IPVS: lookup/out TCP 172.17.0.2:37455-> 172.17.0.16:50130 not hit Jan 24 23:05:44 pc01 kernel: IPVS: lookup/in TCP 172.17.0.2:37455-> 172.17.0.16:50130 hit Jan 24 23:05:44 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit, net/netfilter/ipvs/ip_vs_xmit.c line 1009 Jan 24 23:05:44 pc01 kernel: IPVS: Enter: ip_vs_out, net/netfilter/ipvs/ip_vs_core.c line 1116 Jan 24 23:05:44 pc01 kernel: IPVS: Leave: ip_vs_dr_xmit, net/netfilter/ipvs/ip_vs_xmit.c line 1031 Jan 24 23:05:44 pc01 kernel: IPVS: Enter: ip_vs_out, net/netfilter/ipvs/ip_vs_core.c line 1116 Jan 24 23:05:44 pc01 kernel: IPVS: Enter: ip_vs_out, net/netfilter/ipvs/ip_vs_core.c line 1116 Jan 24 23:05:44 pc01 kernel: IPVS: lookup/out TCP 172.17.0.16:50130-> 172.17.0.2:37455 not hit Jan 24 23:05:44 pc01 kernel: IPVS: lookup/in TCP 172.17.0.16:50130-> 172.17.0.2:37455 not hit Jan 24 23:05:44 pc01 kernel: IPVS: Enter: ip_vs_out, net/netfilter/ipvs/ip_vs_core.c line 1116 Jan 24 23:05:44 pc01 kernel: IPVS: lookup/out TCP 172.17.0.16:50130-> 172.17.0.2:37455 not hit Jan 24 23:05:44 pc01 kernel: IPVS: lookup/in TCP 172.17.0.16:50130-> 172.17.0.2:37455 not hit Jan 24 23:05:44 pc01 kernel: IPVS: Enter: ip_vs_out, net/netfilter/ipvs/ip_vs_core.c line 1116 Jan 24 23:05:44 pc01 kernel: IPVS: lookup/out TCP 172.17.0.2:37455-> 172.17.0.16:50130 not hit Jan 24 23:05:44 pc01 kernel: IPVS: lookup/in TCP 172.17.0.2:37455-> 172.17.0.16:50130 hit Jan 24 23:05:44 pc01 kernel: IPVS: TCP input [.FA.] 172.17.0.17:50130-> 172.17.0.2:37455 state: ESTABLISHED->FIN_WAIT conn->refcnt:2 Debug output on real server 1 after executing the curl command a second time: Jan 24 23:05:45 pc01 kernel: IPVS: ip_vs_rr_schedule(): Scheduling... Jan 24 23:05:45 pc01 kernel: IPVS: RR: server 172.17.0.16:0 activeconns 0 refcnt 1 weight 100 Jan 24 23:05:45 pc01 kernel: IPVS: Bind-dest TCP c:172.17.0.2:37456 v: 172.17.0.16:50130 d:172.17.0.16:50130 fwd:R s:65276 conn->flags:183 conn->refcnt:1 dest->refcnt:2 Jan 24 23:05:45 pc01 kernel: IPVS: Schedule fwd:R c:172.17.0.2:37456 v: 172.17.0.16:50130 d:172.17.0.16:50130 conn->flags:101C3 conn->refcnt:2 Jan 24 23:05:45 pc01 kernel: IPVS: TCP input [S...] 172.17.0.16:50130-> 172.17.0.2:37456 state: NONE->SYN_RECV conn->refcnt:2 Jan 24 23:05:45 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit, net/netfilter/ipvs/ip_vs_xmit.c line 1009 Jan 24 23:05:45 pc01 kernel: IPVS: Enter: ip_vs_out, net/netfilter/ipvs/ip_vs_core.c line 1116 Jan 24 23:05:45 pc01 kernel: IPVS: lookup/out TCP 172.17.0.16:50130-> 172.17.0.2:37456 hit Jan 24 23:05:45 pc01 kernel: IPVS: Leave: handle_response, net/netfilter/ipvs/ip_vs_core.c line 1094 Jan 24 23:05:45 pc01 kernel: IPVS: Enter: ip_vs_out, net/netfilter/ipvs/ip_vs_core.c line 1116 Jan 24 23:05:45 pc01 kernel: IPVS: lookup/out TCP 172.17.0.2:37456-> 172.17.0.16:50130 not hit Jan 24 23:05:45 pc01 kernel: IPVS: lookup/in TCP 172.17.0.2:37456-> 172.17.0.16:50130 hit Jan 24 23:05:45 pc01 kernel: IPVS: TCP input [..A.] 172.17.0.16:50130-> 172.17.0.2:37456 state: SYN_RECV->ESTABLISHED conn->refcnt:2 Jan 24 23:05:45 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit, net/netfilter/ipvs/ip_vs_xmit.c line 1009 Jan 24 23:05:45 pc01 kernel: IPVS: Enter: ip_vs_out, net/netfilter/ipvs/ip_vs_core.c line 1116 Jan 24 23:05:45 pc01 kernel: IPVS: lookup/out TCP 172.17.0.2:37456-> 172.17.0.16:50130 not hit Jan 24 23:05:45 pc01 kernel: IPVS: lookup/in TCP 172.17.0.2:37456-> 172.17.0.16:50130 hit Jan 24 23:05:45 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit, net/netfilter/ipvs/ip_vs_xmit.c line 1009 Jan 24 23:05:45 pc01 kernel: IPVS: Enter: ip_vs_out, net/netfilter/ipvs/ip_vs_core.c line 1116 Jan 24 23:05:45 pc01 kernel: IPVS: lookup/out TCP 172.17.0.16:50130-> 172.17.0.2:37456 hit Jan 24 23:05:45 pc01 kernel: IPVS: Leave: handle_response, net/netfilter/ipvs/ip_vs_core.c line 1094 Jan 24 23:05:45 pc01 kernel: IPVS: Enter: ip_vs_out, net/netfilter/ipvs/ip_vs_core.c line 1116 Jan 24 23:05:45 pc01 kernel: IPVS: lookup/out TCP 172.17.0.16:50130-> 172.17.0.2:37456 hit Jan 24 23:05:45 pc01 kernel: IPVS: TCP output [.FA.] 172.17.0.16:50130-> 172.17.0.2:37456 state: ESTABLISHED->FIN_WAIT conn->refcnt:2 Jan 24 23:05:45 pc01 kernel: IPVS: Leave: handle_response, net/netfilter/ipvs/ip_vs_core.c line 1094 Jan 24 23:05:45 pc01 kernel: IPVS: Enter: ip_vs_out, net/netfilter/ipvs/ip_vs_core.c line 1116 Jan 24 23:05:45 pc01 kernel: IPVS: lookup/out TCP 172.17.0.2:37456-> 172.17.0.16:50130 not hit Jan 24 23:05:45 pc01 kernel: IPVS: lookup/in TCP 172.17.0.2:37456-> 172.17.0.16:50130 hit Jan 24 23:05:45 pc01 kernel: IPVS: TCP input [.FA.] 172.17.0.16:50130-> 172.17.0.2:37456 state: FIN_WAIT->TIME_WAIT conn->refcnt:2 Jan 24 23:05:45 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit, net/netfilter/ipvs/ip_vs_xmit.c line 1009 Jan 24 23:05:45 pc01 kernel: IPVS: Enter: ip_vs_out, net/netfilter/ipvs/ip_vs_core.c line 1116 Jan 24 23:05:45 pc01 kernel: IPVS: lookup/out TCP 172.17.0.16:50130-> 172.17.0.2:37456 hit Jan 24 23:05:45 pc01 kernel: IPVS: Leave: handle_response, net/netfilter/ipvs/ip_vs_core.c line 1094 Jan 24 23:05:45 pc01 kernel: IPVS: Enter: ip_vs_out, net/netfilter/ipvs/ip_vs_core.c line 1116 Jan 24 23:05:45 pc01 kernel: IPVS: lookup/out UDP 172.17.0.16:50014-> 239.192.0.1:50015 not hit Jan 24 23:05:45 pc01 kernel: IPVS: packet type=2 proto=17 daddr=239.192.0.1 ignored in hook 1 Jan 24 23:05:45 pc01 kernel: IPVS: Enter: ip_vs_out, net/netfilter/ipvs/ip_vs_core.c line 1116 Jan 24 23:05:45 pc01 kernel: IPVS: lookup/out UDP 127.0.0.1:45176-> 127.0.0.1:53 not hit Jan 24 23:05:45 pc01 kernel: IPVS: lookup/in UDP 127.0.0.1:45176-> 127.0.0.1:53 not hit Jan 24 23:05:45 pc01 kernel: IPVS: lookup service: fwm 0 UDP 127.0.0.1:53not hit Below is an example of good results when connecting directly to port 50000. For this scenario I removed port 80 and updated iptables with fwmark for port 50000: iptables -t mangle -A PREROUTING -d 172.17.0.24/32 ! -i lo -p tcp -m tcp --dport 50000 -j MARK --set-xmark 0x64/0xffffffff Debug output on real server 1 when not port mapping first test (curl -v 'http://172.17.0.24:50000'): Jan 25 00:19:37 pc01 kernel: IPVS: ip_vs_rr_schedule(): Scheduling... Jan 25 00:19:37 pc01 kernel: IPVS: RR: server 172.17.0.17:0 activeconns 0 refcnt 1 weight 100 Jan 25 00:19:37 pc01 kernel: IPVS: Bind-dest TCP c:172.17.0.2:42815 v: 172.17.0.24:50130 d:172.17.0.17:50130 fwd:R s:4 conn->flags:183 conn->refcnt:1 dest->refcnt:2 Jan 25 00:19:37 pc01 kernel: IPVS: Schedule fwd:R c:172.17.0.2:42815 v: 172.17.0.24:50130 d:172.17.0.17:50130 conn->flags:101C3 conn->refcnt:2 Jan 25 00:19:37 pc01 kernel: IPVS: TCP input [S...] 172.17.0.17:50130-> 172.17.0.2:42815 state: NONE->SYN_RECV conn->refcnt:2 Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit, net/netfilter/ipvs/ip_vs_xmit.c line 1009 Jan 25 00:19:37 pc01 kernel: IPVS: new dst 172.17.0.17, src 172.17.0.16, refcnt=1 Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_out, net/netfilter/ipvs/ip_vs_core.c line 1116 Jan 25 00:19:37 pc01 kernel: IPVS: Leave: ip_vs_dr_xmit, net/netfilter/ipvs/ip_vs_xmit.c line 1031 Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_out, net/netfilter/ipvs/ip_vs_core.c line 1116 Jan 25 00:19:37 pc01 kernel: IPVS: lookup/out TCP 172.17.0.2:42815-> 172.17.0.24:50130 not hit Jan 25 00:19:37 pc01 kernel: IPVS: lookup/in TCP 172.17.0.2:42815-> 172.17.0.24:50130 hit Jan 25 00:19:37 pc01 kernel: IPVS: TCP input [..A.] 172.17.0.17:50130-> 172.17.0.2:42815 state: SYN_RECV->ESTABLISHED conn->refcnt:2 Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit, net/netfilter/ipvs/ip_vs_xmit.c line 1009 Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_out, net/netfilter/ipvs/ip_vs_core.c line 1116 Jan 25 00:19:37 pc01 kernel: IPVS: Leave: ip_vs_dr_xmit, net/netfilter/ipvs/ip_vs_xmit.c line 1031 Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_out, net/netfilter/ipvs/ip_vs_core.c line 1116 Jan 25 00:19:37 pc01 kernel: IPVS: lookup/out TCP 172.17.0.2:42815-> 172.17.0.24:50130 not hit Jan 25 00:19:37 pc01 kernel: IPVS: lookup/in TCP 172.17.0.2:42815-> 172.17.0.24:50130 hit Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit, net/netfilter/ipvs/ip_vs_xmit.c line 1009 Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_out, net/netfilter/ipvs/ip_vs_core.c line 1116 Jan 25 00:19:37 pc01 kernel: IPVS: Leave: ip_vs_dr_xmit, net/netfilter/ipvs/ip_vs_xmit.c line 1031 Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_out, net/netfilter/ipvs/ip_vs_core.c line 1116 Jan 25 00:19:37 pc01 kernel: IPVS: lookup/out TCP 172.17.0.2:42815-> 172.17.0.24:50130 not hit Jan 25 00:19:37 pc01 kernel: IPVS: lookup/in TCP 172.17.0.2:42815-> 172.17.0.24:50130 hit Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit, net/netfilter/ipvs/ip_vs_xmit.c line 1009 Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_out, net/netfilter/ipvs/ip_vs_core.c line 1116 Jan 25 00:19:37 pc01 kernel: IPVS: Leave: ip_vs_dr_xmit, net/netfilter/ipvs/ip_vs_xmit.c line 1031 Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_out, net/netfilter/ipvs/ip_vs_core.c line 1116 Jan 25 00:19:37 pc01 kernel: IPVS: lookup/out TCP 172.17.0.2:42815-> 172.17.0.24:50130 not hit Jan 25 00:19:37 pc01 kernel: IPVS: lookup/in TCP 172.17.0.2:42815-> 172.17.0.24:50130 hit Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit, net/netfilter/ipvs/ip_vs_xmit.c line 1009 Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_out, net/netfilter/ipvs/ip_vs_core.c line 1116 Jan 25 00:19:37 pc01 kernel: IPVS: Leave: ip_vs_dr_xmit, net/netfilter/ipvs/ip_vs_xmit.c line 1031 Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_out, net/netfilter/ipvs/ip_vs_core.c line 1116 Jan 25 00:19:37 pc01 kernel: IPVS: lookup/out TCP 172.17.0.2:42815-> 172.17.0.24:50130 not hit Jan 25 00:19:37 pc01 kernel: IPVS: lookup/in TCP 172.17.0.2:42815-> 172.17.0.24:50130 hit Jan 25 00:19:37 pc01 kernel: IPVS: TCP input [.FA.] 172.17.0.17:50130-> 172.17.0.2:42815 state: ESTABLISHED->FIN_WAIT conn->refcnt:2 Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit, net/netfilter/ipvs/ip_vs_xmit.c line 1009 Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_out, net/netfilter/ipvs/ip_vs_core.c line 1116 Debug output on real server 1 when not port mapping second test (curl -v 'http://172.17.0.24:50000'): Jan 25 00:19:39 pc01 kernel: IPVS: ip_vs_rr_schedule(): Scheduling... Jan 25 00:19:39 pc01 kernel: IPVS: RR: server 172.17.0.16:0 activeconns 0 refcnt 1 weight 100 Jan 25 00:19:39 pc01 kernel: IPVS: Bind-dest TCP c:172.17.0.2:42816 v: 172.17.0.24:50130 d:172.17.0.16:50130 fwd:R s:65276 conn->flags:183 conn->refcnt:1 dest->refcnt:2 Jan 25 00:19:39 pc01 kernel: IPVS: Schedule fwd:R c:172.17.0.2:42816 v: 172.17.0.24:50130 d:172.17.0.16:50130 conn->flags:101C3 conn->refcnt:2 Jan 25 00:19:39 pc01 kernel: IPVS: TCP input [S...] 172.17.0.16:50130-> 172.17.0.2:42816 state: NONE->SYN_RECV conn->refcnt:2 Jan 25 00:19:39 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit, net/netfilter/ipvs/ip_vs_xmit.c line 1009 Jan 25 00:19:39 pc01 kernel: IPVS: new dst 172.17.0.16, src 172.17.0.16, refcnt=1 Jan 25 00:19:39 pc01 kernel: IPVS: Enter: ip_vs_out, net/netfilter/ipvs/ip_vs_core.c line 1116 Jan 25 00:19:39 pc01 kernel: IPVS: lookup/out TCP 172.17.0.24:50130-> 172.17.0.2:42816 not hit Jan 25 00:19:39 pc01 kernel: IPVS: lookup/in TCP 172.17.0.24:50130-> 172.17.0.2:42816 not hit Jan 25 00:19:39 pc01 kernel: IPVS: lookup service: fwm 0 TCP 172.17.0.2:42816 not hit Jan 25 00:19:39 pc01 kernel: IPVS: Enter: ip_vs_out, net/netfilter/ipvs/ip_vs_core.c line 1116 Jan 25 00:19:39 pc01 kernel: IPVS: lookup/out TCP 172.17.0.2:42816-> 172.17.0.24:50130 not hit Jan 25 00:19:39 pc01 kernel: IPVS: lookup/in TCP 172.17.0.2:42816-> 172.17.0.24:50130 hit Jan 25 00:19:39 pc01 kernel: IPVS: TCP input [..A.] 172.17.0.16:50130-> 172.17.0.2:42816 state: SYN_RECV->ESTABLISHED conn->refcnt:2 Jan 25 00:19:39 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit, net/netfilter/ipvs/ip_vs_xmit.c line 1009 Jan 25 00:19:39 pc01 kernel: IPVS: Enter: ip_vs_out, net/netfilter/ipvs/ip_vs_core.c line 1116 Jan 25 00:19:39 pc01 kernel: IPVS: lookup/out TCP 172.17.0.2:42816-> 172.17.0.24:50130 not hit Jan 25 00:19:39 pc01 kernel: IPVS: lookup/in TCP 172.17.0.2:42816-> 172.17.0.24:50130 hit Jan 25 00:19:39 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit, net/netfilter/ipvs/ip_vs_xmit.c line 1009 Jan 25 00:19:39 pc01 kernel: IPVS: Enter: ip_vs_out, net/netfilter/ipvs/ip_vs_core.c line 1116 Jan 25 00:19:39 pc01 kernel: IPVS: lookup/out TCP 172.17.0.24:50130-> 172.17.0.2:42816 not hit Jan 25 00:19:39 pc01 kernel: IPVS: lookup/in TCP 172.17.0.24:50130-> 172.17.0.2:42816 not hit Jan 25 00:19:39 pc01 kernel: IPVS: Enter: ip_vs_out, net/netfilter/ipvs/ip_vs_core.c line 1116 Jan 25 00:19:39 pc01 kernel: IPVS: lookup/out TCP 172.17.0.24:50130-> 172.17.0.2:42816 not hit Jan 25 00:19:39 pc01 kernel: IPVS: lookup/in TCP 172.17.0.24:50130-> 172.17.0.2:42816 not hit Jan 25 00:19:39 pc01 kernel: IPVS: Enter: ip_vs_out, net/netfilter/ipvs/ip_vs_core.c line 1116 Jan 25 00:19:39 pc01 kernel: IPVS: lookup/out TCP 172.17.0.2:42816-> 172.17.0.24:50130 not hit Jan 25 00:19:39 pc01 kernel: IPVS: lookup/in TCP 172.17.0.2:42816-> 172.17.0.24:50130 hit Jan 25 00:19:39 pc01 kernel: IPVS: TCP input [.FA.] 172.17.0.16:50130-> 172.17.0.2:42816 state: ESTABLISHED->FIN_WAIT conn->refcnt:2 Jan 25 00:19:39 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit, net/netfilter/ipvs/ip_vs_xmit.c line 1009 Jan 25 00:19:39 pc01 kernel: IPVS: Enter: ip_vs_out, net/netfilter/ipvs/ip_vs_core.c line 1116 Jan 25 00:19:39 pc01 kernel: IPVS: lookup/out TCP 172.17.0.24:50130-> 172.17.0.2:42816 not hit Jan 25 00:19:39 pc01 kernel: IPVS: lookup/in TCP 172.17.0.24:50130-> 172.17.0.2:42816 not hit Jan 25 00:19:39 pc01 kernel: IPVS: Enter: ip_vs_out, net/netfilter/ipvs/ip_vs_core.c line 1116 Jan 25 00:19:39 pc01 kernel: IPVS: lookup/out TCP 172.17.0.16:39545-> 172.17.0.16:3306 not hit The tcpdump command that was used was as follows on real server 1: tcpdump -iany -nn port 80 or port 50000 I realized later that using 'any' device isn't as helpful when trying to pinpoint loopback traffic, so that's what my follow up email was referring to. Thanks again for the support, feel free to ask for any additional information to help debug. Jacoby On Sat, Jan 25, 2014 at 6:25 AM, Julian Anastasov <j...@ssi.bg> wrote: > > Hello, > > On Thu, 23 Jan 2014, Jacoby Hickerson wrote: > > > Just to clarify the packets are going to the loopback of node 1, when > they > > should be going to node 2. This is shown in the tcpdump output:Here is > the > > output from the lo device of the first node: > > 02:10:51.987030 IP 172.17.0.2.54276 > 172.17.0.16.50000: Flags [.], ack > > 2970678458, win 115, options [nop,nop,TS val 3044575793 ecr 978483], > length > > 0 > > 02:10:51.987079 IP 172.17.0.2.54276 > 172.17.0.16.50000: Flags [P.], seq > > 0:173, ack 1, win 115, options [nop,nop,TS val 3044575793 ecr 978483], > > length 173 > > 02:10:51.987426 IP 172.17.0.2.54276 > 172.17.0.16.50000: Flags [.], ack > 2, > > win 115, options [nop,nop,TS val 3044575793 ecr 978484], length 0 > > 02:10:51.987480 IP 172.17.0.2.54276 > 172.17.0.16.50000: Flags [F.], seq > > 173, ack 2, win 115, options [nop,nop,TS val 3044575793 ecr 978484], > length > > 0 > > ... > > > Packets are being sent from the RIP of the first node only. From my > > understanding when using DR OutPkts should always be zero. > > When LocalNode (local RIP) is used, we can see > the local reply in LOCAL_OUT hook. It happens for NAT but > also for DR. So, it is normal. But we see these replies > after DNAT in LOCAL_OUT, see ip_vs_ops[] for reference. > > > The end result is that the packets are always coming from the first > > node and never balanced to the second node. > > > > Thanks for any further help, seems the solution is really close! > > Can you provide more understandable description > for the test, for example: > > - client box: > IP1: X.X.X.X/N dev DEV > IP2: ... > > - director: > IP1: ... > VIP: XXX > are client and director same box > > - real server: > IP1: ... > > iptable rules used. By this way I can try to > duplicate the problem. Now I see some IPs in tcpdump > output but I'm not sure what kind of traffic is shown, > where is started the tcpdump, on what box, on what > interface, external, internal... > > Regards > > -- > Julian Anastasov <j...@ssi.bg> > _______________________________________________ Please read the documentation before posting - it's available at: http://www.linuxvirtualserver.org/ LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org Send requests to lvs-users-requ...@linuxvirtualserver.org or go to http://lists.graemef.net/mailman/listinfo/lvs-users
