Apologies, the debug output showing port 50130 should be 50000 ex: IPVS: lookup/in TCP 172.17.0.24:*50130*->172.17.0.2:42816 not hit should be: IPVS: lookup/in TCP 172.17.0.24:*50000*->172.17.0.2:42816 not hit
I have attached the file 'ipvs_debug_output' with correct debug so that this thread is not cluttered. Thanks! Jacoby On Mon, Jan 27, 2014 at 4:00 PM, Jacoby Hickerson <hickerso...@gmail.com>wrote: > Certainly and that makes sense, I will consolidate what I've emailed > before with the additional information here. > > # PC info: Linux 3.12.5 for real servers 1 and 2, and Linux 3.9.10 for the > client box. > > There are 3 boxes total, client box, director/RIP1( real server 1) and > RIP2 (real server 2): > - client box: > inet 172.17.0.2/16 brd 172.17.255.255 scope global eth1 #CIP > > - director which is the same as real server 1 (RIP1). The client is on a > separate box. > inet 172.17.0.16/16 brd 172.17.255.255 scope global bond0 > #RIP1 > inet 172.17.0.24/16 brd 172.17.255.255 scope global secondary bond0:2 > #VIP > > - real server 2 (RIP2) > inet 172.17.0.24/32 scope global lo:0 #VIP on > loopback > inet 172.17.0.17/16 brd 172.17.255.255 scope global bond0 #RIP2 > > # ipvs setup on real server 1 (RIP1) only > ipvsadm -C > ipvsadm -A -f 100 -s rr > ipvsadm -a -f 100 -r 172.17.0.16 -w 100 > ipvsadm -a -f 100 -r 172.17.0.17 -w 100 > > # iptable rules (these rules are set for both real server 1 and real > server 2) > iptables -t mangle -A PREROUTING -d 172.17.0.24/32 ! -i lo -p tcp -m tcp > --dport 80 -j MARK --set-xmark 0x64/0xffffffff > iptables -t nat -A PREROUTING -p tcp -m tcp --dport 80 -j REDIRECT > --to-ports 50000 > iptables -t nat -A OUTPUT -o lo -p tcp -m tcp --dport 80 -j REDIRECT > --to-ports 50000 > > The test I'm conducting is an http get from the client box connecting to > the VIP: > - Issue the following command on the client box: > curl -v 'http://172.17.0.24' > > On both real servers there is an nginx webserver listening on port 50000 > > I also turned on debugging and ran the curl command with port mapping > using level 12 debug (this is output when the issue occurs of no load > balancing). > Debug output on real server 1 after executing the curl command the first > time: > > Jan 24 23:05:44 pc01 kernel: IPVS: RR: server 172.17.0.17:0 activeconns 0 > refcnt 1 weight 100 > Jan 24 23:05:44 pc01 kernel: IPVS: Bind-dest TCP c:172.17.0.2:37455 v: > 172.17.0.16:50130 d:172.17.0.17:50130 fwd:R s:65276 conn->flags:183 > conn->refcnt:1 dest->refcnt:2 > Jan 24 23:05:44 pc01 kernel: IPVS: Schedule fwd:R c:172.17.0.2:37455 v: > 172.17.0.16:50130 d:172.17.0.17:50130 conn->flags:101C3 conn->refcnt:2 > Jan 24 23:05:44 pc01 kernel: IPVS: TCP input [S...] 172.17.0.17:50130-> > 172.17.0.2:37455 state: NONE->SYN_RECV conn->refcnt:2 > Jan 24 23:05:44 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit, > net/netfilter/ipvs/ip_vs_xmit.c line 1009 > Jan 24 23:05:44 pc01 kernel: IPVS: Enter: ip_vs_out, > net/netfilter/ipvs/ip_vs_core.c line 1116 > Jan 24 23:05:44 pc01 kernel: IPVS: Leave: ip_vs_dr_xmit, > net/netfilter/ipvs/ip_vs_xmit.c line 1031 > Jan 24 23:05:44 pc01 kernel: IPVS: Enter: ip_vs_out, > net/netfilter/ipvs/ip_vs_core.c line 1116 > Jan 24 23:05:44 pc01 kernel: IPVS: Enter: ip_vs_out, > net/netfilter/ipvs/ip_vs_core.c line 1116 > Jan 24 23:05:44 pc01 kernel: IPVS: lookup/out TCP 172.17.0.16:50130-> > 172.17.0.2:37455 not hit > Jan 24 23:05:44 pc01 kernel: IPVS: lookup/in TCP 172.17.0.16:50130-> > 172.17.0.2:37455 not hit > Jan 24 23:05:44 pc01 kernel: IPVS: lookup service: fwm 0 TCP > 172.17.0.2:37455 not hit > Jan 24 23:05:44 pc01 kernel: IPVS: Enter: ip_vs_out, > net/netfilter/ipvs/ip_vs_core.c line 1116 > Jan 24 23:05:44 pc01 kernel: IPVS: lookup/out TCP 172.17.0.2:37455-> > 172.17.0.16:50130 not hit > Jan 24 23:05:44 pc01 kernel: IPVS: lookup/in TCP 172.17.0.2:37455-> > 172.17.0.16:50130 hit > Jan 24 23:05:44 pc01 kernel: IPVS: TCP input [..A.] 172.17.0.17:50130-> > 172.17.0.2:37455 state: SYN_RECV->ESTABLISHED conn->refcnt:2 > Jan 24 23:05:44 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit, > net/netfilter/ipvs/ip_vs_xmit.c line 1009 > Jan 24 23:05:44 pc01 kernel: IPVS: Enter: ip_vs_out, > net/netfilter/ipvs/ip_vs_core.c line 1116 > Jan 24 23:05:44 pc01 kernel: IPVS: Leave: ip_vs_dr_xmit, > net/netfilter/ipvs/ip_vs_xmit.c line 1031 > Jan 24 23:05:44 pc01 kernel: IPVS: Enter: ip_vs_out, > net/netfilter/ipvs/ip_vs_core.c line 1116 > Jan 24 23:05:44 pc01 kernel: IPVS: Enter: ip_vs_out, > net/netfilter/ipvs/ip_vs_core.c line 1116 > Jan 24 23:05:44 pc01 kernel: IPVS: lookup/out TCP 172.17.0.2:37455-> > 172.17.0.16:50130 not hit > Jan 24 23:05:44 pc01 kernel: IPVS: lookup/in TCP 172.17.0.2:37455-> > 172.17.0.16:50130 hit > Jan 24 23:05:44 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit, > net/netfilter/ipvs/ip_vs_xmit.c line 1009 > Jan 24 23:05:44 pc01 kernel: IPVS: Enter: ip_vs_out, > net/netfilter/ipvs/ip_vs_core.c line 1116 > Jan 24 23:05:44 pc01 kernel: IPVS: Leave: ip_vs_dr_xmit, > net/netfilter/ipvs/ip_vs_xmit.c line 1031 > Jan 24 23:05:44 pc01 kernel: IPVS: Enter: ip_vs_out, > net/netfilter/ipvs/ip_vs_core.c line 1116 > Jan 24 23:05:44 pc01 kernel: IPVS: Enter: ip_vs_out, > net/netfilter/ipvs/ip_vs_core.c line 1116 > Jan 24 23:05:44 pc01 kernel: IPVS: lookup/out TCP 172.17.0.16:50130-> > 172.17.0.2:37455 not hit > Jan 24 23:05:44 pc01 kernel: IPVS: lookup/in TCP 172.17.0.16:50130-> > 172.17.0.2:37455 not hit > Jan 24 23:05:44 pc01 kernel: IPVS: Enter: ip_vs_out, > net/netfilter/ipvs/ip_vs_core.c line 1116 > Jan 24 23:05:44 pc01 kernel: IPVS: lookup/out TCP 172.17.0.16:50130-> > 172.17.0.2:37455 not hit > Jan 24 23:05:44 pc01 kernel: IPVS: lookup/in TCP 172.17.0.16:50130-> > 172.17.0.2:37455 not hit > Jan 24 23:05:44 pc01 kernel: IPVS: Enter: ip_vs_out, > net/netfilter/ipvs/ip_vs_core.c line 1116 > Jan 24 23:05:44 pc01 kernel: IPVS: lookup/out TCP 172.17.0.2:37455-> > 172.17.0.16:50130 not hit > Jan 24 23:05:44 pc01 kernel: IPVS: lookup/in TCP 172.17.0.2:37455-> > 172.17.0.16:50130 hit > Jan 24 23:05:44 pc01 kernel: IPVS: TCP input [.FA.] 172.17.0.17:50130-> > 172.17.0.2:37455 state: ESTABLISHED->FIN_WAIT conn->refcnt:2 > > Debug output on real server 1 after executing the curl command a second > time: > > Jan 24 23:05:45 pc01 kernel: IPVS: ip_vs_rr_schedule(): Scheduling... > Jan 24 23:05:45 pc01 kernel: IPVS: RR: server 172.17.0.16:0 activeconns 0 > refcnt 1 weight 100 > Jan 24 23:05:45 pc01 kernel: IPVS: Bind-dest TCP c:172.17.0.2:37456 v: > 172.17.0.16:50130 d:172.17.0.16:50130 fwd:R s:65276 conn->flags:183 > conn->refcnt:1 dest->refcnt:2 > Jan 24 23:05:45 pc01 kernel: IPVS: Schedule fwd:R c:172.17.0.2:37456 v: > 172.17.0.16:50130 d:172.17.0.16:50130 conn->flags:101C3 conn->refcnt:2 > Jan 24 23:05:45 pc01 kernel: IPVS: TCP input [S...] 172.17.0.16:50130-> > 172.17.0.2:37456 state: NONE->SYN_RECV conn->refcnt:2 > Jan 24 23:05:45 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit, > net/netfilter/ipvs/ip_vs_xmit.c line 1009 > Jan 24 23:05:45 pc01 kernel: IPVS: Enter: ip_vs_out, > net/netfilter/ipvs/ip_vs_core.c line 1116 > Jan 24 23:05:45 pc01 kernel: IPVS: lookup/out TCP 172.17.0.16:50130-> > 172.17.0.2:37456 hit > Jan 24 23:05:45 pc01 kernel: IPVS: Leave: handle_response, > net/netfilter/ipvs/ip_vs_core.c line 1094 > Jan 24 23:05:45 pc01 kernel: IPVS: Enter: ip_vs_out, > net/netfilter/ipvs/ip_vs_core.c line 1116 > Jan 24 23:05:45 pc01 kernel: IPVS: lookup/out TCP 172.17.0.2:37456-> > 172.17.0.16:50130 not hit > Jan 24 23:05:45 pc01 kernel: IPVS: lookup/in TCP 172.17.0.2:37456-> > 172.17.0.16:50130 hit > Jan 24 23:05:45 pc01 kernel: IPVS: TCP input [..A.] 172.17.0.16:50130-> > 172.17.0.2:37456 state: SYN_RECV->ESTABLISHED conn->refcnt:2 > Jan 24 23:05:45 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit, > net/netfilter/ipvs/ip_vs_xmit.c line 1009 > Jan 24 23:05:45 pc01 kernel: IPVS: Enter: ip_vs_out, > net/netfilter/ipvs/ip_vs_core.c line 1116 > Jan 24 23:05:45 pc01 kernel: IPVS: lookup/out TCP 172.17.0.2:37456-> > 172.17.0.16:50130 not hit > Jan 24 23:05:45 pc01 kernel: IPVS: lookup/in TCP 172.17.0.2:37456-> > 172.17.0.16:50130 hit > Jan 24 23:05:45 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit, > net/netfilter/ipvs/ip_vs_xmit.c line 1009 > Jan 24 23:05:45 pc01 kernel: IPVS: Enter: ip_vs_out, > net/netfilter/ipvs/ip_vs_core.c line 1116 > Jan 24 23:05:45 pc01 kernel: IPVS: lookup/out TCP 172.17.0.16:50130-> > 172.17.0.2:37456 hit > Jan 24 23:05:45 pc01 kernel: IPVS: Leave: handle_response, > net/netfilter/ipvs/ip_vs_core.c line 1094 > Jan 24 23:05:45 pc01 kernel: IPVS: Enter: ip_vs_out, > net/netfilter/ipvs/ip_vs_core.c line 1116 > Jan 24 23:05:45 pc01 kernel: IPVS: lookup/out TCP 172.17.0.16:50130-> > 172.17.0.2:37456 hit > Jan 24 23:05:45 pc01 kernel: IPVS: TCP output [.FA.] 172.17.0.16:50130-> > 172.17.0.2:37456 state: ESTABLISHED->FIN_WAIT conn->refcnt:2 > Jan 24 23:05:45 pc01 kernel: IPVS: Leave: handle_response, > net/netfilter/ipvs/ip_vs_core.c line 1094 > Jan 24 23:05:45 pc01 kernel: IPVS: Enter: ip_vs_out, > net/netfilter/ipvs/ip_vs_core.c line 1116 > Jan 24 23:05:45 pc01 kernel: IPVS: lookup/out TCP 172.17.0.2:37456-> > 172.17.0.16:50130 not hit > Jan 24 23:05:45 pc01 kernel: IPVS: lookup/in TCP 172.17.0.2:37456-> > 172.17.0.16:50130 hit > Jan 24 23:05:45 pc01 kernel: IPVS: TCP input [.FA.] 172.17.0.16:50130-> > 172.17.0.2:37456 state: FIN_WAIT->TIME_WAIT conn->refcnt:2 > Jan 24 23:05:45 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit, > net/netfilter/ipvs/ip_vs_xmit.c line 1009 > Jan 24 23:05:45 pc01 kernel: IPVS: Enter: ip_vs_out, > net/netfilter/ipvs/ip_vs_core.c line 1116 > Jan 24 23:05:45 pc01 kernel: IPVS: lookup/out TCP 172.17.0.16:50130-> > 172.17.0.2:37456 hit > Jan 24 23:05:45 pc01 kernel: IPVS: Leave: handle_response, > net/netfilter/ipvs/ip_vs_core.c line 1094 > Jan 24 23:05:45 pc01 kernel: IPVS: Enter: ip_vs_out, > net/netfilter/ipvs/ip_vs_core.c line 1116 > Jan 24 23:05:45 pc01 kernel: IPVS: lookup/out UDP 172.17.0.16:50014-> > 239.192.0.1:50015 not hit > Jan 24 23:05:45 pc01 kernel: IPVS: packet type=2 proto=17 > daddr=239.192.0.1 ignored in hook 1 > Jan 24 23:05:45 pc01 kernel: IPVS: Enter: ip_vs_out, > net/netfilter/ipvs/ip_vs_core.c line 1116 > Jan 24 23:05:45 pc01 kernel: IPVS: lookup/out UDP 127.0.0.1:45176-> > 127.0.0.1:53 not hit > Jan 24 23:05:45 pc01 kernel: IPVS: lookup/in UDP 127.0.0.1:45176-> > 127.0.0.1:53 not hit > Jan 24 23:05:45 pc01 kernel: IPVS: lookup service: fwm 0 UDP 127.0.0.1:53not > hit > > Below is an example of good results when connecting directly to port > 50000. For this scenario I removed port 80 and updated iptables with > fwmark for port 50000: > iptables -t mangle -A PREROUTING -d 172.17.0.24/32 ! -i lo -p tcp -m tcp > --dport 50000 -j MARK --set-xmark 0x64/0xffffffff > > Debug output on real server 1 when not port mapping first test (curl -v > 'http://172.17.0.24:50000'): > > Jan 25 00:19:37 pc01 kernel: IPVS: ip_vs_rr_schedule(): Scheduling... > Jan 25 00:19:37 pc01 kernel: IPVS: RR: server 172.17.0.17:0 activeconns 0 > refcnt 1 weight 100 > Jan 25 00:19:37 pc01 kernel: IPVS: Bind-dest TCP c:172.17.0.2:42815 v: > 172.17.0.24:50130 d:172.17.0.17:50130 fwd:R s:4 conn->flags:183 > conn->refcnt:1 dest->refcnt:2 > Jan 25 00:19:37 pc01 kernel: IPVS: Schedule fwd:R c:172.17.0.2:42815 v: > 172.17.0.24:50130 d:172.17.0.17:50130 conn->flags:101C3 conn->refcnt:2 > Jan 25 00:19:37 pc01 kernel: IPVS: TCP input [S...] 172.17.0.17:50130-> > 172.17.0.2:42815 state: NONE->SYN_RECV conn->refcnt:2 > Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit, > net/netfilter/ipvs/ip_vs_xmit.c line 1009 > Jan 25 00:19:37 pc01 kernel: IPVS: new dst 172.17.0.17, src 172.17.0.16, > refcnt=1 > Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_out, > net/netfilter/ipvs/ip_vs_core.c line 1116 > Jan 25 00:19:37 pc01 kernel: IPVS: Leave: ip_vs_dr_xmit, > net/netfilter/ipvs/ip_vs_xmit.c line 1031 > Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_out, > net/netfilter/ipvs/ip_vs_core.c line 1116 > Jan 25 00:19:37 pc01 kernel: IPVS: lookup/out TCP 172.17.0.2:42815-> > 172.17.0.24:50130 not hit > Jan 25 00:19:37 pc01 kernel: IPVS: lookup/in TCP 172.17.0.2:42815-> > 172.17.0.24:50130 hit > Jan 25 00:19:37 pc01 kernel: IPVS: TCP input [..A.] 172.17.0.17:50130-> > 172.17.0.2:42815 state: SYN_RECV->ESTABLISHED conn->refcnt:2 > Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit, > net/netfilter/ipvs/ip_vs_xmit.c line 1009 > Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_out, > net/netfilter/ipvs/ip_vs_core.c line 1116 > Jan 25 00:19:37 pc01 kernel: IPVS: Leave: ip_vs_dr_xmit, > net/netfilter/ipvs/ip_vs_xmit.c line 1031 > Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_out, > net/netfilter/ipvs/ip_vs_core.c line 1116 > Jan 25 00:19:37 pc01 kernel: IPVS: lookup/out TCP 172.17.0.2:42815-> > 172.17.0.24:50130 not hit > Jan 25 00:19:37 pc01 kernel: IPVS: lookup/in TCP 172.17.0.2:42815-> > 172.17.0.24:50130 hit > Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit, > net/netfilter/ipvs/ip_vs_xmit.c line 1009 > Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_out, > net/netfilter/ipvs/ip_vs_core.c line 1116 > Jan 25 00:19:37 pc01 kernel: IPVS: Leave: ip_vs_dr_xmit, > net/netfilter/ipvs/ip_vs_xmit.c line 1031 > Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_out, > net/netfilter/ipvs/ip_vs_core.c line 1116 > Jan 25 00:19:37 pc01 kernel: IPVS: lookup/out TCP 172.17.0.2:42815-> > 172.17.0.24:50130 not hit > Jan 25 00:19:37 pc01 kernel: IPVS: lookup/in TCP 172.17.0.2:42815-> > 172.17.0.24:50130 hit > Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit, > net/netfilter/ipvs/ip_vs_xmit.c line 1009 > Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_out, > net/netfilter/ipvs/ip_vs_core.c line 1116 > Jan 25 00:19:37 pc01 kernel: IPVS: Leave: ip_vs_dr_xmit, > net/netfilter/ipvs/ip_vs_xmit.c line 1031 > Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_out, > net/netfilter/ipvs/ip_vs_core.c line 1116 > Jan 25 00:19:37 pc01 kernel: IPVS: lookup/out TCP 172.17.0.2:42815-> > 172.17.0.24:50130 not hit > Jan 25 00:19:37 pc01 kernel: IPVS: lookup/in TCP 172.17.0.2:42815-> > 172.17.0.24:50130 hit > Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit, > net/netfilter/ipvs/ip_vs_xmit.c line 1009 > Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_out, > net/netfilter/ipvs/ip_vs_core.c line 1116 > Jan 25 00:19:37 pc01 kernel: IPVS: Leave: ip_vs_dr_xmit, > net/netfilter/ipvs/ip_vs_xmit.c line 1031 > Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_out, > net/netfilter/ipvs/ip_vs_core.c line 1116 > Jan 25 00:19:37 pc01 kernel: IPVS: lookup/out TCP 172.17.0.2:42815-> > 172.17.0.24:50130 not hit > Jan 25 00:19:37 pc01 kernel: IPVS: lookup/in TCP 172.17.0.2:42815-> > 172.17.0.24:50130 hit > Jan 25 00:19:37 pc01 kernel: IPVS: TCP input [.FA.] 172.17.0.17:50130-> > 172.17.0.2:42815 state: ESTABLISHED->FIN_WAIT conn->refcnt:2 > Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit, > net/netfilter/ipvs/ip_vs_xmit.c line 1009 > Jan 25 00:19:37 pc01 kernel: IPVS: Enter: ip_vs_out, > net/netfilter/ipvs/ip_vs_core.c line 1116 > > Debug output on real server 1 when not port mapping second test (curl -v > 'http://172.17.0.24:50000'): > > Jan 25 00:19:39 pc01 kernel: IPVS: ip_vs_rr_schedule(): Scheduling... > Jan 25 00:19:39 pc01 kernel: IPVS: RR: server 172.17.0.16:0 activeconns 0 > refcnt 1 weight 100 > Jan 25 00:19:39 pc01 kernel: IPVS: Bind-dest TCP c:172.17.0.2:42816 v: > 172.17.0.24:50130 d:172.17.0.16:50130 fwd:R s:65276 conn->flags:183 > conn->refcnt:1 dest->refcnt:2 > Jan 25 00:19:39 pc01 kernel: IPVS: Schedule fwd:R c:172.17.0.2:42816 v: > 172.17.0.24:50130 d:172.17.0.16:50130 conn->flags:101C3 conn->refcnt:2 > Jan 25 00:19:39 pc01 kernel: IPVS: TCP input [S...] 172.17.0.16:50130-> > 172.17.0.2:42816 state: NONE->SYN_RECV conn->refcnt:2 > Jan 25 00:19:39 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit, > net/netfilter/ipvs/ip_vs_xmit.c line 1009 > Jan 25 00:19:39 pc01 kernel: IPVS: new dst 172.17.0.16, src 172.17.0.16, > refcnt=1 > Jan 25 00:19:39 pc01 kernel: IPVS: Enter: ip_vs_out, > net/netfilter/ipvs/ip_vs_core.c line 1116 > Jan 25 00:19:39 pc01 kernel: IPVS: lookup/out TCP 172.17.0.24:50130-> > 172.17.0.2:42816 not hit > Jan 25 00:19:39 pc01 kernel: IPVS: lookup/in TCP 172.17.0.24:50130-> > 172.17.0.2:42816 not hit > Jan 25 00:19:39 pc01 kernel: IPVS: lookup service: fwm 0 TCP > 172.17.0.2:42816 not hit > Jan 25 00:19:39 pc01 kernel: IPVS: Enter: ip_vs_out, > net/netfilter/ipvs/ip_vs_core.c line 1116 > Jan 25 00:19:39 pc01 kernel: IPVS: lookup/out TCP 172.17.0.2:42816-> > 172.17.0.24:50130 not hit > Jan 25 00:19:39 pc01 kernel: IPVS: lookup/in TCP 172.17.0.2:42816-> > 172.17.0.24:50130 hit > Jan 25 00:19:39 pc01 kernel: IPVS: TCP input [..A.] 172.17.0.16:50130-> > 172.17.0.2:42816 state: SYN_RECV->ESTABLISHED conn->refcnt:2 > Jan 25 00:19:39 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit, > net/netfilter/ipvs/ip_vs_xmit.c line 1009 > Jan 25 00:19:39 pc01 kernel: IPVS: Enter: ip_vs_out, > net/netfilter/ipvs/ip_vs_core.c line 1116 > Jan 25 00:19:39 pc01 kernel: IPVS: lookup/out TCP 172.17.0.2:42816-> > 172.17.0.24:50130 not hit > Jan 25 00:19:39 pc01 kernel: IPVS: lookup/in TCP 172.17.0.2:42816-> > 172.17.0.24:50130 hit > Jan 25 00:19:39 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit, > net/netfilter/ipvs/ip_vs_xmit.c line 1009 > Jan 25 00:19:39 pc01 kernel: IPVS: Enter: ip_vs_out, > net/netfilter/ipvs/ip_vs_core.c line 1116 > Jan 25 00:19:39 pc01 kernel: IPVS: lookup/out TCP 172.17.0.24:50130-> > 172.17.0.2:42816 not hit > Jan 25 00:19:39 pc01 kernel: IPVS: lookup/in TCP 172.17.0.24:50130-> > 172.17.0.2:42816 not hit > Jan 25 00:19:39 pc01 kernel: IPVS: Enter: ip_vs_out, > net/netfilter/ipvs/ip_vs_core.c line 1116 > Jan 25 00:19:39 pc01 kernel: IPVS: lookup/out TCP 172.17.0.24:50130-> > 172.17.0.2:42816 not hit > Jan 25 00:19:39 pc01 kernel: IPVS: lookup/in TCP 172.17.0.24:50130-> > 172.17.0.2:42816 not hit > Jan 25 00:19:39 pc01 kernel: IPVS: Enter: ip_vs_out, > net/netfilter/ipvs/ip_vs_core.c line 1116 > Jan 25 00:19:39 pc01 kernel: IPVS: lookup/out TCP 172.17.0.2:42816-> > 172.17.0.24:50130 not hit > Jan 25 00:19:39 pc01 kernel: IPVS: lookup/in TCP 172.17.0.2:42816-> > 172.17.0.24:50130 hit > Jan 25 00:19:39 pc01 kernel: IPVS: TCP input [.FA.] 172.17.0.16:50130-> > 172.17.0.2:42816 state: ESTABLISHED->FIN_WAIT conn->refcnt:2 > Jan 25 00:19:39 pc01 kernel: IPVS: Enter: ip_vs_dr_xmit, > net/netfilter/ipvs/ip_vs_xmit.c line 1009 > Jan 25 00:19:39 pc01 kernel: IPVS: Enter: ip_vs_out, > net/netfilter/ipvs/ip_vs_core.c line 1116 > Jan 25 00:19:39 pc01 kernel: IPVS: lookup/out TCP 172.17.0.24:50130-> > 172.17.0.2:42816 not hit > Jan 25 00:19:39 pc01 kernel: IPVS: lookup/in TCP 172.17.0.24:50130-> > 172.17.0.2:42816 not hit > Jan 25 00:19:39 pc01 kernel: IPVS: Enter: ip_vs_out, > net/netfilter/ipvs/ip_vs_core.c line 1116 > Jan 25 00:19:39 pc01 kernel: IPVS: lookup/out TCP 172.17.0.16:39545-> > 172.17.0.16:3306 not hit > > The tcpdump command that was used was as follows on real server 1: > tcpdump -iany -nn port 80 or port 50000 > > I realized later that using 'any' device isn't as helpful when trying to > pinpoint loopback traffic, so that's what my follow up email was referring > to. > > Thanks again for the support, feel free to ask for any additional > information to help debug. > > Jacoby > > > On Sat, Jan 25, 2014 at 6:25 AM, Julian Anastasov <j...@ssi.bg> wrote: > >> >> Hello, >> >> On Thu, 23 Jan 2014, Jacoby Hickerson wrote: >> >> > Just to clarify the packets are going to the loopback of node 1, when >> they >> > should be going to node 2. This is shown in the tcpdump output:Here is >> the >> > output from the lo device of the first node: >> > 02:10:51.987030 IP 172.17.0.2.54276 > 172.17.0.16.50000: Flags [.], ack >> > 2970678458, win 115, options [nop,nop,TS val 3044575793 ecr 978483], >> length >> > 0 >> > 02:10:51.987079 IP 172.17.0.2.54276 > 172.17.0.16.50000: Flags [P.], seq >> > 0:173, ack 1, win 115, options [nop,nop,TS val 3044575793 ecr 978483], >> > length 173 >> > 02:10:51.987426 IP 172.17.0.2.54276 > 172.17.0.16.50000: Flags [.], ack >> 2, >> > win 115, options [nop,nop,TS val 3044575793 ecr 978484], length 0 >> > 02:10:51.987480 IP 172.17.0.2.54276 > 172.17.0.16.50000: Flags [F.], seq >> > 173, ack 2, win 115, options [nop,nop,TS val 3044575793 ecr 978484], >> length >> > 0 >> >> ... >> >> > Packets are being sent from the RIP of the first node only. From my >> > understanding when using DR OutPkts should always be zero. >> >> When LocalNode (local RIP) is used, we can see >> the local reply in LOCAL_OUT hook. It happens for NAT but >> also for DR. So, it is normal. But we see these replies >> after DNAT in LOCAL_OUT, see ip_vs_ops[] for reference. >> >> > The end result is that the packets are always coming from the first >> > node and never balanced to the second node. >> > >> > Thanks for any further help, seems the solution is really close! >> >> Can you provide more understandable description >> for the test, for example: >> >> - client box: >> IP1: X.X.X.X/N dev DEV >> IP2: ... >> >> - director: >> IP1: ... >> VIP: XXX >> are client and director same box >> >> - real server: >> IP1: ... >> >> iptable rules used. By this way I can try to >> duplicate the problem. Now I see some IPs in tcpdump >> output but I'm not sure what kind of traffic is shown, >> where is started the tcpdump, on what box, on what >> interface, external, internal... >> >> Regards >> >> -- >> Julian Anastasov <j...@ssi.bg> >> > >
ipvs_debug_output
Description: Binary data
_______________________________________________ Please read the documentation before posting - it's available at: http://www.linuxvirtualserver.org/ LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org Send requests to lvs-users-requ...@linuxvirtualserver.org or go to http://lists.graemef.net/mailman/listinfo/lvs-users
