Just to clarify the packets are going to the loopback of node 1, when they should be going to node 2. This is shown in the tcpdump output: Here is the output from the lo device of the first node: 02:10:51.987030 IP 172.17.0.2.54276 > 172.17.0.16.50000: Flags [.], ack 2970678458, win 115, options [nop,nop,TS val 3044575793 ecr 978483], length 0 02:10:51.987079 IP 172.17.0.2.54276 > 172.17.0.16.50000: Flags [P.], seq 0:173, ack 1, win 115, options [nop,nop,TS val 3044575793 ecr 978483], length 173 02:10:51.987426 IP 172.17.0.2.54276 > 172.17.0.16.50000: Flags [.], ack 2, win 115, options [nop,nop,TS val 3044575793 ecr 978484], length 0 02:10:51.987480 IP 172.17.0.2.54276 > 172.17.0.16.50000: Flags [F.], seq 173, ack 2, win 115, options [nop,nop,TS val 3044575793 ecr 978484], length 0
Thanks, Jacoby On Wed, Jan 22, 2014 at 7:04 PM, Jacoby Hickerson <hickerso...@gmail.com>wrote: > I spoke to soon about this configuration working, the output of ipvsadm > lead me to believe connections and packets were being load balanced, > however they are now all coming from the real server which also is the > director. I'll call this the 'first node'. > > Here is some more info in addition to the details in my previous emails: > First node routing: > Kernel IP routing table > Destination Gateway Genmask Flags Metric Ref Use > Iface > 0.0.0.0 172.17.0.16 0.0.0.0 UG 0 0 0 > bond0 > 172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 > bond0 > > Second node routing: > Kernel IP routing table > Destination Gateway Genmask Flags Metric Ref Use > Iface > 0.0.0.0 172.17.0.16 0.0.0.0 UG 0 0 0 > bond0 > 172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 > bond0 > > The second node also has the VIP on the loopback device > > How I setup the director > ipvsadm -C > ipvsadm -A -f 100 -s rr > ipvsadm -a -f 100 -r 172.17.0.16 -w 100 > ipvsadm -a -f 100 -r 172.17.0.17 -w 100 > > After conducting a few wget's here are the expected results from ipvsadm > while not port mapping: > IP Virtual Server version 1.2.1 (size=4096) > Prot LocalAddress:Port Conns InPkts OutPkts InBytes > OutBytes > -> RemoteAddress:Port > FWM 100 12 61 0 5416 > 0 > -> 172.17.0.16:0 6 25 0 2422 > 0 > -> 172.17.0.17:0 6 36 0 2994 > 0 > > Actual results from ipvsadm when using port mapping: > IP Virtual Server version 1.2.1 (size=4096) > Prot LocalAddress:Port Conns InPkts OutPkts InBytes > OutBytes > -> RemoteAddress:Port > FWM 100 2 8 4 778 > 216 > -> 172.17.0.16:0 1 4 4 389 > 216 > -> 172.17.0.17:0 1 4 0 389 > 0 > > Packets are being sent from the RIP of the first node only. From my > understanding when using DR OutPkts should always be zero. > > Output from tcpdump is a bit strange (first pass): > 02:10:51.986719 IP 172.17.0.2.54276 > 172.17.0.24.80: Flags [S], seq > 1873056683, win 14600, options [mss 1460,sackOK,TS val 3044575792 ecr > 0,nop,wscale 7], length 0 > 02:10:51.986719 IP 172.17.0.2.54276 > 172.17.0.24.80: Flags [S], seq > 1873056683, win 14600, options [mss 1460,sackOK,TS val 3044575792 ecr > 0,nop,wscale 7], length 0 > 02:10:51.986719 IP 172.17.0.2.54276 > 172.17.0.16.50000: Flags [S], seq > 1873056683, win 14600, options [mss 1460,sackOK,TS val 3044575792 ecr > 0,nop,wscale 7], length 0 > 02:10:51.986814 IP 172.17.0.24.80 > 172.17.0.2.54276: Flags [S.], seq > 2970678457, ack 1873056684, win 14480, options [mss 1460,sackOK,TS val > 978483 ecr 3044575792,nop,wscale 7], length 0 > 02:10:51.986919 IP 172.17.0.24.80 > 172.17.0.2.54276: Flags [S.], seq > 2970678457, ack 1873056684, win 14480, options [mss 1460,sackOK,TS val > 978483 ecr 3044575792,nop,wscale 7], length 0 > 02:10:51.987030 IP 172.17.0.2.54276 > 172.17.0.24.80: Flags [.], ack 1, > win 115, options [nop,nop,TS val 3044575793 ecr 978483], length 0 > 02:10:51.987030 IP 172.17.0.2.54276 > 172.17.0.24.80: Flags [.], ack 1, > win 115, options [nop,nop,TS val 3044575793 ecr 978483], length 0 > *02:10:51.987030 IP 172.17.0.2.54276 > 172.17.0.16.50000: Flags [.], ack > 2970678458, win 115, options [nop,nop,TS val 3044575793 ecr 978483], length > 0* > 02:10:51.987079 IP 172.17.0.2.54276 > 172.17.0.24.80: Flags [P.], seq > 1:174, ack 1, win 115, options [nop,nop,TS val 3044575793 ecr 978483], > length 173 > 02:10:51.987079 IP 172.17.0.2.54276 > 172.17.0.24.80: Flags [P.], seq > 1:174, ack 1, win 115, options [nop,nop,TS val 3044575793 ecr 978483], > length 173 > *02:10:51.987079 IP 172.17.0.2.54276 > 172.17.0.16.50000: Flags [P.], seq > 0:173, ack 1, win 115, options [nop,nop,TS val 3044575793 ecr 978483], > length 173* > 02:10:51.987126 IP 172.17.0.24.80 > 172.17.0.2.54276: Flags [.], ack 174, > win 122, options [nop,nop,TS val 978484 ecr 3044575793], length 0 > 02:10:51.987128 IP 172.17.0.24.80 > 172.17.0.2.54276: Flags [.], ack 174, > win 122, options [nop,nop,TS val 978484 ecr 3044575793], length 0 > 02:10:51.987241 IP 172.17.0.24.80 > 172.17.0.2.54276: Flags [F.], seq 1, > ack 174, win 122, options [nop,nop,TS val 978484 ecr 3044575793], length 0 > 02:10:51.987245 IP 172.17.0.24.80 > 172.17.0.2.54276: Flags [F.], seq 1, > ack 174, win 122, options [nop,nop,TS val 978484 ecr 3044575793], length 0 > 02:10:51.987426 IP 172.17.0.2.54276 > 172.17.0.24.80: Flags [.], ack 2, > win 115, options [nop,nop,TS val 3044575793 ecr 978484], length 0 > 02:10:51.987426 IP 172.17.0.2.54276 > 172.17.0.24.80: Flags [.], ack 2, > win 115, options [nop,nop,TS val 3044575793 ecr 978484], length 0 > *02:10:51.987426 IP 172.17.0.2.54276 > 172.17.0.16.50000: Flags [.], ack > 2, win 115, options [nop,nop,TS val 3044575793 ecr 978484], length 0* > 02:10:51.987480 IP 172.17.0.2.54276 > 172.17.0.24.80: Flags [F.], seq 174, > ack 2, win 115, options [nop,nop,TS val 3044575793 ecr 978484], length 0 > 02:10:51.987480 IP 172.17.0.2.54276 > 172.17.0.24.80: Flags [F.], seq 174, > ack 2, win 115, options [nop,nop,TS val 3044575793 ecr 978484], length 0 > *02:10:51.987480 IP 172.17.0.2.54276 > 172.17.0.16.50000: Flags [F.], seq > 173, ack 2, win 115, options [nop,nop,TS val 3044575793 ecr 978484], length > 0* > 02:10:51.987527 IP 172.17.0.24.80 > 172.17.0.2.54276: Flags [.], ack 175, > win 122, options [nop,nop,TS val 978484 ecr 3044575793], length 0 > 02:10:51.987529 IP 172.17.0.24.80 > 172.17.0.2.54276: Flags [.], ack 175, > win 122, options [nop,nop,TS val 978484 ecr 3044575793], length 0 > > Second pass: > 02:10:59.360433 IP 172.17.0.2.54277 > 172.17.0.24.80: Flags [S], seq > 2134630706, win 14600, options [mss 1460,sackOK,TS val 3044583166 ecr > 0,nop,wscale 7], length 0 > 02:10:59.360433 IP 172.17.0.2.54277 > 172.17.0.24.80: Flags [S], seq > 2134630706, win 14600, options [mss 1460,sackOK,TS val 3044583166 ecr > 0,nop,wscale 7], length 0 > 02:10:59.360519 IP 172.17.0.24.80 > 172.17.0.2.54277: Flags [S.], seq > 2952275478, ack 2134630707, win 14480, options [mss 1460,sackOK,TS val > 985857 ecr 3044583166,nop,wscale 7], length 0 > 02:10:59.360524 IP 172.17.0.24.80 > 172.17.0.2.54277: Flags [S.], seq > 2952275478, ack 2134630707, win 14480, options [mss 1460,sackOK,TS val > 985857 ecr 3044583166,nop,wscale 7], length 0 > 02:10:59.360627 IP 172.17.0.2.54277 > 172.17.0.24.80: Flags [.], ack 1, > win 115, options [nop,nop,TS val 3044583166 ecr 985857], length 0 > 02:10:59.360627 IP 172.17.0.2.54277 > 172.17.0.24.80: Flags [.], ack 1, > win 115, options [nop,nop,TS val 3044583166 ecr 985857], length 0 > 02:10:59.360728 IP 172.17.0.2.54277 > 172.17.0.24.80: Flags [P.], seq > 1:174, ack 1, win 115, options [nop,nop,TS val 3044583166 ecr 985857], > length 173 > 02:10:59.360728 IP 172.17.0.2.54277 > 172.17.0.24.80: Flags [P.], seq > 1:174, ack 1, win 115, options [nop,nop,TS val 3044583166 ecr 985857], > length 173 > 02:10:59.360751 IP 172.17.0.24.80 > 172.17.0.2.54277: Flags [.], ack 174, > win 122, options [nop,nop,TS val 985857 ecr 3044583166], length 0 > 02:10:59.360753 IP 172.17.0.24.80 > 172.17.0.2.54277: Flags [.], ack 174, > win 122, options [nop,nop,TS val 985857 ecr 3044583166], length 0 > 02:10:59.361058 IP 172.17.0.24.80 > 172.17.0.2.54277: Flags [F.], seq 1, > ack 174, win 122, options [nop,nop,TS val 985858 ecr 3044583166], length 0 > 02:10:59.361062 IP 172.17.0.24.80 > 172.17.0.2.54277: Flags [F.], seq 1, > ack 174, win 122, options [nop,nop,TS val 985858 ecr 3044583166], length 0 > 02:10:59.361270 IP 172.17.0.2.54277 > 172.17.0.24.80: Flags [F.], seq 174, > ack 2, win 115, options [nop,nop,TS val 3044583167 ecr 985858], length 0 > 02:10:59.361270 IP 172.17.0.2.54277 > 172.17.0.24.80: Flags [F.], seq 174, > ack 2, win 115, options [nop,nop,TS val 3044583167 ecr 985858], length 0 > 02:10:59.361331 IP 172.17.0.24.80 > 172.17.0.2.54277: Flags [.], ack 175, > win 122, options [nop,nop,TS val 985858 ecr 3044583167], length 0 > 02:10:59.361334 IP 172.17.0.24.80 > 172.17.0.2.54277: Flags [.], ack 175, > win 122, options [nop,nop,TS val 985858 ecr 3044583167], length 0 > > The end result is that the packets are always coming from the first node > and never balanced to the second node. > > Thanks for any further help, seems the solution is really close! > > Jacoby > > > On Sat, Jan 18, 2014 at 12:44 AM, Julian Anastasov <j...@ssi.bg> wrote: > >> >> Hello, >> >> On Fri, 17 Jan 2014, Jacoby Hickerson wrote: >> >> > Thanks Julian! After enabling CONFIG_IP_VS_NFCT and setting conntrack >> to 1 >> > that resolved the problem. >> > However, how leery should I be with it consuming memory? Is there a >> test to >> > monitor this consumption? Currently the nf_conntrack_max is set to the >> > default: 65536 >> >> cat /proc/slabinfo | grep nf_conntrack >> or 'slabtop' can show the object size used by conntracks. >> It should be 240+ bytes. You can expect one conntrack per >> IPVS connection. You can also see conntracks with >> cat /proc/net/nf_conntrack | less >> >> cat /proc/sys/net/netfilter/nf_conntrack_count shows >> the current number of conntracks. You can look for 'count' >> at __nf_conntrack_alloc() and nf_conntrack_free() to see >> how it is implemented. >> >> Regards >> >> -- >> Julian Anastasov <j...@ssi.bg> >> > > _______________________________________________ Please read the documentation before posting - it's available at: http://www.linuxvirtualserver.org/ LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org Send requests to lvs-users-requ...@linuxvirtualserver.org or go to http://lists.graemef.net/mailman/listinfo/lvs-users