LVS users, I have a "one arm NAT" configuration with so-called "local node" that I'm trying to make work with LVS.
I have two servers, syslog servers (call them A and B), and one client (we'll call C). Both A and B also run LVS with keepalived. There is a VIP address managed by keepalived. The idea is that when client C contacts the VIP, it will hit either A or B based on LVS scheduling. I'm doing both UDP and TCP protocols. For purposes of this discussion with respect to packets and the IP addresses, I'll use the format (source IP, destination IP). When client contacts the VIP, a packet is sent with (C-IP, VIP). It arrives at A that holds the VIP at the moment, and proceeds to LOCAL_IN and LVS-NAT changes the packet to either (C-IP, A-IP) or (C-IP, B-IP). In the former case (C-IP, A-IP) case, it gets processed by the local syslog process. In the later case (C-IP, B-IP), it gets sent back out to B. However, because the packet is (C-IP, B-IP), B's replies go to (B-IP, C-IP) and doesn't get processed by the LVS-NAT anymore. Connection hangs... as the C is expecting reply from (VIP, C-IP). I know in reading the various FAQ and postings about "one arm NAT" LVS-NAT situations is to have the "real servers" be the default GW. This won't work for me since the real servers are also the LVS servers. However, I was thinking that if I can use netfilter/iptables to source NAT the load balanced packets, I can change the (C-IP,B-IP) show up as (A-IP, B-IP) instead, and hence B's replies would go back to A (the current LVS-NAT) and sent back to C as (VIP, C-IP). However, the packet doesn't seem to traverse the "netfliter stack" as expected: (i setup a -j LOG rule within every chain [PREROUTING, INPUT, FORWARD, OUTPUT, POSTROUTING] of every table [mangle, nat, filter], and watch the packet traverse the "netfilter stack") Jun 25 22:11:20 sprdrsyas400 kernel: DEBUG_IP-nat-PRER: IN=eth0 OUT= MAC=00:50:56:b1:33:39:84:2b:2b:72:40:bb:08:00 SRC=10.180.88.65 DST=10.180.88.35 LEN=36 TOS=0x00 PREC=0x00 TTL=64 ID=51839 DF PROTO=UDP SPT=39178 DPT=514 LEN=16 Jun 25 22:11:20 sprdrsyas400 kernel: DEBUG_IP-mangle-INPU: IN=eth0 OUT= MAC=00:50:56:b1:33:39:84:2b:2b:72:40:bb:08:00 SRC=10.180.88.65 DST=10.180.88.35 LEN=36 TOS=0x00 PREC=0x00 TTL=64 ID=51839 DF PROTO=UDP SPT=39178 DPT=514 LEN=16 Jun 25 22:11:20 sprdrsyas400 kernel: DEBUG_IP-filter-INPU: IN=eth0 OUT= MAC=00:50:56:b1:33:39:84:2b:2b:72:40:bb:08:00 SRC=10.180.88.65 DST=10.180.88.35 LEN=36 TOS=0x00 PREC=0x00 TTL=64 ID=51839 DF PROTO=UDP SPT=39178 DPT=514 LEN=16 Jun 25 22:11:20 sprdrsyas400 kernel: DEBUG_IP-mangle-OUTP: IN= OUT=eth1 SRC=10.180.88.65 DST=10.180.88.30 LEN=36 TOS=0x00 PREC=0x00 TTL=64 ID=51839 DF PROTO=UDP SPT=39178 DPT=514 LEN=16 Jun 25 22:11:20 sprdrsyas400 kernel: DEBUG_IP-filter-OUTP: IN= OUT=eth1 SRC=10.180.88.65 DST=10.180.88.30 LEN=36 TOS=0x00 PREC=0x00 TTL=64 ID=51839 DF PROTO=UDP SPT=39178 DPT=514 LEN=16 Jun 25 22:11:20 sprdrsyas400 kernel: DEBUG_IP-mangle-POSR: IN= OUT=eth1 SRC=10.180.88.65 DST=10.180.88.30 LEN=36 TOS=0x00 PREC=0x00 TTL=64 ID=51839 DF PROTO=UDP SPT=39178 DPT=514 LEN=16 This is a packet from (C-IP, VIP), goes through NAT/PREROUTING->ROUTING->MANGLE/INPUT->FILTER/INPUT->(LVS-NAT?)->ROUTING->MANGLE/OUTPUT->FILTER/OUTPUT->MANGLE/POSTROUTING. What I see missing are MANGLE/PREROUTING and particularly interested in the missing NAT/POSTROUTING, where I was hoping to SNAT the packet. According to this post: http://archive.linuxvirtualserver.org/html/lvs-users/2011-03/msg00019.html It would appear that at some point, LVS-NAT packets did traverse NAT/POSTROUTING and one could SNAT the LVS outbound packets? I'm using: RHEL 6.5 with RH kernel 2.6.32-431.17.1.el6.x86_64 ipvsadm-1.26-2.el6.x86_64 keepalived-1.2.13-1.el6.x86_64 here are the -j LOG iptables rules I'm using: [root@sprdrsyas400 ~]# iptables -t mangle -vnL Chain PREROUTING (policy ACCEPT 293K packets, 29M bytes) pkts bytes target prot opt in out source destination 4 144 LOG udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:514 LOG flags 0 level 4 prefix `DEBUG_IP-mangle-PRER: ' Chain INPUT (policy ACCEPT 293K packets, 29M bytes) pkts bytes target prot opt in out source destination 4 144 LOG udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:514 LOG flags 0 level 4 prefix `DEBUG_IP-mangle-INPU: ' Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination 0 0 LOG udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:514 LOG flags 0 level 4 prefix `DEBUG_IP-mangle-FORW: ' Chain OUTPUT (policy ACCEPT 362K packets, 18M bytes) pkts bytes target prot opt in out source destination 2 72 LOG udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:514 LOG flags 0 level 4 prefix `DEBUG_IP-mangle-OUTP: ' Chain POSTROUTING (policy ACCEPT 362K packets, 18M bytes) pkts bytes target prot opt in out source destination 2 72 LOG udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:514 LOG flags 0 level 4 prefix `DEBUG_IP-mangle-POSR: ' [root@sprdrsyas400 ~]# iptables -t nat -vnL Chain PREROUTING (policy ACCEPT 8739 packets, 500K bytes) pkts bytes target prot opt in out source destination 7 252 LOG udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:514 LOG flags 0 level 4 prefix `DEBUG_IP-nat-PRER: ' Chain POSTROUTING (policy ACCEPT 25612 packets, 1374K bytes) pkts bytes target prot opt in out source destination 0 0 LOG udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:514 LOG flags 0 level 4 prefix `DEBUG_IP-nat-POSR: ' Chain OUTPUT (policy ACCEPT 25612 packets, 1374K bytes) pkts bytes target prot opt in out source destination 0 0 LOG udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:514 LOG flags 0 level 4 prefix `DEBUG_IP-nat-OUTP: ' [root@sprdrsyas400 ~]# iptables -vnL Chain INPUT (policy ACCEPT 292K packets, 29M bytes) pkts bytes target prot opt in out source destination 4 144 LOG udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:514 LOG flags 0 level 4 prefix `DEBUG_IP-filter-INPU: ' Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination 0 0 LOG udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:514 LOG flags 0 level 4 prefix `DEBUG_IP-filter-FORW: ' Chain OUTPUT (policy ACCEPT 361K packets, 18M bytes) pkts bytes target prot opt in out source destination 2 72 LOG udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:514 LOG flags 0 level 4 prefix `DEBUG_IP-filter-OUTP: ' And this is LVS on A: [root@sprdrsyas400 ~]# ipvsadm -Ln IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn UDP 10.180.88.35:514 rr ops -> 10.180.88.29:514 Local 1 0 0 -> 10.180.88.30:514 Masq 1 0 0 I would appreciate any help... Thanks in advance, -Bond _______________________________________________ Please read the documentation before posting - it's available at: http://www.linuxvirtualserver.org/ LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org Send requests to lvs-users-requ...@linuxvirtualserver.org or go to http://lists.graemef.net/mailman/listinfo/lvs-users
