Re: [LARTC] Balancing multiple connections and NAT
On 3/5/06, Andreas Hasenack [EMAIL PROTECTED] wrote: Em Qui 23 Fev 2006 20:41, Markus Schulz escreveu: you need a patch for NAT processing with multiple gateways. this will then save the routing information for each connection inside NAT structures, so that each packet of an established connection will be get routed over the same gateway. you can find the patches here: http://www.ssi.bg/~ja/#routes please read the guides (nano howto or dgd-usage) carefully. Any idea why these patches are not yet integrated into the upstream kernel? Possibly disagreement on the features included, or any number of other reasons. Personally I had serious problems with the patches in that they didn't fully function. Multi-path routing was only choosing one path, and if too many routes were requested a kernel panic would occur. - Jody ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Re: [LARTC] Balancing multiple connections and NAT
Em Qui 23 Fev 2006 20:41, Markus Schulz escreveu: you need a patch for NAT processing with multiple gateways. this will then save the routing information for each connection inside NAT structures, so that each packet of an established connection will be get routed over the same gateway. you can find the patches here: http://www.ssi.bg/~ja/#routes please read the guides (nano howto or dgd-usage) carefully. Any idea why these patches are not yet integrated into the upstream kernel? ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Re: [LARTC] Balancing multiple connections and NAT
On Fr, 2006-02-24 at 22:22 +0100, Sebastian Bork wrote: Done. It happens here, too. But now it gets really strange: the data (I tried scp) goes out on IF1 with IF2's source address. The ACK packets come in on IF2. The connection works anyway ... *That's* what I'd call really cool load-balancing. It obviously only worked with my setup because two of the three routes end on the same router of the upstream provider. As soon as the other provider is used for a route, everything breaks down. I'll try those patches tomorrow. ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Re: [LARTC] Balancing multiple connections and NAT
On Fr, 2006-02-24 at 00:44 +0530, Raj Mathur wrote: I have a client connected to the 'net through 3 ISP's. Have set up a Linux box to do routing and load sharing for the 3 connections. A fourth interface is connected to the LAN with private IP addresses. Am using iptables to SNAT traffic to the appropriate IP depending on the interface the packet gets routed onto. I use exactly the same setup with a customer's conenction, the only difference: I use MASQUERADE instead of SNAT. I did not see anything like the problem you describe. Maybe because MAQUERADE works stateful, SNAT not? If you do not have a special reason for using SNAT, I think you should try MASQUERADE. If your problem persits, please tell me, as I have to look at my customer's setup very closely then, to catch this before anyone complains. ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Re: [LARTC] Balancing multiple connections and NAT
Sebastian == Sebastian Bork [EMAIL PROTECTED] writes: Sebastian On Fr, 2006-02-24 at 00:44 +0530, Raj Mathur wrote: I have a client connected to the 'net through 3 ISP's. Have set up a Linux box to do routing and load sharing for the 3 connections. A fourth interface is connected to the LAN with private IP addresses. Am using iptables to SNAT traffic to the appropriate IP depending on the interface the packet gets routed onto. Sebastian I use exactly the same setup with a customer's Sebastian conenction, the only difference: I use MASQUERADE Sebastian instead of SNAT. I did not see anything like the Sebastian problem you describe. Maybe because MAQUERADE works Sebastian stateful, SNAT not? If you do not have a special reason Sebastian for using SNAT, I think you should try MASQUERADE. If Sebastian your problem persits, please tell me, as I have to look Sebastian at my customer's setup very closely then, to catch this Sebastian before anyone complains. Well, both MASQUERADE and SNAT are stateful (MASQUERADE is just a special case of SNAT as far as I remember); however it's worth a shot if it's working for you. It's pretty easy to trap the wrong source IP errors -- going back to my example, just run: tcpdump -i intA -q -t -n ! host ipA tcpdump -i intB -q -t -n ! host ipB tcpdump -i intC -q -t -n ! host ipC Any IP packets that get displayed will be those with wrong source IPs. You may need to start some large FTP uploads or similar and watch for a while -- the problem manifests itself for me when the client is uploading 10+ MB files to his public FTP server. Of course, it may be present in other places also, but outgoing FTP comprises the bulk of his traffic so it's most patent there. Digressing a bit, from the responses I've got from this list, it seems that a kernel patch is required to make the whole load sharing + iptables NAT work properly. I'm a bit disappointed that this isn't part of the mainstream kernel -- any chances of it being rolled in upstream? Regards, -- Raju -- Raj Mathur[EMAIL PROTECTED] http://kandalaya.org/ GPG: 78D4 FC67 367F 40E2 0DD5 0FEF C968 D0EF CC68 D17F It is the mind that moves ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
[LARTC] Balancing multiple connections and NAT
Hi, I have a client connected to the 'net through 3 ISP's. Have set up a Linux box to do routing and load sharing for the 3 connections. A fourth interface is connected to the LAN with private IP addresses. Am using iptables to SNAT traffic to the appropriate IP depending on the interface the packet gets routed onto. The setup looks something like this: Interface IP Gateway Table Network - -- --- - --- intAipA gwA tableA netA intBipB gwB tableB netB intCipC gwC tableC netC [intD is the LAN interface] intDipD (private)no gateway global netD This works fine most of the time, except that once in a while (every 5-10 minutes or so) packets going out on (e.g.) intB suddenly start getting NAT'ed to source address ipA (i.e. the address of another interface). Obviously this plays hell with the existing connections on that link! The ip commands I'm using are: /sbin/ip route add netA dev intA src ipA table tableA /sbin/ip route add netA dev intA src ipA /sbin/ip route add default via gwA table tableA /sbin/ip route add netB dev intB src ipB table tableB /sbin/ip route add netB dev intB src ipB /sbin/ip route add default via gwB table tableB /sbin/ip route add netC dev intC src ipC table tableC /sbin/ip route add netC dev intC src ipC /sbin/ip route add default via gwC table tableC /sbin/ip route add default scope global nexthop via gwB dev intB weight 1 nexthop via gwC dev intC weight 2 nexthop via gwA dev intA weight 2 /sbin/ip rule add from ipA table tableA /sbin/ip rule add from ipB table tableB /sbin/ip rule add from ipC table tableC The iptables commands are: /sbin/iptables -P FORWARD DROP # Enable full flow on the LAN /sbin/iptables -I FORWARD -s netD -i intD -j ACCEPT /sbin/iptables -I FORWARD -d netD -o intD -j ACCEPT # Allow all packets to go out /sbin/iptables -I OUTPUT -o intA -j ACCEPT /sbin/iptables -I OUTPUT -o intB -j ACCEPT /sbin/iptables -I OUTPUT -o intC -j ACCEPT /sbin/iptables -I OUTPUT -o intD -j ACCEPT /sbin/iptables -I INPUT -i intD -j ACCEPT /sbin/iptables -I INPUT -i lo -j ACCEPT /sbin/iptables -P INPUT DROP /sbin/iptables -A INPUT -i ! intD -m state --state RELATED,ESTABLISHED -j ACCEPT # Hmmm, why is this one there? /sbin/iptables -A INPUT -i intD -m state --state RELATED,ESTABLISHED -j ACCEPT # NAT depending on outbound interface /sbin/iptables -t nat -A POSTROUTING -s netD -o intA -j SNAT --to-source ipA /sbin/iptables -t nat -A POSTROUTING -s netD -o intB -j SNAT --to-source ipB /sbin/iptables -t nat -A POSTROUTING -s netD -o intC -j SNAT --to-source ipC Any idea why connections that are flowing perfectly would suddenly decide to start getting NAT'ed to the wrong source? Or some place on the 'net I can start looking? Regards, -- Raju -- Raj Mathur[EMAIL PROTECTED] http://kandalaya.org/ GPG: 78D4 FC67 367F 40E2 0DD5 0FEF C968 D0EF CC68 D17F It is the mind that moves ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Re: [LARTC] Balancing multiple connections and NAT
That's because the route to a host X has been changed. There's a routing decision to host X every so or so minutes. Ways to handle that? Boy, that can be tough. I've read of prople who use a different LAN IP for every public connection. So If you have 3 public connections, you should use three different IPs on the lan. Then, when you DNAT packets on the way in (from the internet), you mark those packets with a different fwmark for each internet connection. On POSTROUTING do a SNAT according to this fwmark to a different IP each (remember you have three inner IPs to choose from). Then when packets come back from your servers, they will be sent to three different IPs and you can tell packets that should go out one way or the other. Now, if you don't care about incoming (to your lan) connections but outgoing I guess you are very much out of luck unless you use some policy routing for stateful connections (I think that's the name of connections that DO care if you change the IP the connection is going through) so that they use a single interface all the time. On 2/23/06, Raj Mathur [EMAIL PROTECTED] wrote: Hi, I have a client connected to the 'net through 3 ISP's. Have set up a Linux box to do routing and load sharing for the 3 connections. A fourth interface is connected to the LAN with private IP addresses. Am using iptables to SNAT traffic to the appropriate IP depending on the interface the packet gets routed onto. The setup looks something like this: Interface IP Gateway Table Network - -- --- - --- intAipA gwA tableA netA intBipB gwB tableB netB intCipC gwC tableC netC [intD is the LAN interface] intDipD (private)no gateway global netD This works fine most of the time, except that once in a while (every 5-10 minutes or so) packets going out on (e.g.) intB suddenly start getting NAT'ed to source address ipA (i.e. the address of another interface). Obviously this plays hell with the existing connections on that link! The ip commands I'm using are: /sbin/ip route add netA dev intA src ipA table tableA /sbin/ip route add netA dev intA src ipA /sbin/ip route add default via gwA table tableA /sbin/ip route add netB dev intB src ipB table tableB /sbin/ip route add netB dev intB src ipB /sbin/ip route add default via gwB table tableB /sbin/ip route add netC dev intC src ipC table tableC /sbin/ip route add netC dev intC src ipC /sbin/ip route add default via gwC table tableC /sbin/ip route add default scope global nexthop via gwB dev intB weight 1 nexthop via gwC dev intC weight 2 nexthop via gwA dev intA weight 2 /sbin/ip rule add from ipA table tableA /sbin/ip rule add from ipB table tableB /sbin/ip rule add from ipC table tableC The iptables commands are: /sbin/iptables -P FORWARD DROP # Enable full flow on the LAN /sbin/iptables -I FORWARD -s netD -i intD -j ACCEPT /sbin/iptables -I FORWARD -d netD -o intD -j ACCEPT # Allow all packets to go out /sbin/iptables -I OUTPUT -o intA -j ACCEPT /sbin/iptables -I OUTPUT -o intB -j ACCEPT /sbin/iptables -I OUTPUT -o intC -j ACCEPT /sbin/iptables -I OUTPUT -o intD -j ACCEPT /sbin/iptables -I INPUT -i intD -j ACCEPT /sbin/iptables -I INPUT -i lo -j ACCEPT /sbin/iptables -P INPUT DROP /sbin/iptables -A INPUT -i ! intD -m state --state RELATED,ESTABLISHED -j ACCEPT # Hmmm, why is this one there? /sbin/iptables -A INPUT -i intD -m state --state RELATED,ESTABLISHED -j ACCEPT # NAT depending on outbound interface /sbin/iptables -t nat -A POSTROUTING -s netD -o intA -j SNAT --to-source ipA /sbin/iptables -t nat -A POSTROUTING -s netD -o intB -j SNAT --to-source ipB /sbin/iptables -t nat -A POSTROUTING -s netD -o intC -j SNAT --to-source ipC Any idea why connections that are flowing perfectly would suddenly decide to start getting NAT'ed to the wrong source? Or some place on the 'net I can start looking? Regards, -- Raju -- Raj Mathur[EMAIL PROTECTED] http://kandalaya.org/ GPG: 78D4 FC67 367F 40E2 0DD5 0FEF C968 D0EF CC68 D17F It is the mind that moves ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Re: [LARTC] Balancing multiple connections and NAT
Raj, I use something just like you make and for this problem I have patched my kernel with diff-routes patch (there is a link in LARTC) and I have make a little script that makes just a ping to an outside address every 5 min (cron job) and flush the route cache after this ping, like this: ip route flush cache So all cache will be lost (off course that conections that still exists will remain with their track to the destination). Att, Nataniel Klug - Original Message - From: Raj Mathur [EMAIL PROTECTED] To: lartc@mailman.ds9a.nl Sent: Thursday, February 23, 2006 4:14 PM Subject: [LARTC] Balancing multiple connections and NAT Hi, I have a client connected to the 'net through 3 ISP's. Have set up a Linux box to do routing and load sharing for the 3 connections. A fourth interface is connected to the LAN with private IP addresses. Am using iptables to SNAT traffic to the appropriate IP depending on the interface the packet gets routed onto. The setup looks something like this: Interface IP Gateway Table Network - -- --- - --- intAipA gwA tableA netA intBipB gwB tableB netB intCipC gwC tableC netC [intD is the LAN interface] intDipD (private)no gateway global netD This works fine most of the time, except that once in a while (every 5-10 minutes or so) packets going out on (e.g.) intB suddenly start getting NAT'ed to source address ipA (i.e. the address of another interface). Obviously this plays hell with the existing connections on that link! The ip commands I'm using are: /sbin/ip route add netA dev intA src ipA table tableA /sbin/ip route add netA dev intA src ipA /sbin/ip route add default via gwA table tableA /sbin/ip route add netB dev intB src ipB table tableB /sbin/ip route add netB dev intB src ipB /sbin/ip route add default via gwB table tableB /sbin/ip route add netC dev intC src ipC table tableC /sbin/ip route add netC dev intC src ipC /sbin/ip route add default via gwC table tableC /sbin/ip route add default scope global nexthop via gwB dev intB weight 1 nexthop via gwC dev intC weight 2 nexthop via gwA dev intA weight 2 /sbin/ip rule add from ipA table tableA /sbin/ip rule add from ipB table tableB /sbin/ip rule add from ipC table tableC The iptables commands are: /sbin/iptables -P FORWARD DROP # Enable full flow on the LAN /sbin/iptables -I FORWARD -s netD -i intD -j ACCEPT /sbin/iptables -I FORWARD -d netD -o intD -j ACCEPT # Allow all packets to go out /sbin/iptables -I OUTPUT -o intA -j ACCEPT /sbin/iptables -I OUTPUT -o intB -j ACCEPT /sbin/iptables -I OUTPUT -o intC -j ACCEPT /sbin/iptables -I OUTPUT -o intD -j ACCEPT /sbin/iptables -I INPUT -i intD -j ACCEPT /sbin/iptables -I INPUT -i lo -j ACCEPT /sbin/iptables -P INPUT DROP /sbin/iptables -A INPUT -i ! intD -m state --state RELATED,ESTABLISHED -j ACCEPT # Hmmm, why is this one there? /sbin/iptables -A INPUT -i intD -m state --state RELATED,ESTABLISHED -j ACCEPT # NAT depending on outbound interface /sbin/iptables -t nat -A POSTROUTING -s netD -o intA -j SNAT --to-source ipA /sbin/iptables -t nat -A POSTROUTING -s netD -o intB -j SNAT --to-source ipB /sbin/iptables -t nat -A POSTROUTING -s netD -o intC -j SNAT --to-source ipC Any idea why connections that are flowing perfectly would suddenly decide to start getting NAT'ed to the wrong source? Or some place on the 'net I can start looking? Regards, -- Raju -- Raj Mathur[EMAIL PROTECTED] http://kandalaya.org/ GPG: 78D4 FC67 367F 40E2 0DD5 0FEF C968 D0EF CC68 D17F It is the mind that moves ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Re: [LARTC] Balancing multiple connections and NAT
Am Donnerstag, 23. Februar 2006 20:14 schrieb Raj Mathur: Hi, I have a client connected to the 'net through 3 ISP's. Have set up a Linux box to do routing and load sharing for the 3 connections. A fourth interface is connected to the LAN with private IP addresses. Am using iptables to SNAT traffic to the appropriate IP depending on the interface the packet gets routed onto. The setup looks something like this: Interface IP GatewayTable Network - -- --- - --- intAipA gwAtableA netA intBipB gwBtableB netB intCipC gwCtableC netC [intD is the LAN interface] intDipD (private)no gateway global netD This works fine most of the time, except that once in a while (every 5-10 minutes or so) packets going out on (e.g.) intB suddenly start getting NAT'ed to source address ipA (i.e. the address of another interface). Obviously this plays hell with the existing connections on that link! you need a patch for NAT processing with multiple gateways. this will then save the routing information for each connection inside NAT structures, so that each packet of an established connection will be get routed over the same gateway. you can find the patches here: http://www.ssi.bg/~ja/#routes please read the guides (nano howto or dgd-usage) carefully. -- Markus Schulz ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
