Re: [LARTC] equalize / ecmp not working as expected in 2.6 vs 2.4
Andy Furniss [EMAIL PROTECTED] wrote on 10/04/2007 21:29:33: [EMAIL PROTECTED] wrote: thanks for the info. First of all, in order to use the nth match you need to patch your kernel using patch-o-matic. I think nth is in kernel now as part of the statistic match. Andy. I was searching this yesterday afternoon and could not verify it. Today I found it in the 2.6.18 kernel's changelog: [NETFILTER]: x_tables: add statistic match Add statistic match which is a combination of the nth and random matches. Signed-off-by: Patrick McHardy ... Signed-off-by: David S. Miller ... commit 62b7743483b402f8fb73545d5d487ca714e82766 Author: Patrick McHardy ... Date: Mon May 29 18:20:32 2006 -0700 ¿ Does this match help you to solve the problem ? Regards, Eric -- ADVERTENCIA LEGAL El contenido de este correo es confidencial y dirigido unicamente a su destinatario. Para acceder a su clausula de privacidad consulte http://www.barceloviajes.com/privacy LEGAL ADVISORY This message is confidential and intended only for the person or entity to which it is addressed. In order to read its privacy policy consult it at http://www.barceloviajes.com/privacy ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
RE: [LARTC] equalize / ecmp not working as expected in 2.6 vs 2.4
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Sent: 11 April 2007 09:01 To: lartc@mailman.ds9a.nl Subject: Re: [LARTC] equalize / ecmp not working as expected in 2.6 vs 2.4 Andy Furniss [EMAIL PROTECTED] wrote on 10/04/2007 21:29:33: [EMAIL PROTECTED] wrote: thanks for the info. First of all, in order to use the nth match you need to patch your kernel using patch-o-matic. I think nth is in kernel now as part of the statistic match. Andy. I was searching this yesterday afternoon and could not verify it. Today I found it in the 2.6.18 kernel's changelog: [NETFILTER]: x_tables: add statistic match Add statistic match which is a combination of the nth and random matches. Signed-off-by: Patrick McHardy ... Signed-off-by: David S. Miller ... commit 62b7743483b402f8fb73545d5d487ca714e82766 Author: Patrick McHardy ... Date: Mon May 29 18:20:32 2006 -0700 ¿ Does this match help you to solve the problem ? Regards, Eric-- ADVERTENCIA LEGAL El contenido de este correo es confidencial y dirigido unicamente a su destinatario. Para acceder a su clausula de privacidad consulte http://www.barceloviajes.com/privacy LEGAL ADVISORY This message is confidential and intended only for the person or entity to which it is addressed. In order to read its privacy policy consult it at http://www.barceloviajes.com/privacy Your suggestion pointed me in the right direction, it is now working with the following setup: Kernel 2.6.20-gentoo-r4 x86_64 Iptables 1.3.7 Iptables rules: iptables -t mangle -A OUTPUT -s localip2 -m statistic --mode nth --every 2 --packet 0 -j MARK --set-mark 111 iptables -t mangle -A OUTPUT -s localip2 -m statistic --mode nth --every 2 --packet 1 -j MARK --set-mark 222 localip2 is one of the ip addresses from the /28 range that is routed to both of our lines, in your example you said to add to PREROUTING, but the packets are from the box itself so I changed to OUTPUT, so far that has not caused any problems... any comments on that? We only want to do per-packet load balanced for some local and some routed ips, not all of them, some services cannot cope with the out of order packets that arise from sending outgoing traffic through two different links. IP Rules: ip rule add prio 111 fwmark 111 table ADSLLink1 ip rule add prio 222 fwmark 222 table ADSLLink2 Both ADSLLink1 and ADSLLink2 already existed and contain a default route via the router for line 1 or line 2, they also have routes for other subnets so that for example I can ping our routers from my workstation which has a private IP address, I wont show all the routes as the box has some 10 eth interfaces and it is very long and confusing, but the important bit is: ip route show table ADSLLink1 | grep default default via router1 dev inet0 ip route show table ADSLLink2 | grep default default via router2 dev inet0 This is currently working in combination with the ecmp routes that were already in place, and that is working very well for us, services that suffer when there are lots of OOOP's still get per-flow/cached route load balanced over the two lines, and services that can handle a few OOOP's are getting the full benefit of 2 x upload speed. Iptables also gives me much more fine grained control of the setup, when I have more time I will be making more improvements. A final note, I got very confused for a while last night because whenever I used iptables with -t mangle I got a error like this: iptables --list -t mangle Chain PREROUTING (policy ACCEPT) target prot opt source destination Chain INPUT (policy ACCEPT) target prot opt source destination Chain FORWARD (policy ACCEPT) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination FATAL: Module ip_tables not found. snip It turns out that this is a issue with having ip_tables compiled into the kernel, /sbin/iptables tries to modprobe it regardless of that and then fails because it is not a module, I believe a fix was posted to netfilter mailing list, I got rid of the error by making a dummy kernel module with the name ip_tables, not a nice solution but it does suppress the error. Many thanks for your help Andy JOSEDV001TAG ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Re: [LARTC] equalize / ecmp not working as expected in 2.6 vs 2.4
Hi Andrew, I would use a combination with iptables. You should mark the packets, for example using average or n-th, and then use ip rules to send half of the packets vía one router and the rest to the other router according to the marks you set with iptables. Just a question ¿ dont you have problems with your source IP and the returning responses when you are sending packets from one connection over multiple routers ? ¿ do you have something like an AS ? Best regards, Eric Janz Andrew Lyon [EMAIL PROTECTED] Enviado por: [EMAIL PROTECTED] 10/04/2007 14:36 Para 'lartc@mailman.ds9a.nl' lartc@mailman.ds9a.nl cc Asunto [LARTC] equalize / ecmp not working as expected in 2.6 vs 2.4 Hi, With kernel 2.4 I was able to use equalize to send our outgoing packets to two different routers (our isp supports this setup), like this: ip route add default src ip.a.dd.rr equalize nexthop via router1 weight 1 nexthop via router2 weight 1 The two routes were used equally on a per packet basis, not per flow or per cached route, but per packet, each line has 800k upload and with that route we could upload to a single remote host at 1.6mbit. We replaced the server with a newer one and changed to 2.6 (2.6.20) kernel, I found that equalize no longer works as expected, it does choose a router at random but once a single packet has been sent to a remote host the same route/router is used for all packets going to that remote host. Once the cached route expires a random route is chosen again, but that is not what we want. I had made no changes to the ip route commands, but then I realised that kernel 2.6.20 has options for multipath (IP: equal cost multipath with caching support), I enabled that and now our kernel options are: CONFIG_IP_ROUTE_MULTIPATH=y CONFIG_IP_ROUTE_MULTIPATH_CACHED=y CONFIG_IP_ROUTE_MULTIPATH_RR=m CONFIG_IP_ROUTE_MULTIPATH_RANDOM=m CONFIG_IP_ROUTE_MULTIPATH_WRANDOM=m CONFIG_IP_ROUTE_MULTIPATH_DRR=m But even with these options, and default route set as follows: ip route add default src ip.a.dd.rr mpath rr nexthop via router1 weight 1 nexthop via router2 weight 1 The result is the same, a single upload to a remote host only uses 800k bandwidth on one of the lines, it does not send packets to both lines, although two uploads to two different hosts will usually make use of both lines. It seems to me that the multipath with caching support is broken in 2.6.20? The exact kernel we use is 2.6.20.4, with that kernel how would you specify a remote route such that packets going to a remote host are sent 50/50 ratio to two different routers? Thanks Andy JOSEDV001TAG ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc -- ADVERTENCIA LEGAL El contenido de este correo es confidencial y dirigido unicamente a su destinatario. Para acceder a su clausula de privacidad consulte http://www.barceloviajes.com/privacy LEGAL ADVISORY This message is confidential and intended only for the person or entity to which it is addressed. In order to read its privacy policy consult it at http://www.barceloviajes.com/privacy ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
RE: [LARTC] equalize / ecmp not working as expected in 2.6 vs 2.4
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Sent: 10 April 2007 15:17 To: lartc@mailman.ds9a.nl Subject: Re: [LARTC] equalize / ecmp not working as expected in 2.6 vs 2.4 Hi Andrew, I would use a combination with iptables. You should mark the packets, for example using average or n-th, and then use ip rules to send half of the packets vía one router and the rest to the other router according to the marks you set with iptables. Just a question ¿ dont you have problems with your source IP and the returning responses when you are sending packets from one connection over multiple routers ? ¿ do you have something like an AS ? Best regards, Eric Janz Andrew Lyon [EMAIL PROTECTED] Enviado por: [EMAIL PROTECTED] 10/04/2007 14:36 Para 'lartc@mailman.ds9a.nl' lartc@mailman.ds9a.nl cc Asunto [LARTC] equalize / ecmp not working as expected in 2.6 vs 2.4 Eric, Could you give me a example of how to do that? With nth if possible... It is not common for a ISP to support that sort of setup, but they do http://aaisp.net.uk/aa/aaisp/multiline.html Each line has two ips, one for the router and another for the interface on a linux box or other device, the isp routes a larger /28 down both lines, and allows packets with source address in the /28 range to be sent through both lines. On my linux server I have a routing table for each line with the necessary routes to make each router ip reachable, and a default route that equalizes over both router ips, it worked with 2.4 but with 2.6 it seems to be per-flow instead of per packet. I can login to a control page app on the ISP website and configure which lines a given block is routed down, and they also do really good traffic monitoring etc http://www.aaisp.net.uk/cqm.html PS. Please reply below original posting, not above! http://en.wikipedia.org/wiki/Top-posting Andy JOSEDV001TAG ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
RE: [LARTC] equalize / ecmp not working as expected in 2.6 vs 2.4
Andrew Lyon [EMAIL PROTECTED] wrote on 10/04/2007 16:34:52: From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Sent: 10 April 2007 15:17 To: lartc@mailman.ds9a.nl Subject: Re: [LARTC] equalize / ecmp not working as expected in 2.6 vs 2.4 Hi Andrew, I would use a combination with iptables. You should mark the packets, for example using average or n-th, and then use ip rules to send half of the packets vía one router and the rest to the other router according to the marks you set with iptables. Just a question ¿ dont you have problems with your source IP and the returning responses when you are sending packets from one connection over multiple routers ? ¿ do you have something like an AS ? Best regards, Eric Janz Andrew Lyon [EMAIL PROTECTED] Enviado por: [EMAIL PROTECTED] 10/04/2007 14:36 Para 'lartc@mailman.ds9a.nl' lartc@mailman.ds9a.nl cc Asunto [LARTC] equalize / ecmp not working as expected in 2.6 vs 2.4 Eric, Could you give me a example of how to do that? With nth if possible... It is not common for a ISP to support that sort of setup, but they do http://aaisp.net.uk/aa/aaisp/multiline.html Each line has two ips, one for the router and another for the interface on a linux box or other device, the isp routes a larger /28 down both lines, and allows packets with source address in the /28 range to be sent through both lines. On my linux server I have a routing table for each line with the necessary routes to make each router ip reachable, and a default route that equalizes over both router ips, it worked with 2.4 but with 2.6 it seems to be per-flow instead of per packet. I can login to a control page app on the ISP website and configure which lines a given block is routed down, and they also do really good traffic monitoring etc http://www.aaisp.net.uk/cqm.html PS. Please reply below original posting, not above! http://en.wikipedia.org/wiki/Top-posting Andy JOSEDV001TAG Hi Andy, thanks for the info. First of all, in order to use the nth match you need to patch your kernel using patch-o-matic. After that, the nth match should be available. Try something like this: Supposing that the local traffic is entering into your linux server via eth0: 1. Mark the packets using iptables before the routing decision is done: iptables -t mangle -A PREROUTING -i eth0 -m nth --every 2 --packet 0 -j MARK --set-mark 111 iptables -t mangle -A PREROUTING -i eth0 -m nth --every 2 --packet 1 -j MARK --set-mark 222 2. Setup some rules to jump to the correct routing tables. In this case I will suppose that you are using the tables 111 and 222 ( obviously you can use the ones you like ) ip rule add prio 111 fwmark 111 table 111 ip rule add prio 222 fwmark 222 table 222 ( you can also set the priority of the rules at your convenience ) 3. Setup your routing tables ( in this example 111 and 222 ) to reach each router as you had with the 2.4 kernel. [ ... ] ip route add table 111 default via ROUTER1_IP_ADDRESS ip route add table 222 default via ROUTER2_IP_ADDRESS In this case we need no multipath route. Half of all the packets that come into eth0 are routed using the 111 table and the rest is routed using the 222 table thanks to the marks we set. The problems you are experiencing with the multipath routing are due to that the routing decision uses a cache and after a routing decision to a destination is done, it would always use the same gateway to reach that destination until the routing caché expires. I hope this helps, Regards, Eric Janz -- ADVERTENCIA LEGAL El contenido de este correo es confidencial y dirigido unicamente a su destinatario. Para acceder a su clausula de privacidad consulte http://www.barceloviajes.com/privacy LEGAL ADVISORY This message is confidential and intended only for the person or entity to which it is addressed. In order to read its privacy policy consult it at http://www.barceloviajes.com/privacy ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
