RE: [LARTC] Unequal Multipath Routing?
Luciano Ruete wrote: On Wednesday 28 June 2006 12:01, Andrew Lyon wrote: That's probably an equalize bug, that does not take in account 'weight' for packet balance. Equalize is unmaintained software AFAIK. Perhaps not so much a bug as a lack of functionality.. Back to my original question then, is there anything in 2.6 or a patch for 2.4 that could be used to do that? (4:3 ratio split of outgoing packets over two interfaces/gateways). Why you need the balance in a packet basis? If you use a vanilla kernel and simple multipath, then the radio 4:3 should work ok. Simple.. we use apps that need more bandwidth than a single line can provide. Andy /*JOSEDV001TAG/* ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
RE: [LARTC] Unequal Multipath Routing?
Torsten Luettgert wrote: On Mi, 2006-06-28 at 16:35 +0100, Andrew Lyon wrote: Back to my original question then, is there anything in 2.6 or a patch for 2.4 that could be used to do that? (4:3 ratio split of outgoing packets over two interfaces/gateways). If you aren't afraid of patching compiling kernels, there is one solution. It's a bit ugly, but works (we sell bundled DSL lines using this method). bundled dsl is exactly what this is. The basic idea is to use the iptables ROUTE target to make exemptions from the default gw. It works like this: First make a kernel with the netfilter random and ROUTE targets (can be obtained from patch-o-matic-ng, but they are removed from the current HEAD - you'd have to check out an older revision, or I could send you my copies which work with 2.6.16 and 2.6.17). I will get our box upgraded to a 2.6 kernel and give it a try, there is one feature in 2.4 that we need at the moment but I think I found a 2.6 solution for that a few days ago. Then, point the default route to the bigger pipe, and add an iptables rule like this: iptables -t mangle -A POSTROUTING -o (interface of default route) \ -m random --average 43 \ -j ROUTE --gw (ip of other gateway) Makes sense. I only tried this with different interfaces for different upstreams, but thinking about it, it should also work if they are on the same interface. 43% is about 3/7, so about 3/7 of your packets would use the slower line. Next thing to worry about would be the downstream :) Our isp does the downstream already (its 50/50 but our lines are same speed download), we have a /30 on each line (one ip for router, one for gateway linux box), and a /28 which is routed down both lines, and ipv6 too, our isp is very innovative, they built their own l2pp router which does all this and more AA www.aaisp.net.uk Some remarks: - If you can make the downstream work the same way, you have true packet-based bundling, so single connections will also experience the full bandwidth. Depending on how the downstream is configured, different things can happen (only one line used, downstream bundled per-connection, downstream is 50/50 instead 43/57). - if the lines have different latencies, packets can arrive in different order, so e.g. VoIP won't be pleasure - connecting to the modems from your box will need some more rules (packets would also be sent to the modem you're not talking to) - I'm not 100% sure the random match options are right, I used the nth match for lines of equal sizes (so it's round-robin), not random. One could also use a cascade of nth matches to make it round-robin 4:3 (abababa abababa ...) I've already run into the out of order packets problem, pptpd supports this but it seems to be a little buggy at reassembling packets, basically it doesn't work so I had to move our vpn onto the /30 ips which are not bonded, I used round robin dns to load balance our vpn clients across both lines, crude but better than nothing! Thanks for your help Andy /*JOSEDV001TAG/* ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Re: [LARTC] Unequal Multipath Routing?
On Wednesday 28 June 2006 05:19, Andrew Lyon wrote: Peter Surda wrote: On Tuesday 27 June 2006 15:16, Luciano Ruete wrote: how about: ip route add default nexthop via a.a.a.a weight 4 nexthop via b.b.b.b weight 3 exactly. Not tested but i think it can work. tested, works. Luciano Yours sincerely, Peter It works in so far as the command is accepted and there is no error, but having changed the default route and then doing a upload by ftp I can see that both lines are still getting 1/2 of the outgoing packets. One upload means nothing, plain multipath(vanilla kernel, with multipath cached not set) take in account destination address(DA for convenience) and TOS. For each new pair of DA and TOS it takes the nexthop available. So doing an ftp to a single host will make no difference. Think in connections instead of packets, for a 'per packet multipath' you need to have same source address for all your choosed gw/isps and to patch your kernel. I am fairly sure about this because both lines are adsl, when the upstream is saturated the latency goes up and this is reflected in graphs that our isp make available, the line with 600kbit upload has noticeably higher latency, the line with 800kbit does not. The latency problem is easy to solve, making your linux box to be the one who manage the queue, see section 9.2.2.2 of LARTC HowTo. Have you verified that it does actually distribute the packets in a different ratio? I think multipath is just random. As i say, each time a new pair of DA and TOS arrives it takes the nexthop, in the case mentioned above, it will choose 4 times the hop a.a.a.a and 3 times the hop b.b.b.b and so on. It is possibly that you start a heavy http download and goes by a, and then a ping and goes by b, and then a new heavy http download and it goes by a again. So one line is saturated while the other is empty, thats why multipath works better(fairly) as the number of clients arise. -- Luciano ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
RE: [LARTC] Unequal Multipath Routing?
Luciano Ruete wrote: On Wednesday 28 June 2006 05:19, Andrew Lyon wrote: Peter Surda wrote: On Tuesday 27 June 2006 15:16, Luciano Ruete wrote: how about: ip route add default nexthop via a.a.a.a weight 4 nexthop via b.b.b.b weight 3 exactly. Not tested but i think it can work. tested, works. Luciano Yours sincerely, Peter It works in so far as the command is accepted and there is no error, but having changed the default route and then doing a upload by ftp I can see that both lines are still getting 1/2 of the outgoing packets. One upload means nothing, plain multipath(vanilla kernel, with multipath cached not set) take in account destination address(DA for convenience) and TOS. For each new pair of DA and TOS it takes the nexthop available. So doing an ftp to a single host will make no difference. Think in connections instead of packets, for a 'per packet multipath' you need to have same source address for all your choosed gw/isps and to patch your kernel. It am not using plain multipath, its equalized multipath using the patch eql-patch-2.4.30.gz. My upload was using both lines, our ISP graphs include bandwidth and I can see the upload was approx 550kbit on each line, but the capacity is 600kbit and 800kbit. I am fairly sure about this because both lines are adsl, when the upstream is saturated the latency goes up and this is reflected in graphs that our isp make available, the line with 600kbit upload has noticeably higher latency, the line with 800kbit does not. The latency problem is easy to solve, making your linux box to be the one who manage the queue, see section 9.2.2.2 of LARTC HowTo. I am not trying to solve the latency, I was only using it as a rough guide of whether the data was being sent in the ratio I expected to the two lines. Have you verified that it does actually distribute the packets in a different ratio? I think multipath is just random. As i say, each time a new pair of DA and TOS arrives it takes the nexthop, in the case mentioned above, it will choose 4 times the hop a.a.a.a and 3 times the hop b.b.b.b and so on. Yes, unless you are using equalize patch. It is possibly that you start a heavy http download and goes by a, and then a ping and goes by b, and then a new heavy http download and it goes by a again. So one line is saturated while the other is empty, thats why multipath works better(fairly) as the number of clients arise. JOSEDV001TAG ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Re: [LARTC] Unequal Multipath Routing?
On Wednesday 28 June 2006 11:01, Andrew Lyon wrote: Luciano Ruete wrote: On Wednesday 28 June 2006 05:19, Andrew Lyon wrote: Peter Surda wrote: On Tuesday 27 June 2006 15:16, Luciano Ruete wrote: how about: ip route add default nexthop via a.a.a.a weight 4 nexthop via b.b.b.b weight 3 exactly. Not tested but i think it can work. tested, works. Luciano Yours sincerely, Peter It works in so far as the command is accepted and there is no error, but having changed the default route and then doing a upload by ftp I can see that both lines are still getting 1/2 of the outgoing packets. One upload means nothing, plain multipath(vanilla kernel, with multipath cached not set) take in account destination address(DA for convenience) and TOS. For each new pair of DA and TOS it takes the nexthop available. So doing an ftp to a single host will make no difference. Think in connections instead of packets, for a 'per packet multipath' you need to have same source address for all your choosed gw/isps and to patch your kernel. It am not using plain multipath, its equalized multipath using the patch eql-patch-2.4.30.gz. My upload was using both lines, our ISP graphs include bandwidth and I can see the upload was approx 550kbit on each line, but the capacity is 600kbit and 800kbit. That's probably an equalize bug, that does not take in account 'weight' for packet balance. Equalize is unmaintained software AFAIK. -- Luciano ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
RE: [LARTC] Unequal Multipath Routing?
On Mi, 2006-06-28 at 16:35 +0100, Andrew Lyon wrote: Back to my original question then, is there anything in 2.6 or a patch for 2.4 that could be used to do that? (4:3 ratio split of outgoing packets over two interfaces/gateways). If you aren't afraid of patching compiling kernels, there is one solution. It's a bit ugly, but works (we sell bundled DSL lines using this method). The basic idea is to use the iptables ROUTE target to make exemptions from the default gw. It works like this: First make a kernel with the netfilter random and ROUTE targets (can be obtained from patch-o-matic-ng, but they are removed from the current HEAD - you'd have to check out an older revision, or I could send you my copies which work with 2.6.16 and 2.6.17). Then, point the default route to the bigger pipe, and add an iptables rule like this: iptables -t mangle -A POSTROUTING -o (interface of default route) \ -m random --average 43 \ -j ROUTE --gw (ip of other gateway) I only tried this with different interfaces for different upstreams, but thinking about it, it should also work if they are on the same interface. 43% is about 3/7, so about 3/7 of your packets would use the slower line. Next thing to worry about would be the downstream :) Some remarks: - If you can make the downstream work the same way, you have true packet-based bundling, so single connections will also experience the full bandwidth. Depending on how the downstream is configured, different things can happen (only one line used, downstream bundled per-connection, downstream is 50/50 instead 43/57). - if the lines have different latencies, packets can arrive in different order, so e.g. VoIP won't be pleasure - connecting to the modems from your box will need some more rules (packets would also be sent to the modem you're not talking to) - I'm not 100% sure the random match options are right, I used the nth match for lines of equal sizes (so it's round-robin), not random. One could also use a cascade of nth matches to make it round-robin 4:3 (abababa abababa ...) Regards, Torsten ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Re: [LARTC] Unequal Multipath Routing?
On Tuesday 27 June 2006 15:16, Luciano Ruete wrote: how about: ip route add default nexthop via a.a.a.a weight 4 nexthop via b.b.b.b weight 3 exactly. Not tested but i think it can work. tested, works. Luciano Yours sincerely, Peter -- http://www.shurdix.org - Linux distribution for routers and firewalls ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
