RE: [LARTC] Unequal Multipath Routing?
Torsten Luettgert wrote: > On Mi, 2006-06-28 at 16:35 +0100, Andrew Lyon wrote: >> Back to my original question then, is there anything in 2.6 or a >> patch for >> 2.4 that could be used to do that? (4:3 ratio split of outgoing >> packets over two interfaces/gateways). > > If you aren't afraid of patching & compiling kernels, there is one > solution. It's a bit ugly, but works (we sell "bundled" DSL lines > using this method). "bundled" dsl is exactly what this is. > The basic idea is to use the iptables ROUTE target to make exemptions > from the default gw. > > It works like this: > > First make a kernel with the netfilter random and ROUTE targets (can > be obtained from patch-o-matic-ng, but they are removed from the > current HEAD - you'd have to check out an older revision, or I could > send you my copies which work with 2.6.16 and 2.6.17). I will get our box upgraded to a 2.6 kernel and give it a try, there is one feature in 2.4 that we need at the moment but I think I found a 2.6 solution for that a few days ago. > Then, point the default route to the bigger pipe, and add an iptables > rule like this: > > iptables -t mangle -A POSTROUTING -o (interface of default route) \ > -m random --average 43 \ > -j ROUTE --gw (ip of other gateway) Makes sense. > I only tried this with different interfaces for different upstreams, > but thinking about it, it should also work if they are on the same > interface. > > 43% is about 3/7, so about 3/7 of your packets would use the slower > line. Next thing to worry about would be the downstream :) Our isp does the downstream already (its 50/50 but our lines are same speed download), we have a /30 on each line (one ip for router, one for gateway linux box), and a /28 which is routed down both lines, and ipv6 too, our isp is very innovative, they built their own l2pp router which does all this and more A&A www.aaisp.net.uk > Some remarks: > - If you can make the downstream work the same way, you have true > packet-based bundling, so single connections will also experience > the full bandwidth. Depending on how the downstream is configured, > different things can happen (only one line used, downstream > bundled per-connection, downstream is 50/50 instead 43/57). > - if the lines have different latencies, packets can arrive > in different order, so e.g. VoIP won't be pleasure > - connecting to the modems from your box will need some more rules > (packets would also be sent to the modem you're not talking to) > - I'm not 100% sure the random match options are right, I used the > nth match for lines of equal sizes (so it's round-robin), not > random. One could also use a cascade of nth matches to make > it round-robin 4:3 (abababa abababa ...) I've already run into the out of order packets problem, pptpd supports this but it seems to be a little buggy at reassembling packets, basically it doesn't work so I had to move our vpn onto the /30 ips which are not bonded, I used round robin dns to load balance our vpn clients across both lines, crude but better than nothing! Thanks for your help Andy /*JOSEDV001TAG/* ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
RE: [LARTC] Unequal Multipath Routing?
Luciano Ruete wrote: > On Wednesday 28 June 2006 12:01, Andrew Lyon wrote: >>> That's probably an equalize bug, that does not take in account >>> 'weight' for packet balance. Equalize is unmaintained software >>> AFAIK. >> >> Perhaps not so much a bug as a lack of functionality.. >> >> Back to my original question then, is there anything in 2.6 or a >> patch for >> 2.4 that could be used to do that? (4:3 ratio split of outgoing >> packets over two interfaces/gateways). > > Why you need the balance in a packet basis? > > If you use a vanilla kernel and simple multipath, then the radio 4:3 > should work ok. Simple.. we use apps that need more bandwidth than a single line can provide. Andy /*JOSEDV001TAG/* ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
RE: [LARTC] Unequal Multipath Routing?
On Mi, 2006-06-28 at 16:35 +0100, Andrew Lyon wrote: > Back to my original question then, is there anything in 2.6 or a patch for > 2.4 that could be used to do that? (4:3 ratio split of outgoing packets over > two interfaces/gateways). If you aren't afraid of patching & compiling kernels, there is one solution. It's a bit ugly, but works (we sell "bundled" DSL lines using this method). The basic idea is to use the iptables ROUTE target to make exemptions from the default gw. It works like this: First make a kernel with the netfilter random and ROUTE targets (can be obtained from patch-o-matic-ng, but they are removed from the current HEAD - you'd have to check out an older revision, or I could send you my copies which work with 2.6.16 and 2.6.17). Then, point the default route to the bigger pipe, and add an iptables rule like this: iptables -t mangle -A POSTROUTING -o (interface of default route) \ -m random --average 43 \ -j ROUTE --gw (ip of other gateway) I only tried this with different interfaces for different upstreams, but thinking about it, it should also work if they are on the same interface. 43% is about 3/7, so about 3/7 of your packets would use the slower line. Next thing to worry about would be the downstream :) Some remarks: - If you can make the downstream work the same way, you have true packet-based bundling, so single connections will also experience the full bandwidth. Depending on how the downstream is configured, different things can happen (only one line used, downstream bundled per-connection, downstream is 50/50 instead 43/57). - if the lines have different latencies, packets can arrive in different order, so e.g. VoIP won't be pleasure - connecting to the modems from your box will need some more rules (packets would also be sent to the modem you're not talking to) - I'm not 100% sure the random match options are right, I used the nth match for lines of equal sizes (so it's round-robin), not random. One could also use a cascade of nth matches to make it round-robin 4:3 (abababa abababa ...) Regards, Torsten ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Re: [LARTC] Unequal Multipath Routing?
On Wednesday 28 June 2006 12:01, Andrew Lyon wrote: > > That's probably an equalize bug, that does not take in account > > 'weight' for packet balance. Equalize is unmaintained software AFAIK. > > Perhaps not so much a bug as a lack of functionality.. > > Back to my original question then, is there anything in 2.6 or a patch for > 2.4 that could be used to do that? (4:3 ratio split of outgoing packets > over two interfaces/gateways). Why you need the balance in a packet basis? If you use a vanilla kernel and simple multipath, then the radio 4:3 should work ok. -- Luciano ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
RE: [LARTC] Unequal Multipath Routing?
> That's probably an equalize bug, that does not take in account > 'weight' for packet balance. Equalize is unmaintained software AFAIK. Perhaps not so much a bug as a lack of functionality.. Back to my original question then, is there anything in 2.6 or a patch for 2.4 that could be used to do that? (4:3 ratio split of outgoing packets over two interfaces/gateways). Andy JOSEDV001TAG ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Re: [LARTC] Unequal Multipath Routing?
On Wednesday 28 June 2006 11:01, Andrew Lyon wrote: > Luciano Ruete wrote: > > On Wednesday 28 June 2006 05:19, Andrew Lyon wrote: > >> Peter Surda wrote: > >>> On Tuesday 27 June 2006 15:16, Luciano Ruete wrote: > how about: > ip route add default nexthop via a.a.a.a weight 4 nexthop via > b.b.b.b weight 3 > >>> > >>> exactly. > >>> > Not tested but i think it can work. > >>> > >>> tested, works. > >>> > Luciano > >>> > >>> Yours sincerely, > >>> Peter > >> > >> It works in so far as the command is accepted and there is no error, > >> but having changed the default route and then doing a upload by ftp I > >> can see that both lines are still getting 1/2 of the outgoing > >> packets. > > > > One upload means nothing, plain multipath(vanilla kernel, with > > multipath cached not set) take in account destination address(DA for > > convenience) and TOS. For each new pair of DA and TOS it takes the > > nexthop available. So doing an ftp to a single host will make no > > difference. Think in connections instead of packets, for a 'per > > packet multipath' you need to have same source address for all your > > choosed gw/isps and to patch your kernel. > > It am not using plain multipath, its equalized multipath using the patch > eql-patch-2.4.30.gz. > > My upload was using both lines, our ISP graphs include bandwidth and I can > see the upload was approx 550kbit on each line, but the capacity is 600kbit > and 800kbit. That's probably an equalize bug, that does not take in account 'weight' for packet balance. Equalize is unmaintained software AFAIK. -- Luciano ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
RE: [LARTC] Unequal Multipath Routing?
Luciano Ruete wrote: > On Wednesday 28 June 2006 05:19, Andrew Lyon wrote: >> Peter Surda wrote: >>> On Tuesday 27 June 2006 15:16, Luciano Ruete wrote: how about: ip route add default nexthop via a.a.a.a weight 4 nexthop via b.b.b.b weight 3 >>> >>> exactly. >>> Not tested but i think it can work. >>> >>> tested, works. >>> Luciano >>> >>> Yours sincerely, >>> Peter >> >> It works in so far as the command is accepted and there is no error, >> but having changed the default route and then doing a upload by ftp I >> can see that both lines are still getting 1/2 of the outgoing >> packets. > > One upload means nothing, plain multipath(vanilla kernel, with > multipath cached not set) take in account destination address(DA for > convenience) and TOS. For each new pair of DA and TOS it takes the > nexthop available. So doing an ftp to a single host will make no > difference. Think in connections instead of packets, for a 'per > packet multipath' you need to have same source address for all your > choosed gw/isps and to patch your kernel. It am not using plain multipath, its equalized multipath using the patch eql-patch-2.4.30.gz. My upload was using both lines, our ISP graphs include bandwidth and I can see the upload was approx 550kbit on each line, but the capacity is 600kbit and 800kbit. > >> I am fairly sure about this because both lines are adsl, when the >> upstream is saturated the latency goes up and this is reflected in >> graphs that our isp make available, the line with 600kbit upload has >> noticeably higher latency, the line with 800kbit does not. > > The latency problem is easy to solve, making your linux box to be the > one who manage the queue, see section 9.2.2.2 of LARTC HowTo. I am not trying to solve the latency, I was only using it as a rough guide of whether the data was being sent in the ratio I expected to the two lines. > >> Have you verified that it does actually distribute the packets in a >> different ratio? I think multipath is just random. > > As i say, each time a new pair of DA and TOS arrives it takes the > nexthop, in the case mentioned above, it will choose 4 times the hop > a.a.a.a and 3 times the hop b.b.b.b and so on. > Yes, unless you are using equalize patch. > It is possibly that you start a heavy http download and goes by a, > and then a ping and goes by b, and then a new heavy http download and > it goes by a again. So one line is saturated while the other is > empty, thats why multipath works better(fairly) as the number of > clients arise. JOSEDV001TAG ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Re: [LARTC] Unequal Multipath Routing?
On Wednesday 28 June 2006 05:19, Andrew Lyon wrote: > Peter Surda wrote: > > On Tuesday 27 June 2006 15:16, Luciano Ruete wrote: > >> how about: > >> ip route add default nexthop via a.a.a.a weight 4 nexthop via b.b.b.b > >> weight 3 > > > > exactly. > > > >> Not tested but i think it can work. > > > > tested, works. > > > >> Luciano > > > > Yours sincerely, > > Peter > > It works in so far as the command is accepted and there is no error, but > having changed the default route and then doing a upload by ftp I can see > that both lines are still getting 1/2 of the outgoing packets. One upload means nothing, plain multipath(vanilla kernel, with multipath cached not set) take in account destination address(DA for convenience) and TOS. For each new pair of DA and TOS it takes the nexthop available. So doing an ftp to a single host will make no difference. Think in connections instead of packets, for a 'per packet multipath' you need to have same source address for all your choosed gw/isps and to patch your kernel. > I am fairly sure about this because both lines are adsl, when the upstream > is saturated the latency goes up and this is reflected in graphs that our > isp make available, the line with 600kbit upload has noticeably higher > latency, the line with 800kbit does not. The latency problem is easy to solve, making your linux box to be the one who manage the queue, see section 9.2.2.2 of LARTC HowTo. > Have you verified that it does actually distribute the packets in a > different ratio? I think multipath is just random. As i say, each time a new pair of DA and TOS arrives it takes the nexthop, in the case mentioned above, it will choose 4 times the hop a.a.a.a and 3 times the hop b.b.b.b and so on. It is possibly that you start a heavy http download and goes by a, and then a ping and goes by b, and then a new heavy http download and it goes by a again. So one line is saturated while the other is empty, thats why multipath works better(fairly) as the number of clients arise. -- Luciano ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
RE: [LARTC] Unequal Multipath Routing?
Peter Surda wrote: > On Tuesday 27 June 2006 15:16, Luciano Ruete wrote: >> how about: >> ip route add default nexthop via a.a.a.a weight 4 nexthop via b.b.b.b >> weight 3 > exactly. > >> Not tested but i think it can work. > tested, works. > >> Luciano > Yours sincerely, > Peter It works in so far as the command is accepted and there is no error, but having changed the default route and then doing a upload by ftp I can see that both lines are still getting 1/2 of the outgoing packets. I am fairly sure about this because both lines are adsl, when the upstream is saturated the latency goes up and this is reflected in graphs that our isp make available, the line with 600kbit upload has noticeably higher latency, the line with 800kbit does not. Have you verified that it does actually distribute the packets in a different ratio? I think multipath is just random. andy Registered Office: J.O. Sims Ltd, Pudding Lane, Pinchbeck, Spalding, Lincs. PE11 3TJ Company reg No: 2084187 Vat reg No: GB 437 4621 47 Tel: +44 (0) 1775 842100 Fax: +44 (0) 1775 842101 Web: www.josims.com Email:[EMAIL PROTECTED] The information contained in this e-mail is confidential and is intended for the addressee only. The contents of this e-mail must not be disclosed or copied without the sender's consent. If you are not the intended recipient of the message, please notify the sender immediately, and delete the message. The statements and opinions expressed in this message are those of the author and do not necessarily reflect those of the company. No commitment may be inferred from the contents unless explicitly stated. The company does not take any responsibility for the personal views of the author. This message has been scanned for viruses before sending, but the company does not accept any responsibility for infection and recommends that you scan any attachments.JOSEDV001TAG ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Re: [LARTC] Unequal Multipath Routing?
On Tuesday 27 June 2006 15:16, Luciano Ruete wrote: > how about: > ip route add default nexthop via a.a.a.a weight 4 nexthop via b.b.b.b > weight 3 exactly. > Not tested but i think it can work. tested, works. > Luciano Yours sincerely, Peter -- http://www.shurdix.org - Linux distribution for routers and firewalls ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Re: [LARTC] Unequal Multipath Routing?
On Tuesday 27 June 2006 08:33, Andrew Lyon wrote: > Hi, > > I am currently using a 2.4 kernel with multipath routing patch v5 to send > outgoing packets in a 1:1 ratio over two lines, however the lines do not > actually have equal upstream bandwidth, one has 800kbit and the other > 600kbit, so ideally I would like to send packets in a 4:3 ratio, is there > anything in 2.6 or a patch for 2.4 that could be used to do that? > > I tried this ugly hack but it didn't seem to work: > > ip route add default src my.ip.add.res equalize nexthop via a.a.a.a weight > 1 nexthop via b.b.b.b weight 1 nexthop via a.a.a.a weight 1 exthop via > b.b.b.b weight 1 nexthop via a.a.a.a weight 1 nexthop via b.b.b.b weight 1 > nexthop via a.a.a.a weight 1 how about: ip route add default nexthop via a.a.a.a weight 4 nexthop via b.b.b.b weight 3 Not tested but i think it can work. -- Luciano ___ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
