Re: [aqm] RED implementation on Linux 3.14.22
Hi Jim, Hi Dave, Thanks a lot for the pointers. Very helpful! And happy new year too! It took me some time to catch up here, sorry for that. From the links/options you posted, it looks like there was no major change (bug fix), only refactoring, since 2012: https://github.com/torvalds/linux/commits/master/net/sched/sch_red.c On 15 December 2014 at 17:17, Dave Taht wrote: > On Mon, Dec 15, 2014 at 7:54 AM, Dave Taht wrote: >> On Mon, Dec 15, 2014 at 5:41 AM, Jim Gettys wrote: >>> >>> >>> On Mon, Dec 15, 2014 at 2:51 AM, Simone Ferlin-Oliveira >>> wrote: All, I am doing some work with shared bottleneck detection that requires some evaluation with different AQM, in particular, RED. Since I haven't been following the evolution of the implementation, I would like to ask about your experience with the code on Linux 3.14 (and newer). > > I need to clarify something about "newer". The third parameter in Linux is > for bug fixes only. 3.14 is the major release, a 3.14.22 was 22 bug > fix releases. A -X or 4th parameter, if it exists, is distro specific > changes, which can often, particularly in major distros like redhat or > ubuntu, be quite extensive. > > "New features", such as the ones I mentioned in the previous email, generally > do not make it to the bug fix releases, and I don't know if (for > example) the hystart > change or GSO half cwnd change will make it to the -stable tree for > older releases (without checking), as usually only security or crash > critical bugs make it into stable. > > I mention this in light of a fairly recent DCTCP paper which used a > pre-bufferbloat-fixes kernel of 3.2.something, discussed (Well, ranted > about slightly, apologies) here. > > https://lists.bufferbloat.net/pipermail/bloat/2013-November/001736.html > > (I would dearly like to see that paper's experiments revised and > updated in light of that discussion, now that all these other fixes > have landed, and DCTCP is now in mainline linux.) > > I try to publish a simple debian kernel build script, and my own patch > set of the codel-related research in progress regularly, somewhere: > > http://snapon.lab.bufferbloat.net/~d/codel_patches/ > > and will probably restart publishing a separate debloat-testing tree > for the upcoming make-wifi-fast effort, as that set of changes is > going to be quite extensive, and buggy, for a while. > > -- > Dave Täht > > http://www.bufferbloat.net/projects/bloat/wiki/Upcoming_Talks ___ aqm mailing list aqm@ietf.org https://www.ietf.org/mailman/listinfo/aqm
Re: [aqm] RED implementation on Linux 3.14.22
On Mon, Dec 15, 2014 at 7:54 AM, Dave Taht wrote: > On Mon, Dec 15, 2014 at 5:41 AM, Jim Gettys wrote: >> >> >> On Mon, Dec 15, 2014 at 2:51 AM, Simone Ferlin-Oliveira >> wrote: >>> >>> All, >>> >>> I am doing some work with shared bottleneck detection that requires >>> some evaluation with different AQM, in particular, RED. Since I >>> haven't been following the evolution of the implementation, I would >>> like to ask about your experience with the code on Linux 3.14 (and >>> newer). I need to clarify something about "newer". The third parameter in Linux is for bug fixes only. 3.14 is the major release, a 3.14.22 was 22 bug fix releases. A -X or 4th parameter, if it exists, is distro specific changes, which can often, particularly in major distros like redhat or ubuntu, be quite extensive. "New features", such as the ones I mentioned in the previous email, generally do not make it to the bug fix releases, and I don't know if (for example) the hystart change or GSO half cwnd change will make it to the -stable tree for older releases (without checking), as usually only security or crash critical bugs make it into stable. I mention this in light of a fairly recent DCTCP paper which used a pre-bufferbloat-fixes kernel of 3.2.something, discussed (Well, ranted about slightly, apologies) here. https://lists.bufferbloat.net/pipermail/bloat/2013-November/001736.html (I would dearly like to see that paper's experiments revised and updated in light of that discussion, now that all these other fixes have landed, and DCTCP is now in mainline linux.) I try to publish a simple debian kernel build script, and my own patch set of the codel-related research in progress regularly, somewhere: http://snapon.lab.bufferbloat.net/~d/codel_patches/ and will probably restart publishing a separate debloat-testing tree for the upcoming make-wifi-fast effort, as that set of changes is going to be quite extensive, and buggy, for a while. -- Dave Täht http://www.bufferbloat.net/projects/bloat/wiki/Upcoming_Talks ___ aqm mailing list aqm@ietf.org https://www.ietf.org/mailman/listinfo/aqm
Re: [aqm] RED implementation on Linux 3.14.22
On Mon, Dec 15, 2014 at 5:41 AM, Jim Gettys wrote: > > > On Mon, Dec 15, 2014 at 2:51 AM, Simone Ferlin-Oliveira > wrote: >> >> All, >> >> I am doing some work with shared bottleneck detection that requires >> some evaluation with different AQM, in particular, RED. Since I >> haven't been following the evolution of the implementation, I would >> like to ask about your experience with the code on Linux 3.14 (and >> newer). > > > I know that Dave Taht ran into bugs in RED a while back, which I believe > have been fixed for quite a while. The power of git to answer questions like this is unparalleled. Taking a look at my current kernel tree and doing a: git log net/sched/sch_red.c shows eric fixed 2 bugs in Linux RED in commit 1ee5fa1e9970a16036e37c7b9d5ce81c778252fc Author: Eric Dumazet Date: Thu Dec 1 11:06:34 2011 + sch_red: fix red_change ... http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=1ee5fa1e9970a16036e37c7b9d5ce81c778252fc ARED was added slightly later, and sfqred (a first attempt at blending fq + AQM together) shortly after that. (which doesn't do ared (never made it to mainline as fq_codel landed soon after that) Advice: Keep track of net-next, do git pulls regularly, and watch "git log net" for changes. > You should always be looking at whether code has been patched in the current > kernel.org system for a module like that you are interested in, so do a diff > between 3.14 and the current Linux system. 3.14 is recent enough that it may > be viable for experiments, for the time being. Planning to keep up with > Linux development is wise long term in any case, as the rate of > improvement/change in the networking stack is very high at the moment as > draining the bufferbloat swamp and other performance work continues. Important changes since 3.14: pie added, DCTCP added, gso/tso offloads seriously reworked and made gentler, sch_fq's pacing improved. The last kernel rounds (3.18,3.19) were seriously productive: hystart improved at longer RTTs, still more TSO/gso improvements, and xmit_more support was added for some devices. Also support for per route congestion control settings (primarily targetted at DCTCP) was just added. I believe some of the long RTT falloff we saw in toke's paper was due to hystart issues, as I have been unable to duplicate some of his results with this upcoming release. I have basically thrown out all my 3.14 results at this point and am starting over with the soon-to-stablize 3.19 release. (Well, in fact, I ended up starting over 3 times in the last 2 months as each of the new features above landed in the kernel) (but as for red, no changes except in the underlying TCPs and device drivers) Relevant commits were: Hystart change: http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=42eef7a0bb0989cd50d74e673422ff98a0ce4d7b xmit_more: http://netoptimizer.blogspot.com/2014/10/unlocked-10gbps-tx-wirespeed-smallest.html very good lwn article on it http://lwn.net/Articles/615238/ one of several GSO fixes: commit d649a7a81f3b5bacb1d60abd7529894d8234a666 Author: Eric Dumazet Date: Thu Nov 13 09:45:22 2014 -0800 tcp: limit GSO packets to half cwnd ... etc. Do a git log net. :) preso that convinced systemd to switch to fq_codel: http://lwn.net/Articles/616241/ > Also note that underlying device drivers may have (sometimes lots) of > buffering out of control of the Linux queue discipline. For Ethernet > devices, you should ensure that that the drivers have BQL support > implemented to minimize this buffering. Other classes of drivers are more > problematic, and may have lots of buffering to surprise you. +10 (or rather, -10). It's up to 25 devices now. I note that TSO/GSO used to interact very badly with soft rate limiting (htb), it seems better now. > > Also be aware that ethernet flow control may move the bottleneck from where > you expect to somewhere else, and that switches in networks also have to be > well understood. Most consumer switches have this *on* by default, and > mixed 1G/100Mb networks can be particularly "entertaining" in this regard. > Cable modems, unfortunately, typically do not implement flow control, but > some DSL modems do (putting the bottleneck into your router, rather than in > the modem). I should probably put red back into my test matrixes. I stopped benchmarking it and pfifo_fast a long time ago. A netperf-wrapper data set that predates the hystart fix, testing 3 RTTs: http://snapon.lab.bufferbloat.net/~d/comprehensive.puck/ or: http://snapon.lab.bufferbloat.net/~d/comprehensive_puck.tgz >> >> *Any* help is appreciated. > > > Hope this helps. > >> >> >> Thanks, >> Simone >> >> ___ >> aqm mailing list >> aqm@ietf.org >> https://www.ietf.org/mailman/listinfo/aqm -- Dave Täht thttp://www.bufferbloat.net/projects/bloat/wiki/Upcoming_Talks ___ aqm mailing list aqm@ietf.
Re: [aqm] RED implementation on Linux 3.14.22
On Mon, Dec 15, 2014 at 2:51 AM, Simone Ferlin-Oliveira wrote: > > All, > > I am doing some work with shared bottleneck detection that requires > some evaluation with different AQM, in particular, RED. Since I > haven't been following the evolution of the implementation, I would > like to ask about your experience with the code on Linux 3.14 (and > newer). > I know that Dave Taht ran into bugs in RED a while back, which I believe have been fixed for quite a while. You should always be looking at whether code has been patched in the current kernel.org system for a module like that you are interested in, so do a diff between 3.14 and the current Linux system. 3.14 is recent enough that it may be viable for experiments, for the time being. Planning to keep up with Linux development is wise long term in any case, as the rate of improvement/change in the networking stack is very high at the moment as draining the bufferbloat swamp and other performance work continues. Also note that underlying device drivers may have (sometimes lots) of buffering out of control of the Linux queue discipline. For Ethernet devices, you should ensure that that the drivers have BQL support implemented to minimize this buffering. Other classes of drivers are more problematic, and may have lots of buffering to surprise you. Also be aware that ethernet flow control may move the bottleneck from where you expect to somewhere else, and that switches in networks also have to be well understood. Most consumer switches have this *on* by default, and mixed 1G/100Mb networks can be particularly "entertaining" in this regard. Cable modems, unfortunately, typically do not implement flow control, but some DSL modems do (putting the bottleneck into your router, rather than in the modem). > *Any* help is appreciated. > Hope this helps. > > Thanks, > Simone > > ___ > aqm mailing list > aqm@ietf.org > https://www.ietf.org/mailman/listinfo/aqm > ___ aqm mailing list aqm@ietf.org https://www.ietf.org/mailman/listinfo/aqm
[aqm] RED implementation on Linux 3.14.22
All, I am doing some work with shared bottleneck detection that requires some evaluation with different AQM, in particular, RED. Since I haven't been following the evolution of the implementation, I would like to ask about your experience with the code on Linux 3.14 (and newer). *Any* help is appreciated. Thanks, Simone ___ aqm mailing list aqm@ietf.org https://www.ietf.org/mailman/listinfo/aqm
