On Tue, May 1, 2018 at 11:53 AM, Toke Høiland-Jørgensen <t...@toke.dk> wrote: > Eric Dumazet <eric.duma...@gmail.com> writes: > >> On 04/30/2018 02:27 PM, Dave Taht wrote: >> >>> I actually have a tc - bpf based ack filter, during the development of >>> cake's ack-thinner, that I should submit one of these days. It >>> proved to be of limited use. >>> >>> Probably the biggest mistake we made is by calling this cake feature a >>> filter. It isn't. >>> >>> Maybe we should have called it a "thinner" or something like that? In >>> order to properly "thin" or "reduce" an ack stream >>> you have to have a queue to look at and some related state. TC filters >>> do not operate on queues, qdiscs do. Thus the "ack-filter" here is >>> deeply embedded into cake's flow isolation and queue structures. >> >> A feature eating packets _is_ a filter. >> >> Given that a qdisc only sees one direction, I really do not get why >> ack-filter is so desirable in a packet scheduler. > > The ACK filter in CAKE is there to solve a very particular use case: > Residential internet connections with bandwidths so asymmetric that it > hurts TCP performance. It is not on by default; but rather meant to be > configured by users which suffer from this particular ISP brokenness > (which, sadly, does happen; there are ISPs who believe a 50/1 bandwidth > ratio is reasonable). For those users, the ACK filter can help. > > We certainly do not advise people to turn it on in the general case! As > you point, in general TCP performance is best improved in the TCP stack... > >> You have not provided any numbers to show how useful it is to maintain >> this code > > You mean apart from that in the linked blog post and the paper? > Here's an executive summary: > > On a 30/1 Mbps connection with a bidirectional traffic test (RRUL), > turning on the ACK filter improves downstream throughput by ~20% and > upstream throughput by ~12% in conservative mode and ~40% in aggressive > mode, at the cost of ~5ms of inter-flow latency due to the increased > congestion.
On a simulated 50/1 comcast connection, I got double the throughput on a similar test, with no obvious glitches in the tcp flow, in the early stages of development. http://blog.cerowrt.org/post/ack_filtering/ I was very, very dubious about the value of ack thinning until that point, but it was hard to argue with the data. > In *really* pathological cases, the effect can be a lot more; for > instance, the ACK filter increases the achievable downstream throughput > on a link with 100 Kbps in the upstream direction by an order of > magnitude (from ~2.5 Mbps to ~25 Mbps). > >> (probably still broken btw, considering it is changing some skb >> attributes). > > As you may have noticed over the last few iterations, I've actually been > trying to fix any brokenness. If you could be specific as to what is > still broken, that would be helpful. > > (I'm assuming are referring to the calls to skb_set_inner* - but do you > mean all of them, or just the ones that set inner == outer?) > >> On wifi (or any half duplex medium), you might gain something by >> carefully sending ACK not too often, but ultimately this should be >> done by TCP stack, in well controlled environment [1], instead of >> middle-boxes happily playing/breaking some Internet standards. >> >> [1] TCP stack has the estimations of RTT, RWIN, throughput, and might >> be able to avoid flooding the network with acks under some conditions. > > You are quite right that in general, TCP performance is best improved in > the TCP stack. But home users are not generally in control of that; the > ACK filter helps in those specific deployments (again, it's off by > default, and you shouldn't turn it on in the general case!). > >> Say RTT is 100ms, and we receive 1 packet every 100 usec (no GRO >> aggregation) Maybe we do not really _need_ to send 5000 ack per second >> (or even 10,000 ack per second if a hole needs a repair) > > Yes, please do fix that. :) I really would like to see cake tested at 10GigE and above, and its performance improved overall. I tend to think we need more queues than 1024 at 40GigE+, and we presently run out of cpu (even unshaped) long before we hit that point. > >> Also on wifi, the queue builds in the driver queues anyway, not in the >> qdisc. > > Actually, we've fixed that (for some drivers); there is now no qdisc on > WiFi interfaces, and we've integrated FQ-CoDel'ed queueing into the > stack where it can be effective. But no, running CAKE with an ACK filter > on a WiFi link is not going to be effective. Don't do that. I share the belief with eric that thinning acks (either at the wifi qdisc or in tcp) on wifi interfaces will help - given that the underlying wifi layer is reliable and does retransmits, and the number of packets that can fit into a wifi aggregate limited, you really only need one ack per wifi aggregate per flow to keep the tcp connection running. That said, I'd much rather see fq_codel work with more wifi drivers than pursue this. > >> So ACK filtering, if _really_ successful, would need to be >> modularized. Heh. I keep hoping ISPs will ship symmetric links and wifi antennas > I really hope ACK filtering won't be "_really_ successful". Again, it is > not meant to be a feature that's on everywhere, it's targeting a very > specific use case that CAKE is optimised for: The home router use case. Please note that I find cake far more general purpose than just this, the ease of just slamming: tc qdisc add dev eth0 root cake bandwidth 50mbit on a link that needs it is far easier than the equivalent htb + fq_codel + other filters, and more effective. That mode is with nat on, some diffserv awareness (more correct than pfifo_fast), no link layer compensation, no ack-filter, and "just works". Certainly the major use case to date has been on home routers. Every feature in cake was based on extensive feedback from that part of the field. >> Please split Cake into a patch series. >> Presumably putting the ack-filter on a patch of its own. > > *sigh* - can do, I guess. Though I'm not sure what that is going to > accomplish, exactly? > > -Toke -- Dave Täht CEO, TekLibre, LLC http://www.teklibre.com Tel: 1-669-226-2619
