Re: [Cake] cake's ack-filter vs GSO
> On 18 Feb, 2024, at 2:01 am, Dave Taht via Cake > wrote: > > Does anyone remember why we do not ack-filter a gso-split? That may actually be unintentional. The original code self-recursed, IIRC, so individual segments would then go through the non-gso-split path. At some point we inlined it for efficiency. The thing to try might be to copy the relevant code (both the search and the drop) into the gso-split path and see if anything breaks, and if the ack-dropping statistics tend to increase. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] Ubiquity (Unifi ) Smart Queues
> On 2 Jan, 2024, at 8:59 pm, dave seddon via Cake > wrote: > > • I'm not really sure what "overlimits" means or what that does, and > tried looking this up, but I guess the kernel source is likely the "best" > documentation for this. Maybe this means it's dropping? Or is it ECN? Overlimit just means the shaper in HTB is restricting the flow, thus doing something useful. Drop means the AQM is working to provide congestion information to a Not-ECT flow. Those numbers look normal (about 0.2% drop rate, and most packets hitting "overlimit" when the link is saturated). Using fq_codel's default parameters is not a bad thing at all. I'd much rather they did that, thereby using numbers that are tuned for general Internet conditions, than try to change that tuning in ignorance of the end-user's actual network environment. Most end-users have their WAN port facing "general Internet conditions" anyway. > Apparently rather than applying the tc qdsic on the outbound path on the LAN > side ( eth0 ), they are applying it inbound on the the eth2 via ifb_eth2. That's the correct place to do it, so that the qdisc applies specifically to traffic coming from the WAN. If you apply it to the LAN egress, it tends to affect traffic coming through the router from the WiFi or its internal servers, which is not desirable. These are all questions we've considered at length in the process of developing and deploying Cake and other SQM solutions. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] some comprehensive arm64 w/cake results
> On 15 Oct, 2023, at 11:29 pm, David P. Reed via Cake > wrote: > > Of course, Internet congestion control, in general, is still stuck in the > original Van Jacobsen sawtooth era. My guess is it won't get fixed, though I > applaud Cake, and despair the hardware folks who keep adding buffers. I am still working in this area, including on a revised version of SCE which might regain traction in the IETF. One of the more immediate results of this is a new AQM algorithm which builds on the success of Codel, and a family of qdiscs I'm building around it. These range from a queueless traffic policer to a "Cake version 2", with an interesting approximate-fairness approach for the middle child. The AQM itself is already working, though not yet documented in a public-facing form. I'm also taking a new approach to overlaying the "small" congestion response of SCE over the "big" response of conventional congestion control, which I think is capable of solving several long-standing problems in one go. I'll say more when we have a working prototype - but think in terms of "no sawtooth" and "naturally max-min fair". TCP Prague will look positively primitive compared to this - if it works (and it should). - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] some comprehensive arm64 w/cake results
> On 28 Sep, 2023, at 3:15 pm, Sebastian Moeller wrote: > > This promises even better performance for loads like cake than the already > pretty nifty pi4B Well, increased computing performance is always welcome - but as I've said before, in most cases I don't think CPU performance is the limiting factor for CAKE. When the CPU load goes up as networking throughput reaches the physical limit of the interface (or the I/O subsystem), what you're seeing is the CPU just spinning its wheels while waiting for a mutex to unblock. Spinning faster doesn't make the mutex unblock sooner! - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] some comprehensive arm64 w/cake results
> On 19 Sep, 2023, at 1:07 am, Jonathan Morton wrote: > >> Raspberry Pi 4's just aren't very good at networking because of their I/O >> architecture on the board, just as they are slow at USB in general. That's >> why the CM4 is interesting. It's interesting that the PiHole has gotten so >> popular - it would run better on an Pi with a better network architecture. > > On the contrary, the Pi 4 has an excellent I/O architecture compared to most > of its peers, and especially compared to the previous Pis. The built-in NIC > is internal to the SoC and *NOT* attached via USB any more, so it can > genuinely support gigabit speeds. The USB interface is also fast enough to > support a second GigE NIC, though the latency wouldn't be as good as one > attached over PCIe. That's with a standard, off-the-shelf Pi 4B. Timely breaking news: the Raspberry Pi 5 has just been announced. The important new feature here (for us) is that it exposes a PCIe bus lane on the standard model, so you don't have to mess around with the Compute Module just to get access to that. The built-in Ethernet port is now implemented in a PCIe-attached "southbridge" chip, and the WiFi performance has been improved by accelerating the interface by which the radio is attached. On the downside, the price has gone up. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] some comprehensive arm64 w/cake results
> On 19 Sep, 2023, at 1:52 am, dave seddon wrote: > > I'd love to understand the difference between the tests I've been doing and > your tests. > • How many TCP flows did you have please ( cake performance seems to > drop significantly with increased number of TCP flows, although I need to do > more testing to understand why )? It's almost certainly related to the extremely tight AQM settings you imposed on it, and on no other AQM you tested. > • What was the RTT? I think this was just in a LAN environment, to gauge the capabilities of the machine just as you're doing. We insert a delay in the middle when testing other things, such as the performance and fine-scale behaviour of a qdisc/CC combination. > • Load tool? I believe Pete was using iperf3 for this, or possibly Flent which delegates to netperf. These days he's developing something new along the same lines. A simple test using one or more of Flent's standard tests should be enough to replicate these results, at least approximately. I would also recommend using ECN, but you should get at least reasonable results without it, provided the AQM is set sanely. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] some comprehensive arm64 w/cake results
> On 18 Sep, 2023, at 10:50 pm, dave seddon via Cake > wrote: > > The cake tests so far had rtt 1ms and rtt 3ms, which might be too low. ( If > it is too low, then maybe it would make sense to remove "rtt lan = rtt 1ms" > option, as it's a misleading configuration option? ) If all your traffic is over the LAN, and you have a machine and application tuned for the extra-low latencies that a LAN can offer, then setting LAN-grade targets for Cake might make sense. But most people's traffic is a mixture, with the performance of Internet traffic being more important, and that is better served by the *default* settings. You ran fq_codel at its default settings. These are equivalent to Cake's default settings, so far as the AQM activity is concerned. I'm just asking for a like-to-like comparison. You could be pleasantly surprised. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] some comprehensive arm64 w/cake results
> On 18 Sep, 2023, at 11:24 pm, David P. Reed via Cake > wrote: > > I'm curious, too! We know that on older home routers, with really slow MIPS > processors, Cake struggles with GigE. As these old MIPS designs get phased > out and replaced by ARM designs, it will matter. All qdiscs struggle with gigabit speeds on these SoCs. It's because their networking architecture is optimised for passing packets straight through on the switch fabric, not pulling them into the CPU on the way. The bandwidth to the CPU is comparable to an old PCI 32b 33MHz bus. At 100Mbps or so each way, they can keep up just fine. > Raspberry Pi 4's just aren't very good at networking because of their I/O > architecture on the board, just as they are slow at USB in general. That's > why the CM4 is interesting. It's interesting that the PiHole has gotten so > popular - it would run better on an Pi with a better network architecture. On the contrary, the Pi 4 has an excellent I/O architecture compared to most of its peers, and especially compared to the previous Pis. The built-in NIC is internal to the SoC and *NOT* attached via USB any more, so it can genuinely support gigabit speeds. The USB interface is also fast enough to support a second GigE NIC, though the latency wouldn't be as good as one attached over PCIe. That's with a standard, off-the-shelf Pi 4B. Here are some single-armed (just the internal NIC) results we ran a while ago. Note that Cake is actually faster than fq_codel - or rather, Cake's internal shaper is faster than HTB shaping with an fq_codel leaf queue - and (not shown here) both are faster on the Pi 4 than on an x86-based APU2. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] some comprehensive arm64 w/cake results
On Monday, 18 September 2023, Dave Taht via Cake wrote: > A huge thanks to dave seddon for buckling down and doing some comprehensive > testing of a variety of arm64 gear! > > https://docs.google.com/document/d/1HxIU_TEBI6xG9jRHlr8rzyyxFEN43zMcJXUFlRuhiUI/edit#heading=h.bpvv3vr500nw I'm really not sure it's valid to compare fq_codel on default settings (equivalent to rtt 100ms) to Cake on 1ms and 3ms rtt settings. At such tight settings you're asking Cake to hold the queue down to a fifth of a millisecond, and we already know from L4S that's a fool's errand. The internal overheads of the Linux kernel will give you delays of that order already. At least include a run with Cake similarly on rtt 100ms, which is the default and what most Internet facing CPE would run. You may well find that throughput improves significantly. - Jonathan Morton Sent from my Sailfish device ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] Anybody has contacts at Dropbox?
> On 25 Jun, 2023, at 12:00 am, Sebastian Moeller via Cake > wrote: > > Is dropbox silently already using an L4S-style CC for their TCP? It should be possible to distinguish this by looking at the three-way handshake at the start of the connection. This will show a different set of TCP flags and ECN field values depending on whether RFC-3168 or AccECN is being attempted. Without AccECN, you won't have functioning L4S on a TCP stream. But I think it is more likely that it's a misapplied DSCP. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] big tcp
> On 23 Feb, 2023, at 3:35 am, Dave Taht via Cake > wrote: > > does this break cake? > > https://lore.kernel.org/netdev/de811bf3-e2d8-f727-72bc-c8a754a9d...@tessares.net/T/ It looks like they've included patches *to* Cake to handle anticipated breakage. This is one of the great things about having it upstreamed. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] sometimes I worry about cobalt's effectiveness
>> On 14 Dec, 2021, at 11:59 am, Sebastian Moeller wrote: >> >> Could we maybe introduce a no-ecn keyword to switch cake to drop only mode? >> If only to help diagnose ECN issues? > > Inserting a firewall rule *before* Cake to force the ECN field to Not-ECT > should suffice for this purpose. Addendum: either this firewall rule should ignore packets that are already CE-marked, *or* a second firewall rule which *drops* CE-marked packets should be inserted as well. This avoids erasing congestion information from any upstream AQM. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] sometimes I worry about cobalt's effectiveness
> On 14 Dec, 2021, at 11:59 am, Sebastian Moeller wrote: > > Could we maybe introduce a no-ecn keyword to switch cake to drop only mode? > If only to help diagnose ECN issues? Inserting a firewall rule *before* Cake to force the ECN field to Not-ECT should suffice for this purpose. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] sometimes I worry about cobalt's effectiveness
> On 14 Dec, 2021, at 8:06 am, Dave Taht wrote: > > ok, it looks like ecn and perhaps dscp is busted on this mikrotik > release. Ton more plots, false starts, and packet captures here. > > https://forum.mikrotik.com/viewtopic.php?p=897892#p897892 > > Also well, codel is doing better than cobalt, and SFQ at least at > these RTTs is doing really, really well. Codel *with ECN disabled* is doing better under these conditions, based on what I can see via the Google Drive links. This makes some sense if the ECN CE marks are being silently erased (which is *very* bad behaviour), rather than the packets carrying them being treated as dropped (as I'd expect from a wrong checksum). Under this particular pathology, COBALT is still able to act via the BLUE algorithm, but in Cake this kicks in only when the queue first reads as full. In other implementations of COBALT, it also triggers when the sojourn time reaches 400ms (by default). Mikrotik - or whoever is responsible for this - needs to fix their crap so that the ECN field is processed correctly. End of discussion. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] sometimes I worry about cobalt's effectiveness
> On 12 Dec, 2021, at 9:34 pm, Dave Taht wrote: > > LONG string of tests on mikrotik at 19Mbit here: > https://forum.mikrotik.com/viewtopic.php?p=897554#p897554 I can't see any of the image attachments in that thread, since I'm not a member of the forum. Is there anything that screams of being something I should be aware of? - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] Understanding the Achieved Rate Multiplication Effect in FlowQueue-based AQM Bottleneck
> On 4 Dec, 2021, at 12:27 am, Dave Taht wrote: > > https://jonathankua.github.io/preprints/jkua-ieeelcn2021_understanding_ar_preprint-20jul2021.pdf > > I would love it if somehow the measured effects of chunklets against cake's > per-host/per flow fq was examined one day. I haven't actually measured it, but based on what the above paper says, I can make some firm predictions: 1: When competing against traffic to the same local host, the performance effects they describe will be present. 2: When competing against traffic to a different local-network host, the performance effects they describe will be attenuated or even entirely absent. 3: They noted one or two cases of observable effects of hash collisions in their tests with FQ-Codel. These will be greatly reduced in prevalence with Cake, due to the set-associative hash function which specifically addresses that phenomenon. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] [Ecn-sane] l4s kernel submission
> On 16 Oct, 2021, at 11:38 am, Sebastian Moeller wrote: > > I also think that some one subscribed should weight in on the exempt one > packet from marking, especially in the light of GRO/GSO. I do have some experience with this from Cake, but Cake has the distinct benefit of (usually) knowing what the dequeue rate is, without having to estimate it. Without that knowledge, I'm not sure it's profitable to guess - except to specifically avoid tail *loss*, which is not at all the same as *marking* the last packet. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] [Ecn-sane] l4s kernel submission
> On 15 Oct, 2021, at 1:17 am, Dave Taht wrote: > >> You can also subscribe to Linux lists by importing the mails from Lore, >> as one of the replies in the thread above pointed out. Been meaning to >> switch to that myself, but haven't gotten around to it yet... > > I attempted to subscribe again, nothing happened. > > figuring out lore... is too much work for today. I'd rather hammer > small things into oblivion on my boat. > > please feel free to pass along my comments and the sce teams findings > into that thread. https://lore.kernel.org/all/308c88c6-d465-4d50-8038-416119a35...@gmail.com/ I haven't yet posted a link to the WGLC Objections document. I will if it seem s necessary to do so. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] Cake: how know 'framing compensation'?
> On 3 Sep, 2021, at 6:16 pm, Dave Taht wrote: > > the cake mailing list contains answers to your questions. > > On Fri, Sep 3, 2021 at 7:05 AM Marco Gaiarin wrote: >> >> >> I'm thinking about give Cake a try: >> >>https://www.bufferbloat.net/projects/codel/wiki/Cake/ >> >> >> How can i determine my 'framing compensation'? I'm using now an >> wireless link, terminating the PPPoE link directly on my linux router, >> via PPPD/PPPOE. >> >> So, i'm surely using PPPoE, but pppoe-vcmux or pppoe-llcsnap? How >> determine it? In general, you need to set the framing compensation according to the bottleneck link. If your wireless link is typically faster than your ADSL line, then the ADSL is the right focus. This is probably helpful to most ADSL users in figuring out their framing type: https://www.routertech.org/guides/info/settings-by-country/ - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] [Bloat] [Make-wifi-fast] [Starlink] [Cerowrt-devel] Due Aug 2: Internet Quality workshop CFP for the internet architecture board
> On 8 Aug, 2021, at 9:36 pm, Aaron Wood wrote: > > Less common, but something I still see, is that a moving station has > continual issues staying in proper MIMO phase(s) with the AP. Or I think > that's what's happening. Slow, continual movement of the two, relative to > each other, and the packet rate drops through the floor until they stop > having relative motion. And I assume that also applies to time-varying > path-loss and path-distance (multipath reflections). So is it time to mount test stations on model railway wagons? - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] cake srchost/dsthost stopped working?
Those results look good to me as well. The difference between the dual modes and the pure host modes is visible in terms of looser fairness in the latter case. Don't worry too much about triple-isolate, as it is significantly more complicated to test comprehensively. I'm satisfied that if the dual modes are working, we probably haven't broken triple-isolate either. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] cake srchost/dsthost stopped working?
On Wed, 4 Aug 2021 at 14:14, Toke Høiland-Jørgensen via Cake
wrote:
>
> Pete Heist writes:
>
> > One more tip, reverting this commit seems to fix it:
> >
> > https://github.com/torvalds/linux/commit/b0c19ed6088ab41dd2a727b60594b7297c15d6ce
>
> Ah, I think I see what the problem is; could you please try the patch
> below?
>
> -Toke
>
> diff --git a/net/sched/sch_cake.c b/net/sched/sch_cake.c
> index 951542843cab..a83c4d4326da 100644
> --- a/net/sched/sch_cake.c
> +++ b/net/sched/sch_cake.c
> @@ -720,7 +720,7 @@ static u32 cake_hash(struct cake_tin_data *q, const
> struct sk_buff *skb,
> skip_hash:
> if (flow_override)
> flow_hash = flow_override - 1;
> - else if (use_skbhash)
> + else if (use_skbhash && flow_mode & CAKE_FLOW_FLOWS)
> flow_hash = skb->hash;
> if (host_override) {
> dsthost_hash = host_override - 1;
Good catch - I was going to have to wade in and remind myself how this
lump of code worked. But shouldn't the masking operation be in
brackets, to make the precedence explicit?
- Jonathan Morton
___
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] [Make-wifi-fast] [Bloat] Little's Law mea culpa, but not invalidating my main point
> On 12 Jul, 2021, at 11:04 pm, Bob McMahon via Make-wifi-fast > wrote: > > "Flow control in store-and-forward computer networks is appropriate for > decentralized execution. A formal description of a class of "decentralized > flow control algorithms" is given. The feasibility of maximizing power with > such algorithms is investigated. On the assumption that communication links > behave like M/M/1 servers it is shown that no "decentralized flow control > algorithm" can maximize network power. Power has been suggested in the > literature as a network performance objective. It is also shown that no > objective based only on the users' throughputs and average delay is > decentralizable. Finally, a restricted class of algorithms cannot even > approximate power." > > https://ieeexplore.ieee.org/document/1095152 > > Did Jaffe make a mistake? I would suggest that if you model traffic as having no control feedback, you will inevitably find that no control occurs. But real Internet traffic *does* have control feedback - though it was introduced some time *after* Jaffe's paper, so we can forgive him for a degree of ignorance on that point. Perhaps Jaffe effectively predicted the ARPANET congestion collapse events with his analysis. > Also, it's been observed that latency is non-parametric in it's distributions > and computing gaussians per the central limit theorem for OWD feedback loops > aren't effective. How does one design a control loop around things that are > non-parametric? It also begs the question, what are the feed forward knobs > that can actually help? Control at endpoints benefits greatly from even small amounts of information supplied by the network about the degree of congestion present on the path. This is the role played first by packets lost at queue overflow, then deliberately dropped by AQMs, then marked using the ECN mechanism rather than dropped. AQM algorithms can be exceedingly simple, or they can be rather sophisticated. Increased levels of sophistication in both the AQM and the endpoint's congestion control algorithm may be used to increase the "network power" actually obtained. The required level of complexity for each, achieving reasonably good results, is however quite low. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] [Bloat] Little's Law mea culpa, but not invalidating my main point
> On 10 Jul, 2021, at 2:01 am, Leonard Kleinrock wrote: > > No question that non-stationarity and instability are what we often see in > networks. And, non-stationarity and instability are both topics that lead to > very complex analytical problems in queueing theory. You can find some > results on the transient analysis in the queueing theory literature > (including the second volume of my Queueing Systems book), but they are > limited and hard. Nevertheless, the literature does contain some works on > transient analysis of queueing systems as applied to network congestion > control - again limited. On the other hand, as you said, control theory > addresses stability head on and does offer some tools as well, but again, it > is hairy. I was just about to mention control theory. One basic characteristic of Poisson traffic is that it is inelastic, and assumes there is no control feedback whatsoever. This means it can only be a valid model when the following are both true: 1: The offered load is *below* the link capacity, for all links, averaged over time. 2: A high degree of statistical multiplexing exists. If 1: is not true and the traffic is truly inelastic, then the queues will inevitably fill up and congestion collapse will result, as shown from ARPANET experience in the 1980s; the solution was to introduce control feedback to the traffic, initially in the form of TCP Reno. If 2: is not true then the traffic cannot be approximated as Poisson arrivals, regardless of load relative to capacity, because the degree of correlation is too high. Taking the iPhone introduction anecdote as an illustrative example, measuring utilisation as very close to 100% is a clear warning sign that the Poisson model was inappropriate, and a control-theory approach was needed instead, to capture the feedback effects of congestion control. The high degree of statistical multiplexing inherent to a major ISP backhaul is irrelevant to that determination. Such a model would have found that the primary source of control feedback was human users giving up in disgust. However, different humans have different levels of tolerance and persistence, so this feedback was not sufficient to reduce the load sufficiently to give the majority of users a good service; instead, *all* users received a poor service and many users received no usable service. Introducing a technological control feedback, in the form of packet loss upon overflow of correctly-sized queues, improved service for everyone. (BTW, DNS becomes significantly unreliable around 1-2 seconds RTT, due to protocol timeouts, which is inherited by all applications that rely on DNS lookups. Merely reducing the delays consistently below that threshold would have improved perceived reliability markedly.) Conversely, when talking about the traffic on a single ISP subscriber's last-mile link, the Poisson model has to be discarded due to criterion 2 being false. The number of flows going to even a family household is probably in the low dozens at best. A control-theory approach can also work here. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] STEAM tcp algo from CDN?
> On 9 Mar, 2021, at 10:20 pm, Dave Taht wrote: > > 10-20 flows, cubic, last I looked. It's ugly. I can't confirm CUBIC from here, but it seems to be 4-8 flows in parallel now. Latency to the national CDN is about 22ms over LTE, so it's hard to distinguish CUBIC from anything else in particular; in this range it would look a lot like NewReno. It seems to shut down each flow and start a fresh one after about a minute. It does have ECN enabled and seems to be producing CWR flags in response at the correct times. With this small a BDP (I've throttled down quite hard due to the amount of videoconferencing I'm doing at peak hours this week), it quickly ends up getting CE marks on almost every data segment. This is actually fine, since it's then controlled nicely by ack-clocking and FQ. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] [Bloat] how to ecn again on osx and ios!!!
> On 9 Mar, 2021, at 10:38 pm, Dave Taht wrote: > > sudo sysctl -w net.inet.tcp.disable_tcp_heuristics=1 Now that might well be the missing link. I think we missed it before since it doesn't have "ecn" in its name. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] ISP Implementation
> On 4 Mar, 2021, at 8:31 am, Thomas Croghan wrote: > > So this would be preferable right? -- > -- to 100 Mbps> -- -- <10 > x 25 Mbps Customers> Yes, putting the Cake instances associated with the backhaul link upstream of the link in both directions like that is better for a number of reasons. You can still have the instances managing individual customers on the right-hand side, or even further to the right. If the customer links are physically wider in the upstream direction than is made available to the customer, then there's no problem in doing all the per-customer work in an aggregated position. The difference (in the long run) between the traffic transmitted by the customer and that released to traverse the backhaul is limited to AQM activity on Not-ECT traffic, which will be small, unless they start flooding in which case the overload protection will kick in and start dropping a lot of packets. This is also what you'd expect to see with a well-behaved policer in the same position. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] ISP Implementation
> On 4 Mar, 2021, at 5:14 am, Dave Taht wrote: > > yes, that. can it be made to work with cake? The README says there is experimental support. I haven't looked at it closely. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] ISP Implementation
> On 4 Mar, 2021, at 4:51 am, Dave Taht wrote: > > recently there was a thread on another bufferbloat list about a very > interesting ISP approach using massively hashed tc filters + fq_codel > or cake that has code in github. I cannot for the life of me remember > the name of the thread or the github right. This, surely? https://github.com/rchac/LibreQoS/ - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] ISP Implementation
> On 4 Mar, 2021, at 3:54 am, Thomas Croghan wrote: > > So, a beta of Mikrotik's RouterOS was released some time ago which finally > has Cake built into it. > > In testing everything seems to be working, I just am coming up with some > questions that I haven't been able to answer. > > Should there be any special considerations when Cake is being used in a > setting where it's by far the most significant limiting factor to a > connection? For example: --10 Gbps Fiber -- --10 Gbps > Fiber -- [ISP Switch] -- 1 Gbps Fiber -- <500 Mbps Customer> > In this situation very frequently the "" could be running Cake > and do the bandwidth limiting of the customer down to 1/2 (or even less) of > the physical connectivity. A lot of the conversations here revolve around > Cake being set up just below the Bandwidth limits of the ISP, but that's not > really going to be the case in a lot of the ISP world. There shouldn't be any problems with that. Indeed, Cake is *best* used as the bottleneck inducer with effectively unlimited inbound bandwidth, as is typically the case when debloating a customer's upstream link at the CPE. In my own setup, I currently have GigE LAN feeding into a 2Mbps Cake instance in that direction, to deal with a decidedly variable LTE last-mile; this is good enough to permit reliable videoconferencing. All you should ned to do here is to filter each subscriber's traffic into a separate Cake instance, configured to the appropriate rate, and ensure that the underlying hardware has enough throughput to keep up. > Another question would be based on the above: > > How well does Cake do with stacking instances? In some cases our above > example could look more like this: -- [Some sort of limitation to > 100 Mbps] -- -- 1 Gbps connection- <25 Mbps Customer X 10> > > In this situation, would it be helpful to Cake to have a "Parent Queue" that > limits the total throughput of all customer traffic to 99-100 Mbps then > "Child Queues" that respectively limit customers to their 25 Mbps? Or would > it be better to just setup each customer Queue at their limit and let Cake > handle the times when the oversubscription has reared it's ugly head? Cake is not specifically designed to handle this case. It is designed around the assumption that there is one bottleneck link to manage, though there may be several hosts who have equal rights to use as much of it as is available. Ideally you would put one Cake or fq_codel instance immediately upstream of every link that may become saturated; in practice you might not have access to do so. With that said, for the above topology you could use an ingress Cake instance to manage the backhaul bottleneck (using the "dual-dsthost" mode to more-or-less fairly share this bandwidth between subscribers), then a per-subscriber array of Cake instances on egress to handle that side, as above. In the reverse direction you could invert this, with a per-subscriber tree on ingress and a backhaul-generic instance (using "dual-srchost" mode) on egress. The actual location where queuing and ECN marking occurs would shift dynamically depending on where the limit exists, and that can be monitored via the qdisc stats. This sort of question has come up before, which sort-of suggests that there's room for a qdisc specifically designed for this family of use cases. Indeed I think HTB is designed with stuff like this in mind, though it uses markedly inferior shaping algorithms. At this precise moment I'm occupied with the upcoming IETF (and my current project, Some Congestion Experienced), but there is a possibility I could adapt some of Cake's technology to a HTB-like structure, later on. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] [Make-wifi-fast] [Bloat] Fwd: [Galene] Dave on bufferbloat and jitter at 8pm CET Tuesday 23
> On 24 Feb, 2021, at 5:19 pm, Taraldsen Erik wrote: > > Do you have a subscription with rate limitations? The PGW (router which > enforces the limit) is a lot more latency friendly than if you are radio > limited. So it may be beneficial to have a "slow" subscription rather than > "free speed" then it comes to latency. Slow meaning lower subscrption rate > than radio rate. This is actually something I've noticed in Finland with DNA. The provisioning shaper they use for the "poverty tariff" is quite well debloated (which was very much not the case some years ago). However, there's no tariff at any convenient level between 1Mbps (poverty tariff) and 50Mbps (probably radio limited on a single carrier). - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] bringing up a new router/connection
> On 3 Feb, 2021, at 11:24 pm, David Lang wrote: > > when plugged directly into the ISP router, I am getting the advertised > speeds, when going through the c2600 I top out at 200-300Mb download and > 10-15mb upload That sounds about right for a consumer CPE router. I believe there is usually a hardware bottleneck between the SoC and the Ethernet complex that is significantly narrower than the Ethernet ports and switch fabric. Once the downstream gets saturated there is also no room for upstream traffic. You could set it up for 100Mbit down, 25Mbit up using Cake, and see how that works. It'll be a major improvement over 8/1 DSL, even if it isn't using the full advertised capacity of the cable. One device that should be able to keep up is a Raspberry Pi 4 (not the earlier versions) supplemented with a USB3-attached GigE dongle. Pete Heist has established that it can sustain 600Mbit through Cake without much CPU load or added latency. Above that level the characteristics do degrade a bit. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] ECN not working?
> On 22 Dec, 2020, at 10:06 pm, xnor wrote: > > The client initiates the IPv4 TCP connection with: > IP Differentiated Services Field: 0x02 (DSCP: CS0, ECN: ECT(0)) > TCP Flags: 0x0c2 (SYN, ECN, CWR) > Seq=0 Win=64240 Len=0 MSS=1460 WS=256 SACK_PERM=1 > > The server responds: > Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT) > Flags: 0x012 (SYN, ACK) > Seq=0 Ack=1 Win=64240 Len=0 MSS=1460 SACK_PERM=1 WS=128 > > Shouldn't the server respond with ECT set in the SYN ACK packet > and possibly also have ECN-related flags set in the TCP header? Not all servers have ECN support enabled. A SYN-ACK without the ECE bit set indicates it does not. The connection then proceeds as Not-ECT. I'm reasonably sure Akamai has specifically enabled ECN support. A lot of smaller webservers are probably running with the default passive-mode ECN support as well (ie. will negotiate inbound but not initiate outbound). - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] NLA_F_NESTED is missing
> On 1 Nov, 2020, at 12:15 pm, Dean Scarff wrote: > > Error: NLA_F_NESTED is missing. Since you're running an up-to-date kernel, you should check you are also running up-to-date userspace tools. That flag is associated with the interface between the two. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] Cake, low speed ADSL & fwmark
> On 28 Jul, 2020, at 7:51 pm, Jim Geo wrote: > > Thanks for the info! I've noticed that by using 0xF, marks 1-4 become > tins 0-3. Tin 0 is special? I assumed it's for bulk traffic. I use > diffserv8. Mark 0 (not tin 0) is special because it corresponds to "no mark set". Otherwise, what you see is what you get, and mark N goes into tin N-1. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] Cake, low speed ADSL & fwmark
> On 28 Jul, 2020, at 12:41 am, Jim Geo wrote: > > Thank you for all the efforts you have done to make internet usable. > > I currently use htb & fq_codel in my low speed ADSL 6Mbps downlink/1 Mbps > uplink. I use fwmark to control both uplink and downlink with good results in > terms of bandwidth allocation. Streaming video is chopping bulk traffic > successfully. > > Is setting up cake worth the effort at such low speeds? Would it reduce > latency? Cake has a better-quality shaper than HTB does, and a more sophisticated flow-isolation scheme than fq_codel does. These tend to matter more at low speeds, not less. It's also generally easier to set up than a compound qdisc scheme. > Regarding fwmark can you please elaborate more on the calculations performed? > Man page is not that helpful. > > My understanding is this: > > I use 1,2,3,4 as marks of traffic. > If I set the mask to 0xff[..] the marks will remain unchanged. Then right > shifting will occur for the unset bits, so they will land on tins > 1,1,3,1 > > Can you please correct me? If logical and performed between mask and mark > value? Since there's only a few "tins" at a time used in Cake, and the fwmark is a direct mapping into those tins, a narrow mask is probably safer to use than a wide one. The reason for the mask is so you can encode several values into different parts of the mark value. The shift is simply to move the field covered by the mask to the low end of the word, so that it is useful to Cake. For your use case, a mask of 0xF will be completely sufficient. It would allow you to specify mark values of 1-15, to map directly in the first 15 tins used by Cake, or a mark value of 0 to fall back to Cake's default Diffserv handling. None of Cake's tin setups use more than 8 tins, and most use fewer. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] diffserv3 vs diffserv4
> On 25 Jul, 2020, at 10:35 pm, David P. Reed wrote: > > And to be clear, AQM (cake, being an example) is not about bandwidth > allocation. It does focus on latency/queueing-delay, for the most part. Cake is not *just* an AQM, though I understand your point. It is a qdisc with many interwoven functions. Cake's Diffserv support is neither a pure priority scheme nor a pure bandwidth allocation. By using a hybrid of the two for bandwidth allocation, I was hoping to avoid the main pitfalls that the simple Bell-headed approaches routinely encounter. Each tin also has its own AQM parameters, which feed into the distinction between high-throughput and low-latency classes of traffic. There are doubtless other approaches that could be tried, of course. And there might be endless debate over exactly how many traffic classes are actually needed; I don't think five is the right number, and the symmetry argument is not persuasive. But can we at least agree that Cake's attempt is a step in the right direction? - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] diffserv3 vs diffserv4
> On 25 Jul, 2020, at 8:18 pm, Sebastian Moeller wrote: > > I am confused... but I am also confused by cake's output: > > Bulk Best EffortVoice > thresh 3062Kbit 49Mbit12250Kbit" > > as far as I can tell, Bulk's 3062Kbit must be the minimum, while BE and Voice > give their maxima... That, or I am missing something important... > (I wonder whether it would not be clearer to give both min and max for each > tin, then again I probably missing all the deyails of the actual > implementation...) Cake delivers from the highest-priority tin that both has data to send and is "behind" its local schedule, defined by the threshold rate. If no tin with data to send is behind schedule, then some tin that does have data to send is chosen (so Cake as a whole is work-conserving, modulo its global shaper). IIRC, it'll be the highest priority such tin. The notion of which tin is highest priority is a little counter-intuitive. One tin must be at the global shaper rate, and will be the lowest priority tin - and normally that is the "best effort" tin. So the BK tin is actually at a higher priority, but only up to its very limited threshold rate. To avoid starving the best effort tin under all possible combinations of traffic, it is necessary and sufficient to ensure that the sum of all higher-priority tins' threshold rates is less than the global rate. In the case of Diffserv3, the BK and VO tins both have higher priority than BE and sum to 5/16ths of the global rate. So with all tins saturated, the BE traffic gets 11/16ths which is pretty respectable. If the BE and VO traffic goes away, BK is then able to use all available bandwidth. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] diffserv3 vs diffserv4
> On 24 Jul, 2020, at 6:56 pm, Justin Kilpatrick wrote: > > "sqm-scripts used 3 tiers of priority pretty successfully as does free.fr. - > de-prioritization seems a good idea, prioritization not so much." > > This is the best comment on why diffserv3 is the default that I could find on > bufferbloat.net. I'm interested in hearing about what data (anecdotes > welcome) lead to this conclusion. In Cake, Diffserv4 maps conceptually (but not in detail) to the four priority buckets in Wifi - BK, BE, VI, VO. In Diffserv3 the VI bucket is dropped, because Cake's flow isolation within BE is already good enough to give decent video streaming performance. The BK and VO buckets are still useful to deal with certain specific problems; BK is the place to put "swarm" protocols which intend to be scavengers but which flow-isolation tends to prioritise, and VO is for latency-sensitive protocols which the user wants to specifically protect from random traffic fluctuations. Thinking more broadly, I believe Diffserv would have been far more successful if it had replaced Precedence/TOS with a simple two-bit, four-way set of PHBs: 00: High Throughput - equivalent to traditional Best Effort service. 01: High Reliability - "Every Packet's Sacred". 10: Low Cost - a scavenging service for altruistic applications. 11: Low Latency - for the many protocols that are sensitive to delays more than throughput. It may also have been reasonable to include a couple of extra bits for uses internal to an AS, on the understanding that the basic two bits would be preserved end-to-end as an indication of application intent. Of the above four classes, Diffserv3 provides three - omitting only the High Reliability class. But that is a class most useful within a datacentre, where it is actually practical to implement a lossless backplane with backpressure signals instead of loss. What we *actually* have is a six-bit field with ill-defined semantics, that is neither preserved nor respected end-to-end, is consequently ignored by most applications, and consumes all the space in the former TOS byte that is not specifically set aside for ECN (a field which could profitably have been larger). It is a serious problem. Implementations of PHBs still tend to think in terms of bandwidth reservation (a Bell-head paradigm) and/or strict priority (like the Precedence system which was lifted directly from telegraphy practice). Both approaches are inefficient, and go along with the misconception that if we can merely categorise traffic on the fly into a large enough number of pigeonholes, some magical method of dealing with the pigeonholes will make itself obvious. However, both the easy, universal method of categorisation and the magical delivery strategy have failed to materialise. It rather suggests that they're doing it wrong. So that is why Diffserv3 is the default in Cake. It offers explicit "low cost" and "low latency" service for suitably marked traffic, and for everything else the Best Effort service uses flow and host isolation strategies to maintain good behaviour. It usually works well. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] [PATCH net-next 1/5] sch_cake: fix IP protocol handling in the presence of VLAN tags
> On 26 Jun, 2020, at 5:59 pm, Sebastian Moeller wrote: > > thinking this over, I wonder whether a hypothetical carrier grade cake, might > not actually grow a classify-by-vlan-priority keyword to allow switching over > to using VLAN priority tags instead of dscps? That would avoid tempting > carriers to re-map deeep-encapsulated dscps if they can just ignore them for > good. And it scratches my pet itch, that 3 bits of classification should be > enough for >80 % of the cases ;) > > What do you think? If carriers could use Ethernet VLANs for internal purposes instead of DSCPs, I would count that as progress towards allowing DSCPs to carry end-to-end information. And if there's a desire for a software qdisc which fits that paradigm, then we can do a requirements analysis which might well lead to something useful being developed. But that isn't going to be Cake. It'll be a different qdisc which might share some features and technology with Cake, but definitely arranged in a different order and with a different focus. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] [PATCH net-next 1/5] sch_cake: fix IP protocol handling in the presence of VLAN tags
Toke has already replied, but: > Sure, my proposal does not cover the problem of mangling the CE bit inside > VLAN-tagged packets, i.e. if we should understand if qdiscs should allow > it or not. This is clearly wrong-headed by itself. Everything I've heard about VLAN tags thus far indicates that they should be *transparent* to nodes which don't care about them; they determine where the packet goes within the LAN, but not how it behaves. In particular this means that AQM should be able to apply congestion control signals to them in the normal way, by modifying the ECN field of the IP header encapsulated within. The most I would entertain is to incorporate a VLAN tag into the hashes that Cake uses to distinguish hosts and/or flows. This would account for the case where two hosts on different VLANs of the same physical network have the same IP address. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] Why are target & interval increased on the reduced bandwidth tins?
> On 25 Jun, 2020, at 4:40 pm, Kevin Darbyshire-Bryant > wrote: > > So the scenario I have in my head says that BK traffic could burst at full > bandwidth rate (or higher) filling upstream ISP buffers and thus inducing > delays on all other traffic because "we think it’s a slow link and have high > interval and target values” delaying our response to the burst. Whereas if > we retain the default interval & target from the true link capacity > calculation we’ll jump on it in time. You might be forgetting about ack clocking. This gives the sender information about how quickly data is reaching the receiver, and normally the sender will generate either one or two packets for each packet acked. So even without an immediate AQM action, there is still *some* restraint on the sender's behaviour within approximately one RTT. This is one of the many subtle factors that Codel relies on. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] [CAKE] Rate is much lower than expected - CPU load is higher than expected
> On 23 Jun, 2020, at 7:08 pm, Sebastian Moeller wrote: > > But I assume that you bound the bursts somehow, do you remember your bust > sizing method by chance? It bursts exactly enough to catch up to the schedule. No more, no less - unless the queue is emptied in the process. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] [CAKE] Rate is much lower than expected - CPU load is higher than expected
> On 23 Jun, 2020, at 5:41 pm, Toke Høiland-Jørgensen wrote: > > Right, well if you're not running out of CPU I guess it could be a > timing issue. The CAKE shaper relies on accurate timestamps and the > qdisc watchdog timer to schedule the transmission of packets. A loaded > system can simply miss deadlines, which would be consistent with the > behaviour you're seeing. > > In fact, when looking into this a bit more, I came across this commit > that seemed to observe the same behaviour in sch_fq: > https://git.kernel.org/torvalds/c/fefa569a9d4b > > So I guess we could try to do something similar in CAKE. Actually, we already do. The first version of Cake's shaper was based closely on the one in sch_fq at the time, and I modified it after noticing that it had a very limited maximum throughput when timer resolution was poor (eg. at 1kHz on an old PC without HPET hardware, we could only get 1k pps). Now, any late timer event will result in a burst being issued to catch up with the proper schedule. The only time that wouldn't work is if the queue is empty. If the patches currently being trialled are not sufficient, then perhaps we could try something counter-intuitive: switch on "flows" instead of "flowblind", and enable the ack-filter. That should result in fewer small packets to process, as the ack-filter will coalesce some acks, especially under load. It might also help to select "satellite" AQM parameters, reducing the amount of processing needed at that layer. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] Query on ACK
> On 14 Jun, 2020, at 3:43 pm, Avakash bhat wrote: > > I wanted another clarification on the results obtained by the Ack filtering > experiment( Fig 6) . > Was the experiment conducted with only ack filtering enabled? > Or was set associative hash and the other modules of Cake enabled along with > Ack filtering while running this experiment ? The test was run on a complete implementation of Cake, set up in the normal way. I think we kept the configuration simple for this test, so everything at defaults except for choosing the shaped bandwidth in each direction. The ack-filter relies on having fairly good flow isolation, so that consecutive packets in the appropriate queue belong to the same ack stream. So at minimum it is appropriate to have the set-associative flow hash enabled. The host-fairness and Diffserv features were probably enabled, but did not have relevant effects in this case, since only one pair of hosts and the Best Effort DSCP were used in the traffic. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] Query on ACK
> On 25 May, 2020, at 8:17 am, Avakash bhat wrote: > > We had another query we would like to resolve. We wanted to verify the > working of ack filter in ns-3, > so we decided to replicate the Fig 6 graph in the CAKE > paper(https://ieeexplore.ieee.org/document/8475045). > While trying to build the topology we realized that we do not know the number > of packets or bytes sent from > the source to the destination for each of the TCP connections ( We are > assuming it is a point to point connection with 4 TCP flows). > > Could we get a bit more details about how the experiment was conducted? I believe this was conducted using the RRUL test in Flent. This opens four saturating TCP flows in each direction, and also sends a small amount of latency measuring traffic. On this occasion I don't think we added any simulated path delays, and only imposed the quoted asymmetric bandwidth limits (30Mbps down, 1Mbps up). > Also is this the best way to verify the correctness of our implementation? Obviously with limited space in our paper, we could only include a small selection of test results. Many other tests were run in practice, and we have expanded our test repertoire since. In particular, we now routinely run tests with a simulated typical Internet path delay inserted, eg. 20ms, 80ms, 160ms baseline RTTs to represent reaching a local-ish CDN, across the Atlantic, and from Europe to the US West Coast. You will also want to include multiple traffic mixes in the analysis, in particular different congestion control algorithms (at least Reno and CUBIC), and running with ECN both enabled and disabled at the endpoints. A useful torture test we used was to send many bulk flows up the narrow side of the link and a single bulk flow down the wide side. For example, 50:1 flow counts with 1:10, 1:20 and 1:30 bandwidth asymmetries. The acks of the single flow then have to compete with the heavy load of the many flows, and the total goodput of that single flow is an important metric, along with both the total goodput and the Jain's fairness of the upload traffic. This should show a particularly strong effect of the ack filter, as otherwise individual acks have to be dropped by the AQM, which Codel is not very good at adapting to quickly. In evaluating the above, you will want to be vigilant not only for observed gross performance, but also the extent to which the ack filter preserves or loses information from the ack stream. This is particularly true in the runs without ECN, in which congestion signals can only be applied through packet loss, and the feedback of that signal is through dup-acks and SACK. I think you will find that the "aggressive" setting loses some information, and its performance suffers accordingly in some cases. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] Query on ACK
>> The ACK filter runs on enqueue, so if a queue has only ACKs in it, it >> will never accumulate anything in the first place... > > but the side effect is that on dequeue, it flips it into the fast > queue drr rotation, not the slow, so it can't accumulate > as many acks before delivering the one it has left. > > Or so I thought, way back when The ack filter converts a stream of acks that might be treated as a bulk flow into a sparse flow, which is delivered promptly. This is a good thing; an ack should not be held back solely to see whether another one will arrive. I think of it as an optimisation to reduce delay of the information in the ack stream, not solely as a way to reduce the bandwidth consumed by the ack stream; the latter is a happy side effect. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] Latency target curiosity
> On 7 May, 2020, at 10:58 am, Kevin Darbyshire-Bryant > wrote: > > A curiosity has arisen: I use diffserv4 mode on a 20Mbit egress link. Bulk > tin has ‘capacity’ threshold of 1.2Mbit and because it’s a slow ’tin', the > default target & interval values get overridden to 14.6ms and 109.6ms > respectively. The 3 other tins are 5ms & 100ms defaults. > > I have a backup job that bulk uploads 5 simultaneous flows to Onedrive. The > sparse_delay, average_delay & peak_delay figures settle on 32, 38 & 43 ms > respectively with around 9 drops per second on that tin. > > I’m curious as to why the reported delays are over double the target latency? It's likely that there's a minimum cwnd in your sender's TCP stack, which may be as large as 4 segments. In Linux it is 2 segments. No matter how much congestion signalling is asserted, the volume of data in flight (including retransmissions of dropped packets) will always correspond to at least that minimum per flow. If the path is short, most of that volume will exists in queues instead of on the wire. Fortunately, backups are unlikely to suffer from a small amount of extra latency, and Cake will isolate their influence from other flows that may be more sensitive. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] Query on ACK
> On 7 May, 2020, at 9:44 am, Avakash bhat wrote: > > Thanks for the quick response. I also had a followup question. > > If the ack filter adds the new ack to the tail of the queue after removing an > ack from the queue, won't it be starving the ack? > The replaced ack was much ahead in the queue than the ack we replaced at the > tail right? No, if you are doing this on enqueue, then you are comparing the new ack with an ack immediately preceding it in the same queue, which will also be at the tail. And if you are doing it on dequeue then both packets were enqueued some time ago, and both are already due for delivery very soon. In general, the second packet is delivered sooner, in place of the first one that was removed. This means it reduces feedback latency to the (forward path) sender. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] Query on ACK
> On 6 May, 2020, at 9:43 pm, Avakash bhat wrote: > > We are trying to implement the ACK filtering module of CAKE in ns-3 (Network > Simulator). Ah yes. Sorry I didn't respond to the introduction earlier - we were right in the middle of preparing for an IETF virtual meeting. The debris is still falling from orbit… > We had a question on the working of ack filtering. > If an incoming ack which can replace an eligible ack in the queue is about to > be enqueued, do we replace the ack in the queue with the incoming ack > or do we enqueue the ack to the tail of the queue and remove the eligible ack > from the queue? That sounds like an implementation detail. But what we do in Cake is to simply enqueue all the packets, and deal with everything complicated on dequeue. At that point, we check whether the two packets at the head of the queue are acks for the same flow, and if so, we further check whether the information in the first packet is redundant given the presence of the second packet. If there is information in the first packet that is not also provided by the second packet, the first packet is delivered. Otherwise the first packet is dropped, and the second packet moves to the head of the queue. This process may repeat several times if there are several consecutive, redundant acks in the queue. The important part is the set of rules determining whether the ack is redundant. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] cake on linux 5.6 32 bit x86 might be broken
> On 25 Apr, 2020, at 10:09 pm, Dave Taht wrote: > >~# tc qdisc add dev eth1 root cake bandwidth 160mbps For tc, the "mbps" suffix is interpreted as megaBYTES per second. For megaBITS, use Mbit. The output and behaviour is consistent with that. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] Cake tin behaviour - discuss....
> On 25 Apr, 2020, at 2:07 pm, Kevin Darbyshire-Bryant > wrote: > > Download from ‘onedrive’ from 1 box, using 5 flows, classified as Bulk. > Little other traffic going on, sits there at circa 70Mbit, no problem. > > If I started another download on another box, say 5 flows, classified as Best > Effort, what rates would you expect the Bulk & Best effort tins to flow at? Approximately speaking, Cake should give the Best Effort traffic priority over Bulk, until the latter is squashed down to its tin's capacity. So you may see 5-10Mbps of Bulk and 65-70Mbps of Best Effort, depending on some short-term effects. This assumes that the Diffserv marking actually reaches Cake, of course. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] Advantages to tightly tuning latency
> On 22 Apr, 2020, at 1:50 am, Dave Taht wrote: > > Jon, how's SCE looking? ready for a backport yet? We can't do any sort of wide deployment of SCE until it's been approved as an Internet experiment by the IETF. Interested individuals can already compile the SCE-enabled kernel and jump through the hoops to try things out - carefully. There's a bit of infrastructure code to go with the new TCP algorithms and qdiscs, so I'm not certain how easy a backport would be; better to just build the (relatively) current code for now. IETF TSVWG interim meeting next week (the second of two replacing planned in-person sessions at Vancouver) will discuss the big ECT(1) issue, which is hotly disputed between SCE and L4S. The key question is whether ECT(1) should become a classifier input to the network (orthogonal to Diffserv but with some of the same basic problems), or an additional congestion signal output from the network (indicating a lesser degree of congestion, to which a smaller and more nuanced response is desired). It's anyone's guess how that will turn out, but the technical merit is on our side and that really should count for something. If you're keeping an eye on the TSVWG list, expect a major bombshell to drop there in the next few days. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] Advantages to tightly tuning latency
> On 22 Apr, 2020, at 1:25 am, Thibaut wrote: > > My curiosity is piqued. Can you elaborate on this? What does free.fr do? They're a large French ISP. They made their own CPE devices, and debloated both them and their network quite a while ago. In that sense, at least, they're a model for others to follow - but few have. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] Advantages to tightly tuning latency
> On 21 Apr, 2020, at 9:22 pm, Justin Kilpatrick wrote: > > I have a frequently changing link I'm using automated tools to monitor and > tune using Cake. Currently I'm only tuning bandwidth parameter using latency > and packet loss data. > > My reading of the codel RFC seems to say that trying to tune the 'interval' > value using known path and link latency won't provide any advantages over > just tuning the bandwidth parameter. > > Obviously codel is just one part of the Cake setup and I'm wondering if there > are any advantages I'm missing by not providing this extra input using data I > already gather. The default latency parameters are tuned well for general Internet paths. The median path length on the public Internet is about 80ms, for which the default interval of 100ms and target of 5ms works well. Codel is also designed to accommodate a significant deviation from the expected path length without too much difficulty. I think it's only worth trying to adjust this if your typical path is substantially different from that norm. If all your traffic goes over a satellite link, for example, the default parameters might be too tight. If the vast majority of it goes to a local CDN, you could try the "metro" keyword to tighten things up a bit. Otherwise, you'll be fine. Also, most protocols are actually not very sensitive to how tight the AQM is set in the first place. Either they don't really care about latency at all (eg. bulk downloads) or they are latency-sensitive but also sparse (eg. DNS, NTP, VoIP). So they are more interested in being isolated from the influence of other flows, which Cake does pretty well regardless of the AQM settings. It's *considerably* more important to ensure that your shaper is configured correctly. That means setting not only the bandwidth parameter, but the overhead parameters as well. A bad shaper setting could result in some or all of your traffic not seeing Cake as the effective bottleneck, and thus not receiving its care. This can be an orders-of-magnitude effect, depending on just how bloated the underlying hardware is. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] Thinking about ingress shaping & cake
> On 12 Apr, 2020, at 11:23 am, Kevin Darbyshire-Bryant > wrote: > > I’m wondering what the relationship between actual incoming rate vs shaped > rate and latency peaks is? My brain can’t compute that but I suspect is > related to the rtt of the flow/s and hence how quickly the signalling manages > to control the incoming rate. There are two important cases to consider here: the slow-start and congestion-avoidance phases of TCP. But in general, the bigger the difference between the link rate and Cake's shaped rate, the less latency peaks you will notice. Slow-start basically doubles the send rate every RTT until terminated by a congestion signal. It's therefore likely that you'll get a full RTT of queued data at the moment of slow-start exit, which then has to drain - and most of this will occur in the dumb FIFO upstream of you. Typical Internet RTTs are about 80ms. You should expect a slow-start related latency spike every time you start a bulk flow, although some of them will be avoided by the HyStart algorithm, which uses increases in latency as a congestion signal specifically for governing slow-start exit. In congestion avoidance, TCP typically adds one segment to the congestion window per RTT. If you assume the shaper is saturated, you can calculate the excess bandwidth caused by this "Reno linear growth" as 8 bits per byte * 1500 bytes * flow count / RTT seconds. For a single flow at 80ms, that's 150 Kbps. At 20ms it would be 600 Kbps. If that number totals less than the margin you've left, then the peaks of the AIMD sawtooth should not collect in the dumb FIFO and will be handled entirely by Cake. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] Thinking about ingress shaping & cake
> On 10 Apr, 2020, at 4:16 pm, Kevin Darbyshire-Bryant > wrote: > > I have a 80/20mbit FTTC line into the house. Egress shaping/control with > cake is simple, easy, beautiful and it works. Tell it to use 19900Kbit, set > some min packet size, a bit of overhead and off you go. Ingress has more > problems: > > Assuming I do actually get 80Mbit incoming then the naive bandwidth setting > for CAKE would be 80Mbit. Cake internally dequeues at that 80Mbit rate and > therefore the only way any flows can accumulate backlog is when they’re > competing with each other in terms of fairness(Tin/Host) and quantums become > involved…I think. No. If the dequeue rate is never less than the enqueue rate, then the backlog remains at zero pretty much all the time. There are some short-term effects which can result in transient queuing of a small number of packets, but these will all drain out promptly. For Cake to actually gain control of the bottleneck queue, it needs to *become* the bottleneck - which, when downstream of the nominal bottleneck, can only be achieved by shaping to a slower rate. I would try 79Mbit for your case. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] [Make-wifi-fast] Cake in mac80211
> On 5 Feb, 2020, at 6:06 pm, Dave Taht wrote: > >>D) "cobalt" is proving out better in several respects than pure >>codel, >>and folding in some of that makes sense, except I don't know which >>things are the most valuable considering wifi's other problems >> >> Reading paper now. Thanks for the pointer. > > I tend to think out that fq_codel is "good enough" in most > circumstances. The edge cases that cake handles better are a matter of a > few percentage points, vs orders of magnitude that we get with fq_codel > alone vs a vs a FIFO, and my focus of late has been to make things that > ate less cpu or were better offloadable than networked better. Others differ. I think COBALT might be worth putting in, as it should have essentially no net cost and does behave a little better than stock Codel. It's better at handling unresponsive traffic, in particular. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] Cake in mac80211
> On 4 Feb, 2020, at 5:20 pm, Bjørn Ivar Teigen wrote: > > Are there any plans, work or just comments on the idea of implementing cake > in mac80211 as was done with fq_codel? To consider doing that, there'd have to be a concrete benefit to doing so. Most of Cake's most useful features, beyond what fq_codel already supports, are actually implied or even done better by the WiFi environment and the mac80211 layer adaptation (particularly airtime fairness). You can of course attach a Cake instance to the wifi interface as well, if you have a need to do so. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] [PATCH] sch_cake: avoid possible divide by zero in cake_enqueue()
> On 2 Jan, 2020, at 11:21 am, Wen Yang wrote: > > The variables 'window_interval' is u64 and do_div() > truncates it to 32 bits, which means it can test > non-zero and be truncated to zero for division. > The unit of window_interval is nanoseconds, > so its lower 32-bit is relatively easy to exceed. > Fix this issue by using div64_u64() instead. That might actually explain a few things. I approve. Honestly the *correct* fix is for the compiler to implement division in a way that doesn't require substituting it with function calls. As this shows, it's error-prone to do this manually. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] Trouble with CAKE
> On 14 Dec, 2019, at 1:59 pm, Thibaut wrote: > > I’m wondering if this could be an “use of uninitialized data” type of bug. This is why I wouldn't keep working on an old kernel that's full of vendor patches. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] Trouble with CAKE
> On 14 Dec, 2019, at 1:52 am, Thibaut wrote: > > Culprit turned out to be easy to identify: it’s the current master HEAD. > > Reverting 183b320 fixed the issue. That's extremely odd. That commit should only affect traffic carrying the LE DSCP, which is not the default. Perhaps it was not actually the code change, but triggering a rebuild of the module? - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] Trouble with CAKE
> On 13 Dec, 2019, at 3:43 pm, Thibaut wrote: > > I've been using CAKE on my DSL-connected Linux router for the last few years, > and it worked well until very recently. Two things happened: > > 1) My ISP (French "Free") switched my DSLAM to native IPv6, which for the > time being means that I had to revert to using their set-top-box (Freebox) > instead of the VDSL2 model I was using in bridge mode until then (CAKE in > "bridged-ptm ether-vlan" mode) > 2) I upgraded my router from 3.16 (Devuan Jessie) to 4.9 (Devuan ASCII) > > Since then, no matter which setup I use, I cannot get CAKE to work as > intended. Specifically, any long-standing best effort stream (such as a > remote rsync) will be throttled to a near grinding halt even though there is > no other significant traffic going on. Some random bursts can be seen (with > iftop) but nothing ever gets close to half the maximum bandwidth. This is > notably affecting the OpenWRT buildbots I'm hosting on this link. Old kernels, including 4.9 series, tend to be more problematic than the latest ones. If you can, I would recommend updating to a 5.x series kernel, in which Cake is an upstream feature. I won't presume to guess how best to achieve that with your distro. The good news is that Free.fr is among the relatively few ISPs who have actively tackled bufferbloat themselves. As a workaround while you sort this out, you should get reasonable performance just from using the Freebox directly. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] Cake implementations
> On 22 Nov, 2019, at 9:33 pm, Adam Moffett wrote: > > Is the software more or less CPU independent? Would we run into any known > problems with a 72-core Tilera platform? I don't know how well it scales to many cores (though that patch Toke mentioned will certainly help), but it should compile for just about any Linux-supported CPU. We know it works on ARM, MIPS, PowerPC, AMD64… - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] Fighting bloat in the face of uncertinty
> On 3 Oct, 2019, at 8:52 pm, Justin Kilpatrick wrote: > > I've developed a rough version of this and put it into production Monday. > After a few tweaks we're seeing a ~10x reduction in the magnitude of latency > spikes at high usage times. Sounds promising. Keep it up! - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] Frontier FIOS Framing
> I have searched every nook and cranny of the bloated internet looking for any > information I can find on whether Frontier/Verizon FIOS (assuming the only > difference between the service offered by both Frontier and Verizon is in > name only) requires any special framing parameters passed on to sch_cake's > overhead settings. Most mentions of cake/fq/scm/etc and FIOS are ether very > dated and inconclusive or I find messages and forum posts asking questions a > lot like this one. I don't know precisely what framing FIOS uses. However, most provisioning shapers used by cable/fibre ISPs operate on Ethernet frames, so if you use the "ethernet" keyword you should match what the shaper is doing. The proof of the pudding is in the eating, of course. > Currently this is what I have and am also curious if I should be using the > "nat" keyword for both ingress and egress? I'm not entirely sure - see below: If your box is doing NAT *and* you are using a flow-mode that depends on accurate internal host information, then you should have the "nat" keyboard on in both directions. Otherwise it's more efficient to switch it off, though leaving it on does no harm otherwise. The default flow-mode is "triple-isolate", which does use internal host information. So do the "dual-srchost" and "dual-dsthost" modes, which are more precise but need you to specify which direction the traffic is flowing. The "besteffort" and "flows" modes do not, but you should only use those if you're deliberately experimenting with something. > In absence of framing compensation I figured I should just go extreme by > reserving more bandwidth than the qdisc needs because I also read somewhere I > think that mentioned that if you don't compensate and are incorrect > everything stops working as opposed to if you over compensate you might lose > out on bandwidth but you'll still win in the latency department. That's approximately correct, close enough for actual practice. It's also why we included the "conservative" keyword, which applies the maximum amount of framing compensation that is ever likely to be seen in the wild - much more than you'd expect to see on a cable/fibre link, but only slightly more than on most ADSL lines. The overhead compensation matters more with small packets than with the larger ones used for bulk transfers; for the latter, reserving a little more bandwidth will appear to make everything work. For fibre I would try "ethernet" and reserve about 1% bandwidth each way, then if possible test to see whether there is any bloat. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] cake memory consumption
> On 17 Sep, 2019, at 8:31 am, Sebastian Gottschall > wrote: > > according to the output there is a flaw/bug in the memory limit calculation > cake_reconfigure may set buffer_limit to ~0 if no rate is set. > > the following line "min(buffer_limit, max(sch->limit * > psched_mtu(qdisc_dec(sch), q->buffer_config_limit))" doesnt make it better > since buffer_config_limit is not configured > so we got a possible memory overuse here. In C, ~0 means "as near to infinity as an unsigned integer can get", or effectively 4GB. That construct is used to get that part of the calculation out of the way, so that it has no effect in the following nested max() and min() macros. What actually happens here is that the "packet limit" property of the interface becomes governing, and is recalculated in terms of a byte count by multiplying it by the MTU. So the limit configured for each Cake instance in your particular case is 15MB, corresponding to 10,000 packets: > memory used: 0b of 15140Kb With so many Cake instances loaded (very much *not* the normal configuration!) and only 128MB total RAM, 15MB is obviously too high a limit to be completely safe - even though Cake's AQM action will keep the *average* queue depth well below that limit. The correct fix here is not to change the code, but to use the memlimit parameter to override the default. These unusual configurations, where the default logic breaks, are precisely why it was added. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] cake memory consumption
If you're able to log in as root, what does "tc -s qdisc | fgrep memory" tell you? Cake actually does very little dynamic memory allocation. There's a small amount of memory used per queue and per tin, which should total less than 100KB in "besteffort" mode (which you should be using if you have manual traffic classification). All other memory consumption is due to packets in the queue, which are allocated by the kernel when they are received, and deallocated when transmitted or dropped. Cake applies a limit to the memory used by queued packets, generally 4MB by default. The only way this can be exceeded by more than one packet (transiently, when a packet is enqueued and Cake has to drop other packets to make room) is if there's an unaccounted memory leak somewhere. If you can find such a leak in Cake, we'll fix it. But I think it is probably elsewhere. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] cake memory consumption
> On 16 Sep, 2019, at 4:28 pm, Justin Kilpatrick wrote: > > OpenWRT 18.06.4 on a glb1300 and 10+ virtual interfaces with cake. Total > memory usage is 70MB for everything. My IQrouter, which is Archer C7 hardware, is presently running with 73MB free out of 128MB, after nearly 43 days uptime with heavy usage. It has at least two Cake instances running, on a recent kernel. I see from the forum logs that kernel 3.18.x is in use there. That's very old indeed, and I believe there were some fairly big differences in packet memory management since then. It would be entirely possible for some memory management bug to be introduced by a vendor patch, for example. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] Fighting bloat in the face of uncertinty
>> You could also set it back to 'internet' and progressively reduce the >> bandwidth parameter, making the Cake shaper into the actual bottleneck. >> This is the correct fix for the problem, and you should notice an >> instant improvement as soon as the bandwidth parameter is correct. > > Hand tuning this one link is not a problem. I'm searching for a set of > settings that will provide generally good performance across a wide range of > devices, links, and situations. > > From what you've indicated so far there's nothing as effective as a correct > bandwidth estimation if we consider the antenna (link) a black box. Expecting > the user to input expected throughput for every link and then managing that > information is essentially a non-starter. > > Radio tuning provides some improvement, but until ubiquiti starts shipping > with Codel on non-router devices I don't think there's a good solution here. > > Any way to have the receiving device detect bloat and insert an ECN? That's what the qdisc itself is supposed to do. > I don't think the time spent in the intermediate device is detectable at the > kernel level but we keep track of latency for routing decisions and could > detect bloat with some accuracy, the problem is how to respond. As long as you can detect which link the bloat is on (and in which direction), you can respond by reducing the bandwidth parameter on that half-link by a small amount. Since you have a cooperating network, maintaining a time standard on each node sufficient to observe one-way delays seems feasible, as is establishing a normal baseline latency for each link. The characteristics of the bandwidth parameter being too high are easy to observe. Not only will the one-way delay go up, but the received throughput in the same direction at the same time will be lower than configured. You might use the latter as a hint as to how far you need to reduce the shaped bandwidth. Deciding when and by how much to *increase* bandwidth, which is presumably desirable when link conditions improve, is a more difficult problem when the link hardware doesn't cooperate by informing you of its status. (This is something you could reasonably ask Ubiquiti to address.) I would assume that link characteristics will change slowly, and run an occasional explicit bandwidth probe to see if spare bandwidth is available. If that probe comes through without exhibiting bloat, *and* the link is otherwise loaded to capacity, then increase the shaper by an amount within the probe's capacity of measurement - and schedule a repeat. A suitable probe might be 100x 1500b packets paced out over a second, bypassing the shaper. This will occupy just over 1Mbps of bandwidth, and can be expected to induce 10ms of delay if injected into a saturated 100Mbps link. Observe the delay experienced by each packet *and* the quantity of other traffic that appears between them. Only if both are favourable can you safely open the shaper, by 1Mbps. Since wireless links can be expected to change their capacity over time, due to eg. weather and tree growth, this seems to be more generally useful than a static guess. You could deploy a new link with a conservative "guess" of say 10Mbps, and just probe from there. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] Fighting bloat in the face of uncertinty
> On 8 Sep, 2019, at 3:03 am, Justin Kilpatrick wrote:
>
> So you believe that setting the target RTT closer to the path latency was not
> the main contributor to reducing bloat? Is there a configuration I could use
> to demonstrate that one way or the other?
The second-order effect I mentioned is related to the 'target' parameter.
Checking the code, I am reminded that while Cake itself can have 'target' set
from userspace, there actually isn't a parameter to the tc module which allows
setting it independently of 'rtt'. But there *is* a table in q_cake.c (in tc)
which you can temporarily extend with the following entries for experimentation:
static struct cake_preset presets[] = {
{"datacentre", 5, 100},
{"lan", 50, 1000},
{"metro", 500,1},
{"regional",1500, 3},
{"internet",5000, 10},
{"oceanic", 15000, 30},
{"satellite", 5, 100},
{"interplanetary", 5000, 10},
+
+ {"metro-loose", 5000, 1},
+ {"internet-tight", 500,10},
};
If the effect is genuinely due to marking rate, then 'metro-loose' should
behave like 'metro' and 'internet-tight' should behave like 'internet', to a
first-order approximation. If, on the other hand, it's due to the second-order
interaction with CPU scheduling latency, the reverse may be true. The latter
is not something you should be counting on, as it will insert random AQM
marking even when the link is not actually saturated.
You could also set it back to 'internet' and progressively reduce the bandwidth
parameter, making the Cake shaper into the actual bottleneck. This is the
correct fix for the problem, and you should notice an instant improvement as
soon as the bandwidth parameter is correct.
- Jonathan Morton
___
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] Fighting bloat in the face of uncertinty
> On 8 Sep, 2019, at 2:31 am, Justin Kilpatrick wrote: > > If I set a throughput that's 50% too high should it still help? In my testing > it didn't seem to. In that case you would be relying on backpressure from the network interface to cause queuing to actually occur in Cake rather than in the driver or hardware (which would almost certainly be a dumb FIFO). If the driver doesn't implement BQL, that would easily explain 300ms of bloat. In fact I'm unsure as to why changing the AQM parameters would cure it. You may have benefited from an unintentional second-order effect which we normally try to eliminate, when the 'target' parameter gets too close to the CPU scheduling latency of the kernel. I generally find it's better to *underestimate* the bandwidth parameter by 50% than the reverse, simply to keep the queue out of the dumb hardware. But if you want to try implementing BQL in the relevant drivers, go ahead. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] Fighting bloat in the face of uncertinty
> On 8 Sep, 2019, at 1:42 am, Justin Kilpatrick wrote: > > I'm using Cake on embedded OpenWRT devices. You probably saw this video on > the list a month or two ago. > > https://www.youtube.com/watch?v=G4EKbgShyLw I haven't actually watched that one yet... > Anyways up until now I've left cake totally untuned and had pretty great > results. But we've finally encountered a scenario where untuned Cake allowed > for unacceptable bufferbloat on a link. > > Hand configuration in accordance with the best practices provided in the RFC > works out perfectly, but I need a set of settings I can ship with any device > with the expectation that it will be used and abused in many non-standard > situations. Producing non-optimal outcomes is fine, producing dramatically > degraded outcomes is unacceptable. What was the scenario that gave you trouble? I note that Cake is not defined in an RFC. Were you referring to a Codel RFC? Cake is a bit more sophisticated, with the aim of making it easier to configure. > Which leads to a few questions > > 1) What happens if the target is dramatically too low? > > Most of our links can expect latency between 1-10ms, but they may > occasionally go much longer than that. What are the consequences of having a > 100ms link configured with a target of 10ms? The default 'target' parameter is normally 5ms, which goes with a default 'rtt' and 'interval' parameter of 100ms. You shouldn't normally need to set 'target' and 'interval' manually, only 'rtt', and there are various keywords to assist with choosing an appropriate 'rtt'. The default of 100ms is provided by the 'internet' keyword, and this should be able to cope reasonably well with paths down to 10ms. You could also try "regional" which gives you tuning for 30ms, or "metro" which gives you 10ms, with good behaviour on paths within about an order of magnitude of that. Remember, it's the path RTT that matters for this, not the link itself. Should the bandwidth setting correspond to a serialisation delay per packet that approaches the 'target' implied by the above, 'target' will automatically be tuned to avoid the nasty effects that might cause - *unless* you manually override it. So don't do that. ECN enabled flows should not easily notice an 'rtt' setting that's too small. RFC-3168 compliant transports only care about how many RTTs contain at least one CE mark. Non-ECN flows may see elevated packet loss, however, and thus more retransmissions, but the same congestion control behaviour. Cake protects these flows from experiencing "tail loss" which could lead to an RTO that the end-user would notice. > 2) If interval is dramatically unpredictable is it best to err on the side of > under or over estimating? > > The user may select an VPN/exit server of their own choosing, the path to it > over the network may change or the exit may be much further away. Both 10ms > and 80ms would be sane choices of target depending on factors that may change > on the fly. Generally the default 'rtt' of 100ms is suitable for generic Internet paths, including nearby 10ms hops and 500ms satellite-linked islands. The default 5ms target actually puts a floor on the minimum effective RTT that the marking schedule has to cope with. There's also a good chance that the "hystart" algorithm in CUBIC will drop it out of slow-start on very short-RTT paths before the AQM triggers. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] cake in dd-wrt
> On 21 Aug, 2019, at 1:21 pm, Toke Høiland-Jørgensen wrote: > >> Also it seems like a good idea to also submit the NS bit >> exclusion from the ack filter to mainline as well. > > What's that? The ack-filter has rules about which bits and options changing need to be preserved to avoid losing information from the ack stream, and therefore prevent two acks from being coalesced. The version of Cake in the SCE tree adds the former NS bit (which SCE uses as ESCE) to that list. SCE still works fine without that patch, it just preserves more information on the reverse path and thus makes the behaviour a bit smoother. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] cake in dd-wrt
> On 20 Aug, 2019, at 9:39 pm, Sebastian Gottschall > wrote: > > …a heavy bittorrent downloader will still steal the bandwidth of my scp > session. If you can identify the Bittorrent packets, you can mark them CS1, and switch on Cake's "diffserv3" mode (as it is by default). Then the Bittorrent packets will still be able to use full bandwidth if it's available, but will be limited to 1/16th of the total if there is contention. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] [Cerowrt-devel] althea presentation on isp in a box at nanog 76
> On 24 Jun, 2019, at 2:48 pm, Maciej Sołtysiak wrote: > > On Getting Started Page [1] they suggest TP Link C7 for CPEs. Is that still > one of the best suited home routers for getting make-wifi-fast and > bufferbloat-combat in? If nothing else, it's what the IQrouter is based on. Custom firmware, sticker over the logo… it works well. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] Possible conntrack lookup improvements
> On 3 May, 2019, at 4:55 pm, Kevin Darbyshire-Bryant > wrote: > > Two patches attached - one is a simple variable elimination with no > functional change. The second changes/simplifies the conntrack tuple lookup > & usage. I’ve had a play and I don’t think I’ve broken any of the host > fairness BUT it could do with some more testing, that’s where you come in… Looks like sound logic, as long as it does actually work. It could be a useful speedup for those small CPE devices which need NAT and host-fairness working. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] Testing Scenarios for Validation of Ack_Filter implementation in ns-3
> On 22 Apr, 2019, at 2:45 pm, Shefali Gupta wrote: > > It would really help if you could suggest the testing scenarios so that I can > validate Ack_Filter implementation in ns-3. It may help to understand the design intentions of the ack-filter: 1: Reduce the bandwidth consumed by pure acks, and possibly also the latency of the ack stream. 2: Avoid losing any relevant information from the ack stream. This particularly includes not erasing any "tail" acks and not interfering with any SYN, FIN or RST packets. To test the latter, you really need a correctness analysis, but it should be sufficient to observe no reduction in performance across a wide range of scenarios, including those involving application-limited workloads (with a lot of tail acks), many short flow starts, continuous transfers with various amounts of packet loss in the forward direction, and cases where ack-filter activity is intense. That last scenario will involve a demonstration of point 1, which is best achieved by setting up an asymmetric link, then running one flow over the "wide" direction but many flows over the "narrow" direction. A 10:1 bandwidth ratio with a 1:50 flow setup should do the trick. The ack filter should greatly improve the throughput of the single, fast flow, and also free up a little bandwidth on the slow side for the many flows. As an aside, in the current Cake version we treat the former NS bit the same way as the ECE and CWR bits in the ack filter, so you may want to ensure this tweak is also made in your version. This is to improve its behaviour on SCE traffic. The change should have no effect on non-SCE traffic. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] The two SCE tests I have in mind
> On 24 Mar, 2019, at 12:05 pm, Pete Heist wrote: > > tcpdump -r file.pcap udp port 2112 and greater 80 and "ip[1] != 0x1” > > “greater 80” ignores the handshake packets and 0x1 is whatever TOS value we > want to make sure the packets contain. We can use different filters for other > traffic. Bear in mind that the TOS byte contains ECN as well as DSCP fields, and the latter is left-justified. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] Sce In cake testers wanted
> On 24 Mar, 2019, at 8:37 am, Pete Heist wrote: > > I should theoretically arrive at the boat today some time after 3pm, having > picked up a mini HDMI to HDMI adapter, which we can use with the cable that’s > there… Awesome. I'm also setting up a Linux VM on my Mac, which should help things along. We're bringing up some actual hardware with the SCE-enabled Cake on it now. Dave wants to investigate various theoretical phenomena the Internet might exhibit with a mixture of ECN codepoints; I just want to be sure it actually works as intended, before I move on to fiddling with TCP. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] Sce In cake testers wanted
> On 24 Mar, 2019, at 12:54 am, Dave Taht wrote: > > There are two openwrt routers on the boat. I mean as a complete system, with a TCP that responds to the signal. As well as the compatibility cases where a TCP doesn't respond, or one that does respond doesn't get a signal, or a mixture of aware and ignorant flows share the queue. Can't do any of that without a working TCP. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] Sce In cake testers wanted
> On 23 Mar, 2019, at 10:28 pm, Kevin Darbyshire-Bryant > wrote: > > Sent a github PR to make it build - not run it yet. LGTM, so I approved it. I've reached the "bufferboat" and started to settle in, met Rod and Pete… I brought a Pi Zero W with me as a basic Linux test environment, but I need to find a mini to macro HDMI adapter to make it work with this TV. Pete says he'll look for one to bring over tomorrow. So now we probably have *two* working middlebox implementations, using different approaches, but we need a visualisation tool (hi Pete) and some sort of working TCP that uses it (whether using the sqrt rule or the 1/n rule, doesn't really matter - but preferably examples of both). For the latter, we can temporarily get away with inferring stuff from standard tcptrace. If there's a working implementation of AccECN for the feedback path, we might as well use that for demo purposes. I want to be able to demonstrate SCE actually working as designed, in a single-queue implementation, to answer the main argument that some of the L4S folks have latched onto. But I don't know how soon we'll be able to manage that. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] [Ecn-sane] [Bloat] The "Some Congestion Experienced" ECN codepoint - a new internet draft -
> On 11 Mar, 2019, at 5:29 pm, Holland, Jake wrote:
>
> The key difference to me is that L4S doesn't work without a dual queue.
> It embeds a major design decision ("how many queues should there be at
> a middle box?") into the codepoint, and comes with very narrow requirements
> on the sender's congestion control.
It's certainly unclear to me what happens when an L4S flow passes through a
Classic ECN middlebox, especially one with only a single queue. I get the
impression that a DCTCP-like congestion response would tend to starve out
conventional flows competing with it. Unless you have data showing otherwise?
That has serious implications for incremental deployability, because you can't
safely deploy L4S endpoints until single-queue Classic ECN middleboxes have all
been replaced with L4S or flow-isolating types. And without L4S endpoints in
use, where's the impetus to do that? Chicken and egg.
Conversely, an SCE-aware flow passing through a Classic ECN middlebox behaves
just like a Classic ECN flow. The only question arises when a single-queue AQM
is made SCE-aware, and in that case the SCE-aware flows are the ones that might
get outcompeted (ie. they are friendlier, not more aggressive). If necessary,
it would be easy to specify that single-queue AQMs "should not" produce SCE
marks, only the flow-isolating types - which in any case are easier to deploy
at edge devices where statistical multiplexing works less well.
Incremental deployability is very important when tackling a project as big as
this. SCE takes it as a central tenet. L4S appears, in practice, to have
overlooked it. That's my objection to L4S.
- Jonathan Morton
___
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] profiling using perf
> On 11 Mar, 2019, at 4:49 pm, Adrian Popescu wrote: > > This doesn't mean cake is poorly optimized or poorly > implemented. It's not a good fit for small embedded systems with small > CPU caches. More importantly, how much CPU time is spent elsewhere than in sch_cake.c? I suspect there's a lot of overhead that we have only indirect control over. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] [Bloat] The "Some Congestion Experienced" ECN codepoint - a new internet draft -
> On 11 Mar, 2019, at 11:07 am, Mikael Abrahamsson wrote: > > Well, I am not convinced blowing the last codepoint on SCE has enough merit. I will make a stronger statement: I am convinced that blowing the last codepoint on L4S does *not* have enough merit. Meanwhile, work continues. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] [Bloat] The "Some Congestion Experienced" ECN codepoint - a new internet draft -
serted until the link is full. SCE-aware AQMs and TCPs are not difficult to describe and should not be very difficult to implement. Though not described explicitly in the SCE I-D, I'm working on descriptions of how to do so, which should become additional I-Ds. We should then be able to write working code and run simulations in ns-3, and maybe also in Linux. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] [Bloat] The "Some Congestion Experienced" ECN codepoint - a new internet draft -
> On 10 Mar, 2019, at 9:08 pm, Holland, Jake wrote: > > It's easy to accidently read section 5 as underspecified concrete > proposals instead of rough sketches for future direction that might > be worth investigating. This is something I noticed as well, and have edited to match the intended tone of the document (which is to leave specific implementation details to other documents, and to focus on the information conveyed). > New SCE-aware receivers and transport protocols must continue to I had already changed this to SHOULD, among other edits. > "Some" in "Some Congestion Experienced" is maybe misleading, and > arguably has the same meaning as "Congestion Experienced". > > I was thinking maybe "Pre-Congestion Experienced" or "Queue > Utilization Observed", or if you want to preserve "SCE" and the > link to CE (which I do agree is nice), maybe "Slight" or "Sub" > instead of "Some", just to try to make it more obvious it's > flagging a lesser situation than "there is some congestion". An interesting idea, but SCE marks will appear even when there's a lot of congestion (at high rates, ie. probably every packet that doesn't carry CE), as well as showing up at low frequency when the level of congestion only warrants reducing the growth rate. I think the word "Some" is sufficiently descriptive, while "Slight" might cause people to ignore it completely. I think other points are less urgent to address in the first submission. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] Bug or not Cake + hfsc or Cake + Drr
> On 7 Mar, 2019, at 8:23 pm, Martin Zaharinov wrote: > > tc qdisc add dev eth1 root handle 1: hfsc default 1 > tc class add dev eth1 parent 1: classid 1:1 hfsc ls rate 7mbit ul rate 7mbit > tc qdisc add dev eth1 parent 1:1 cake besteffort bandwidth 5mbit datacentre > nat memlimit 32m > > user speed is 5mbit and on hfsc i add extra 2 mbit to give free limit for > cake. > and work fine but in dmesg get : > > [44835.530255] qdisc_peek_len: cake qdisc 8017: is non-work-conserving? > [44840.689403] HFSC: cake qdisc 8012: is non-work-conserving? Here's a setup that should work: > tc qdisc add dev eth1 root handle 1: hfsc default 1 > tc class add dev eth1 parent 1: classid 1:1 hfsc ls rate 5mbit ul rate 5mbit > tc qdisc add dev eth1 parent 1:1 cake unlimited besteffort internet nat > memlimit 32m Why don't you try that instead of editing the kernel? (Also, don't use "datacentre" with bandwidths this low, it doesn't magically do what you're probably thinking of.) - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] Using firewall connmarks as tin selectors
> On 4 Mar, 2019, at 7:37 am, Ryan Mounce wrote: > > The overwriting of the DSCP bits from fwmark could be called > 'staining', as opposed to DSCP 'bleaching'. But this doesn't sound > very appealing when we're baking delicious CAKE - we're in the kitchen > not the laundry! So how about dyeing, or colouring? …icing? https://www.youtube.com/watch?v=ouRcDRpsRsA - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] Upstream submission of dual-mode fairness patch
> On 4 Mar, 2019, at 4:55 am, Georgios Amanakis wrote: > > …the fairness logic wouldn't account for that "ingress traffic" and would > yield fairer results. Well there we have a quandary, since presently it enforces fairness of *load* on the bottleneck link, but the change would alter that to fairness of *delivered* traffic. The current setup is arguably more robust against adversarial traffic, don't you think? - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] Upstream submission of dual-mode fairness patch
> On 3 Mar, 2019, at 6:35 pm, Pete Heist wrote: > > It almost seems like in ingress mode, dropped packets are being counted as > bytes transferred, when they weren’t. Well yes, that's the whole point - at least as far as the shaper is concerned. Ingress mode is supposed to deal with the case where inbound packets have already traversed the bottleneck link *before* Cake gets to choose what to do with them. But that shouldn't affect fairness, only total throughput. Fairness is not handled by the shaper, but by the (very) extended DRR++ algorithm. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] Upstream submission of dual-mode fairness patch
> On 3 Mar, 2019, at 6:07 pm, Pete Heist wrote: > > Does it say anything to you that enabling ECN makes the asymmetry go away? Yes - it tells me that it has something directly to do with how dropped packets are accounted for, not with the congestion response by TCP. ECN changes the number of dropped packets, and shouldn't change the congestion response. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] Upstream submission of dual-mode fairness patch
> On 3 Mar, 2019, at 1:26 pm, Sebastian Moeller wrote: > >>> Doesn't this look like ingress magic being applied selectively to the users >>> based on number of flows? I thought that the idea behind the ingress >>> keyword is to effectively shape harder the more bulk flows are coming in. >> >> No, it simply counts dropped packets against the shaper, as well as those >> actually transmitted. > > Sure, but the question is how is the resulting "pressure" to drop/mark > distributed over the existing (bulk) flows. > >> There shouldn't be that many packets being dropped to make this much of a >> difference. > > My intuition (probably wrong) is that is not the few packets dropped, but the > fact that the the dropping does seem to be restricted to the flows of the IP > with more flows, no? As long as there is a backoff response to a dropped packet, the extra back-pressure should be contained to the flows experiencing drops, and other flows should see no difference - so you should see a slight reduction in goodput on the 16 flows, but *no increase* on the single flow in parallel. Even if that were not the case, the single flow should take longer on average to recover from a cwnd reduction (even in CUBIC) to the fair BDP. That should result in a greater reduction in goodput on the single flow than the many flows - but we see the reverse here. So I'm not entirely sure what's happening here, but at least the asymmetry isn't too bad; it's achieving significantly better host-fairness than a pure flow-fair system would. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] Upstream submission of dual-mode fairness patch
> On 3 Mar, 2019, at 11:53 am, Sebastian Moeller wrote: > > Doesn't this look like ingress magic being applied selectively to the users > based on number of flows? I thought that the idea behind the ingress keyword > is to effectively shape harder the more bulk flows are coming in. No, it simply counts dropped packets against the shaper, as well as those actually transmitted. There shouldn't be that many packets being dropped to make this much of a difference. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] progress? dual-src/dsthost unfairness
>> I tried Toke's and Jonathan's suggestion, dropping the >> sparse_flow_count. Tthe results are the same (fairness). >> In a hash collision in this patch the host_bulk_flow_count is not updated, >> does this make sense? > > Yeah, think so; it should be updated later when that flow transitions to > bulk. > > Care to resend with a proper commit message + signed-off-by line (or > open a pull request on github)? I figure we can put it into the github > repo for a bit more testing before submitting a patch upstream :) The important thing is that the host_bulk_flow_count values match the actual bulk-status and host-reference properties assigned to each flow queue. When a hash collision occurs, the bulk-status of the affected flow doesn't change but the hosts to which it refers might do. In that case the host_bulk_flow_count values must be decremented on the old hosts and incremented on the new ones *if* the queue is in the bulk set. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] progress? dual-src/dsthost unfairness
> On 13 Feb, 2019, at 9:26 pm, > wrote: > > Because the flow for which cake_enqueue() is run is by definition a sparse > one. So the host_load should be adjusted according to the sparse flow count > for the particular host. That's my reasoning behing this, would be happy to > hear other opinions. On the contrary, even if a particular flow is sparse, the relevant quantum calculation should involve the number of *bulk* flows attached to the same host. Though there is possibly an argument for making it the *sum* of the sparse and bulk flows, making it just the sparse ones is wrong. I need to think about this a bit more. But meanwhile, try it with the sum. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] progress? dual-src/dsthost unfairness
> On 13 Feb, 2019, at 8:31 pm, George Amanakis wrote: > > I recently rewrote the patch (out-of-tree cake) so as to keep track of the > bulk/sparse flow-count per host. I have been testing it for about a month > on a WRT1900ACS and it runs fine. > > Would love to hear if Jonathan or anybody else has thought of > implementing something different. This does actually look more reasonable. However: 1: Why is host_load calculated using host_sparse_flow_count instead of host_bulk_flow_count in cake_enqueue()? 2: Do we actually need to maintain host_sparse_flow_count at all, or only the bulk one? If the above is corrected, I don't think there are any uses left. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] [Cerowrt-devel] https://tools.ietf.org/html/draft-ietf-tsvwg-le-phb-06 is in last call
alid traffic classes such as Network Control and Isochronous Audio/Video, which might reasonably have strict priority over the global classes but which Should Not be sent over the core network, and Should be interpreted as Low Priority if encountered by a device not specifically configured to understand them. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] separate shaping of wifi
> On 4 Feb, 2019, at 12:04 am, Jonas Mårtensson > wrote: > > I'm running OpenWrt with sqm on my home router. Is there any potential > problem with enabling sqm with cake on both eth0 (wan) and wlan0? The reason > for doing this is that if I only do shaping on the wan interface I still get > bad uplink bufferbloat on wifi. I assume this is because the bottleneck in > this case is actually the wifi and the bloat is in the end device (laptop, > phone, etc.). By shaping to a rate lower than the wifi rate, the bloat > disappears. Is there any better way to accomplish this than to enable a > second sqm instance on wlan0 if I don't want to sacrifice speed for wired > devices? Aside from fixing the end device so it isn't bloated in the first place, I think you have a reasonable solution there. You might only need to limit the ingress rate on wlan0, if your router's wifi stack is debloated. - Jonathan Morton ___ Cake mailing list Cake@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cake
Re: [Cake] [Cerowrt-devel] https://tools.ietf.org/html/draft-ietf-tsvwg-le-phb-06 is in last call
> On 3 Feb, 2019, at 8:39 pm, Dave Taht wrote:
>
> it's 01 which I guess is:
>
> diff --git a/sch_cake.c b/sch_cake.c
> index 3a26db0..67263b3 100644
> --- a/sch_cake.c
> +++ b/sch_cake.c
> @@ -343,7 +343,7 @@ static const u8 diffserv4[] = {
> };
>
> static const u8 diffserv3[] = {
> - 0, 0, 0, 0, 2, 0, 0, 0,
> + 0, 1, 0, 0, 2, 0, 0, 0,
>1, 0, 0, 0, 0, 0, 0, 0,
>0, 0, 0, 0, 0, 0, 0, 0,
>0, 0, 0, 0, 0, 0, 0, 0,
>
> (or is that reversed? my big endian/little endian chops scuks, and
> nobody modified the gen_cake_const tool to match what cake expects
> now)
That looks correct to me, though it should also be added to the other Diffserv
modes including a bulk tin. Fields in TCP/IP are all laid out in "network
order" which is most-significant bit/byte first, the same way you'd *normally*
write a number down.
The TOS field can sometimes be confusing because the DSCP field is the upper 6
bits and the ECN field the lower 2, and the BSD Sockets API gives you the while
byte to work with while DSCPs are quoted as just their own 6 bits. So you have
to shift the latter left two bits before applying them to the former.
- Jonathan Morton
___
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake
