Re: [Cerowrt-devel] [Cake] active sensing queue management
Hi Daniel, thanks for the clarifications. On Jun 11, 2015, at 02:10 , Daniel Havey dha...@gmail.com wrote: Hmmm, maybe I can help clarify. Bufferbloat occurs in the slowest queue on the path. This is because the other queues are faster and will drain. AQM algorithms work only if they are placed where the packets pile up (e.g. the slowest queue in the path). This is documented in Kathy and Van's CoDel paper. I am with you so far. This is usually all well and good because we know where the bottleneck (the slowest queue in the path) is. It is the IP layer in the modem where the ISP implements their rate limiter. That is why algorithms such as PIE and CoDel are implemented in the IP layer on the modem. Okay. Suppose the full committed rate of the token bucket rate limiter is 8 Mbps. This means that the queue at the IP layer in the modem is capable of emitting packets at 8 Mbps sustained rate. The problem occurs during peak hours when the ISP is not providing the full committed rate of 8 Mbps or that some queue in the system (probably in the access link) is providing something less than 8 Mbps (say for sake of discussion that the number is 7.5 Mbps). We know that (see Kathy and Van's paper) that AQM algorithms only work when they are placed at the slowest queue. However, the AQM is placed at the queue that is capable of providing 8 Mbps and this is not the slowest queue. The AQM algorithm will not work in these conditions. This is what is shown in the paper where the CoDel and PIE performance goes to hell in a handbasket. The ASQM algorithm is designed to address this problem. Except that DOCSIS 3.1 pie in the modem does not work that way. As I understand http://www.cablelabs.com/wp-content/uploads/2014/06/DOCSIS-AQM_May2014.pdf section 3.2 MAP PROCESSING REQUIREMENTS, cable-modem pie will not stuff data into the lower layers until it has received a grant to actually send that data, hence no uncontrolled sub-IP layer buffer bloat possible (unless there are severe RF issues that take out data during transmission). So at least for the upstream direction docsis 3.1 pie will not suffer from buffer displacement, if I understand the cable labs white paper correctly. Your solution still might be a valuable add-on to control the downstream buffer bloat in addition. I also believe that free in france had a modified DSL driver for their box that made sure sub-IP buffering was bound to a low number of packets as well, so no displaced buffers there as well. Now it seems that this solution for DSL was unique so far and has not caught on, but once docsis3.1 modems hit the market upstream PIE in the modems will be reality. On Wed, Jun 10, 2015 at 1:54 PM, Sebastian Moeller moell...@gmx.de wrote: Hi Dave, On Jun 10, 2015, at 21:53 , Dave Taht dave.t...@gmail.com wrote: http://dl.ifip.org/db/conf/networking/networking2015/1570064417.pdf gargoyle's qos system follows a similar approach, using htb + sfq, and a short ttl udp flow. Doing this sort of measured, then floating the rate control with cake would be fairly easy (although it tends to be a bit more compute intensive not being on a fast path) What is sort of missing here is trying to figure out which side of the bottleneck is the bottleneck (up or down). Yeah, we never did figure out how to separate the up from the downlink. However, we just consider the access link as a whole (up + down) and mark/drop according to ratios of queuing time. This is a bit sad; why reduce say the effective uplink bandwidth if only the downstream is contended? Not that I have a good alternative solution that will not require help from outside boxes. Overall it seems to work well, but, we never did a mathematical analysis. Kind of like saying it's not a bug, it is a feature. And it this case it is true since both sides can experience bloat. Yes, but you only want to throttle traffic on the congested leg of the link, otherwise bandwidth efficiency goes to hell if you look at bi-direction link-saturating traffic. Yeah, they relay on having a reliable packet reflector upstream of the “bottleneck” so they get their timestamped probe packets returned. In the paper they used either uplink or downlink traffic so figuring where the bottleneck was easy at least this is how I interpret “Experiments were performed in the upload (data flowing from the users to the CDNs) as well as in the download direction.. At least this is what I get from their short description in glossing over the paper. Nice paper, but really not a full solution either. Unless the ISPs cooperate in supplying stable reflectors powerful enough to support all downstream customers. But if the ISPs cooperate, I would guess, they could eradicate downstream buffer bloat to begin with. Or the ISPs could have the reflector also add its own
Re: [Cerowrt-devel] [Cake] active sensing queue management
On Wed, 10 Jun 2015, Daniel Havey wrote: We know that (see Kathy and Van's paper) that AQM algorithms only work when they are placed at the slowest queue. However, the AQM is placed at the queue that is capable of providing 8 Mbps and this is not the slowest queue. The AQM algorithm will not work in these conditions. so the answer is that you don't deploy the AQM algorithm only at the perimeter, you deploy it much more widely. Eventually you get to core devices that have multiple routes they can use to get to a destination. Those devices should notice that one route is getting congested and start sending the packets through alternate paths. Now, if the problem is that the aggregate of inbound packets to your downstreams where you are the only path becomes higher than the available downstream bandwidth, you need to be running an AQM to handle things. David Lang ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] [Bloat] [Cake] active sensing queue management
On Thu, 11 Jun 2015, Sebastian Moeller wrote: On Jun 11, 2015, at 03:05 , Alan Jenkins alan.christopher.jenk...@gmail.com wrote: On 10/06/15 21:54, Sebastian Moeller wrote: One solution would be if ISPs made sure upload is 100% provisioned. Could be cheaper than for (the higher rate) download. Not going to happen, in my opinion, as economically unfeasible for a publicly traded ISP. I would settle for that approach as long as the ISP is willing to fix its provisioning so that oversubscription episodes are reasonable rare, though. not going to happen on any network, publicly traded or not. The question is not can the theoretical max of all downstream devices exceed the upstream bandwidth because that answer is going to be yes for every network built, LAN or WAN, but rather does the demand in practice of the combined downstream devices exceed the upstream bandwidth for long enough to be a problem it's not even a matter of what percentage are they oversubscribed. someone with 100 1.5Mb DSL lines downstream and a 50Mb upstream (30% of theoretical requirements) is probably a lot worse than someone with 100 1G lines downstream and a 10G upstream (10% of theoretical requirements) because it's far less likely that the users of the 1G lines are actually going to saturate them (let alone simultaniously for a noticable timeframe), while it's very likely that the users of the 1.5M DSL lines are going to saturate their lines for extended timeframes. The problem shows up when either usage changes rapidly, or the network operator is not keeping up with required upgrades as gradual usage changes happen (including when they are prevented from upgrading because a peer won't cooperate) As for the 100% provisioning ideal, think through the theoretical aggregate and realize that before you get past very many layers, you get to a bandwidh requirement that it's not technically possible to provide. David Lang ___ Cerowrt-devel mailing list Cerowrt-devel@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/cerowrt-devel
Re: [Cerowrt-devel] [Cake] active sensing queue management
Hi Alan, On Jun 11, 2015, at 03:05 , Alan Jenkins alan.christopher.jenk...@gmail.com wrote: On 10/06/15 21:54, Sebastian Moeller wrote: Hi Dave, On Jun 10, 2015, at 21:53 , Dave Taht dave.t...@gmail.com wrote: http://dl.ifip.org/db/conf/networking/networking2015/1570064417.pdf gargoyle's qos system follows a similar approach, using htb + sfq, and a short ttl udp flow. Doing this sort of measured, then floating the rate control with cake would be fairly easy (although it tends to be a bit more compute intensive not being on a fast path) What is sort of missing here is trying to figure out which side of the bottleneck is the bottleneck (up or down). Yeah, they relay on having a reliable packet reflector upstream of the “bottleneck” so they get their timestamped probe packets returned. They copy frob real IP headers. They don't _say_ how the reflection works, but I guess low TTL - ICMP TTL exceeded, like traceroute. Then I read Gargoyle also use ICMP TTL exceeded and I thought my guess is quite educated 8). Daniel elucidated their magic packets: they create self-addressed IP packets at the simulated CPE and inject them in the simulated cable link; the other end will pass the data through its stack and once the sender-self-addressed packet reaches the IP-layer of the simulated CMTS it gets send back, since that IP layer sees the CPE’s IP address as the to address. @Daniel, this trick can only work if a) the magic packets are only passed one IP-hop since the first upstream IP-layer will effectively bounce them back (so the injector in the docsis case needs to be the cable modem) b) the CPE actually has an IP that can be reached from the outside and that is known to the person setting up your AQM, is that correct? How does this work if the CPE acts as an ethernet bridge without an external IP? Note the size of the timestamp, a generous 8 bytes. It just happens that ICMP responses are required to include the first 8 bytes of the IP payload 8). In the paper they used either uplink or downlink traffic so figuring where the bottleneck was easy at least this is how I interpret “Experiments were performed in the upload (data flowing from the users to the CDNs) as well as in the download direction.. At least this is what I get from their short description in glossing over the paper. Ow! I hadn't noticed that. You could reduce both rates proportionally but the effect is inelegant. I think that it what they do, as long as one only measures uni-directional saturating traffic this approach will work fine as the bandwidth loss in the opposite direction simply does not materialize. I wonder what Gargoyle does... 2012 gargoyle developer comment says There are not settings for active congestion control on the uplink side. ACC concentrats on the download side only. Random blog post points out this is sufficient to fix prioritization v.s. bufferbloat. In upstream direction this is not a big problem because your router can still prioritize which packet should be sent first. (Yay, I get to classify every application I care about /s and still get killed by uploads in http). Not fully convinced that this is fully sane, as in cable systems the upstream bandwidth can fluctuate significantly depending on how many people are active. Actually scratch the “cable” since most customer links have shared oversubscribed links somewhere between the CPE and the internet that will make static bandwidth shaping mis-behave some of the time. A good ISP just manages the oversubscription well enough that this issue only occurs transiently… (I hope). One solution would be if ISPs made sure upload is 100% provisioned. Could be cheaper than for (the higher rate) download. Not going to happen, in my opinion, as economically unfeasible for a publicly traded ISP. I would settle for that approach as long as the ISP is willing to fix its provisioning so that oversubscription episodes are reasonable rare, though. Nice paper, but really not a full solution either. Unless the ISPs cooperate in supplying stable reflectors powerful enough to support all downstream customers. I think that's a valid concern. Is TTL Exceeded rate-limited like Echo (because it may be generated outside the highest-speed forwarding path?), and would this work as tested if everyone did it? I thing Daniel agrees and that is why they came up with the “magic” packet approach (that drags in its own set of challenges as far as I can see). But if the ISPs cooperate, I would guess, they could eradicate downstream buffer bloat to begin with. Or the ISPs could have the reflector also add its own UTC time stamp which would allow to dissect the RTT into its constituting one-way delays to detect the currently bloated direction. (Think ICMP type 13/14 message pairs on steroids, with higher
