Re: [Bloat] number of home routers with ingress AQM
On Wed, 3 Apr 2019 at 00:05, Sebastian Moeller wrote: > > > > > On Apr 2, 2019, at 15:15, Ryan Mounce wrote: > > > > On Tue, 2 Apr 2019 at 22:08, Sebastian Moeller wrote: > >> > >> I just wondered if anybody has any reasonable estimate how many end-users > >> actually employ fair-queueing AQMs with active ECN-marking for ingress > >> traffic @home? I am trying to understand whether L4S approach to simply > >> declare these as insignificant in number is justifiable? > > > > L4S people are concerned by RFC 3168 / "classic" ECN bottlenecks > > *without* fq. > > I know, but I believe that they misunderstand the issues resulting > from post-bottleneck shaping, like ingress shaping on the remote side of the > true bottleneck. The idea seems that sending at too high a rate is > unproblematic if the AQM can simply queue up these packets and delay them > accordingly. But in the ingress shaper case these packets already traversed > the bottleneck and aone has payed the bandwidth price to hoist them to the > home, delaying or even dropping on the AQM side will not magically get the > time back the packets took traversing the link. My understanding is that with an AQM performing RFC 3168 style ECN marking, the L4S flows will build a standing queue within their flow queue before the AQM starts ECN marking. At this point the L4S flow will be less responsive to marks with a linear (?) decrease rather than treating it like loss (multiplicative decrease), however the AQM will just keep on marking to keep in in check. The fq shaper will continue to isolate it from other flows at the bottleneck and enforce fairness between flows (or in the case of an ingress shaper after the true bottleneck, continue to selectively apply drops/marks that effectively 'nudge' flows towards fairness). If there's no fq and there is RFC 3168 style ECN marking, the L4S flows assume that they're receiving aggressive DCTCP-style marking, respond less aggressively to each mark, and will starve conventional Reno friendly flows. L4S people are relying on there being almost no such bottlenecks on the internet, and to be honest I think this is a fair assumption. The most deployed single queue AQM may be PIE as mandated by DOCSIS 3.1, which had ECN marking disabled before the whole DualQ/L4S thing. Bob Briscoe thinks the main concern is people that have found old Cisco routers with RED and re-commissioned them... As L4S flows would still be still be getting ECN marked in either case, there would be no dropping of packets that have already traversed the bottleneck in order to signal congestion. So long as there is fq to enforce fairness, I can't see any problem. Sure, signalling congestion without loss doesn't mean that the packets haven't already spent more than their fair share of time at the previous bottleneck. This is a more general problem with shaping ingress further downstream from the true bottleneck - and I don't see that it's any worse here than with any other TCP overshooting during slow start. There are certainly more real threats such as Akamai's FAST TCP. It's like BBR in that it attempts to detect and respond to congestion based on queuing latency, and tries to ignore low levels of random packet loss that don't occur due to congestion. Their implementation definitely presents issues for ingress shaping: https://forums.whirlpool.net.au/thread/2530363 I saw that thread years ago, and then eventually saw the issue for myself after changing ISP. Suddenly Windows Update was downloading from my ISP's embedded Akamai cache using FAST TCP, and was effectively unresponsive to cake's ingress shaping, instead filling the queue in my ISP's BNG. Fact is... ingress shaping is a hack. The real problem needs to be solved in the BNG, in the CMTS, in the DSLAM, etc. L4S is just one proposal and of course it deserves scrutiny before gobbling up the precious ECT(1) codepoint, however it seems to have some traction with vendors and a chance at wide deployment with a view to address address exactly this problem *at* the bottleneck. For that, it also deserves consideration. > Why do I care, because that ingress shaping setup is what I use at > home, and I have zero confidence that ISPs will come up with a solution to my > latency desires that I am going to be happy with... And what I see from the > L4S mixes light with a lot of shadows. > > > > I don't think there would be any such ingress shapers > > configured on home gateways. Certainly not by anyone on this list... > > anyone running non-fq codel or flowblind cake for ingress shaping? > > As stated above, I believe fq to not be a reliable safety valve for > the ingress shaping case. -Ryan ___ Bloat mailing list Bloat@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/bloat
Re: [Bloat] number of home routers with ingress AQM
On Wed, 3 Apr 2019 at 00:41, Jonathan Foulkes wrote: > > Hi Ryan, > > See below > > > On Apr 2, 2019, at 9:33 AM, Ryan Mounce wrote: > > > > On Tue, 2 Apr 2019 at 23:35, Mikael Abrahamsson wrote: > >> > >> I've read rumours about some ISPs implementing interaction with the VDSL > >> DSLAM where there is an estimation of the current link-speed for each > >> individual customer and then it tries to set the BNG egress shaper > >> appropriately. > > > > NBN here in Australia do this for their wholesale FTTN/B VDSL2 > > product, injecting the downstream and upstream sync speed into > > PPPoE/DHCP requests. This still depends on the retail service > > providers to make use of these attributes from their BNG to configure > > the shaper. > > Very interesting, and extremely useful that they surface the sync rate in the > PPPoE/DHCP exchanges. How are those externalized in something like OpenWRT at > this time? These attributes are tacked onto the requests towards the ISP - so unfortunately they're not of any use to configure a shaper on the home gateway side. > FYI- The IQrouter supports the notion of a tightly coupled modem from which > current sync rates (as well as other SNR / Atten stats) can be dynamically > queried, which in turn are used in the AQM settings computations. Works > wonders on flaky DSL lines that vary sync rate with time of day and weather. > If that VDSL2 sync rate relayed is exposed in the system somehow, it would > make dynamic adjustments possible. Yeah, I did something similar for a family member stuck on ADSL. Scraped the sync speed from the cheap bridged TP-Link modem and used it to configure cake - with periodic updates in case of a resync. > > NBN is an example again. Their documented behaviour is to police > > traffic in both directions. Most ISPs then shape in the downstream - > > and it's up to a tightly managed (TR-069 ?) ISP HGW or a diligent end > > user to shape traffic in the upstream. Many - probably even most users > > have no shaping whatsoever in the upstream, only a policer. > > > > And then there is their new FTTdp product, where it is not currently > > possible to determine the real VDSL2 sync speed. If there's a drop of > > rain it will resync at a lower speed in the upstream, and then > > everything ends up queuing inside the supplied modem… > > Oh, so they regressed from their other offering, not exposing sync rates? Yep. The wholesaler provides the modem (also integrating a reverse power supply to the mini DSLAM / "DPU") so the customer demarcation point is a gigabit ethernet port - abstracting the xDSL side of things from the customer's equipment. Apparently they're working on making real time sync speeds available ISPs that want to query them as part of service diagnostics, and haven't even got that contrived solution working yet. And then you still need to get that info from your ISP, fortunately mine has just added the unique ability to trigger your own service diagnostics and retrieve the results so it may be possible to scrape it this way. Quite the kludge at that point. Fortunately the mini DSLAM is typically less than 50m from your property boundary and spliced directly into the lead-in cable, so it is normally capable of syncing above the maximum 100/40 rate that's offered and thus limited by the consistent policer rather than the variable line rate. Unless it rains - then all bets are off. Isn't copper great? > > BTW- this hole notion of the BGM relaying the provisioned or current sync > rates should be a mandated requirement, as it has huge benefits for AQMs. Not > that it can be relied on 100%, but at least it makes a good starting point. ___ Bloat mailing list Bloat@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/bloat
Re: [Bloat] number of home routers with ingress AQM
Hi Jonathan, Thanks for the data. On April 2, 2019 4:14:34 PM GMT+02:00, Jonathan Foulkes wrote: >Responses below > >> On Apr 2, 2019, at 8:10 AM, Mikael Abrahamsson >wrote: >> >> On Tue, 2 Apr 2019, Sebastian Moeller wrote: >> >>> I just wondered if anybody has any reasonable estimate how many >end-users actually employ fair-queueing AQMs with active ECN-marking >for ingress traffic @home? I am trying to understand whether L4S >approach to simply declare these as insignificant in number is >justifiable? >> >> If more than 0.01% of HGWs did this I'd be extremely surprised. > >My observation is that the number is very small, even devices with SQM >services, rarely see them enabled, and when they are, are set to >sub-optimal values. >I see Sebastian doing a valiant, even heroic effort at addressing >technical users questions on forums, but even those users seem confused >at times. > >> >>> I know in openwrt with sqm that is the default, but I have no idea >about >> >> To configure ingress shaping you actually have to know the speed and >configure it. It's not the default. Also, it's useless if the transport >network queues the packets at lower rate than at what you receive it. >When I used my DOCSIS connection it routinely forwarded packets at >lower rates than what I bought (and had configured the ingress shaper >for). > >As noted in other responses, the actual throughput needs to be measured >and then monitored to ensure the ingress shaping is aligned with >current capacity of the link. And not just the HGW to BNG, but just as >importantly, account for any constraints in backhaul from the BNG. Sure, but for the most part I have been limited either by the access link/BNG-shaper or limited peering/transit between my ISP and specific target servers, and in the second case I would hate to slow everything to a crawl just because my ISP and say YouTube try to Duke out who should pay for access, content to eyeballs or vice versa... > >> >>> the number of devices that actually use sqm in the field; @Jonathan: >does evenroute have numbers you are willing to share, like total >numbers or % of iqrouters with ecn-marking ingress routing active? > >@Sebastian, 100% of IQrouters running firmware 3.x (which uses Cake as >the default AQM) respect/use ECN. This has been shipping since >September, 2018. All existing v2 IQrouters (first ship January 2017) >may upgrade to 3x (user initiated, but one-click). >As for split, 70% of deployed IQrouters are doing ECN today. Excellent, thanks so most of them >As for count, well, that’s private. I had a hunch, that would be the case ;) . Understandable, though. >But the good new is we have ISP customers >rolling them out at a good clip. >Turns out that having a sane traffic manager at the HGW on every node >of a DSLAM is very good for the DSLAM, the backhaul and the actual >users, who quit screaming at the ISP ;-) Oh, nice, I fully agree upstream AQM is a case where the goals and incentives for end-users and ISPs seem well aligned. > >> >> ISP networks typically looks like this in the ISP->HGW direction: >> >> BNG->L2->L2->HGW >> >> This is the same regardless if it's DSL, DOCSIS, FTTH/PON or >whatever. So shaping is done egress on BNG and it tries to send at >lower rate than any of the L2 devices. Generally there is no ingress >shaping of any kind on the HGW, it doesn't even know what speed the >subscription is. >> >> -- >> Mikael Abrahamssonemail: swm...@swm.pp.se >> ___ >> Bloat mailing list >> Bloat@lists.bufferbloat.net >> https://lists.bufferbloat.net/listinfo/bloat -- Sent from my Android device with K-9 Mail. Please excuse my brevity. ___ Bloat mailing list Bloat@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/bloat
Re: [Bloat] number of home routers with ingress AQM
> On 2 Apr, 2019, at 5:14 pm, Jonathan Foulkes wrote: > > But the good new is we have ISP customers rolling them out at a good clip. > Turns out that having a sane traffic manager at the HGW on every node of a > DSLAM is very good for the DSLAM, the backhaul and the actual users, who quit > screaming at the ISP ;-) This is very encouraging to hear, not least that there are measurable technical benefits to the ISP as well as (more obviously) to the immediate user. How big are your marketing plans? :-) - Jonathan Morton ___ Bloat mailing list Bloat@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/bloat
Re: [Bloat] number of home routers with ingress AQM
> On Apr 2, 2019, at 16:14, Jonathan Foulkes wrote: > > Hi Ryan, > > See below > >> On Apr 2, 2019, at 9:33 AM, Ryan Mounce wrote: >> >> On Tue, 2 Apr 2019 at 23:35, Mikael Abrahamsson wrote: >>> >>> I've read rumours about some ISPs implementing interaction with the VDSL >>> DSLAM where there is an estimation of the current link-speed for each >>> individual customer and then it tries to set the BNG egress shaper >>> appropriately. >> >> NBN here in Australia do this for their wholesale FTTN/B VDSL2 >> product, injecting the downstream and upstream sync speed into >> PPPoE/DHCP requests. This still depends on the retail service >> providers to make use of these attributes from their BNG to configure >> the shaper. > > Very interesting, and extremely useful that they surface the sync rate in the > PPPoE/DHCP exchanges. How are those externalized in something like OpenWRT at > this time? > > FYI- The IQrouter supports the notion of a tightly coupled modem from which > current sync rates (as well as other SNR / Atten stats) can be dynamically > queried, which in turn are used in the AQM settings computations. Works > wonders on flaky DSL lines that vary sync rate with time of day and weather. > If that VDSL2 sync rate relayed is exposed in the system somehow, it would > make dynamic adjustments possible. IMHO the ISP should be mandated make machine readable dynamically updated information about a links limits available. Grossrate, unavoidable overhead and potentially additional encodings like (ATM's 48/53, or PTM's 64/65). With BNG-shapers/policers the synch-rate alone does not help except for allowing an estimate of the upper bound. Best Regards Sebastian > > >> >>> >>> I am very happy about my FTTH solution I have now since from what I can >>> see the L2 network is almost never a limiting factor (much better than my >>> DOCSIS connection) so my bidirectional SQM with CAKE seems to work very >>> well. >>> >>> Still, the HGW can never solve these problems properly, the egress shaping >>> in the BNG needs to do a proper job. From what I have been told, there has >>> been improvements here in the past few years. >>> >>> What I am more worried about is the egress shaping from the HGW. I talked >>> to several vendors at BBF and they talked about ingress policing being >>> commonly used on the BNG. This means no ingress shaping at all (just >>> packet drops if they exceed the configured rate). I don't know about >>> buffering on the HGW though and how the policed rate is set compared to >>> the L2 rate the HGW can send via. >> >> NBN is an example again. Their documented behaviour is to police >> traffic in both directions. Most ISPs then shape in the downstream - >> and it's up to a tightly managed (TR-069 ?) ISP HGW or a diligent end >> user to shape traffic in the upstream. Many - probably even most users >> have no shaping whatsoever in the upstream, only a policer. >> >> And then there is their new FTTdp product, where it is not currently >> possible to determine the real VDSL2 sync speed. If there's a drop of >> rain it will resync at a lower speed in the upstream, and then >> everything ends up queuing inside the supplied modem… > > Oh, so they regressed from their other offering, not exposing sync rates? > > BTW- this hole notion of the BGM relaying the provisioned or current sync > rates should be a mandated requirement, as it has huge benefits for AQMs. Not > that it can be relied on 100%, but at least it makes a good starting point. > > >> ___ >> Bloat mailing list >> Bloat@lists.bufferbloat.net >> https://lists.bufferbloat.net/listinfo/bloat > > ___ > Bloat mailing list > Bloat@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/bloat ___ Bloat mailing list Bloat@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/bloat
Re: [Bloat] number of home routers with ingress AQM
Hi Ryan, See below > On Apr 2, 2019, at 9:33 AM, Ryan Mounce wrote: > > On Tue, 2 Apr 2019 at 23:35, Mikael Abrahamsson wrote: >> >> I've read rumours about some ISPs implementing interaction with the VDSL >> DSLAM where there is an estimation of the current link-speed for each >> individual customer and then it tries to set the BNG egress shaper >> appropriately. > > NBN here in Australia do this for their wholesale FTTN/B VDSL2 > product, injecting the downstream and upstream sync speed into > PPPoE/DHCP requests. This still depends on the retail service > providers to make use of these attributes from their BNG to configure > the shaper. Very interesting, and extremely useful that they surface the sync rate in the PPPoE/DHCP exchanges. How are those externalized in something like OpenWRT at this time? FYI- The IQrouter supports the notion of a tightly coupled modem from which current sync rates (as well as other SNR / Atten stats) can be dynamically queried, which in turn are used in the AQM settings computations. Works wonders on flaky DSL lines that vary sync rate with time of day and weather. If that VDSL2 sync rate relayed is exposed in the system somehow, it would make dynamic adjustments possible. > >> >> I am very happy about my FTTH solution I have now since from what I can >> see the L2 network is almost never a limiting factor (much better than my >> DOCSIS connection) so my bidirectional SQM with CAKE seems to work very >> well. >> >> Still, the HGW can never solve these problems properly, the egress shaping >> in the BNG needs to do a proper job. From what I have been told, there has >> been improvements here in the past few years. >> >> What I am more worried about is the egress shaping from the HGW. I talked >> to several vendors at BBF and they talked about ingress policing being >> commonly used on the BNG. This means no ingress shaping at all (just >> packet drops if they exceed the configured rate). I don't know about >> buffering on the HGW though and how the policed rate is set compared to >> the L2 rate the HGW can send via. > > NBN is an example again. Their documented behaviour is to police > traffic in both directions. Most ISPs then shape in the downstream - > and it's up to a tightly managed (TR-069 ?) ISP HGW or a diligent end > user to shape traffic in the upstream. Many - probably even most users > have no shaping whatsoever in the upstream, only a policer. > > And then there is their new FTTdp product, where it is not currently > possible to determine the real VDSL2 sync speed. If there's a drop of > rain it will resync at a lower speed in the upstream, and then > everything ends up queuing inside the supplied modem… Oh, so they regressed from their other offering, not exposing sync rates? BTW- this hole notion of the BGM relaying the provisioned or current sync rates should be a mandated requirement, as it has huge benefits for AQMs. Not that it can be relied on 100%, but at least it makes a good starting point. > ___ > Bloat mailing list > Bloat@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/bloat ___ Bloat mailing list Bloat@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/bloat
Re: [Bloat] number of home routers with ingress AQM
Responses below > On Apr 2, 2019, at 8:10 AM, Mikael Abrahamsson wrote: > > On Tue, 2 Apr 2019, Sebastian Moeller wrote: > >> I just wondered if anybody has any reasonable estimate how many end-users >> actually employ fair-queueing AQMs with active ECN-marking for ingress >> traffic @home? I am trying to understand whether L4S approach to simply >> declare these as insignificant in number is justifiable? > > If more than 0.01% of HGWs did this I'd be extremely surprised. My observation is that the number is very small, even devices with SQM services, rarely see them enabled, and when they are, are set to sub-optimal values. I see Sebastian doing a valiant, even heroic effort at addressing technical users questions on forums, but even those users seem confused at times. > >> I know in openwrt with sqm that is the default, but I have no idea about > > To configure ingress shaping you actually have to know the speed and > configure it. It's not the default. Also, it's useless if the transport > network queues the packets at lower rate than at what you receive it. When I > used my DOCSIS connection it routinely forwarded packets at lower rates than > what I bought (and had configured the ingress shaper for). As noted in other responses, the actual throughput needs to be measured and then monitored to ensure the ingress shaping is aligned with current capacity of the link. And not just the HGW to BNG, but just as importantly, account for any constraints in backhaul from the BNG. > >> the number of devices that actually use sqm in the field; @Jonathan: does >> evenroute have numbers you are willing to share, like total numbers or % of >> iqrouters with ecn-marking ingress routing active? @Sebastian, 100% of IQrouters running firmware 3.x (which uses Cake as the default AQM) respect/use ECN. This has been shipping since September, 2018. All existing v2 IQrouters (first ship January 2017) may upgrade to 3x (user initiated, but one-click). As for split, 70% of deployed IQrouters are doing ECN today. As for count, well, that’s private. But the good new is we have ISP customers rolling them out at a good clip. Turns out that having a sane traffic manager at the HGW on every node of a DSLAM is very good for the DSLAM, the backhaul and the actual users, who quit screaming at the ISP ;-) > > ISP networks typically looks like this in the ISP->HGW direction: > > BNG->L2->L2->HGW > > This is the same regardless if it's DSL, DOCSIS, FTTH/PON or whatever. So > shaping is done egress on BNG and it tries to send at lower rate than any of > the L2 devices. Generally there is no ingress shaping of any kind on the HGW, > it doesn't even know what speed the subscription is. > > -- > Mikael Abrahamssonemail: swm...@swm.pp.se > ___ > Bloat mailing list > Bloat@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/bloat ___ Bloat mailing list Bloat@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/bloat
Re: [Bloat] number of home routers with ingress AQM
On Wed, 3 Apr 2019, Ryan Mounce wrote: On Wed, 3 Apr 2019 at 00:04, Mikael Abrahamsson wrote: What you described is probably on 95% or more of egress shaping on the BNG and on egress shaping on HGWs in the field. How many of these single queue deployments actually have ECN marking enabled? This is the really dangerous case with L4S. I don't have numbers, but from looking at the platforms I have available to me, very few do ECN-marking. Considering that Apple has been beating the ECN-enablement drum for a few years now, there might actually be ECN-marking deployment soon. I'm in process of trying to figure out support for ingress/egress shaping and ECN marking on the most common BNG platforms, we'll see what that will result in. -- Mikael Abrahamssonemail: swm...@swm.pp.se ___ Bloat mailing list Bloat@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/bloat
Re: [Bloat] number of home routers with ingress AQM
On Wed, 3 Apr 2019 at 00:04, Mikael Abrahamsson wrote: > > What you described is probably on 95% or more of egress shaping on the BNG > and on egress shaping on HGWs in the field. How many of these single queue deployments actually have ECN marking enabled? This is the really dangerous case with L4S. ___ Bloat mailing list Bloat@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/bloat
Re: [Bloat] number of home routers with ingress AQM
> On Apr 2, 2019, at 15:15, Ryan Mounce wrote: > > On Tue, 2 Apr 2019 at 22:08, Sebastian Moeller wrote: >> >> I just wondered if anybody has any reasonable estimate how many end-users >> actually employ fair-queueing AQMs with active ECN-marking for ingress >> traffic @home? I am trying to understand whether L4S approach to simply >> declare these as insignificant in number is justifiable? > > L4S people are concerned by RFC 3168 / "classic" ECN bottlenecks > *without* fq. I know, but I believe that they misunderstand the issues resulting from post-bottleneck shaping, like ingress shaping on the remote side of the true bottleneck. The idea seems that sending at too high a rate is unproblematic if the AQM can simply queue up these packets and delay them accordingly. But in the ingress shaper case these packets already traversed the bottleneck and aone has payed the bandwidth price to hoist them to the home, delaying or even dropping on the AQM side will not magically get the time back the packets took traversing the link. Why do I care, because that ingress shaping setup is what I use at home, and I have zero confidence that ISPs will come up with a solution to my latency desires that I am going to be happy with... And what I see from the L4S mixes light with a lot of shadows. > I don't think there would be any such ingress shapers > configured on home gateways. Certainly not by anyone on this list... > anyone running non-fq codel or flowblind cake for ingress shaping? As stated above, I believe fq to not be a reliable safety valve for the ingress shaping case. Best Regards Sebastian > > -Ryan ___ Bloat mailing list Bloat@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/bloat
Re: [Bloat] number of home routers with ingress AQM
On Tue, 2 Apr 2019, Ryan Mounce wrote: L4S people are concerned by RFC 3168 / "classic" ECN bottlenecks *without* fq. I don't think there would be any such ingress shapers configured on home gateways. Certainly not by anyone on this list... anyone running non-fq codel or flowblind cake for ingress shaping? What you described is probably on 95% or more of egress shaping on the BNG and on egress shaping on HGWs in the field. -- Mikael Abrahamssonemail: swm...@swm.pp.se ___ Bloat mailing list Bloat@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/bloat
Re: [Bloat] number of home routers with ingress AQM
On Tue, 2 Apr 2019 at 23:35, Mikael Abrahamsson wrote: > > I've read rumours about some ISPs implementing interaction with the VDSL > DSLAM where there is an estimation of the current link-speed for each > individual customer and then it tries to set the BNG egress shaper > appropriately. NBN here in Australia do this for their wholesale FTTN/B VDSL2 product, injecting the downstream and upstream sync speed into PPPoE/DHCP requests. This still depends on the retail service providers to make use of these attributes from their BNG to configure the shaper. > > I am very happy about my FTTH solution I have now since from what I can > see the L2 network is almost never a limiting factor (much better than my > DOCSIS connection) so my bidirectional SQM with CAKE seems to work very > well. > > Still, the HGW can never solve these problems properly, the egress shaping > in the BNG needs to do a proper job. From what I have been told, there has > been improvements here in the past few years. > > What I am more worried about is the egress shaping from the HGW. I talked > to several vendors at BBF and they talked about ingress policing being > commonly used on the BNG. This means no ingress shaping at all (just > packet drops if they exceed the configured rate). I don't know about > buffering on the HGW though and how the policed rate is set compared to > the L2 rate the HGW can send via. NBN is an example again. Their documented behaviour is to police traffic in both directions. Most ISPs then shape in the downstream - and it's up to a tightly managed (TR-069 ?) ISP HGW or a diligent end user to shape traffic in the upstream. Many - probably even most users have no shaping whatsoever in the upstream, only a policer. And then there is their new FTTdp product, where it is not currently possible to determine the real VDSL2 sync speed. If there's a drop of rain it will resync at a lower speed in the upstream, and then everything ends up queuing inside the supplied modem... ___ Bloat mailing list Bloat@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/bloat
Re: [Bloat] number of home routers with ingress AQM
Hi Mikael, > On Apr 2, 2019, at 15:04, Mikael Abrahamsson wrote: > > On Tue, 2 Apr 2019, Sebastian Moeller wrote: > >> See above how Deutsche Telekom deals with that issue, at least in the >> German market. > > I've read rumours about some ISPs implementing interaction with the VDSL > DSLAM where there is an estimation of the current link-speed for each > individual customer and then it tries to set the BNG egress shaper > appropriately. I believe this is what Deutsche Telekom actually does already. > > I am very happy about my FTTH solution I have now since from what I can see > the L2 network is almost never a limiting factor (much better than my DOCSIS > connection) so my bidirectional SQM with CAKE seems to work very well. I envy you ;) that said, I have little issue with my VDSL2 link, but at 50/10 it is hardly pushing the limit. DOCSIS with its often huge segments is a mixed bag, I have head od segments with 1000 customers and no issue even saturating a DOCSIS 3.1 1000/50 plan, and also people with severe issues with more modes 100/20 plans. > > Still, the HGW can never solve these problems properly, Assuming fixed bandwidth, it can do a pretty good approximation of properly though ;) > the egress shaping in the BNG needs to do a proper job. I agree, but then my wishlist includes flow-queueing and then reality intervenes, and I keep having to do this on my side as BNGs do not offer fq for customer bandwidth shaping/policing, and might never do. > From what I have been told, there has been improvements here in the past few > years. I agree, when I started with Deutsche Telekom latency under saturating load spikes (in ICMP probes) were above 1000ms in 2015, but now with the Juniper BNG shaper solution I saw around 300ms (I switched ISP, but the cap is still around 300ms). IMHO this is much better, but also far away from good enough, so I keep shaping on my end to keep latency acceptable (with a 50/10 link saturating loads are common enough to justify the time spent for configuring the home ingress shaper IMHO). > > What I am more worried about is the egress shaping from the HGW. If you ask me that is going to come, all shared media links (docsis, G-PON, ...) already need this to keep customer modems from shouting over each other. > I talked to several vendors at BBF and they talked about ingress policing > being commonly used on the BNG. This means no ingress shaping at all (just > packet drops if they exceed the configured rate). I wonder how that rhymes with the 300-1000ms added latency I see under load, assuming the BNG-limiter to be the bottleneck. > I don't know about buffering on the HGW though and how the policed rate is > set compared to the L2 rate the HGW can send via. Typical DSL modem-routers and DOCSIS modems (presumably < 3.1) often show pretty manly buffering (that is to say probably a bit too much with too little brains attached ;) ). I note that the L4S-project already identified CPEs as the next target to get upstream queueing tackled, I see no chance for doing that effectively without getting the ISPs on board. I have no idea how well the heavy cable labs involvement will sit with the old telco incumbents and whether any non-ITU standard will fly. But let's see... Best Regards Sebastian > > -- > Mikael Abrahamssonemail: swm...@swm.pp.se ___ Bloat mailing list Bloat@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/bloat
Re: [Bloat] number of home routers with ingress AQM
On Tue, 2 Apr 2019 at 22:08, Sebastian Moeller wrote: > > I just wondered if anybody has any reasonable estimate how many end-users > actually employ fair-queueing AQMs with active ECN-marking for ingress > traffic @home? I am trying to understand whether L4S approach to simply > declare these as insignificant in number is justifiable? L4S people are concerned by RFC 3168 / "classic" ECN bottlenecks *without* fq. I don't think there would be any such ingress shapers configured on home gateways. Certainly not by anyone on this list... anyone running non-fq codel or flowblind cake for ingress shaping? -Ryan ___ Bloat mailing list Bloat@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/bloat
Re: [Bloat] number of home routers with ingress AQM
On Tue, 2 Apr 2019, Sebastian Moeller wrote: See above how Deutsche Telekom deals with that issue, at least in the German market. I've read rumours about some ISPs implementing interaction with the VDSL DSLAM where there is an estimation of the current link-speed for each individual customer and then it tries to set the BNG egress shaper appropriately. I am very happy about my FTTH solution I have now since from what I can see the L2 network is almost never a limiting factor (much better than my DOCSIS connection) so my bidirectional SQM with CAKE seems to work very well. Still, the HGW can never solve these problems properly, the egress shaping in the BNG needs to do a proper job. From what I have been told, there has been improvements here in the past few years. What I am more worried about is the egress shaping from the HGW. I talked to several vendors at BBF and they talked about ingress policing being commonly used on the BNG. This means no ingress shaping at all (just packet drops if they exceed the configured rate). I don't know about buffering on the HGW though and how the policed rate is set compared to the L2 rate the HGW can send via. -- Mikael Abrahamssonemail: swm...@swm.pp.se ___ Bloat mailing list Bloat@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/bloat
Re: [Bloat] number of home routers with ingress AQM
HI Mikael, > On Apr 2, 2019, at 14:10, Mikael Abrahamsson wrote: > > On Tue, 2 Apr 2019, Sebastian Moeller wrote: > >> I just wondered if anybody has any reasonable estimate how many end-users >> actually employ fair-queueing AQMs with active ECN-marking for ingress >> traffic @home? I am trying to understand whether L4S approach to simply >> declare these as insignificant in number is justifiable? > > If more than 0.01% of HGWs did this I'd be extremely surprised. Me too, but still I want to see numbers, plus those are likely those end-users that could be won-over by L4S as theses obvipusly would be receptive for the "low-latency" for all pitch of the L4S project. > >> I know in openwrt with sqm that is the default, but I have no idea about > > To configure ingress shaping you actually have to know the speed and > configure it. As a first approximation knowing your speed just requires to run a single (decent) speedtest, as the reported TCP/IPv[4|6]/HTTP[s]-Goodput numbers can be feed directly into any shaper (shapers typically desire gross rates, but since net < gross net rates will just do fine if the aim is to keep latency under control). Deutsche Telekom actually sends crude estimates of the achievable goodput via PPPoE-ACK messages that AVM's FritzBox routers (quite popular in Germany for those unhappy with the ISP supplied gear) evaluate to configure up- and downstream shaping. Which due to lack of proper documentation of the PPPoE-ACK message is a bit of a hit and miss job. > It's not the default. Also, it's useless if the transport network queues the > packets at lower rate than at what you receive it. Sure, that is not a solution for congestion outside of the access link, but that is an impossible problem to solve. But getting the access link debloated often is an important first step even if it does not fix the whole internet ;) > When I used my DOCSIS connection it routinely forwarded packets at lower > rates than what I bought (and had configured the ingress shaper for). Gotta love oversubscribed segments ;) Gargoyle firmware has an adaptive method that tries to tackl;e that poroblem, and eventouter, IIUC, actually runs multiple bandwidth measures per day to track the actually achievable bandwidth. But these are just work-arounds. Now if your cable ISP acctually had used a competent QoS system you might not have had to suffer bad latency with bad bandwidth. But I believe cablelabs has identified that as a problem and is actively working on it (and sooner or later their flow-queueing in disguise will do the right thing ;) ). Flow-queueing in disguise, I hear nobody ask, well sure they mandate a DualPI2 solution but also mandate in Annex P Queue Protection Algorithm (Normative) of CM-SP-MULPIv3.1-I17-190121 something that must come close to flow queuueing in cost, to cite from the source "The algorithm maintains per-Microflow state that holds a “queuing score” representing how much each Microflow was responsible for recent queuing." I asked about the cost but have not heard an answer yet. > >> the number of devices that actually use sqm in the field; @Jonathan: does >> evenroute have numbers you are willing to share, like total numbers or % of >> iqrouters with ecn-marking ingress routing active? > > ISP networks typically looks like this in the ISP->HGW direction: > > BNG->L2->L2->HGW > > This is the same regardless if it's DSL, DOCSIS, FTTH/PON or whatever. So > shaping is done egress on BNG and it tries to send at lower rate than any of > the L2 devices. Yes, I know, and there are good reasons for doing it that way. > Generally there is no ingress shaping of any kind on the HGW, That is part of my question > it doesn't even know what speed the subscription is. See above how Deutsche Telekom deals with that issue, at least in the German market. Thanks for your insights, much appreciated Best Regards Sebastian > > -- > Mikael Abrahamssonemail: swm...@swm.pp.se ___ Bloat mailing list Bloat@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/bloat
Re: [Bloat] number of home routers with ingress AQM
On Tue, 2 Apr 2019, Sebastian Moeller wrote: I just wondered if anybody has any reasonable estimate how many end-users actually employ fair-queueing AQMs with active ECN-marking for ingress traffic @home? I am trying to understand whether L4S approach to simply declare these as insignificant in number is justifiable? If more than 0.01% of HGWs did this I'd be extremely surprised. I know in openwrt with sqm that is the default, but I have no idea about To configure ingress shaping you actually have to know the speed and configure it. It's not the default. Also, it's useless if the transport network queues the packets at lower rate than at what you receive it. When I used my DOCSIS connection it routinely forwarded packets at lower rates than what I bought (and had configured the ingress shaper for). the number of devices that actually use sqm in the field; @Jonathan: does evenroute have numbers you are willing to share, like total numbers or % of iqrouters with ecn-marking ingress routing active? ISP networks typically looks like this in the ISP->HGW direction: BNG->L2->L2->HGW This is the same regardless if it's DSL, DOCSIS, FTTH/PON or whatever. So shaping is done egress on BNG and it tries to send at lower rate than any of the L2 devices. Generally there is no ingress shaping of any kind on the HGW, it doesn't even know what speed the subscription is. -- Mikael Abrahamssonemail: swm...@swm.pp.se ___ Bloat mailing list Bloat@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/bloat
[Bloat] number of home routers with ingress AQM
Dear all, I just wondered if anybody has any reasonable estimate how many end-users actually employ fair-queueing AQMs with active ECN-marking for ingress traffic @home? I am trying to understand whether L4S approach to simply declare these as insignificant in number is justifiable? I know in openwrt with sqm that is the default, but I have no idea about the number of devices that actually use sqm in the field; @Jonathan: does evenroute have numbers you are willing to share, like total numbers or % of iqrouters with ecn-marking ingress routing active? Best Regards Sebastian ___ Bloat mailing list Bloat@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/bloat
