Re: [Bloat] [Starlink] Of interest: Comcast AQM Paper
Jonathan Bennett wrote: > Isn't that what CZnic has done with the Turris router? > > https://www.turris.com/en/mox/overview/ > I hadn't seen the Mox, that is clever. The downside is the price, and that > they're hard to get outside Europe. Yup. The MOX is very nice, but maybe a bit too modular... :-) (but: they had their reasons) I found it difficult to figure out where to put the antennae. CIRALabs got several dozen for the SecureHomeRouter project, but the pandemic kept us from getting them into alpha testing. I think we bought direct. -- ] Never tell me the odds! | ipv6 mesh networks [ ] Michael Richardson, Sandelman Software Works|IoT architect [ ] m...@sandelman.ca http://www.sandelman.ca/| ruby on rails[ signature.asc Description: PGP signature ___ Bloat mailing list Bloat@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/bloat
Re: [Bloat] [Starlink] Of interest: Comcast AQM Paper
Hi Jonathan, > On Aug 4, 2021, at 20:28, Jonathan Morton wrote: > > I firmly believe this is due to an I/O bottleneck in the SoC between > the network complex and the CPU complex, not due to any limitation of > the CPU itself. It stems from the reliance on accelerated forwarding > hardware to achieve full line-rate throughput. Even so, I'd much > rather have 40Mbps with Cake than 400Mbps with a dumb FIFO. (Heck, > 40Mbps would be a big upgrade from what I currently have.) I think > some of the newer Atheros chipsets are less constrained in this > respect. > > There are two reasonably good solutions to this problem in the hands > of the SoC vendors: > > 1: Relieve that I/O bottleneck, so that the CPU can handle packets at > full line rate. I assume this is not hugely complicated to implement, > and just requires a sufficient degree of will to select the right > option from the upstream fabless IP vendor's design library. > > 2: Implement good shaping, FQ, and AQM within the network complex. At > consumer broadband/LAN speeds, this shouldn't be too difficult (unlike > doing the same at 100+ Gbps), but it does require a significant amount > of hardware design and validation, and that tends to have long lead > times. > > There is a third solution in the hands of us mere mortals: > > 3: Leverage the Raspberry Pi ecosystem to build a CPE device that > meets our needs. This could be a Compute Module 4 (which has the > necessary I/O throughput) mounted on a custom PCB that provides > additional Ethernet ports and some reasonable Wifi AP. It could > alternatively be a standard Pi 4B with some USB Ethernet and Wifi > hardware plugged into it. Either will do the job withhout any > Ethernet bottlenecks, although the capabilities of USB Wifi dongles > are usually quite limited. Have a look at https://www.dfrobot.com/product-2242.html, which is a small carrier for the raspberry pi4 compute module that offers two gigabit ethernet ports, the one from the CM and an additional RTL8 one connected via the PCIe lanes. At $45 it is a bit pricy, but it sure is small and elegant. People on the OpenWrt Forum report traffic shaping at 1Gbps rates without overloading the CPUs (this needs minimal configuration for receive side packet steering, otherwise all network IRQ and qdisc processing sticks to CPU0, in which case shaping does not reach Gbps speeds, at least not for bi-directonal traffic). That still leaves the need for an AP and a switch (one might be able to "abuse" an AP for switch ports if in a pinch). Regards Sebastian > > - Jonathan Morton ___ Bloat mailing list Bloat@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/bloat
Re: [Bloat] [Starlink] Of interest: Comcast AQM Paper
On Wed, 4 Aug 2021, Jonathan Morton wrote: 3: Leverage the Raspberry Pi ecosystem to build a CPE device that meets our needs. This could be a Compute Module 4 (which has the necessary I/O throughput) mounted on a custom PCB that provides additional Ethernet ports and some reasonable Wifi AP. It could alternatively be a standard Pi 4B with some USB Ethernet and Wifi hardware plugged into it. Either will do the job withhout any Ethernet bottlenecks, although the capabilities of USB Wifi dongles are usually quite limited. I'd buy it, you could also have a carrier board with some sockets that can take the wifi modules that laptops use. David Lang ___ Bloat mailing list Bloat@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/bloat
Re: [Bloat] [Starlink] Of interest: Comcast AQM Paper
Isn't that what CZnic has done with the Turris router? https://www.turris.com/en/mox/overview/ On Wed, Aug 4, 2021 at 1:08 PM Jonathan Bennett < jonathanbenn...@hackaday.com> wrote: > > > >> >> >> The Compute Module 4 exposes the same integrated Ethernet port, and a >> PCIe lane in place of the USB 3 chipset (the latter being attached to >> the former in the standard Pi 4B). This obviously allows attaching at >> least one real GigE port (with a free choice of PCIe-based chipset) at >> full line rate, without the intermediate step of USB. I think it >> would be reasonable to include a small Ethernet switch downstream of >> this, matching the connectivity of typical CPE on the LAN side. If a >> PCIe switch is inserted, then a choice of Mini-PCIe Wifi cards can be >> installed, with cables running to the normal array of external >> antennae, sidestepping the problem of USB Wifi dongles. >> > > I would hype the heck out of a router-style carrier board. I'd even buy a > bunch myself, for that matter. I used to make good money by putting OpenWRT > on cheap routers for small businesses. > > I've had good success chatting with Eben about Hackaday articles. If this > were to become more than a pipe dream, we could reach out to him, to see if > the Pi Foundation had any interest in backing a Pi router carrier board > that beats bufferbloat. > > --Jonathan Bennett > ___ > Starlink mailing list > starl...@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/starlink > ___ Bloat mailing list Bloat@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/bloat
Re: [Bloat] [Starlink] Of interest: Comcast AQM Paper
On Wed, 4 Aug 2021 at 21:31, Juliusz Chroboczek wrote: > A Cortex-A53 SoC at 1GHz with correctly designed Ethernet (i.e. not the > Raspberry Pi) can push 1Gbit from userspace without breaking a sweat. That was true of the earlier Raspberry Pis (eg. the Pi 3 uses a brace of Cortex-A53s) which use Ethernet chipsets attached over USB 2, but the Pi 4B has a directly integrated Ethernet port and two of the external USB ports are USB 3, giving enough bandwidth to attach a second GigE port. We have tested this in practice, and got full line rate throughput through Cake (though the CPU usage went up fairly sharply after about halfway). The Compute Module 4 exposes the same integrated Ethernet port, and a PCIe lane in place of the USB 3 chipset (the latter being attached to the former in the standard Pi 4B). This obviously allows attaching at least one real GigE port (with a free choice of PCIe-based chipset) at full line rate, without the intermediate step of USB. I think it would be reasonable to include a small Ethernet switch downstream of this, matching the connectivity of typical CPE on the LAN side. If a PCIe switch is inserted, then a choice of Mini-PCIe Wifi cards can be installed, with cables running to the normal array of external antennae, sidestepping the problem of USB Wifi dongles. ___ Bloat mailing list Bloat@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/bloat
Re: [Bloat] [Starlink] Of interest: Comcast AQM Paper
Hi Mikael, > If it's not hw accelerated, it sucks. Fortunately, that hasn't been true in a long time. Two data points. The WND3700v2/WNDR3800, which is now over ten years old, can easily forward 400Mbit/s NATed IPv4 (max-sized packets) in software. To be fair, it can saturate 1Gbit/s with hardware offload. A Cortex-A53 SoC at 1GHz with correctly designed Ethernet (i.e. not the Raspberry Pi) can push 1Gbit from userspace without breaking a sweat. I agree with you that hardware acceleration has a number of advantages, but given the speed of modern embedded chips, there is no longer much reason to give up the flexibility and ease of development of pure software solutions just to save a couple hundred mW, at least not until we get 10Gbit/s CPEs. -- Juliusz ___ Bloat mailing list Bloat@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/bloat
Re: [Bloat] [Starlink] Of interest: Comcast AQM Paper
I firmly believe this is due to an I/O bottleneck in the SoC between the network complex and the CPU complex, not due to any limitation of the CPU itself. It stems from the reliance on accelerated forwarding hardware to achieve full line-rate throughput. Even so, I'd much rather have 40Mbps with Cake than 400Mbps with a dumb FIFO. (Heck, 40Mbps would be a big upgrade from what I currently have.) I think some of the newer Atheros chipsets are less constrained in this respect. There are two reasonably good solutions to this problem in the hands of the SoC vendors: 1: Relieve that I/O bottleneck, so that the CPU can handle packets at full line rate. I assume this is not hugely complicated to implement, and just requires a sufficient degree of will to select the right option from the upstream fabless IP vendor's design library. 2: Implement good shaping, FQ, and AQM within the network complex. At consumer broadband/LAN speeds, this shouldn't be too difficult (unlike doing the same at 100+ Gbps), but it does require a significant amount of hardware design and validation, and that tends to have long lead times. There is a third solution in the hands of us mere mortals: 3: Leverage the Raspberry Pi ecosystem to build a CPE device that meets our needs. This could be a Compute Module 4 (which has the necessary I/O throughput) mounted on a custom PCB that provides additional Ethernet ports and some reasonable Wifi AP. It could alternatively be a standard Pi 4B with some USB Ethernet and Wifi hardware plugged into it. Either will do the job withhout any Ethernet bottlenecks, although the capabilities of USB Wifi dongles are usually quite limited. - Jonathan Morton ___ Bloat mailing list Bloat@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/bloat
Re: [Bloat] [Starlink] Of interest: Comcast AQM Paper
Hi Mikael, > On Aug 4, 2021, at 15:06, Mikael Abrahamsson wrote: > > On Wed, 4 Aug 2021, Sebastian Moeller wrote: > >> I guess the point is AQM is not really that expensive, even FQ AQM, traffic >> shaping however is expensive. But for wifi shaping is not required so AQM >> became feasible. > > My point is that CPU based forwarding has very bad performance on some > platforms, regardless if you're doing shaping, AQM or none of them (FIFO). > > If it's not hw accelerated, it sucks. > > When I did tests on MT7621 it did ~100 meg/s without flow-offload, and full > gig with it. Reminds me of the joke of the gent sitting in a pub boasting the size of his property being so immense it takes a full week to ride around the perimeter... To which another guest responds wryly, he used to have a horse that slow in the past as well Point being a number of commercially sold CPE are not fit for the job of doing anything at Gbps link rates... sure accelerators can be nice, but they all come at the expense of reduced generality... Give that a raspberry Pi 4B has no trouble doing traffic shaping a 1Gbps (plus firewalling, NAT, PPPoE, ... and all of this in "software" with full generality) there is little excuse of still selling deploying CPE that were a decent fit for DSL link rates... But I live in a rich country where ISP take 2-5 EUR rent per month for CPE, so money is not a reasonable excuse for an ISP over here to cut corners Regards Sebastian > > -- > Mikael Abrahamssonemail: swm...@swm.pp.se ___ Bloat mailing list Bloat@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/bloat
Re: [Bloat] [Starlink] Of interest: Comcast AQM Paper
On Wed, 4 Aug 2021, Sebastian Moeller wrote: I guess the point is AQM is not really that expensive, even FQ AQM, traffic shaping however is expensive. But for wifi shaping is not required so AQM became feasible. My point is that CPU based forwarding has very bad performance on some platforms, regardless if you're doing shaping, AQM or none of them (FIFO). If it's not hw accelerated, it sucks. When I did tests on MT7621 it did ~100 meg/s without flow-offload, and full gig with it. -- Mikael Abrahamssonemail: swm...@swm.pp.se ___ Bloat mailing list Bloat@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/bloat
Re: [Bloat] [Starlink] Of interest: Comcast AQM Paper
I guess the point is AQM is not really that expensive, even FQ AQM, traffic shaping however is expensive. But for wifi shaping is not required so AQM became feasible. Regards Sebastian On 4 August 2021 14:46:30 CEST, Mikael Abrahamsson via Bloat wrote: >On Wed, 4 Aug 2021, Jonathan Morton wrote: > >> Linux-based CPE devices have AQM functionality integrated into the Wifi >> stack. The AQM itself operates at layer 3, but the Linux Wifi stack >> implementation uses information from layers 2 and 4 to improve >> scheduling decisions, eg. airtime-fairness and flow-isolation (FQ). This >> works best on soft-MAC Wifi hardware, such as ath9k/10k and MT76, where >> this information is most readily available to software. In principle it >> could also be implemented in the MAC, but I don't know of any vendor >> that's done that yet. > >Does this work also with flowoffload enabled, or is that not accelerated >on for instance MT76? I'm surprised since MT76 can barely do 100 meg/s of >large packets using only CPU? > >-- >Mikael Abrahamssonemail: swm...@swm.pp.se >___ >Bloat mailing list >Bloat@lists.bufferbloat.net >https://lists.bufferbloat.net/listinfo/bloat -- Sent from my Android device with K-9 Mail. Please excuse my brevity.___ Bloat mailing list Bloat@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/bloat
Re: [Bloat] [Starlink] Of interest: Comcast AQM Paper
On Wed, 4 Aug 2021, Jonathan Morton wrote: Linux-based CPE devices have AQM functionality integrated into the Wifi stack. The AQM itself operates at layer 3, but the Linux Wifi stack implementation uses information from layers 2 and 4 to improve scheduling decisions, eg. airtime-fairness and flow-isolation (FQ). This works best on soft-MAC Wifi hardware, such as ath9k/10k and MT76, where this information is most readily available to software. In principle it could also be implemented in the MAC, but I don't know of any vendor that's done that yet. Does this work also with flowoffload enabled, or is that not accelerated on for instance MT76? I'm surprised since MT76 can barely do 100 meg/s of large packets using only CPU? -- Mikael Abrahamssonemail: swm...@swm.pp.se ___ Bloat mailing list Bloat@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/bloat
Re: [Bloat] [Starlink] Of interest: Comcast AQM Paper
> I assume by WiFi what is really meant is devices that have at least one WiFi > (layer 1/layer 2) interface. While there are queues in the MAC sublayer, > there is really no queue management functionality ... yet ... AFAIK. I know > IEEE P802.11bd in conjunction w/ IEEE 1609 is working on implementing a few > rudimentary queue mgmt functions. > > That said, seems any AQM in such devices would more than likely be in layer 3 > and above. Linux-based CPE devices have AQM functionality integrated into the Wifi stack. The AQM itself operates at layer 3, but the Linux Wifi stack implementation uses information from layers 2 and 4 to improve scheduling decisions, eg. airtime-fairness and flow-isolation (FQ). This works best on soft-MAC Wifi hardware, such as ath9k/10k and MT76, where this information is most readily available to software. In principle it could also be implemented in the MAC, but I don't know of any vendor that's done that yet. - Jonathan Morton ___ Bloat mailing list Bloat@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/bloat
Re: [Bloat] [Starlink] Of interest: Comcast AQM Paper
I assume by WiFi what is really meant is devices that have at least one WiFi (layer 1/layer 2) interface. While there are queues in the MAC sublayer, there is really no queue management functionality ... yet ... AFAIK. I know IEEE P802.11bd in conjunction w/ IEEE 1609 is working on implementing a few rudimentary queue mgmt functions. That said, seems any AQM in such devices would more than likely be in layer 3 and above. RR -Original Message- From: Starlink [mailto:starlink-boun...@lists.bufferbloat.net] On Behalf Of Livingood, Jason Sent: Sunday, August 1, 2021 1:20 PM To: Simon Barber Cc: starl...@lists.bufferbloat.net; bloat Subject: Re: [Starlink] [Bloat] Of interest: Comcast AQM Paper WiFi is a different challenge as you know. In this case it varies depending on the radio chipset vendor and is on my list of things to work on... JL On 7/31/21, 13:50, "Simon Barber" wrote: Awesome to hear that you are turning this on both upstream and downstream. Do you know if the wifi stacks in your home routers also have AQM? Simon > On Jul 30, 2021, at 10:28 PM, Livingood, Jason via Bloat wrote: > > FYI that I will be presenting a lightning talk at the IRTF MAPRG meeting today at 17:30 ET (https://urldefense.com/v3/__https://datatracker.ietf.org/meeting/111/materials/agenda-111-maprg__;!!CQl3mcHX2A!XLFMYPw-gnJgHzz_1nF-N7dNeIeT4QD-5wQny8vdAfYE6bzHtVQD3-lqiQI9YwAncrZUew$ ). The talk links to a just-published paper at https://urldefense.com/v3/__https://arxiv.org/abs/2107.13968__;!!CQl3mcHX2A!XLFMYPw-gnJgHzz_1nF-N7dNeIeT4QD-5wQny8vdAfYE6bzHtVQD3-lqiQI9YwCePfNyng$ (click PDF link in upper right of page) that will likely be of interest to these two lists. > > High-level: turning on AQM in the cable modem (upstream queue) took working latency from around 250 ms to between 15-30 ms, which is actually kind of cool. ;-) AQM is turned on in all of our CMTSes (downstream queue) and in DOCSIS 3.1 modems (upstream queue). > > Have a nice weekend, > Jason > > ___ > Bloat mailing list > Bloat@lists.bufferbloat.net > https://urldefense.com/v3/__https://lists.bufferbloat.net/listinfo/bloat__;!!CQl3mcHX2A!XLFMYPw-gnJgHzz_1nF-N7dNeIeT4QD-5wQny8vdAfYE6bzHtVQD3-lqiQI9YwCIw2ZGww$ ___ Starlink mailing list starl...@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/starlink ___ Bloat mailing list Bloat@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/bloat
