Hi Erik, I really appreciate that you are reaching out to the bufferbloat community for this real-life 5G mobile testing. Lets all help out Erik.
From your graphs, it does look like you are measuring latency under-load, e.g. while the curl download/upload is running. This is great as this is the first rule of bufferbloat measuring :-) (and Luca hinted to this) The Huawei policer/shaper sounds scary. And 1000 packets deep queue also sound like a recipe for bufferbloat. I would of-cause like to re-write the Huawei policer/shaper with the knowledge and techniques we know from our bufferbloat work in the Linux Kernel. (If only I knew someone that coded on 5G solutions that could implement this on their hardware solution, and provide a better product Cc. Carlo) Are you familiar with Toke's (cc) work/PhD on handling bufferbloat on wireless networks? (Hint: Airtime fairness) Solving bufferbloat in wireless networks require more than applying fq_codel on the bottleneck queue, it requires Airtime fairness. Doing scheduling based Clients use of Radio-time and transmit-opportunities (TXOP), instead of shaping based on bytes. (This is why it can (if you are very careful) make sense to "holding back packets a bit" to generate a packet aggregate that only consumes one TXOP). The culprit is that each Client/MobilePhone will be sending at different rates, and scheduling based on bytes, will cause a Client with a low rate to consume a too large part of the shared radio airtime. That basically sums up Toke's PhD ;-) -- Best regards, Jesper Dangaard Brouer MSc.CS, Principal Kernel Engineer at Red Hat LinkedIn: http://www.linkedin.com/in/brouer Cc. Marek due to his twitter post[1] and link to 5G-BBR blogpost[2]: [1] https://twitter.com/majek04/status/1329708548297732097 [2] https://blog.acolyer.org/2020/10/05/understanding-operational-5g/ On Thu, 19 Nov 2020 14:35:27 +0000 <erik.tarald...@telenor.com> wrote: > Hello Luca > > > The current PGW is a policer. What the next version will be, I'm not sure. > > > However on parts of the Huawei RAN the policing rate is set to be a > shaper speed on the eNodeB (radio antenna). 1000 packets deep. And > it not only shapes down to 30Mb, but tries to aggregate packets to > keep a level speed whenever using the radio interface. Meaning > holding back packets a bit to try and get to 30Mbit when sending in > bulk in case of less than 30Mbit user traffic. 30Mbit being an > example subscription speed. > > > We are rolling out a fix to turn of that Huawei shaper, but it is not > done nation wide yet. > > The test device is in a lab area, using close to, but not entirely > the same as production 5G setup from Ericsson. Here there should not > be any shapers involved in the downstream path here. There is > however a bloated buffer on the upstream path which we are working on > correcting. > > > The curl graphs are "time to complete a curl download of x file > size", using a apache webserver running bbr. > > > -Erik > > > ________________________________ > Fra: Luca Muscariello <muscarie...@ieee.org> > Sendt: 19. november 2020 14:32 > Til: Taraldsen Erik > Kopi: Jesper Dangaard Brouer; priyar...@google.com; bloat; Luca Muscariello > Emne: Re: [Bloat] BBR implementations, knobs to turn? > > Hi Erick, > > one question about the PGW: is it a policer or a shaper that you have > installed? > Also, have you tried to run a ping session before and in parallel to the curl > sessions? > > Luca > > > > On Thu, Nov 19, 2020 at 2:15 PM > <erik.tarald...@telenor.com<mailto:erik.tarald...@telenor.com>> wrote: > Update: > The 5G router was connected to a new base station. Now the limiting factor > of throughput is the policer on the PGW in mobile core, not the radio link > itself. The SIM card used is limited to 30Mbit/s. This scenario favours the > new server. I have attached graphs comparing radio link limited vs PGW > policer results, and a zoomed in graph of the policer > > > We have Huawei RAN and Ericsson RAN, rate limited and not rate limited > subscriptions, 4G and 5G access, and we are migrating to a new core with new > PGW (policer). Starting to be a bit of a matrix to set up tests for. > > > -Erik > > > ________________________________________ > Fra: Jesper Dangaard Brouer <bro...@redhat.com<mailto:bro...@redhat.com>> > Sendt: 17. november 2020 16:07 > Til: Taraldsen Erik; Priyaranjan Jha > Kopi: bro...@redhat.com<mailto:bro...@redhat.com>; > ncardw...@google.com<mailto:ncardw...@google.com>; > bloat@lists.bufferbloat.net<mailto:bloat@lists.bufferbloat.net> > Emne: Re: [Bloat] BBR implementations, knobs to turn? > > On Tue, 17 Nov 2020 10:05:24 +0000 > <erik.tarald...@telenor.com<mailto:erik.tarald...@telenor.com>> wrote: > > > Thank you for the response Neal > > Yes. And it is impressive how many highly qualified people are on the > bufferbloat list. > > > old_hw # uname -r > > 5.3.0-64-generic > > (Ubuntu 19.10 on xenon workstation, integrated network card, 1Gbit > > GPON access. Used as proof of concept from the lab at work) > > > > > > new_hw # uname -r > > 4.18.0-193.19.1.el8_2.x86_64 > > (Centos 8.2 on xenon rack server, discrete 10Gbit network card, > > 40Gbit server farm link (low utilization on link), intended as fully > > supported and run service. Not possible to have newer kernel and > > still get service agreement in my organization) > > Let me help out here. The CentOS/RHEL8 kernels have a huge amount of > backports. I've attached a patch/diff of net/ipv4/tcp_bbr.c changes > missing in RHEL8. > > It looks like these patches are missing in CentOS/RHEL8: > [1] https://git.kernel.org/torvalds/c/78dc70ebaa38aa3 > [2] https://git.kernel.org/torvalds/c/a87c83d5ee25cf7 > > Could missing patch [1] result in the issue Erik is seeing? > (It explicitly mentions improvements for WiFi...) > > -- > Best regards, > Jesper Dangaard Brouer > MSc.CS, Principal Kernel Engineer at Red Hat > LinkedIn: http://www.linkedin.com/in/brouer > _______________________________________________ > Bloat mailing list > Bloat@lists.bufferbloat.net<mailto:Bloat@lists.bufferbloat.net> > https://lists.bufferbloat.net/listinfo/bloat _______________________________________________ Bloat mailing list Bloat@lists.bufferbloat.net https://lists.bufferbloat.net/listinfo/bloat
