Re: [Bloat] A Transport Protocol's View of Starlink

[Stephen Hemminger via Bloat]

On Wed, 22 May 2024 06:16:17 -0700
Kenneth Porter via Bloat  wrote:

> This technical paper on Starlink by the chief scientist at APNIC crossed my 
> feed this week. [I thought I'd share it to the Starlink list here but my 
> application to join that list seems to have gotten stuck so I'll share it 
> here for now.]
> 
> 
> 
> >From the end of the paper:  
> 
> > While earlier TCP control protocols, such as Reno, have been observed to
> > perform poorly on Starlink connections, more recent TCP counterparts,
> > such as CUBIC, perform more efficiently. The major TCP feature that makes
> > these protocols viable in Starlink contexts is the use of Selective
> > Acknowledgement [11], that allows the TCP control algorithm to
> > distinguish between isolated packet loss and loss-inducing levels of
> > network congestion.
> >
> > TCP control protocols that attempt to detect the onset of network queue
> > formation can do so using end-to-end techniques by detecting changes in
> > end-to-end latency during intermittent periods of burst, such as BBR.
> > These protocols need to operate with a careful implementation of their
> > sensitivity to latency, as the highly unstable short-term latency seen on
> > Starlink connections, coupled with the 15-second coarse level latency
> > shifts have the potential to confuse the queue onset detection algorithm.
> >
> > It would be interesting to observe the behaviour of an ECN-aware TCP
> > protocol behaviour if ECN were to enabled on Starlink routing devices.
> > ECN has the potential to provide a clear signal to the endpoints about
> > the onset of network-level queue formation, as distinct from latency
> > variation.  

It frustrates me that all research still looks primarily at Reno, rather
than the congestion controls that are actually implemented in Linux and Windows
which are used predominately on the Internet.
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] Trying to *really* understand Linux pacing

[Stephen Hemminger via Bloat]

On Wed, 7 Feb 2024 07:05:27 -0500
Dave Taht via Bloat  wrote:

> I also tend to think that attempting it in various cloudy
> environments and virtualization schemes, and with certain drivers, the
> side effects are not as well understood as I would like. For example,
> AWS's nitro lacks BQL as does virtio-net.

FYI - Microsoft Azure drivers don't do BQL either.
Both netvsc and mana.

The problem is that implementing BQL is not as easy as it looks
and has possibility of introducing bugs. One big customer with a stuck
connection totally negates any benefit!

For virtio, there have been several attempts at BQL and they never worked
perfectly.
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] Best approach for debloating Airbnb host?

[Stephen Hemminger via Bloat]

On Tue, 17 Oct 2023 06:26:17 -0700
Dave Taht via Bloat  wrote:

> > 2. What would I recommend? Obviously, inserting something with cake into 
> > the mix would help a lot. Even if they were willing to let me examine their 
> > entire network (Comcast router, Apple Airport in our Airbnb unit, other 
> > router?) I have no idea what kind of tar baby I would be touching. I don't 
> > want to become their network admin for the rest of time.

By now most home routers do have option for some form of QoS but may default to 
disabled.

The problem with interacting with mortals on technical topics is that you 
create a debt.
It is often a multi-hour effort to resolve and the expectation is that this is 
free.
So doing it for relatives (of course), fixing a local coffee shop where you 
know the owner (sure),
but dealing with a one time AirBnb doesn't seem worth it.
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] Easiest/most effective way to test software against adverse networks?

[Stephen Hemminger via Bloat]

On Sat, 5 Aug 2023 13:35:40 -0400
Sean DuBois via Bloat  wrote:

> I am working on improving Pion's Google Congestion Control algorithm
> https://github.com/pion/interceptor/tree/master/pkg/gcc. As I start to use
> it in more real world networks I find flaws.
> 
> How are people testing software today? Is 'Traffic Control' the best option?


Netem works but there are artifacts from the emulation.

But my view on congestion control is that this sounds like Google
doing NIH reinvention. Happens when you hire a lot of smart people

"Those who don't know history are doomed to repeat it"
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] [Cake] Two questions re high speed congestionmanagement anddatagram protocols

[Stephen Hemminger via Bloat]

On Mon, 10 Jul 2023 23:27:46 +0200
Sebastian Moeller  wrote:

> For what it is worth, the tsv working group is considering whether to process 
> mp-dccp on the standards track, but then the IETF seems not to care too 
> deeply about open-source licence compliance. Or recent kernel implementations 
> or implementations that have a realistic path towards mainline inclusion... 
> but I digress.

The issue is the MP-DCCP implementation is derived from the existing kernel 
DCCP which is GPL-v2.
You can't go randomly mixing proprietary code and GPL code in the kernel.
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] [Cake] Two questions re high speed congestionmanagement anddatagram protocols

[Stephen Hemminger via Bloat]

On Wed, 28 Jun 2023 10:32:32 -0400 (EDT)
"David P. Reed"  wrote:

> How to find a kernel maintainer to care about DCCP, seems to be the question 
> for Linux.
> I am tempted... Not to get involved with IETF "barriers" (what a mess, given 
> the folks in IETF who resisted in AQM, I wouldn't last a minute), but to keep 
> DCCP support alive.
> The barrier here is getting accepted as a Linux maintainer, which is a 
> different issue entirely, looking at my last two experiences with submitting 
> simple bug fixes to the kernel, which were nightmares. I don't have the 
> commitment to become accepted as a maintainer.
> But it seems good to maintain DCCP, despite its lack of popularity as an IETF 
> standard. It does deal with CC in a way that simplifies use of UDP for 
> serious work.

Interesting that there is an out of tree DCCP, complete with likely GPL license 
violation.
https://github.com/telekom/mp-dccp
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] Two questions re high speed congestion management anddatagram protocols

[Stephen Hemminger via Bloat]

On Tue, 27 Jun 2023 12:47:01 -0700 (PDT)
David Lang  wrote:

> On Mon, 26 Jun 2023, David P. Reed via Bloat wrote:
> 
> > Sorry for top posting, but ... Bigger question:
> > Why would DCCP be deprecated by Linux kernel?
> > Who makes that decision? Who argues against it?  
> 
> Linus or the networking maintaners make the decision.
> 
> Usually things get pulled from the kernel because there are updates that need 
> to 
> be made to the code (to match changes elsewhere in the kernel or because of 
> security issues) and there isn't a maintainer who works on the code in a 
> resonable time. This means that the maintainers for the general code area (in 
> this case networking maintainers) will need to do extra work in an area they 
> aren't that interested in (and, especially in the case of hardware, may not 
> have 
> the ability to test). They do some of it, especially if it's commonly used, 
> but 
> eventually either another maintainer steps up, or it goes away
> 
> David Lang

See 
https://patchwork.kernel.org/project/netdevbpf/patch/20230614194705.90673-3-kun...@amazon.com/
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] Two questions re high speed congestion management anddatagram protocols

[Stephen Hemminger via Bloat]

On Mon, 26 Jun 2023 23:41:59 -0400 (EDT)
"David P. Reed"  wrote:

> Sorry for top posting, but ... Bigger question:
> Why would DCCP be deprecated by Linux kernel?
> Who makes that decision? Who argues against it?

No one uses it, and unused protocols are targeted by hackers.
And there are few tests and no maintainer.
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] Two questions re high speed congestion management and datagram protocols

[Stephen Hemminger via Bloat]

On Sat, 24 Jun 2023 14:41:52 -0400 (EDT)
"David P. Reed via Bloat"  wrote:

> I also was looking back to DCCP as a useful way to get a UDP that handled 
> congestion without engaging the higher layers, and preserving the other 
> flexibility of UDP.

DCCP never got widely used, and Linux is on the path of deprecating it.
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] why do people still study RED?

[Stephen Hemminger via Bloat]

On Wed, 31 May 2023 16:28:14 -0600
Dave Taht via Bloat  wrote:

> An otherwise good paper on RED vs drop tail here:
> 
> https://arxiv.org/pdf/2303.10220.pdf
> 
> Is it because we completely lack any analytical methods for taking a
> hard look at drop or marking from the head, as codel does, or the
> effect of the fq in fq_codel?
> 
> I keep hoping somehow, somewhere, someone would show up with the tools
> to explain the results I got here:
> 
> https://blog.cerowrt.org/post/juniper/
> 

Probably because that is what all the big router vendors had available in 2010.
Hopefully, they have started do something smarter in HW.
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] hystart++

[Stephen Hemminger via Bloat]

On Sun, 28 May 2023 06:46:48 -0600
Dave Taht via Bloat  wrote:

> Does a linux implementation of this exist? It looks promising...
> 
> https://datatracker.ietf.org/doc/html/rfc9406
> 

Not that I know of. Contact the authors and they may be able to help.
Microsoft is more open source friendly now, but obviously all the research is
on Windows implementation.
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

[Bloat] Traffic shaping for evil?

[Stephen Hemminger via Bloat]

This showed up on Human Infrastructure mailing list and others might be
interested.

https://adriano.fyi/post/2023/2023-04-16-att-traffic-shaping-makes-websites-unusable/
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] WSL2 + fq_codel

[Stephen Hemminger via Bloat]

Should be whatever the Linux distro default is. Most people use Ubuntu

On Mon, Feb 27, 2023, 4:11 AM Maximilian Bachl 
wrote:

> Yes, the default interface of WSL2 seems to use fq_codel:
>
> $ tc qdisc show dev eth0
> qdisc mq 0: root
> qdisc fq_codel 0: parent :8 limit 10240p flows 1024 quantum 1514 target
> 5.0ms interval 100.0ms memory_limit 32Mb ecn
> qdisc fq_codel 0: parent :7 limit 10240p flows 1024 quantum 1514 target
> 5.0ms interval 100.0ms memory_limit 32Mb ecn
> qdisc fq_codel 0: parent :6 limit 10240p flows 1024 quantum 1514 target
> 5.0ms interval 100.0ms memory_limit 32Mb ecn
> qdisc fq_codel 0: parent :5 limit 10240p flows 1024 quantum 1514 target
> 5.0ms interval 100.0ms memory_limit 32Mb ecn
> qdisc fq_codel 0: parent :4 limit 10240p flows 1024 quantum 1514 target
> 5.0ms interval 100.0ms memory_limit 32Mb ecn
> qdisc fq_codel 0: parent :3 limit 10240p flows 1024 quantum 1514 target
> 5.0ms interval 100.0ms memory_limit 32Mb ecn
> qdisc fq_codel 0: parent :2 limit 10240p flows 1024 quantum 1514 target
> 5.0ms interval 100.0ms memory_limit 32Mb ecn
> qdisc fq_codel 0: parent :1 limit 10240p flows 1024 quantum 1514 target
> 5.0ms interval 100.0ms memory_limit 32Mb ecn
>
> On Sat, Feb 25, 2023 at 11:07 PM Dave Taht via Bloat <
> bloat@lists.bufferbloat.net> wrote:
>
>> On Sat, Feb 25, 2023 at 12:54 PM Stephen Hemminger
>>  wrote:
>> >
>> > On Sat, 25 Feb 2023 15:37:02 -0500
>> > Michael Richardson via Bloat  wrote:
>> >
>> > > Dave Taht via Bloat  wrote:
>> > > > I so want to believe... I so want to believe... can anyone
>> confirm?
>> > >
>> > > >
>> https://raw.githubusercontent.com/microsoft/WSL2-Linux-Kernel/linux-msft-wsl-5.15.y/Microsoft/config-wsl
>> > >
>> > > But, AFAIK, WSL isn't a kernel.  It's an implementation of the Linux
>> ABI on
>> > > top of Windows service(s).  If you told me that they build some of it
>> from
>> > > actual Linux kernel sources, I'd believe you.  (Rather like
>> User-Mode-Linux)
>> > >
>> > > If you told me that they have a kernel that they build for when they
>> actually
>> > > spin up an actual VM (such as to run containers) that would also be
>> unsurprising.
>> > >
>> > > > ...
>> > >
>> > > > CONFIG_NET_SCH_DEFAULT=y
>> > > > CONFIG_DEFAULT_FQ_CODEL=y
>> > > > # CONFIG_DEFAULT_PFIFO_FAST is not set
>> > > > CONFIG_DEFAULT_NET_SCH="fq_codel"
>> > >
>> > > It would be nice if the billion windows desktops started doing
>> > > something better, but I don't think it will help observed latency.
>> > > The real question is what the default schedule for the default Azure
>> Linux VM
>> > > is.
>> > >
>> >
>> > I think WSL2 is actually a full Linux VM running in Hyper-V.
>>
>> Yes, it is. But it is kind of unknown how the underlying network
>> interface is behaving in this case, as well as what the actual default
>> qdisc is, and this not just a random gist. It was VERY exciting to see
>> that gist go by...
>>
>> are there no windows users on this list? :/ We long ago should have
>> pursued at least flent, and a TCP_INFO equivalent sampling method for
>> windows. The closest thing we have for windows is the rust-based:
>> https://github.com/Zoxc/crusader
>>
>> --
>> A pithy note on VOQs vs SQM: https://blog.cerowrt.org/post/juniper/
>> Dave Täht CEO, TekLibre, LLC
>> ___
>> Bloat mailing list
>> Bloat@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/bloat
>>
>
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] WSL2 + fq_codel

[Stephen Hemminger via Bloat]

On Sat, 25 Feb 2023 15:37:02 -0500
Michael Richardson via Bloat  wrote:

> Dave Taht via Bloat  wrote:
> > I so want to believe... I so want to believe... can anyone confirm?  
> 
> > 
> https://raw.githubusercontent.com/microsoft/WSL2-Linux-Kernel/linux-msft-wsl-5.15.y/Microsoft/config-wsl
>   
> 
> But, AFAIK, WSL isn't a kernel.  It's an implementation of the Linux ABI on
> top of Windows service(s).  If you told me that they build some of it from
> actual Linux kernel sources, I'd believe you.  (Rather like User-Mode-Linux)
> 
> If you told me that they have a kernel that they build for when they actually
> spin up an actual VM (such as to run containers) that would also be 
> unsurprising.
> 
> > ...  
> 
> > CONFIG_NET_SCH_DEFAULT=y
> > CONFIG_DEFAULT_FQ_CODEL=y
> > # CONFIG_DEFAULT_PFIFO_FAST is not set
> > CONFIG_DEFAULT_NET_SCH="fq_codel"  
> 
> It would be nice if the billion windows desktops started doing
> something better, but I don't think it will help observed latency.
> The real question is what the default schedule for the default Azure Linux VM
> is.
> 

I think WSL2 is actually a full Linux VM running in Hyper-V.


pgpk90lJoAiuz.pgp
Description: OpenPGP digital signature
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] Fwd: [New post] How Good is FWA Wireless?

[Stephen Hemminger via Bloat]

On Wed, 8 Feb 2023 16:43:59 -0800
Dave Taht via Bloat  wrote:

> On Wed, Feb 8, 2023 at 2:41 PM Luis A. Cornejo  
> wrote:
> >
> > As a former T-mobile HSI customer, I can attest the horrible queue 
> > management. The deprioritization was so bad that it became basically 
> > unusable, even a ping or DNS lookup just lagged. I latched to a tower near 
> > the interstate about 2 miles away. I knew when there was an accident at the 
> > interstate, my online experience correlated very well.  
> 
> I ranted over here about how bad just the depriotization could become,
> and also pointed to a flaw, I believe, in how long devices hold onto
> packets in that case. I fear that when the network is getting toasty,
> and comes back online that there is a thundering herd of devices,
> enormous backlogs of stale syns, and so on that hit it, and what is in
> place is some sort of timed rate limiter to deal with it, not very
> well. It is the only explanation I have for seeing pings come back on
> 1s intervals...
> 
> https://blog.cerowrt.org/post/flaws_in_flent/
> 

I think this is leftover circuit switch telco mindset. Everyone 
needs/wants/will buy their own slice.
It looks like they keep trying and trying to see value added networking. But 
well-designed best effort
is always cheaper, more efficient and overall faster. We heard this story 
before with MPLS, ATM, etc.
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] [Rpm] the grinch meets cloudflare's christmas present

[Stephen Hemminger via Bloat]

On Wed, 4 Jan 2023 15:11:34 -0500
David Collier-Brown via Bloat  wrote:

> On 1/4/23 15:02, rjmcmahon via Bloat wrote:
> > Curious to why people keep calling capacity tests speed tests? A semi 
> > at 55 mph isn't faster than a porsche at 141 mph because its load 
> > volume is larger.  
> 
> They're trying to make the results sound more attractive. They know the 
> customer would prefer a porsche, but what they have to sell them are 
> semi-trailers.
> 
> --dave
> 
> 
> ___
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Interesting, the cloudflare test gets much worse results than others.
It maybe because it defaults to IPV6 by default and ISP does not support
real IPV6.
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] Fwd: NANOG 87: Call for Presentations | N86 PC Picks + More

[Stephen Hemminger via Bloat]

On Thu, 10 Nov 2022 08:58:39 -0800
Dave Taht via Bloat  wrote:

> I keep hoping that someone from starlink will start showing up at
> traditional venues such as this.

How is Starlink going, now that Elon has shown how he can screw up a company.
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] [Make-wifi-fast] The most wonderful video ever about bufferbloat

[Stephen Hemminger via Bloat]

On Wed, 19 Oct 2022 14:33:28 -0700 (PDT)
David Lang via Bloat  wrote:

> On Wed, 19 Oct 2022, Stuart Cheshire via Bloat wrote:
> 
> > On Mon, Oct 17, 2022 at 5:02 PM Stuart Cheshire  wrote:
> >  
> >> Accuracy be damned. The analogy to common experience resonates more.  
> >
> > I feel it is not an especially profound insight to observe that, “people 
> > don’t like waiting in line.” The conclusion, “therefore privileged people 
> > should get to go to the front,” describes an airport first class checkin 
> > counter, Disney Fastpass, and countless other analogies from everyday life, 
> > all of which are the wrong solution for packets in a network.  
> 
> the 'privileged go first' is traditional QoS, and it can work to some extent, 
> but is a nightmare to maintain and gets the wrong result most of the time.

A lot of times when this is proposed it has some business/political motivation.
It is like "priority boarding" for Global Services customers.
Not solving a latency problem, instead making stakeholders happy.

> AQM (fw_codel and cake) are more the 'cash only line' and '15 items or less' 
> line, they speed up the things that can be fast a LOT, while not 
> significantly 
> slowing down the people with a full baskets (but in the process, it shortens 
> the 
> lines for those people with full baskets)
> 
> >> I think the person with the cheetos pulling out a gun and shooting 
> >> everyone in front of him (AQM) would not go down well.  
> >
> > Which is why starting with a bad analogy (people waiting in a grocery 
> > store) inevitably leads to bad conclusions.
> >
> > If we want to struggle to make the grocery store analogy work, perhaps we 
> > show 
> > people checking some grocery store app on their smartphone before they 
> > leave 
> > home, and if they see that a long line is beginning to form they wait until 
> > later, when the line is shorter. The challenge is not how to deal with a 
> > long 
> > queue when it’s there, it is how to avoid a long queue in the first place.  
> 
> only somewhat, you aren't going to have people deciding not to click on a 
> link 
> because the network is busy, and if you did try to go that direction, I would 
> fight you. the prioritization is happening at a much lower level, which is 
> hard 
> to put into an analogy
> 
> even with the 'slowing' of bulk traffic, no traffic is prevented, it's just 
> that 
> they aren't allowed to monopolize the links.
> 
> This is where the grocery store analogy is weak, the reality would be more 
> like 
> 'the cashier will only process 30 items before you have to step aside and let 
> someone else in', but since no store operates that way, it would be a bad 
> analogy.

Grocery store analogies also breakdown because packets are not "precious"
it is okay to drop packets. A lot of AQM works by doing "drop early and often"
instead of "drop late and collapse".

> 
> >> Actually that analogy is fairly close to fair queuing. The multiple 
> >> checker analogy is one of the most common analogies in queue theory 
> >> itself.  
> >
> > I disagree. You are describing the “FQ” part of FQ_CoDel. It’s the “CoDel” 
> > part of FQ_CoDel that solves bufferbloat. FQ has been around for a long 
> > time, 
> > and at best it partially masked the effects of bufferbloat. Having more 
> > queues 
> > does not solve bufferbloat. Managing the queue(s) better solves bufferbloat.
> >  
> >> I like the idea of a guru floating above a grocery cart with a better 
> >> string of explanations, explaining
> >>
> >>   - "no, grasshopper, the solution to bufferbloat is no line... at all".  
> >
> > That is the kind of thing I had in mind. Or a similar quote from The 
> > Matrix. 
> > While everyone is debating ways to live with long queues, the guru asks, 
> > “What 
> > if there were no queues?” That is the “mind blown” realization.  
> 
> In a world where there is no universal scheduler (and no universal knowlege 
> to 
> base any scheduling decisions on), and where you are going to have malicious 
> actors trying to get more than their fair share, you can't rely on voluntary 
> actions to eliminate the lines.
> 
> There are data transportation apps that work by starting up a large number of 
> connections in parallel for the highest transfer speeds (shortening slow 
> start, 
> reducing the impact of lost packets as they only affect one connection, etc). 
> This isn't even malicious actors, but places like Hollywood studios sending 
> the raw movie footage around over dedicated leased lines and wanting to get 
> every bps of bandwidth that they are paying for used.
> 
> David Lang

___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] Researchers discover major roadblock in alleviating network congestion

[Stephen Hemminger via Bloat]

On Fri, 5 Aug 2022 00:45:12 +0300
Jonathan Morton via Bloat  wrote:

> > On 4 Aug, 2022, at 3:21 pm, Bjørn Ivar Teigen via Bloat 
> >  wrote:
> > 
> > Main take-away (as I understand it) is something like "In real-world 
> > networks, jitter adds noise to the end-to-end delay such that any algorithm 
> > trying to infer congestion from end-to-end delay measurements will 
> > occasionally get it wrong and this can lead to starvation". Seems related 
> > to Jaffe's work on network power (titled "Flow control power is 
> > non-decentralizable").   
> 
> Hasn't this been known for many years, as a consequence of experience with 
> TCP Vegas?
> 
>  - Jonathan Morton

It seems like BBR developers thought they could do better. Unfortunately, 
papers with negative
results never seem to get written or published ;-(
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] gaming latencies

[Stephen Hemminger]

Apparently audiophiles are willing to spend extra money on a switch
with better power supply and carbon frame to reduce noise in digital signals.

https://www.tomshardware.com/news/ethernet-switch-for-audiophiles
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] Bechtolschiem

[Stephen Hemminger]

On Fri, 2 Jul 2021 09:42:24 -0700
Dave Taht  wrote:

> "Debunking Bechtolsheim credibly would get a lot of attention to the
> bufferbloat cause, I suspect." - dpreed
> 
> "Why Big Data Needs Big Buffer Switches" -
> http://www.arista.com/assets/data/pdf/Whitepapers/BigDataBigBuffers-WP.pdf
> 

Also, a lot depends on the TCP congestion control algorithm being used.
They are using NewReno which only researchers use in real life.

Even TCP Cubic has gone through several revisions. In my experience, the
NS-2 models don't correlate well to real world behavior.

In real world tests, TCP Cubic will consume any buffer it sees at a
congested link. Maybe that is what they mean by capture effect.

There is also a weird oscillation effect with multiple streams, where one
flow will take the buffer, then see a packet loss and back off, the
other flow will take over the buffer until it sees loss.

___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] Really getting 1G out of ISP?

[Stephen Hemminger]

On Tue, 22 Jun 2021 22:51:18 +
"Livingood, Jason"  wrote:

> > It doesn't help that all the local ISP's claim 10Mbit upload even with 1G 
> > download. Is this a head end provisioning problem or related to Docsis 3.0 
> > (or later) modems?  
> 
> I'll cover this in an upcoming technical paper (mid-July I hope). Depending 
> on the DOCSIS version, CMTS, and cable modems involved you may have no AQM, 
> or buffer controls for the cable modem, or AQM (sort of) on the CMTS and in 
> the cable modem. In the Comcast network you should find AQM in the upstream 
> queue on the cable modems for which we have deployed RDK-B software (XB6 and 
> XB7), while other devices would have buffer controls.
> 
> JL
> 

Just a short update. The cable modem matters. Updated from Docsis 3.0 modem 
with bad Intel Puma chipset
to new model with Docsis 3.1 and Broadcom and things are much more stable.

As far as AQM, in this setup; fq_codel does much better than the Cake 
configuration. With fq_codel can
see 700Mbit download speed. It looks like Cake is using more CPU especialy 
since the Cake configuration
is using an ifb ingress queue discipline as well.
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] Really getting 1G out of ISP?

[Stephen Hemminger]

On Tue, 22 Jun 2021 23:04:18 +
"Livingood, Jason"  wrote:

> > For DOCSIS the issue seems to be an unfortunate frequency split between up 
> > and downstream and use of lower efficiency coding schemes.  
> 
> Performance really takes a big step forward once a person has a D3.1 modem in 
> their home, bringing OFDM and OFDMA as key advancements. Also in flux at the 
> moment is where the upstream split is in cable networks, which are moving to 
> mid-split or high-split designs that bring more upstream bandwidth. As well, 
> over the past 18+ months, most cable networks have added substantially more 
> upstream channels as well as performed quite a number of fiber node splits. 
> And that is just below the physical layer stuff - there's also a lot of work 
> at the software layer for modems and CMTSes that is quite interesting.
> 
> JL
> 

Also discovered that the cable modem I am using is one of the ones with the
broken Intel Puma chipset (even a lawsuit about that). Will be changing it 
today.
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

[Bloat] Really getting 1G out of ISP?

[Stephen Hemminger]

Is there any consumer hardware that can actually keep up and do AQM at 1Gbit.
It seems everyone seems obsessed with gamer Wifi 6. But can only do 300Mbit 
single
stream with any kind of QoS.

It doesn't help that all the local ISP's claim 10Mbit upload even with 1G 
download.
Is this a head end provisioning problem or related to Docsis 3.0 (or later) 
modems?

___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] virtio_net: BQL?

[Stephen Hemminger]

On Mon, 17 May 2021 16:32:21 -0700
Dave Taht  wrote:

> On Mon, May 17, 2021 at 4:00 PM Stephen Hemminger
>  wrote:
> >
> > On Mon, 17 May 2021 14:48:46 -0700
> > Dave Taht  wrote:
> >  
> > > On Mon, May 17, 2021 at 1:23 PM Willem de Bruijn
> > >  wrote:  
> > > >
> > > > On Mon, May 17, 2021 at 2:44 PM Dave Taht  wrote:  
> > > > >
> > > > > Not really related to this patch, but is there some reason why virtio
> > > > > has no support for BQL?  
> > > >
> > > > There have been a few attempts to add it over the years.
> > > >
> > > > Most recently, 
> > > > https://lore.kernel.org/lkml/20181205225323.12555-2-...@redhat.com/
> > > >
> > > > That thread has a long discussion. I think the key open issue remains
> > > >
> > > > "The tricky part is the mode switching between napi and no napi."  
> > >
> > > Oy, vey.
> > >
> > > I didn't pay any attention to that discussion, sadly enough.
> > >
> > > It's been about that long (2018) since I paid any attention to
> > > bufferbloat in the cloud and my cloudy provider (linode) switched to
> > > using virtio when I wasn't looking. For over a year now, I'd been
> > > getting reports saying that comcast's pie rollout wasn't working as
> > > well as expected, that evenroute's implementation of sch_cake and sqm
> > > on inbound wasn't working right, nor pf_sense's and numerous other
> > > issues at Internet scale.
> > >
> > > Last week I ran a string of benchmarks against starlink's new services
> > > and was really aghast at what I found there, too. but the problem
> > > seemed deeper than in just the dishy...
> > >
> > > Without BQL, there's no backpressure for fq_codel to do its thing.
> > > None. My measurement servers aren't FQ-codeling
> > > no matter how much load I put on them. Since that qdisc is the default
> > > now in most linux distributions, I imagine that the bulk of the cloud
> > > is now behaving as erratically as linux was in 2011 with enormous
> > > swings in throughput and latency from GSO/TSO hitting overlarge rx/tx
> > > rings, [1], breaking various rate estimators in codel, pie and the tcp
> > > stack itself.
> > >
> > > See:
> > >
> > > http://fremont.starlink.taht.net/~d/virtio_nobql/rrul_-_evenroute_v3_server_fq_codel.png
> > >
> > > See the swings in latency there? that's symptomatic of tx/rx rings
> > > filling and emptying.
> > >
> > > it wasn't until I switched my measurement server temporarily over to
> > > sch_fq that I got a rrul result that was close to the results we used
> > > to get from the virtualized e1000e drivers we were using in 2014.
> > >
> > > http://fremont.starlink.taht.net/~d/virtio_nobql/rrul_-_evenroute_v3_server_fq.png
> > >
> > > While I have long supported the use of sch_fq for tcp-heavy workloads,
> > > it still behaves better with bql in place, and fq_codel is better for
> > > generic workloads... but needs bql based backpressure to kick in.
> > >
> > > [1] I really hope I'm overreacting but, um, er, could someone(s) spin
> > > up a new patch that does bql in some way even half right for this
> > > driver and help test it? I haven't built a kernel in a while.
> > >  
> >
> > The Azure network driver (netvsc) also does not have BQL. Several years ago
> > I tried adding it but it benchmarked worse and there is the added complexity
> > of handling the accelerated networking VF path.  
> 
> I certainly agree it adds complexity, but the question is what sort of
> network behavior resulted without backpressure inside the
> vm?
> 
> What sorts of benchmarks did you do?
> 
> I will get setup to do some testing of this that is less adhoc.

Less of an issue than it seems for must users.

For the most common case, all transmits are passed through to the underlying
VF network device (Mellanox). So since Mellanox supports BQL, that works.
The special case is if accelerated networking is disabled or host is being
serviced and the slow path is used. Optimizing the slow path is not that
interesting.

I wonder if the use of SRIOV with virtio (which requires another layer
with the failover device) behaves the same way?
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] virtio_net: BQL?

[Stephen Hemminger]

On Mon, 17 May 2021 14:48:46 -0700
Dave Taht  wrote:

> On Mon, May 17, 2021 at 1:23 PM Willem de Bruijn
>  wrote:
> >
> > On Mon, May 17, 2021 at 2:44 PM Dave Taht  wrote:  
> > >
> > > Not really related to this patch, but is there some reason why virtio
> > > has no support for BQL?  
> >
> > There have been a few attempts to add it over the years.
> >
> > Most recently, 
> > https://lore.kernel.org/lkml/20181205225323.12555-2-...@redhat.com/
> >
> > That thread has a long discussion. I think the key open issue remains
> >
> > "The tricky part is the mode switching between napi and no napi."  
> 
> Oy, vey.
> 
> I didn't pay any attention to that discussion, sadly enough.
> 
> It's been about that long (2018) since I paid any attention to
> bufferbloat in the cloud and my cloudy provider (linode) switched to
> using virtio when I wasn't looking. For over a year now, I'd been
> getting reports saying that comcast's pie rollout wasn't working as
> well as expected, that evenroute's implementation of sch_cake and sqm
> on inbound wasn't working right, nor pf_sense's and numerous other
> issues at Internet scale.
> 
> Last week I ran a string of benchmarks against starlink's new services
> and was really aghast at what I found there, too. but the problem
> seemed deeper than in just the dishy...
> 
> Without BQL, there's no backpressure for fq_codel to do its thing.
> None. My measurement servers aren't FQ-codeling
> no matter how much load I put on them. Since that qdisc is the default
> now in most linux distributions, I imagine that the bulk of the cloud
> is now behaving as erratically as linux was in 2011 with enormous
> swings in throughput and latency from GSO/TSO hitting overlarge rx/tx
> rings, [1], breaking various rate estimators in codel, pie and the tcp
> stack itself.
> 
> See:
> 
> http://fremont.starlink.taht.net/~d/virtio_nobql/rrul_-_evenroute_v3_server_fq_codel.png
> 
> See the swings in latency there? that's symptomatic of tx/rx rings
> filling and emptying.
> 
> it wasn't until I switched my measurement server temporarily over to
> sch_fq that I got a rrul result that was close to the results we used
> to get from the virtualized e1000e drivers we were using in 2014.
> 
> http://fremont.starlink.taht.net/~d/virtio_nobql/rrul_-_evenroute_v3_server_fq.png
> 
> While I have long supported the use of sch_fq for tcp-heavy workloads,
> it still behaves better with bql in place, and fq_codel is better for
> generic workloads... but needs bql based backpressure to kick in.
> 
> [1] I really hope I'm overreacting but, um, er, could someone(s) spin
> up a new patch that does bql in some way even half right for this
> driver and help test it? I haven't built a kernel in a while.
> 

The Azure network driver (netvsc) also does not have BQL. Several years ago
I tried adding it but it benchmarked worse and there is the added complexity
of handling the accelerated networking VF path.

___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] Questions for Bufferbloat Wikipedia article

[Stephen Hemminger]

On Tue, 6 Apr 2021 23:59:53 +0200
Erik Auerswald  wrote:

> Hi,
> 
> On Tue, Apr 06, 2021 at 10:02:21PM +0200, Bless, Roland (TM) wrote:
> > On 06.04.21 at 20:50 Erik Auerswald wrote:  
> > >On Tue, Apr 06, 2021 at 08:31:01AM +0200, Sebastian Moeller wrote:  
> > >>>On Apr 6, 2021, at 02:47, Erik Auerswald  
> > >>>wrote:
> > >>>On Mon, Apr 05, 2021 at 11:49:00PM +0200, Sebastian Moeller wrote:  
> > >On Apr 5, 2021, at 14:46, Rich Brown  wrote:
> > >
> > >Dave Täht has put me up to revising the current Bufferbloat article
> > >on Wikipedia (https://en.wikipedia.org/wiki/Bufferbloat)
> > >[...]  
> > >Yes, large unmanaged buffers are at the core of the bufferbloat problem.  
> > 
> > I disagree here: it is basically the combination
> > of loss-based congestion control with unmanaged
> > tail-drop buffers.  
> 
> That worked for decades, then stopped working as well as before.
> What changed?
> 
> Yes, there are complex interactions with how packet switched networks
> are used.  Otherwise we would probably not find ourselves in the current
> situation.
> 
> To me, the potential of having to wait minutes (yes, minutes!) for
> the result of a key stroke over an SSH session is not worth the potential
> throughput performance gain of buffers that cannot be called small.
> 
> > There are at least two solutions
> > to the bufferbloat problem
> > 1) better congestion control algorithms
> > 2) active queue management (+fq maybe)  
> 
> Both approaches aim to not use all of the available buffer space, if
> there are unreasonably large buffers, i.e., they aim to not build a
> large standing queue.
> 
> > [...]
> > Small buffers definitely limit the queuing delay as well as
> > jitter. However, how much performance is potentially lost due to
> > the small buffer depends a lot on the arrival distribution.  
> 
> Could the better congestion control algorithms avoid the potential
> performance loss by not requiring large buffers for high throughput?
> Might small buffers incentivise to not send huge bursts of data and hope
> for the best?
> 
> FQ with AQM aims to allow the absorption of large traffic bursts (i.e.,
> use of large buffers) without affecting _other_ flows too much.
> 
> I would consider the combination of FQ+AQM, better congestion control
> algorithms, and large buffers as an optimization, but using just large
> buffers without any of the other two approaches as a mistake currently
> called bufferbloat.  As such I see large unmanaged buffers at the core
> of the bufferbloat problem.
> 
> FQ+AQM for every large buffer may solve the bufferbloat problem by
> attacking the "unmanaged" part of the problem.  Small buffers may solve
> it by attacking the "large" part of the problem.  Small buffers may
> bring their own share of problems, but IMHO those are much less than
> those of bufferbloat.
> 
> I do not see TCP congestion control improvements, even combining
> sender-side improvements with receiver-side methods as in rLEDBAT[0],
> as a solution to bufferbloat, but rather as a mitigation.
> 
> [0] https://datatracker.ietf.org/doc/draft-irtf-iccrg-rledbat/
> 
> Anyway, I think it is obvious that I am willing to sacrifice more
> throughput for better latency than others.
> 

For Wikipedia it is important to make clear:
  * the symptoms = large latency
  * the cause = large buffers and aggressive protocols
  * the solutions = AQM, smaller buffers, pacing, better congestion control, 
etc.

People can argue over best combination of solutions but the symptoms and
causes should be defined, and non-contentious.

It is too easy to go off in the weeds and have the solution of the day.


___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] Questions for Bufferbloat Wikipedia article

[Stephen Hemminger]

On Mon, 5 Apr 2021 08:46:15 -0400
Rich Brown  wrote:

> Dave Täht has put me up to revising the current Bufferbloat article on 
> Wikipedia (https://en.wikipedia.org/wiki/Bufferbloat)
> 
> Before I get into it, I want to ask real experts for some guidance... Here 
> goes:
> 
> 1) What is *our* definition of Bufferbloat? (We invented the term, so I think 
> we get to define it.) 
> 
> a) Are we content with the definition from the bufferbloat.net site, 
> "Bufferbloat is the undesirable latency that comes from a router or other 
> network equipment buffering too much data." (This suggests bufferbloat is 
> latency, and could be measured in seconds/msec.)
> 
> b) Or should we use something like Jim Gettys' definition from the Dark 
> Buffers article (https://ieeexplore.ieee.org/document/5755608), "Bufferbloat 
> is the existence of excessively large (bloated) buffers in systems, 
> particularly network communication systems." (This suggests bufferbloat is an 
> unfortunate state of nature, measured in units of "unhappiness" :-) 
> 
> c) Or some other definition?
> 
> 2) All network equipment can be bloated. I have seen (but not really 
> followed) controversy regarding the amount of buffering needed in the Data 
> Center. Is it worth having the Wikipedia article distinguish between Data 
> Center equipment and CPE/home/last mile equipment? Similarly, is the "bloat 
> condition" and its mitigation qualitatively different between those 
> applications? Finally, do any of us know how frequently data centers/backbone 
> ISPs experience buffer-induced latencies? What's the magnitude of the impact?
> 
> 3) The Wikipedia article mentions guidance that network gear should 
> accommodate buffering 250 msec of traffic(!) Is this a real "rule of thumb" 
> or just an often-repeated but unscientific suggestion? Can someone give 
> pointers to best practices?
> 
> 4) Meta question: Can anyone offer any advice on making a wholesale change to 
> a Wikipedia article? Before I offer a fork-lift replacement I would a) 
> solicit advice on the new text from this list, and b) try to make contact 
> with some of the reviewers and editors who've been maintaining the page to 
> establish some bona fides and rapport...
> 
> Many thanks!
> 
> Rich

I like to think of Bufferbloat as a combination of large buffers and how 
algorithms react to those buffers.
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] [Cerowrt-devel] a start at the FCC filing

[Stephen Hemminger]

Start with Ron Wyden

On Thu, Mar 4, 2021, 7:54 PM Dave Taht  wrote:

> I am planning to take my time on this. I would like for example, to
> at least communicate well with a republican senator and a democratic one.
>
> Admittedly, if we can upgrade everybody to 100Mbit, everybody can have
> all 4 home members being couch potatoes in front of HD netflix and
> there won't be much motivation to do anything else.
>
>
> https://news.slashdot.org/story/21/03/04/1722256/senators-call-on-fcc-to-quadruple-base-high-speed-internet-speeds
>
> Anybody know these guys?
>
> On Sun, Feb 21, 2021 at 8:50 AM David P. Reed  wrote:
> >
> > This is an excellent proposal. I am happy to support it somehow.
> >
> >
> >
> > I strongly recommend trying to find a way to make sure it doesn't become
> a proposal put forward by "progressive" potlitical partisans. (this is hard
> for me, because my politics are more aligned with the Left than with the
> self-described conservatives and right-wing libertarians.
> >
> >
> >
> > This is based on personal experience starting in 2000 and continuing
> through 2012 or so with two issues:
> >
> >
> >
> > 1. Open Spectrum (using computational radio networking to make a
> scalable framework for dense wireless extremely wideband internetworking).
> I along with a small number of others started this as a non-partisan
> effort. It became (due to lobbyists and "activists") considered to be a
> socialist taking of property from spectrum "owners". After that, it became
> an issue where a subset of the Democratic Party (progressives) decided to
> make it a wedge issue in political form. (It should be noted that during
> this time, a Republican Secretary of Commerce took up the idea of making
> UWB legal, and fought off lobbyists to some extent, though the resulting
> regulation was ineffective because it was too weak to be usable).
> >
> >
> >
> > 2. Network Neutrality or Open Internet. Here the key issue was really
> about keeping Internet routing intermediaries from being selective about
> what packets they would deliver and what ones they would not. The design of
> the Internet was completely based on open carriage of all packets without
> the routers billing for or metering based on end-to-end concerns. Again,
> for a variety of reasons, this simple idea got entangled with partisanship
> politically - such that advocates for an Open Internet were seen to be
> promoting both Democratic Party and Silicon Valley Tech interests. In fact,
> the case for Open Internet is not primarily political. It's about
> scalability of the infrastructure and the ability to carry Internet packets
> over any concatenation of paths, for mutual benefit to all users. (That
> "mutual benefit" concept does seem to be alien to a certain kind of
> individualist libertarian cult thinking that is a small subset of
> Republican Party membership).
> >
> >
> >
> > If this becomes yet another Democratic Party initiative, it will
> encounter resistance, both from Republican-identified polarizing reaction,
> and also from the corporate part of the Democratic Party (so called Blue
> Dog Democrats where telecom providers provide the largest quantity of
> funding to those Democrats).
> >
> >
> >
> > Some "progressive" Democrats will reach out to add this to their
> "platform" as a partisan issue.
> >
> >
> >
> > It may feel nice to have some of them on your side. Like you aren't
> alone. But by accepting this "help" on this issue, you may be guaranteeing
> its failure.
> >
> >
> >
> > In a world where compromise is allowed to generate solutions to
> problems, polarizing would not be effective to kill a good idea, rather
> merely raising the issue would lead to recognizing the problem is important
> and joint work to create a solution. In 1975, the Internet was not
> partisan. Its designers weren't party members or loyalists. We were solving
> a problem of creating a scalable, efficient alternative to the "Bell
> System" model of communications where every piece of gear got involved in
> deciding what to do with each bit of information, where there were "voice
> bits" and "data bits", "business bits" and "residential bits", and every
> piece of equipment had to be told everything about each bits (through call
> setup).
> >
> >
> >
> > But today, compromise is not considered possible, even at the level of
> defining the problem!
> >
> >
> >
> > So this simple architectural approach to clearing out the brush that has
> grown like weeds throughout the Internet, especially at the "access
> provider" will become political.
> >
> >
> >
> > Since in the end of the day it threatens to reduce control and revenues
> to edge "access providers" that come from selling higher-rate pipes, the
> natural opposition will likely come from lobbyists for telecom incumbents,
> funded by equipment providers for those incumbents (Cisco, Alcatel Lucent
> and their competitors), with Republicans and Blue-Dog Democrats carrying
> their water. That's tthe likely 

Re: [Bloat] [Cake] Fwd: [Galene] Dave on bufferbloat and jitter at 8pm CET Tuesday 23

[Stephen Hemminger]

On Tue, 23 Feb 2021 17:14:31 -0800
Dave Taht  wrote:

> wow, that is (predictably) miserable, even with cake. The only
> solution that is going to
> work is to somehow actively monitor your link quality and adjust cake
> to suit. Or we can start trying to use kathie's passive ping tools.
> 
> On Tue, Feb 23, 2021 at 1:30 PM Nils Andreas Svee  wrote:
> >
> > Thanks for the talk Dave and it was nice meeting you all!
> >
> > Never really did much in the way of Flent tests after moving from ADSL to 
> > Telenor's "wireless broadband" aka. 4G. So I ran some after leaving the 
> > meeting, with CAKE on or off, and let me tell you - it's terrifying, 4G 
> > sucks indeed., not as bad as DSL without SQM mind, but still
> >
> > Avg. latency without SQM at some points close to 800 ms or above. Had to 
> > sacrifice a lot of bandwidth to get it to sane levels when doing RRUL tests.
> >
> > Dumped all the files over here: https://dl.lochnair.net/Bufferbloat/Tests/
> > Oh btw I promise I'll try to not break things when you need to access 
> > something on that box again Dave...
> >
> > Best Regards
> > Nils  
> 
> 
> 

Because cable was down, used a burner SIM and a spare phone over LTE for a day.
And agree it is awful. Couldn't bring myself to run Flent over this pig with 
wings.
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] OpenWrt Stability?

[Stephen Hemminger]

On Tue, 5 Jan 2021 12:08:56 -0500
Rich Brown  wrote:

> > On Jan 5, 2021, at 12:00 PM, Stephen Hemminger wrote:
> > 
> > I having lots of issues with openwrt stability.  
> 
> I know the OpenWrt dev's are going hammer and tongs to produce a 2x.0x 
> release (which will be out soon-ish...)
> 
> What release and what hardware are you using? (There may be fixes in the 
> newest snapshots, and if not, you may be able to provide valuable debugging 
> info.)
> 
> Rich

I am running latest stable 19.07.4
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] Openwrt stability?

[Stephen Hemminger]

On Tue, 5 Jan 2021 11:49:58 -0500
Daniel Sterling  wrote:

> On Tue, Jan 5, 2021 at 11:10 AM Michael Richardson  wrote:
> >
> > Stephen Hemminger  wrote:  
> > > Any idea how to debug this? Is there a way to get serial console?  
> >
> > It depends upon your device, but in most cases, open the case, find the 
> > pins,
> > attach USB/TTL interface.  Some newish devices (Turris MOX, for instance),
> > annoyingly need 1.8V TTL interfaces, which are harder to find and more
> > expensive. ($20 vs $3)  
> 
> Best option may be to just buy better supported hardware :)
> 
> I've been running openwrt on a small mobile-class PC (sandy bridge
> celeron) and then just using stock UBNT SOHO (amplifi HD) access
> points for the wifi itself. This is working rather well. The amplifi
> units have buffers that are too big but that's mitigated on the
> openwrt side with cake

It is Linksys WRT3200ACM which seemed the best at the time.

Did get the problem down to one device on home network is doing something
that causes crash. It happens now about 20sec after startup.
I suspect a kernel bug and magic bad packet (fragmentation?).
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

[Bloat] Openwrt stability?

[Stephen Hemminger]

I having lots of issues with openwrt stability.
Today's issue seems to be some device on one leg of the home LAN causing 
OpenWrt router
to crash. That leg has Xbox and other audio gear.

Any idea how to debug this? Is there a way to get serial console?

___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

[Bloat] NeST: Network Stack Tester

[Stephen Hemminger]

This looks interesting: 
https://blog.apnic.net/2020/09/18/nest-a-simpleefficient-tool-to-study-congestion-control/

Good - using namespaces for testing is a good idea.
 - testbed is GPLv2

Bad - using iperf as your main test tool encourages buffer bloat

- Does 5G really need yet another congestion control algorithm?
  Or it this just a way to get the grant money?
  Or worse, the start of proprietary TCP congestion control.
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] the future belongs to pacing

[Stephen Hemminger]

On Sun, 05 Jul 2020 13:43:27 -0400
Michael Richardson  wrote:

> Sebastian Moeller  wrote:
> > of the sending rate, no? BBRv2 as I understand it will happily run
> > roughshod over any true rfc3168 AQM on the path, I do not have the
> > numbers, but I am not fully convinced that typically the most
> > significant throttling on a CDN to end-user path happens still inside
> > the CDN's data center...  
> 
> That's an interesting claim. I'm in no position to defend or refute it.
> 
> If it's true, though, it suggests some interesting solutions, because one can
> more easily establish trust relationships within the data-center.
> 
> I'm specifically imagining a clock signal from the Top-of-Rack switch to the
> senders.
> 
> Actually, it's not a clock, so much as a automotive style timing shaft
> running down through all the 1-U servers, with fine vernier adjustments :-)
> I'm also certain we've seen such technology before.
> 
> --
> ]   Never tell me the odds! | ipv6 mesh networks [
> ]   Michael Richardson, Sandelman Software Works|IoT architect   [
> ] m...@sandelman.ca  http://www.sandelman.ca/|   ruby on rails
> [
> 
> 
> 
> 

I keep wondering how BBR will respond to intermediaries that aggregate packets.
At higher speeds, won't packet trains happen and would it not get confused
by this? Or is its measurement interval long enough that it doesn't matter.


pgpr27vG3SeKd.pgp
Description: OpenPGP digital signature
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] intel gives up on home gateways

[Stephen Hemminger]

On Tue, 28 Apr 2020 09:57:26 -0700
Dave Taht  wrote:

> H/T sebastian:
> 
> https://investors.maxlinear.com/press-releases/detail/395/maxlinear-to-acquire-intels-home-gateway-platform
> 
> Gawd knows what this means.
> 

It means somebody is getting a bonus, and somebody is out of a job
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] Rigorous Coffee Shop Bloat Testing

[Stephen Hemminger]

On Wed, 4 Sep 2019 06:24:07 +0300
Jonathan Morton  wrote:

> > On 4 Sep, 2019, at 6:17 am, Dave Taht  wrote:
> > 
> > …and people jump off of cable as soon as
> > they can get fiber... but that's due to the RTT more than the bandwidth.  
> 
> And also due to the downright predatory pricing/billing practices of the 
> typical cable ISP.  I think an awful lot of people would take any viable 
> alternative at this point, purely due to that factor.  Which is why cable 
> industry lobbyists are doing their level best to kill municipal fibre.

I am convinced that all the hype about 5G will die when people see how the
ISP's in the US will want to price it. They will certainly want to charge
a premium for 5G to avoid under cutting their current overpriced service.
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] Rigorous Coffee Shop Bloat Testing

[Stephen Hemminger]

On Tue, 3 Sep 2019 10:29:51 -0700
Dave Taht  wrote:

> On Tue, Sep 3, 2019 at 8:57 AM Rich Brown  wrote:
> >  
> > > On Sep 3, 2019, at 11:22 AM, bloat-requ...@lists.bufferbloat.net wrote:
> > >
> > > The coffee shop tests were fun, but I(we) needed more rigor when doing
> > > them. What I'd typically do is go in,
> > > get on the wifi, start 6 minutes worth of tests, get in line, get
> > > coffee...  
> >
> > OK. I'll bite. What "six minutes of tests" do you queue up? What do you 
> > record?  
> 
> Still working out what the "right thing" would be. The ecn syn failure
> thing was new (didn't know OSX had that stat), getting some caps,
> seeing if ipv6 was available, all seem like good ideas, in addition to
> bloat protection.
> 
> the most recent script was this - (the data is in my blog-cerowrt repo
> on github, also).
> 
> We could turn this into a batch file and try to get more rigorous
> about also getting a packet cap.
> 
> Is there a decent android or IOS tool yet?
> 
> #!/bin/sh
> 
> T="Los_Gatos_Starbucks"
> F="flent -x -H flent-fremont.bufferbloat.net -t $T"
> 
> $F --te=download_streams=4 tcp_ndown
> $F --te=upload_streams=4 --socket-stats tcp_nup
> # $F --te=upload_streams=4 tcp_2up_square # not useful enough
> $F rrul
> $F rrul_be
> 
> 
> > And how do you broach the subject with the owner? Something like...  
> 
> Carefully. The owners are always ready to take a complement or
> complaint, and, so long as you catch 'em when
> not too busy are usually pretty social in general.
> 
> In many cases they share the wifi with their credit card reader and
> when I say that fixing bufferbloat helps,
> their eyes light up. That was a specific problem that at least one had
> - demonstrable - I saw it take forever
> to clear a transaction (and the bloat was 2+ seconds long at the time
> - NOT triggered by me) once... he had a synology router, and "applying
> QoS" "just worked", and we did other things like reposition the
> antenna, also. got me lunch
> that did and he punched a whole bunch of holes in my "repeat business" 
> card
> 
> However the employees usually lack the router's password and clue.
> 
> IF we were to make this a thing (and it does invoke fond memories of
> how we first spread wifi around the bay area
> and then the world in the early 2000s - I was part of a group called
> thirdbreak that generally lept across the counter to help a lot, back
> in the days we were so eager to get out of the office and onto wifi
> that we were mapping all the locations available - example:
> http://www.wififreespot.com/ca.php from those days...), perhaps carry
> a portable printer for the test output, biz cards,  and so on for the
> no-owner-present case.
> 
> I recently hit on the idea of creating stickers - attached is the one
> I'm using on my guitar...
> 
> which is a bit too much over the top, I think, generally.
> 
> but plan to give out 0.0.0.0/8 and 240.0.0.0/4 ones next time.
> 
> They are cheap (I used stickermule) and with a cool logo, folk dig
> adorning their laptops with them, in general. Still,
> we've never found that logo/slogan for fixing bufferbloat - the word
> is too long and too negative, though I thought
> the inverted wifi logo we use on the cerowrt site a good start. "Better wifi".
> 
> With more folk gathering data...
> 
> Maybe (for example) we'd play off starbucks vs peets - attached is
> starbucks (using google wifi) vs another coffeeshop, coffeecat - sigh.
> but weirdly enough starbucks's packet cap - although very close to
> what a fq_codel'd trace would look like, doesn't actually seem to be a
> fq_codeled trace. Still puzzled, need to go back and try that spot
> again.
> 
> And I gotta say, it's *really good* to get out of the lab once in a
> while and see people, and sometimes, actually fix something, trying a
> different coffee shop every week.
> 
> I guess, in the cases where the coffee doesn't become free, I could
> deduct it as a business expense. :)
> 
> > "Uh, I think I know why all those heads are popping up..." OR
> > "This is a nice network you have here. It'd be a shame if something 
> > happened to it..." OR  
> 
> Oh, that's great! Goes with my costume, too. "Hey buddy, got an ipv6 address?"
> 
> > "I know I look like [don't look like] a pointy-headed geek, but there's 
> > this thing called bufferbloat..." OR
> > "Do you ever get complaints that your wifi is really slow?"  
> 
> I might also have an agenda in trying to see how much ipv6 is out
> there, and the syn thing is bugging me, too.
> So with a more organized set of tests, we could fan out to the coffee
> shops of the world and forment another
> wifi revolution and turn that world upside down! Who's with me!?

There was a recent Wall Street Journal article that faster Internet doesn't 
mean anything.
https://www.wsj.com/graphics/faster-internet-not-worth-it/

I just thought "faster Internet just exposes your existing Bufferbloat"
___
Bloat mailing list

Re: [Bloat] [Cake] Fwd: Re: Unable to create htb tc classes more than 64K

[Stephen Hemminger]

On Mon, 26 Aug 2019 09:35:14 +0200
Toke Høiland-Jørgensen  wrote:

> Turns out that with the "earliest departure time" support in sched_fq,
> it is now possible to write a shaper in eBPF, thus avoiding the global
> qdisc lock in sched_htb. This is pretty cool, if you ask me! :)
> 
> -Toke
> 

Thanks, I may use this to revisit doing netem in eBPF (xnetem).
Not having this feature was a show stopper at the time.
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] How can I tell if fq_codel is running?

[Stephen Hemminger]

On Tue, 30 Jul 2019 15:34:21 -0700
Kenneth Porter  wrote:

> I'm running CentOS 7 on a Dell R230 as a small office router. It's on a 
> slow ADSL link, 1.5 Mbps down and 150 kbps up. The kernel supports 
> fq_codel, as does the tg3 driver used by the interfaces.
> 
> In the past I've run the Wondershaper script and now I'm hoping the new 
> codel improves on that. I think it's installed but I don't see its name 
> in a "tc show". Should I see it there? Is there additional setup I need 
> to do?
> 
> The system seems responsive but I've been tasked with prioritizing the 
> network for certain users. Before, I set up some buckets in the 
> wondershaper for that. I'm now trying to figure out how that's supposed 
> to work with codel, which is ostensibly "knob-free".
> 
> [root@saruman ~]# tc -s class show dev em2
> class mq :1 root
>   Sent 127961591021 bytes 412946653 pkt (dropped 0, overlimits 0 
> requeues 2726)
>   backlog 0b 0p requeues 2726
> class mq :2 root
>   Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>   backlog 0b 0p requeues 0
> class mq :3 root
>   Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>   backlog 0b 0p requeues 0
> class mq :4 root
>   Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>   backlog 0b 0p requeues 0
> class mq :5 root
>   Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
>   backlog 0b 0p requeues 0
> [root@saruman ~]# sysctl net.core.default_qdisc
> net.core.default_qdisc = fq_codel

Try:
root@OpenWrt:~# tc qdisc show
qdisc noqueue 0: dev lo root refcnt 2 
qdisc mq 0: dev eth1 root 
qdisc fq_codel 0: dev eth1 parent :8 limit 10240p flows 1024 quantum 1514 
target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc fq_codel 0: dev eth1 parent :7 limit 10240p flows 1024 quantum 1514 
target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc fq_codel 0: dev eth1 parent :6 limit 10240p flows 1024 quantum 1514 
target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc fq_codel 0: dev eth1 parent :5 limit 10240p flows 1024 quantum 1514 
target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc fq_codel 0: dev eth1 parent :4 limit 10240p flows 1024 quantum 1514 
target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc fq_codel 0: dev eth1 parent :3 limit 10240p flows 1024 quantum 1514 
target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc fq_codel 0: dev eth1 parent :2 limit 10240p flows 1024 quantum 1514 
target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc fq_codel 0: dev eth1 parent :1 limit 10240p flows 1024 quantum 1514 
target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc mq 0: dev eth0 root 
qdisc fq_codel 0: dev eth0 parent :8 limit 10240p flows 1024 quantum 1514 
target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc fq_codel 0: dev eth0 parent :7 limit 10240p flows 1024 quantum 1514 
target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc fq_codel 0: dev eth0 parent :6 limit 10240p flows 1024 quantum 1514 
target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc fq_codel 0: dev eth0 parent :5 limit 10240p flows 1024 quantum 1514 
target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc fq_codel 0: dev eth0 parent :4 limit 10240p flows 1024 quantum 1514 
target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc fq_codel 0: dev eth0 parent :3 limit 10240p flows 1024 quantum 1514 
target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc fq_codel 0: dev eth0 parent :2 limit 10240p flows 1024 quantum 1514 
target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1514 
target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc noqueue 0: dev br-lan root refcnt 2 
qdisc noqueue 0: dev eth0.1 root refcnt 2 
qdisc noqueue 0: dev eth1.2 root refcnt 2 
qdisc mq 0: dev wlan1 root 
qdisc fq_codel 0: dev wlan1 parent :4 limit 10240p flows 1024 quantum 1514 
target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc fq_codel 0: dev wlan1 parent :3 limit 10240p flows 1024 quantum 1514 
target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc fq_codel 0: dev wlan1 parent :2 limit 10240p flows 1024 quantum 1514 
target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc fq_codel 0: dev wlan1 parent :1 limit 10240p flows 1024 quantum 1514 
target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc mq 0: dev wlan0 root 
qdisc fq_codel 0: dev wlan0 parent :4 limit 10240p flows 1024 quantum 1514 
target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc fq_codel 0: dev wlan0 parent :3 limit 10240p flows 1024 quantum 1514 
target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc fq_codel 0: dev wlan0 parent :2 limit 10240p flows 1024 quantum 1514 
target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
qdisc fq_codel 0: dev wlan0 parent :1 limit 10240p flows 1024 quantum 1514 
target 5.0ms interval 100.0ms memory_limit 4Mb ecn 
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

[Bloat] Blog post: "Do packet drops matter for TCP"

[Stephen Hemminger]

Good summary of bufferbloat from network operator point of view.

https://blog.ipspace.net/2019/06/do-packet-drops-matter-for-tcp.html?ck_subscriber_id=182966812
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague] Implementation and experimentation of TCP Prague/L4S hackaton at IETF104

[Stephen Hemminger]

On Wed, 20 Mar 2019 19:04:17 +
"Holland, Jake"  wrote:

> Hi Bob & Greg,
> 
> I agree there has been a reasonably open conversation about the L4S
> work, and thanks for all your efforts to make it so.
> 
> However, I think there's 2 senses in which "private" might be fair that
> I didn't see covered in your rebuttals (merging forks and including
> Greg's rebuttal by reference from here:
> https://lists.bufferbloat.net/pipermail/bloat/2019-March/009038.html )
> 
> Please note:
> I don't consider these senses of "private" a disqualifying argument
> against the use of L4S, though I do consider them costs that should be
> taken into account (and of course opinions may differ here).
> 
> With that said, I wondered whether either of you have any responses that
> speak to these points:
> 
> 
> 1. the L4S use of the ECT(1) codepoint can't be marked CE except by a
> patent-protected AQM scheduler.
> 
> I understand that l4s-id suggests the possibility of an alternate
> scheme.  However, comparing the MUSTs of the section 5 requirements
> with the claims made by the patent seems to leave no room for an
> alternate that would not infringe the patent claims, unless I'm missing
> something?
> 
> https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-06#section-5
> https://patents.google.com/patent/US20170019343A1/en
> 

Has anyone done a detailed investigation for prior art?
The patent office does not do a good job of looking for prior art,
and the parties in the patent process are not motivated to look.

Other vendors often are not interested either because their own house of
cards built on patents of previous research might come falling down.

___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] transparenty bridge/tap with fq_codel

[Stephen Hemminger]

On Mon, 11 Mar 2019 16:01:36 -0700
Dev  wrote:

> I built a transparent bridge on a Debian platform earlier running fq_codel 
> between eth2 and eth3 as br0 which seemed to improve throughput. 
> 
> Now, I’m wondering if there’s a way to copy some of those packets to another 
> onboard NIC eth4 for analysis on another box on the network. How significant 
> of a performance hit will this be on commodity hardware on my bridge 
> throughput, and/or what is best practices? Has anyone already done this and 
> made it work?

It is possible to do this with ingress qdisc and a mirred filter to mirror 
packets to another device. That is the Linux way to create SPAN ports on a 
bridge.
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] fq_codel on bridge throughput test/config

[Stephen Hemminger]

On Fri, 4 Jan 2019 12:33:28 -0800
Dev  wrote:

> Okay, thanks to some help from the list, I’ve configured a transparent bridge 
> running fq_codel which works for multiple subnet traffic. Here’s my setup:
> 
> Machine A ——— 192.168.10.200 — — bridge fq_codel machine B —— laptop C 
> 192.168.10.150
> Machine D ——— 192.168.3.50 — —| 
> 
> On Machine A:
> 
> straight gigE interface 192.168.10.200
> 
> Bridge Machine B: enp3s0 mgmt interface
>   enp2s0 bridge interface 1
>   enp1s0 bridge interface 2
>   br0 bridge for 1 and 2
>   
>   # The loopback network interface 
>   auto lo br0 
>   iface lo inet loopback 
> 
>   # The primary network interface 
>   allow-hotplug enp3s0 
>   iface enp3s0 inet static 
>address 172.16.0.5/24 
>gateway 172.16.0.5 
>   dns-nameservers 8.8.8.8
> 
>   iface enp1s0 inet manual 
>tc qdisc add dev enp1s0 root fq_codel 
> 
>iface enp2s0 inet manual 
>   tc qdisc add dev enp2s0 root fq_codel 
> 
># Bridge setup 
>   iface br0 inet static 
>   bridge_ports enp1s0 enp2s0 
>   address 192.168.3.75 
>   broadcast 192.168.3.255 
>   netmask 255.255.255.0 
>   gateway 192.168.3
> 
> note: I still have to run this command later, will troubleshoot at some point 
> (unless you have suggestions to make it work):
> 
> tc qdisc add dev enp1s0 root fq_codel
> 
> To start, my pings from Machine A to Laptop C were around 0.75 msec, then I 
> flooded the link from Machine A to Laptop C using:
> 
> dd if=/dev/urandom | ssh user@192.168.10.150 dd of=/dev/null
> 
> Then my pings went up to around 170 msec. Once I enabled fq_codel on the 
> bridge machine B, my pings dropped to around 10 msec.
> 
> Hope this helps someone else working on a similar setup.
> 
> - Dev
> 
> ___
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Applying a qdisc to a bridge device only impacts the local traffic
going to that bridge (ie br0). It has no impact on traffic transiting
through the bridge. Since normally bridge pseudo device is queueless
putting qdisc on br0 has no impact. In other words packets being transmitted
on br0 go direct to the underlying device, therefore even if you put a
qdisc on br0 it isn't going to do what you expect (unless you layer some
rate control into the stack).
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] ExLL: An Extremely Low-latency Congestion Control for Mobile Cellular Networks

[Stephen Hemminger]

On Wed, 5 Dec 2018 14:56:35 +0100
"Bless, Roland (TM)"  wrote:

> Hi,
> 
> Am 04.12.18 um 19:03 schrieb Dave Taht:
> > I guess someone could modify quic to do this also.
> > 
> > https://sci-hub.tw/https://dl.acm.org/citation.cfm?doid=3281411.3281430  
> 
> It seems that it isn't "extremely low-latency" anymore if the bottleneck
> is not the LTE link...
> Moreover, is FAST TCP meanwhile free to use?

FAST TCP is patent encumbered.
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] when does the CoDel part of fq_codel help in the real world?

[Stephen Hemminger]

On Fri, 30 Nov 2018 06:51:34 +0100 (CET)
Mikael Abrahamsson  wrote:

> On Thu, 29 Nov 2018, Stephen Hemminger wrote:
> 
> > The problem is that any protocol is mostly blind to the underlying 
> > network (and that can change).  To use dave's analogy it is like being 
> > put in the driver seat of a vehicle blind folded.  When you step on the 
> > gas you don't know if it is a dragster, jet fighter, or a soviet 
> > tractor. The only way a protocol can tell is based on the perceived 
> > inertia and when it runs into things...  
> 
> Actually, I've made the argument to IETF TCPM that this is not true. You 
> can be able to communicate earlier data from previous flows on the same 
> connection so that new flows can re-learn this.
> 
> If no flow the past hour has been able to run faster than 1 megabit/s and 
> always PMTUD to 1460 bytes MTU outbound, then there is good chance that 
> the next flow will encounter the same thing. Why not use this information 
> when guessing how things will behave going forward?
> 

Since a majority of the flows in the world are coming from mobile, this both
makes sense but is hard.  Linux used to remember TCP metrics but this was
removed with the flow cache.
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] when does the CoDel part of fq_codel help in the real world?

[Stephen Hemminger]

On Wed, 28 Nov 2018 23:35:53 -0800
Dave Taht  wrote:

> > As someone who works with moving packets, it's perplexing to me to
> > interact with transport peeps who seem enormously focused on
> > "goodput". My personal opinion is that most people would be better off
> > with 80% of their available bandwidth being in use without any
> > noticable buffer induced delay, as opposed to the transport protocol
> > doing its damndest to fill up the link to 100% and sometimes failing
> > and inducing delay instead.  

The problem is that any protocol is mostly blind to the underlying network
(and that can change).  To use dave's analogy it is like being put in
the driver seat of a vehicle blind folded.  When you step on the gas you
don't know if it is a dragster, jet fighter, or a soviet tractor. The only
way a protocol can tell is based on the perceived inertia and when
it runs into things...
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] one benefit of turning off shaping + fq_codel

[Stephen Hemminger]

On Tue, 27 Nov 2018 18:14:01 +
"Holland, Jake"  wrote:

> On 2018-11-23, 08:33, "Dave Taht"  wrote:
> Back in the day, I was a huge fan of async logic, which I first
> encountered via caltech's cpu and later the amulet.
> 
> https://en.wikipedia.org/wiki/Asynchronous_circuit#Asynchronous_CPU
> 
> ...
> 
> I've never really understood why it didn't take off, I think, in part,
> it doesn't scale to wide busses well, and that centrally clocked designs
> are how most engineers and fpgas and code got designed since. Anything
> with delay built into it seems hard for EEs to grasp but I wish I
> knew why, or had the time to go play with circuits again at a reasonable
> scale.
> 
> At the time, I was told the objections they got were that it uses about 2x 
> the space for the same functionality, and space usage is approximately linear 
> with the chip cost, and when under load you still need reasonable cooling, so 
> it was only considered maybe worthwhile for some narrow use cases.
> 
> I don't really know enough to confirm or deny the claim, and the use cases 
> may have gotten a lot closer to a good match by now, but this was the opinion 
> of at least some of the people involved with the work, IIRC.
> 
> 
> ___
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

With asynchronous circuits there is too much unpredictablity and instability.
Seem to remember there are even cases where two inputs arrive at once and 
output is non-determistic.
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] [Codel] found another good use for a queue today, possibly

[Stephen Hemminger]

On Mon, 26 Nov 2018 18:17:32 -0800
Dave Taht  wrote:

> I had been investigating various hashing schemes for speeding up the
> babeld routing protocol daemon, and dealing with annoying bursty cpu
> behavior (resizing memory, bursts of packets, thundering herds of
> retractions), and, although it's a tough slog of a read, this adds a
> queue to cuckoo hashing to good effect in flattening out insertion
> time.
> 
> https://arxiv.org/pdf/0903.0391.pdf
> 
> But for all I know it's dependent on angels dancing on saddles mounted
> on unicorns. I skip to the graphs for insertion time and go back to
> the text for another round...
> 
> "polylog(n)-wise Independent Hash Function". OK, my google-foo fails
> me: The authors use sha1, would something lighter weight suit?
> 
> 

The current favorite in DPDK land seems to be Cuckoo hashing.
It has better cache behavior than typical chaining.
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] vyatta in AT 5G gear

[Stephen Hemminger]

On Tue, 23 Oct 2018 09:38:10 +0200
Jonas Mårtensson  wrote:

> >
> > Linux Foundation announced Danos, but no code is yet available.
> >  
> 
> Yeah I know, but you seemed to know it is written in Go, which I can't find
> any information about, so I thought maybe you had more information.
> 
> /Jonas

I used to work at Brocade. Didn't work directly on the control handler but did
write lots of yang models for it. Yes it was in Go at the time.
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] vyatta in AT 5G gear

[Stephen Hemminger]

On Tue, 16 Oct 2018 17:46:44 -0700
Dave Taht  wrote:

> Jan Ceuleers  writes:
> 
> > On 16/10/2018 17:14, Dave Taht wrote:  
> >> And what was so wrong with the "everything as a file" model??  
> >
> > On a single box - not a lot.
> >
> > On a 5G network, which might consist of 10^3 - 10^5 boxes you need
> > aggregation and abstraction layers.  
> 
> I was mostly just venting. Still... I'd hoped that the next generation
> fios gear would get bufferbloat right... and with all the 5G boasting
> about the 1ms mac, that they'd get queuing delay below 20ms also.
> 
> I was also hoping the DOCSIS 3.1 line cards would get it right...
> 
> It has been a disappointing day.

At scale, you need a usable programming model; on box CLI or files don't work 
well.
Netconf/yang was built for that.
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] vyatta in AT 5G gear

[Stephen Hemminger]

On Tue, 16 Oct 2018 08:14:36 -0700
Dave Taht  wrote:

> On Tue, Oct 16, 2018 at 8:06 AM Stephen Hemminger
>  wrote:
> >
> > On Tue, 16 Oct 2018 11:59:18 +0200
> > Stefan Alfredsson  wrote:
> >  
> > > On 2018-10-16 11:31, Mikael Abrahamsson wrote:
> > >  
> > > > On Mon, 15 Oct 2018, Dave Taht wrote:
> > > >  
> > > >> Vyos (the open source fork of vyatta) was one of the first to add
> > > >> fq_codel support... I wonder
> > > >>
> > > >> http://linuxgizmos.com/att-releases-white-box-spec-for-its-linux-based-5g-routers/
> > > >>  
> > > >
> > > > Isn't Vyos just running the Linux kernel for forwarding? So they
> > > > received fq_codel for free when the Linux kernel got support for it?
> > > > They just had to make it configurable?
> > > >  
> > >
> > > Yes, according to this blog post,
> > > http://www.five-ten-sg.com/mapper/blog/Bufferbloat%20solved%20with%20Vyos
> > >
> > > "Now that Vyos "helium" is available with a Linux 3.13 kernel, the
> > > fq_codel queueing discipline can be used to solve many bufferbloat
> > > issues. The nightly "lithium" builds contain my patches that allow
> > > fq_codel to be used via the native Vyos configuration system."
> > >
> > > Anyway it's nice to see the Vyatta heritage living on in it's various
> > > forms (the AT "production hardened" Vyatta, to the Ubiquity EdgeMax
> > > and some UniFi devices, to the VyOS open version and now the future
> > > plans with dNOS -> DANOS.
> > >
> > > /Stefan
> > >
> > >
> > >  
> >
> > There are two basic components to network OS, the control plane and the
> > data plane.  VyOs has the old V1 which is filesystem based control plane
> > and kernel dataplane. DaNoS has yang/netconf database based control plane
> > (in Go) and DPDK (or switch offload?) based dataplane.  Ubiquity redid
> > the control plane as well, and uses their own hardware for dataplane.
> >
> > So more of "my grandfather's ax"...  
> 
> so in other words we should have done a dpdk version long ago. And even then,
> this white box brags of the "deep buffering" in the switch chip.
> 
> ... and I have an initial report of 2 seconds of buffering on one of
> the first 5G devices.
> 
> Sigh.
> 
> And what was so wrong with the "everything as a file" model??

Two things were an issue. At scale, the filesystem was a bottleneck and it
would have been harder to implement netconf/yang model as required by
the big boy market.

Filesystem works as toy. (see plan 9) 
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] vyatta in AT 5G gear

[Stephen Hemminger]

On Tue, 16 Oct 2018 11:59:18 +0200
Stefan Alfredsson  wrote:

> On 2018-10-16 11:31, Mikael Abrahamsson wrote:
> 
> > On Mon, 15 Oct 2018, Dave Taht wrote:
> >  
> >> Vyos (the open source fork of vyatta) was one of the first to add
> >> fq_codel support... I wonder
> >>
> >> http://linuxgizmos.com/att-releases-white-box-spec-for-its-linux-based-5g-routers/
> >>  
> >>  
> >
> > Isn't Vyos just running the Linux kernel for forwarding? So they 
> > received fq_codel for free when the Linux kernel got support for it? 
> > They just had to make it configurable?
> >  
> 
> Yes, according to this blog post, 
> http://www.five-ten-sg.com/mapper/blog/Bufferbloat%20solved%20with%20Vyos
> 
> "Now that Vyos "helium" is available with a Linux 3.13 kernel, the 
> fq_codel queueing discipline can be used to solve many bufferbloat 
> issues. The nightly "lithium" builds contain my patches that allow 
> fq_codel to be used via the native Vyos configuration system."
> 
> Anyway it's nice to see the Vyatta heritage living on in it's various 
> forms (the AT "production hardened" Vyatta, to the Ubiquity EdgeMax 
> and some UniFi devices, to the VyOS open version and now the future 
> plans with dNOS -> DANOS.
> 
> /Stefan
> 
> 
> 

There are two basic components to network OS, the control plane and the
data plane.  VyOs has the old V1 which is filesystem based control plane
and kernel dataplane. DaNoS has yang/netconf database based control plane
(in Go) and DPDK (or switch offload?) based dataplane.  Ubiquity redid
the control plane as well, and uses their own hardware for dataplane.

So more of "my grandfather's ax"...
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] lwn.net's tcp small queues vs wifi aggregation solved

[Stephen Hemminger]

On Thu, 21 Jun 2018 10:31:18 -0500
Caleb Cushing  wrote:

> actually... all of my devices, including my desktop connect through wifi
> these days... and only one of them isn't running some variant of linux.
> 

Sigh. My experience with wifi is that it is not stable enough for that.
Both AP's I have tried Linksys ACM3200 or Netgear WNDR3800 I still see random 
drop outs.
Not sure if these are device resets (ie workarounds) or other issues.

These happen independent of firmware (vendor, OpenWRT, or LEDE).
So my suspicion is the that Wifi hardware is shite and that firmware is trying
to workaround and mask the problem.
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] benefits of ack filtering

[Stephen Hemminger]

On Wed, 29 Nov 2017 10:41:41 -0800
Dave Taht  wrote:

> On Wed, Nov 29, 2017 at 10:21 AM, Juliusz Chroboczek  wrote:
> >> The better solution would of course be to have the TCP peeps change the
> >> way TCP works so that it sends fewer ACKs.  
> >
> > Which tends to perturb the way the TCP self-clocking feedback loop works,
> > and to break Nagle.  
> 
> Linux TCP is no longer particularly ack-clocked. In the post pacing,
> post sch_fq world, packets are released (currently) on a 1ms schedule.
> Support was recently released for modifying that schedule on a per
> driver basis, which turns out to be helpful for wifi.
> 
> see: https://www.spinics.net/lists/netdev/msg466312.html

Also TCP BBR has lost its initial luster since it is unfair and ignores
losses and ECN (see recent netdev paper).

___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] emulating non-duplex media in linux qdiscs

[Stephen Hemminger]

On Sun, Oct 8, 2017 at 6:54 PM, Dave Taht  wrote:

> I have been hacking away at netem for a while now in the hope that
> eventually - with a great deal more hacking - it could be used to more
> accurately emulate shared media like wifi and lte.
>
> (Some people try to describe these as simplex (which is not true
> because you can have multiple destinations), and they certainly are
> not duplex, so I tend to say non-duplex and still hope some better
> word emerges)
>
> So... one sticking point for me has been wanting to emulate the fact
> that on shared media, that you cannot transmit and receive at the same
> time; that these are "coupled" events, and what I'd like to be able to
> express might be something like:
>
> tc qdisc add dev eth0 root netem rate 100mbit coupled some_identifier
> ... some tc mirred magic for ifb here ...
> tc qdisc add dev ifb0 root netem rate 10mbit coupled the_same_identifier
>
> "some_identifier" would be a mutex of some sort, and I confess to
> not having much grip on the kernel outside of the net/sched directory.
>
> What facility would be best to try and leverage? It would be created
> (globally) on first use, ref-counted (thus destroyed when it goes to
> zero), atomically updated... posix shared memory seems too heavyweight
> to use
>
> --
>
> Dave Täht
> CEO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-669-226-2619
> ___
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat


Since qdisc only see output packets, maybe a overlay device (like a tunnel
or vlan),
would be closer to what you want.
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] fq_codel is *five* years old today

[Stephen Hemminger]

Rich, you gave a good "glass is half full" story but my experience is currently
much worse. Here is the "glass is mostly empty" story.

> There have been so many improvements in so many ways...
> 
> - Cake is the acknowledged winner in SQM

But is still not upstream and requires tuning. In real world, SQM needs to be
adaptive and/or require no tuning. Fq_codel is still way ahead on this.

> 
> - Make Wi-Fi Fast has removed a couple orders of magnitude of latency in the 
> Wi-Fi stack, and Airtime Fairness has eliminated the Wi-Fi Paradox.
But still only works with one chipset, and not in consumer products.

> 
> - All this is Implemented in LEDE, so it's available as a (somewhat) 
> straightforward install on hundreds of different kinds of routers. As a side 
> note, LEDE and OpenWrt are working toward a merger, with a unification of the 
> development efforts.
The merger is wonderful. But my experiments with OpenWRT/LEDE and DD-Wrt all 
show that the OEM firmware has
more stable Wifi on Linksys ACM3200. It seems current code base is about where 
Cerowrt was 2 years ago.


> - A few commercial products - notably IQrouter and Ubiquiti - are shipping a 
> good SQM implementation in their products.
But these are are not the consumer products people get from Comcast, Fry's or 
Best Buy.

Don't confuse developing with deploying. The biggest challenge in the bloat 
world now is getting solutions deployed.

I hope the world will get better, it just seems to take longer than expected.
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

[Bloat] TCP BBR over LTE

[Stephen Hemminger]

Interesting talk at Netdev 2.1

Verizon testing TCP BBR over LTE. 
  https://www.netdevconf.org/2.1/session.html?chung

Compared TCP Cubic vs BBR while driving on highway from Boston
to New Jersey. 

Will post paper and youtube link when available
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] [Make-wifi-fast] [Cake] flent testers wanted prior to next release

[Stephen Hemminger]

On Tue, 31 Jan 2017 17:35:40 +0100
Toke Høiland-Jørgensen <t...@toke.dk> wrote:

> Stephen Hemminger <step...@networkplumber.org> writes:
> 
> > On Tue, 20 Dec 2016 11:02:44 -0800
> > Dave Taht <dave.t...@gmail.com> wrote:
> >  
> >> Toke has been busy adding new features to the flent network test tool.
> >> I consider it *almost* stable enough for a new release. Some of the
> >> development has been focused on making the flent-gui much faster and
> >> more responsive (as our data sets have got larger), others on
> >> providing better default command line output, and there's other fixes
> >> across the board, including QT5 support.
> >> 
> >> In particular, I fear we've broken windows users of flent. I would
> >> dearly like it if some more folk out there using flent could pull the
> >> latest git version and see if there are any new bugs or regressions in
> >> it, any of the the 87 tests, and the plotters, before freezing the
> >> code for a new year's release.
> >> 
> >> github: https://github.com/tohojo/flent
> >> main site: https://flent.org/
> >> 
> >> While you are at it, please feel free to stress out any of the flent
> >> servers as a target, give the new cake a shot and compare it against
> >> htb+fq_codel or your aqm of choice, or fiddle with the new wifi code,
> >> and share your data. tcp_nup, tcp_ndown, rrul, rrul_be remain the main
> >> tests, but the square wave one is turning out interesting :)
> >> 
> >> And if you have any feature requests or bugs to file, please get them
> >> in soon to the github!
> >> 
> >> We could also use better documentation and tutorials for use... some
> >> more example scripts leveraging things like the cpu_stats and
> >> qdisc_stats tools, and so on,
> >> 
> >> Active public servers include:
> >> 
> >> flent-freemont.bufferbloat.net
> >> ( this is colocated with flent-bbr-west which has bbr on by default - an
> >> interesting test might be testing both these servers at the same time
> >> via the rtt_fair* tests from your location)
> >> 
> >> flent-dallas.bufferbloat.net
> >> flent-london.bufferbloat.net
> >> flent-tokyo.bufferbloat.net
> >> flent-newark.bufferbloat.net
> >> 
> >> There are also netperf-west and netperf-east and netperf-eu and no
> >> doubt a few others.
> >> 
> >> We plan to add a few BBR enabled servers over the holidays.
> >> 
> >> The changelog so far:
> >> 
> >> 
> >> - Support PyQt5 in the GUI (and prefer it over PyQt4). If PyQt5 is not
> >> found, fall back to PyQt4.
> >> 
> >> 
> >> - Add new SummaryFormatter that outputs mean and median values for
> >> each data series. This is the new default formatter, meaning that its
> >> output will be shown after a test run if no other formatter (or plot)
> >> is specified.
> >> 
> >> - Support multiprocessing in the GUI. When loading several plots at
> >> once, plotting will now be passed off to separate worker processes.
> >> 
> >>   This allows plotting to use all the available processors on the
> >> machine, and speeds up loading of many plots tremendously (initial
> >> load is sped up by an order of magnitude). This change also means that
> >> re-plotting on config changes will be done dynamically in the
> >> background, which makes the GUI more responsive.
> >> 
> >> - Make text completely black in the default colour scheme. This
> >> increases contrast, and helps legibility, especially on printed
> >> figures.
> >> 
> >> 
> >> - Some internal code changes: Port command line parser from the old
> >> optparse class to the newer argparse, and fix a bunch of linter
> >> 
> >>   
> >
> > Has anyone automated or orchestrated flent? I would love to get several
> > projects doing daily build flent runs. Both upstream kernel, net-next,
> > and Intel Clear Linux has nightly build and test.  
> 
> Flent has a built-in batch mode that lets you automate running several
> tests in a row including setup/teardown scripts etc... Is that what you
> mean?
> 
> -Toke

That is start, was hoping someone had already done the nightly run kind of 
environment.
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] [Cake] flent testers wanted prior to next release

[Stephen Hemminger]

On Tue, 20 Dec 2016 11:02:44 -0800
Dave Taht  wrote:

> Toke has been busy adding new features to the flent network test tool.
> I consider it *almost* stable enough for a new release. Some of the
> development has been focused on making the flent-gui much faster and
> more responsive (as our data sets have got larger), others on
> providing better default command line output, and there's other fixes
> across the board, including QT5 support.
> 
> In particular, I fear we've broken windows users of flent. I would
> dearly like it if some more folk out there using flent could pull the
> latest git version and see if there are any new bugs or regressions in
> it, any of the the 87 tests, and the plotters, before freezing the
> code for a new year's release.
> 
> github: https://github.com/tohojo/flent
> main site: https://flent.org/
> 
> While you are at it, please feel free to stress out any of the flent
> servers as a target, give the new cake a shot and compare it against
> htb+fq_codel or your aqm of choice, or fiddle with the new wifi code,
> and share your data. tcp_nup, tcp_ndown, rrul, rrul_be remain the main
> tests, but the square wave one is turning out interesting :)
> 
> And if you have any feature requests or bugs to file, please get them
> in soon to the github!
> 
> We could also use better documentation and tutorials for use... some
> more example scripts leveraging things like the cpu_stats and
> qdisc_stats tools, and so on,
> 
> Active public servers include:
> 
> flent-freemont.bufferbloat.net
> ( this is colocated with flent-bbr-west which has bbr on by default - an
> interesting test might be testing both these servers at the same time
> via the rtt_fair* tests from your location)
> 
> flent-dallas.bufferbloat.net
> flent-london.bufferbloat.net
> flent-tokyo.bufferbloat.net
> flent-newark.bufferbloat.net
> 
> There are also netperf-west and netperf-east and netperf-eu and no
> doubt a few others.
> 
> We plan to add a few BBR enabled servers over the holidays.
> 
> The changelog so far:
> 
> 
> - Support PyQt5 in the GUI (and prefer it over PyQt4). If PyQt5 is not
> found, fall back to PyQt4.
> 
> 
> - Add new SummaryFormatter that outputs mean and median values for
> each data series. This is the new default formatter, meaning that its
> output will be shown after a test run if no other formatter (or plot)
> is specified.
> 
> - Support multiprocessing in the GUI. When loading several plots at
> once, plotting will now be passed off to separate worker processes.
> 
>   This allows plotting to use all the available processors on the
> machine, and speeds up loading of many plots tremendously (initial
> load is sped up by an order of magnitude). This change also means that
> re-plotting on config changes will be done dynamically in the
> background, which makes the GUI more responsive.
> 
> - Make text completely black in the default colour scheme. This
> increases contrast, and helps legibility, especially on printed
> figures.
> 
> 
> - Some internal code changes: Port command line parser from the old
> optparse class to the newer argparse, and fix a bunch of linter
> 
> 

Has anyone automated or orchestrated flent? I would love to get several
projects doing daily build flent runs. Both upstream kernel, net-next,
and Intel Clear Linux has nightly build and test.

Also Microsoft's internal Linux testing of kernel devices, maybe
even the Azure development cycle. Though I doubt those results could
be shared.
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] Fixing bufferbloat in 2017

[Stephen Hemminger]

My experience has been that the media and developer attention span is short
lived, and maybe that is part of the problem. Gaming is a niche market, and 
therefore is easily ignored; plus the classic gaming market is dying and I am 
not sure anyone is really investing in it. 

The current hot topic use case seems to be machine learning and voice 
interaction. I wonder if we could build a use case something related to that? 
"Ok Google, my house is on fire!!" -- "Sorry can not call fire department, 
network is congested".


___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] fixing bufferbloat in 2017

[Stephen Hemminger]

> BQL for vmxnet3 (if possible). Virtual router are becoming common.

BQL could be implemented in vmxnet3, but probably not in virtio.
virtio defers freeing packets to try and have better cache behavior
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] QCut - "Understanding on-device bufferbloat for cellular upload", Guo et al

[Stephen Hemminger]

On Tue, 15 Nov 2016 12:00:29 -0500
Rich Brown  wrote:

> Google Alerts sent a link to this paper: "Understanding on-device bufferbloat 
> for cellular upload" available at: http://dl.acm.org/citation.cfm?id=2987490 
> (ACM Paywall...)
> 
> In my two-minute skim of the paper, I see they describe how latency gets bad 
> on cellular devices (no kidding) and then propose QCut, a queue inserted 
> between the qdisc and the hardware. They say:
> 
> ---
> ... QCut operates in the kernel space and takes as input only information of 
> buffer occupancy and transmission statistics, which is exposed by most 
> cellular radio firmware from Qualcomm and likely other vendors.
> 
> Since directly limiting the firmware buffer occupancy is difficult, QCut 
> controls the firmware queuing delay indirectly in the kernel by controlling 
> how fast packets from Qdisc flow into the firmware buffer. QCUT estimates the 
> radio firmware buffer occupancy and queuing delay to decide the transmission 
> of packets to the firmware dynamically...
> ---
> 
> Their analysis and charts show that QCut helps a lot.
> 
> I found it interesting that they make measurements with both weak and strong 
> signal strength, to indicate the hits that are caused by differing signal 
> strength.
> 
> Enjoy!
> 
> Rich
> ___
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Reinvention of BQL?
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] DualPI2 qdisc implementation available for Linux

[Stephen Hemminger]

Why not submit it upstream where it can get more review and will get picked
up by more users?


On Fri, Jul 29, 2016 at 2:08 AM, De Schepper, Koen (Nokia - BE) <
koen.de_schep...@nokia-bell-labs.com> wrote:

> Hi all,
>
> For those who missed the L4S BoF, the PI2 AQM with DualQ option is
> available on the following git: https://github.com/olgabo/dualpi2
>
> PI2 (PI Improved with a Square) is a simplification of PIE (PI Enhanced)
> with the advantage of also supporting L4S congestion controls (like DCTCP).
> PI2 controls by default a single queue, with a common target (default
> 20ms). To get the most out of the L4S traffic you need a second L4S queue
> and a coupled AQM. PI2 supports this by specifying the "dualq" option. The
> L4S queue is having an immediate step ECN marker at 1ms, while the classic
> queue still has the 20ms target (with mark/drop coupled back to both L4S
> and Classic).
>
> Note that PI2 supports DCTCP, but DCTCP is not the target to be used on
> the internet. The TCP-Prague requirements (https://tools.ietf.org/html/
> draft-briscoe-tsvwg-aqm-tcpm-rmcat-l4s-problem-02#appendix-A) defines
> what is needed to have an end-system CC implementation which behaves
> similar to what FQ-X achieves in the network. FQ-X is a way the network can
> correct the behavior of classic TCP CCs, TCP-Prague is the end-system
> alternative that keeps the network simple and transport layer independent.
>
> Feel free to try it out, and join in developing a TCP CC that meets the
> TCP-Prague requirements.
>
> Regards,
> Koen.
>
>
> ___
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] [Cake] are anyone playing with dpdk and vpp?

[Stephen Hemminger]

DPDK gets impressive performance on large systems (like 14M packets/sec per
core), but not convinced on smaller systems.
Performance depends on having good CPU cache. I get poor performance on
Atom etc.
Also driver support is limited (mostly 10G and above)

On Wed, Apr 27, 2016 at 12:28 PM, Aaron Wood  wrote:

> I'm looking at DPDK for a project, but I think I can make substantial
> gains with just AF_PACKET + FANOUT and SO_REUSEPORT.  It's not clear to my
> yet how much DPDK is going to gain over those (and those can go a long way
> on higher-powered platforms).
>
> On lower-end systems, I'm more suspicious of the memory bus (and the cache
> in particular), than I am the raw CPU power.
>
> -Aaron
>
> On Wed, Apr 27, 2016 at 11:57 AM, Dave Taht  wrote:
>
>> https://fd.io/technology seems to have come a long way.
>>
>> --
>> Dave Täht
>> Let's go make home routers and wifi faster! With better software!
>> http://blog.cerowrt.org
>> ___
>> Bloat mailing list
>> Bloat@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/bloat
>>
>
>
> ___
> Cake mailing list
> c...@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cake
>
>
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] Hardware upticks

[Stephen Hemminger]

On Tue, 5 Jan 2016 11:37:02 -0800
Dave Täht  wrote:

> 
> 
> On 1/5/16 11:29 AM, Steinar H. Gunderson wrote:
> > On Tue, Jan 05, 2016 at 10:57:13AM -0800, Dave Täht wrote:
> >> Context switch time is probably one of the biggest hidden nightmares in
> >> modern OOO cpu architectures - they only go fast in a straight line. I'd
> >> love to see a 1ghz processor that could context switch in 5 cycles.
> > 
> > It's called hyperthreading? ;-)
> > 
> > Anyway, the biggest cost of a context switch isn't necessarily the time used
> > to set up registers and such. It's increased L1 pressure; your CPU is now
> > running different code and looking at (largely) different data.
> 
> +10.
> 
> A L1/L2 Icache dedicated to interrupt processing code could make a great
> deal of difference, if only cpu makers and benchmarkers would make
> CS time something we valued.
> 
> Dcache, not so much, except for the intel architectures which are now
> doing DMA direct to cache. (any arms doing that?)
> 
> > /* Steinar */
> > 
> ___
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Intel has some new Cache QoS stuff that allows configuring how much
cache is allowed per context.  But of course it is only on the 
newest/latest/unoptinium
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] Hardware upticks

[Stephen Hemminger]

On Wed, 6 Jan 2016 01:22:13 +0100
"Steinar H. Gunderson" <sgunder...@bigfoot.com> wrote:

> On Tue, Jan 05, 2016 at 04:06:03PM -0800, Stephen Hemminger wrote:
> > The expensive part is often having to save and restore all the state in
> > registers and other bits on context switch.
> 
> Are you sure? There's not really all that much state to save, and all I've
> been taught before says the opposite.
> 
> Also, I've never ever seen the actual context switch turn up high in a perf
> profile.  Is this because of some sampling artifact?

Yes, especially with Intel processors getting more and more SSE/floating point
registers.
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] Hardware upticks

[Stephen Hemminger]

On Wed, 6 Jan 2016 01:55:37 +0100
"Steinar H. Gunderson" <sgunder...@bigfoot.com> wrote:

> On Tue, Jan 05, 2016 at 04:53:14PM -0800, Stephen Hemminger wrote:
> >> Also, I've never ever seen the actual context switch turn up high in a perf
> >> profile.  Is this because of some sampling artifact?
> > Yes, especially with Intel processors getting more and more SSE/floating 
> > point
> > registers.
> 
> But those are not saved on context switch to the kernel, no? (Thus the rule
> of no floating-point in the kernel.) Only if you switch between userspace
> processes

Right that just punts the work to the kernel when it context switches
to next process.
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] bufferbloat email list server upgrade going slow and badly

[Stephen Hemminger]

On Mon, 4 Jan 2016 15:38:05 -0800
Dave Täht  wrote:

> all the bufferbloat.net servers are in the process of migrating to a
> new co-location facility. the lists - if not the archives - should be
> alive again, at least.
> 
> There is a stupid bug somewhere stopping the archived pages from
> making the web.
> 
> I am concerned if anyone's bloat email is now auto-ending up in a spam
> folder? (if so, please send response privately) (this email is also a
> test)
> 
> The redmine webserver move is troublesome, also.
> 
> Further more, the mailman web service is under persistent attack by
> a set of 286 (so far) ips flooding it with subscription requests, and
> linode itself has been under enormous attack over the holidays...
> 
> there will be ssl updates and the like once all this is sorted but it
> might take a week or two more. I might give up and try another co-lo also.
> 
> In the meantime, why not top it all off with breakfast at milliways?
> 
> ___
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

Talk to davem, maybe kernel.org would be safer/better more robust?
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] TCP congestion detection - random thoughts

[Stephen Hemminger]

You just reinvented delay based congestion control. 
This has been tried int many forms dating back to TCP Vegas.
  https://en.wikipedia.org/wiki/TCP_Vegas

Unfortunately, it often failed in practice (which no one ever
wanted to publish), Some of the reasons are:
* Delay based CC is sensitive to cross traffic congestion
  where the perceived congestion event was not caused by
  that flow. I.e other elephant stomps on ant.
* Delay based CC is not aggressive enough to compete with
  loss-based CC. Vegas flows lose to Reno.
* Delay based CC requires careful tuning, one variant was FAST TCP
  which was highly tuned for 1G networks in research. It went proprietary
  never heard how well it works in modern networks.
* Delay based CC was sensitive to middleboxes, polling intervals
  and other timing effects in the wild.
* RTT data has a noise to signal ratio, a flow has to be
  consistently maintaining a given rate in order to get
  consistent feedback.

Google has some delay based congestion control that is promised
to be released some time, I am waiting.
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] General Bufferbloat Testing Document.

[Stephen Hemminger]

On Fri, 15 May 2015 12:16:56 -0400
Jim Gettys j...@freedesktop.org wrote:

 Even before I knew about the wonderful DSLreports bufferbloat test, I had
 started working on a document to help people like that (e.g. Ookla)
 understand how to do bufferbloat testing.  The document also grew a bit
 beyond that topic, by the time it was done
 
 The document is at:
 
 https://docs.google.com/document/d/1z5NN4WRKQKK-RtxtKR__XIwkybvsKEmunek2Ezdw_90/edit?usp=sharing
 
 Comments welcome.
 
 It's intended long term home is the bufferbloat.net wiki, but I've found
 Google doc's commenting feature really useful.
- Jim

Great to see, I think it does a good job of being detailed without 
overwhelmingly
research oriented.

What makes you believe SPDY and QUIC will be better than TCP? I know they do
pacing but if they get the rate estimation wrong or get hit by transient 
congestion
it could have same failing that doomed TCP Vegas.
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] Question about Buffer Bloat and Verizon Fios

[Stephen Hemminger]

On Wed, 08 Apr 2015 09:56:48 -0700
Alex Elsayed eternal...@gmail.com wrote:

 Dave Taht wrote:
 
  On Wed, Apr 8, 2015 at 5:05 AM,  ga...@djlatino.com wrote:
 snip
  Reason is Verizon has a combo router/modem.
  
  They CAN be configured in a bridge mode if you desire, but you can
  usually just put cero in front of it to do everything else
  (shaping/wifi) and be golden.
  
  See the intertubes for how to switch to bridging verizon gear (if you
  want to). It is overly complex.
 
 snip
 
 There's actually even more nuance to it than that:
 
 FiOS (although, I thought that was owned by Frontier now? It certainly was 
 when I had it up here [.wa.us]) with TV service requires MoCA (Multimedia 
 over CoAxial), and thus requires the use of their (awful) combined 
 router/wifi/MoCA bridge that connects to the ONT (Optical Network Terminal) 
 over MoCA. In that case, you need to set it up to bridge to your own router 
 behind it, c.
 
 However, if you are using internet/phone-only FiOS service, you can get the 
 service technician to hook your router directly to the ONT via Ethernet, as 
 it supports both.
 
 ___
 Bloat mailing list
 Bloat@lists.bufferbloat.net
 https://lists.bufferbloat.net/listinfo/bloat

I am using the weird double bridge through the FioS router into a
Netgear WNDR3800 and it is stable. The issue is that the Actiontec
box does not bring everything back up after power cycle (so it is on
a UPS).

An alternative is to use a pair of MOCA modems to do MOCA to Ethernet
(and back to get to the DVR box). But it requires a bunch of changing
of MOCA frequencies; so I haven't gotten around to setting it up.
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] Question about Buffer Bloat and Verizon Fios

[Stephen Hemminger]

I got a pair at Frys. The issue is that MOCA frequency from the ONT to the
Router is different than the frequency from the Router to the DVR.
To change the frequency requires creating a standalone network and doing
the 192.168 web access thing.

On Wed, Apr 8, 2015 at 6:24 PM, Antonio Ortiz ga...@djlatino.com wrote:

 I'm going to order them from Amazon. A pair of moca is $110.00. Thanks

 Sent from my iPhone

  On Apr 8, 2015, at 6:18 PM, Stephen Hemminger 
 step...@networkplumber.org wrote:
 
  On Wed, 08 Apr 2015 09:56:48 -0700
  Alex Elsayed eternal...@gmail.com wrote:
 
  Dave Taht wrote:
 
  On Wed, Apr 8, 2015 at 5:05 AM,  ga...@djlatino.com wrote:
  snip
  Reason is Verizon has a combo router/modem.
 
  They CAN be configured in a bridge mode if you desire, but you can
  usually just put cero in front of it to do everything else
  (shaping/wifi) and be golden.
 
  See the intertubes for how to switch to bridging verizon gear (if you
  want to). It is overly complex.
 
  snip
 
  There's actually even more nuance to it than that:
 
  FiOS (although, I thought that was owned by Frontier now? It certainly
 was
  when I had it up here [.wa.us]) with TV service requires MoCA
 (Multimedia
  over CoAxial), and thus requires the use of their (awful) combined
  router/wifi/MoCA bridge that connects to the ONT (Optical Network
 Terminal)
  over MoCA. In that case, you need to set it up to bridge to your own
 router
  behind it, c.
 
  However, if you are using internet/phone-only FiOS service, you can get
 the
  service technician to hook your router directly to the ONT via
 Ethernet, as
  it supports both.
 
  ___
  Bloat mailing list
  Bloat@lists.bufferbloat.net
  https://lists.bufferbloat.net/listinfo/bloat
 
  I am using the weird double bridge through the FioS router into a
  Netgear WNDR3800 and it is stable. The issue is that the Actiontec
  box does not bring everything back up after power cycle (so it is on
  a UPS).
 
  An alternative is to use a pair of MOCA modems to do MOCA to Ethernet
  (and back to get to the DVR box). But it requires a bunch of changing
  of MOCA frequencies; so I haven't gotten around to setting it up.
  ___
  Bloat mailing list
  Bloat@lists.bufferbloat.net
  https://lists.bufferbloat.net/listinfo/bloat
 

___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] The Dark Problem with AQM in the Internet?

[Stephen Hemminger]

On Sat, 30 Aug 2014 09:05:58 +0300
Jonathan Morton chromati...@gmail.com wrote:

 
 On 29 Aug, 2014, at 5:37 pm, Jerry Jongerius wrote:
 
  did you check to see if packets were re-sent even if they weren't lost? on 
  of
  the side effects of excessive buffering is that it's possible for a packet 
  to
  be held in the buffer long enough that the sender thinks that it's been
  lost and retransmits it, so the packet is effectivly 'lost' even if it 
  actually
  arrives at it's destination.
  
  Yes.  A duplicate packet for the missing packet is not seen.
  
  The receiver 'misses' a packet; starts sending out tons of dup acks (for all
  packets in flight and queued up due to bufferbloat), and then way later, the
  packet does come in (after the RTT caused by bufferbloat; indicating it is
  the 'resent' packet).  
 
 I think I've cracked this one - the cause, if not the solution.
 
 Let's assume, for the moment, that Jerry is correct and PowerBoost plays no 
 part in this.  That implies that the flow is not using the full bandwidth 
 after the loss, *and* that the additive increase of cwnd isn't sufficient to 
 recover to that point within the test period.
 
 There *is* a sequence of events that can lead to that happening:
 
 1) Packet is lost, at the tail end of the bottleneck queue.
 
 2) Eventually, receiver sees the loss and starts sending duplicate acks (each 
 triggering CA_EVENT_SLOW_ACK path in the sender).  Sender (running Westwood+) 
 assumes that each of these represents a received, full-size packet, for 
 bandwidth estimation purposes.
 
 3) The receiver doesn't send, or the sender doesn't receive, a duplicate ack 
 for every packet actually received.  Maybe some firewall sees a large number 
 of identical packets arriving - without SACK or timestamps, they *would* be 
 identical - and filters some of them.  The bandwidth estimate therefore 
 becomes significantly lower than the true value, and additionally the RTO 
 fires and causes the sender to reset cwnd to 1 (CA_EVENT_LOSS).
 
 4) The retransmitted packet finally reaches the receiver, and the ack it 
 sends includes all the data received in the meantime (about 3.5MB).  This is 
 not sufficient to immediately reset the bandwidth estimate to the true value, 
 because the BWE is sampled at RTT intervals, and also includes low-pass 
 filtering.
 
 5) This ends the recovery phase (CA_EVENT_CWR_COMPLETE), and the sender 
 resets the slow-start threshold to correspond to the estimated 
 delay-bandwidth product (MinRTT * BWE) at that moment.
 
 6) This estimated DBP is lower than the true value, so the subsequent 
 slow-start phase ends with the cwnd inadequately sized.  Additive increase 
 would eventually correct that - but the key word is *eventually*.
 
  - Jonathan Morton

Bandwidth estimates by ack RTT is fraught with problems. The returning ACK can 
be
delayed for any number of reasons such as other traffic or aggregation. This 
kind
of delay based congestion control suffers badly from any latency induced in the 
network.
So instead of causing bloat, it gets hit by bloat.

___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

[Bloat] T-Mobile LTE buffer bloat

[Stephen Hemminger]

I got one of these Samsung LTE hotspot.
Not surprisingly it has huge bloat and a stupid http proxy
that netalyzer claims rewrites images.
 
Bandwidth: Up 1.6 Mbit/sec Down 4.3Mbit/sec
Latency: 140ms 0% loss
Buffering: Uplink 5100ms Down 1800ms

How can the uplink side be so bad! 5 seconds???

Might even return it as defective. You can't even add a review on their
website. Probably they would end taking down like Apple.
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] T-Mobile LTE buffer bloat

[Stephen Hemminger]

On Thu, 31 Oct 2013 00:34:12 +0100 (CET)
Mikael Abrahamsson swm...@swm.pp.se wrote:

 On Wed, 30 Oct 2013, Stephen Hemminger wrote:
 
  Not surprisingly it has huge bloat and a stupid http proxy that 
  netalyzer claims rewrites images.
 
 This could be done in the provider network, did you try it without the 
 thingie?

There is zero configuration of proxy, it is just a LTE - wifi hotspot.
I suspect some caching (or commercial interest) at their end.



___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] sweet tcp

[Stephen Hemminger]

On Tue, 9 Jul 2013 19:38:40 +0200
Jaume Barcelo jaume.barc...@upf.edu wrote:

 To be more specific, TCP would be at any time increasing or decreasing
 the congestion window. In other words, it will be moving in one
 direction (right or left) along the x axis of Fig. 1 of Getty's paper.
 Each RTT, the performance is measured in terms of delay and
 throughput. If there is a performance improvement, we keep moving in
 the same direction. If there is a performance loss, we change the
 direction.

TCP can not get a reliable measure of RTT. This is the whole reason
that delay based algorithms are not widely deployed and don't work
outside the lab. The issue is that other traffic perturbs RTT measurements
have a huge variance on a real network  (more noise than signal).
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] Google's experiments with QUIC and SPDY

[Stephen Hemminger]

On Fri, 28 Jun 2013 01:33:22 -0700
Hal Murray hmur...@megapathdsl.net wrote:

 Has anybody tried this stuff in a bloat sensitive environment?
 
 Experimenting with QUIC
 http://blog.chromium.org/2013/06/experimenting-with-quic.html
 
QUIC (Quick UDP Internet Connections) is an early-stage network
 protocol we are experimenting with that runs a stream multiplexing
 protocol over a new flavor of Transport Layer Security (TLS) on top of
 UDP instead of TCP. QUIC combines a carefully selected collection of
 techniques to reduce the number of round trips we need as we surf the
 Internet. You can learn more in the design document, but here are some
 of the highlights: ...
 
 
 SPDY: An experimental protocol for a faster web 
   http://www.chromium.org/spdy/spdy-whitepaper
 
 As part of the Let's make the web faster initiative, we are experimenting 
 with alternative protocols to help reduce the latency of web pages. One of 
 these experiments is SPDY (pronounced SPeeDY), an application-layer 
 protocol for transporting content over the web, designed specifically for 
 minimal latency.  In addition to a specification of the protocol, we have 
 developed a SPDY-enabled Google Chrome browser and open-source web server. In 
 lab tests, we have compared the performance of these applications over HTTP 
 and SPDY, and have observed up to 64% reductions in page load times in SPDY. 
 We hope to engage the open source community to contribute ideas, feedback, 
 code, and test results, to make SPDY the next-generation application protocol 
 for a faster web.
 
 
From an application point of view TCP is just a latency inducer.
Because end-to-end system engineering is hard, it natural to focus
on the problem at hand.

Is this a repeat of the story, we don't like slow start and flow control so 
let's
open lots of connections.
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] tc linklayer ADSL calc broken after commit 56b765b79 (htb: improved accuracy at high rates)

[Stephen Hemminger]

On Wed, 29 May 2013 08:52:04 -0700
Eric Dumazet eric.duma...@gmail.com wrote:

 On Wed, 2013-05-29 at 15:13 +0200, Jesper Dangaard Brouer wrote:
  I recently discovered that the (traffic control) tc linklayer
  calculations for ATM/ADSL have been broken by:
   commit 56b765b79 (htb: improved accuracy at high rates).
  
  Thus, people shaping on ADSL links, using e.g.:
   tc class add ... htb rate X ceil Y linklayer atm overhead 10
  
  Will no-longer get ATM cell tax/overhead adjusted.
  
  How can we solve/fix this?
  Perhaps we can change to use the stab system instead (as it does
  not seem to be broken by the commit).
  
  But how do we facilitate a change to use stab system (for all the
  scripts using the old option)?
  
  Can we change the iproute2/tc command to handle this transparently, or
  should we give an error/warning if someone uses tc and linklayer on
  a kernel above v.3.8. ?
  
  
  History:
   - My linklayer ATM changes appeared in kernel 2.6.24 (and iproute2 2.6.25)
   - The STAB changes appeared in kernel 2.6.27
  
 
 Hi Jesper
 
 stab suffers from the same problem : its table driven, so works only for
 packet smaller than a given size.
 
 I am not sure it will solve the ATM logic (with the 5 bytes overhead per
 48 bytes cell)
 
 btw, even on old kernels :


How bad is the failure? If it is fixed, will it break existing installations?

Which probably means, is anyone but the original developers ever using it
and therefore likely to notice?
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] Network test tools for many parallel/concurrent connections?

[Stephen Hemminger]

On Tue, 14 May 2013 15:48:38 +0200
Jesper Dangaard Brouer jbro...@redhat.com wrote:

 
 (I'm testing fq_codel and codel)
 
 I need a test tool that can start many TCP streams (1024).
 During/after the testrun I want to know if the connections got a fair
 share of the bandwidth.
 
 Can anyone recomment tools for this?
 
 After the test I would also like to, deep-dive analyse one of the TCP
 streams to see how the congestion window, outstanding-win/data is
 behaving.  Back in 2005 I used-to-use a tool  called
 tcptrace (http://www.tcptrace.org).
 Have any better tools surfaced?
 


You may want to look at some of the realistic load tools since
in real life not all flows are 100% of bandwidth and long lived.
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] Bufferbloat Paper

[Stephen Hemminger]

The tone of the paper is a bit of if academics don't analyze it to death
it must not exist. The facts are interesting, but the interpretation ignores
the human element. If human's perceive delay Daddy the Internet is slow, then
they will change their behavior to avoid the problem: it hurts when I download,
so I will do it later.

___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] [Codel] better testing, linux 3.6.1, cerowrt credits, other stuff

[Stephen Hemminger]

On Wed, 10 Oct 2012 19:27:11 -0400
Michael Richardson m...@sandelman.ca wrote:

 
  But: It became obvious fast that long RTT tests were needed,
  which I've been trying to establish the infrastructure to do
 
 toke I assume that by infrastructure you mean (netperf) servers
 toke far away? What would be needed for a test server in terms of
 toke resources (bandwidth and otherwise)? I could try and persuade
 toke my university to let me setup a test server on their network
 toke (which is in Denmark)...
 
 I interpret the question to mean networks where is there significant
 actual delay along them.  I seem to recall that there are some ways to
 do this Linux machines, but most commercial test equipment can simulate
 things, including dropping packets.
 I think, however, that we do not want/need and packets dropped, as then
 the bandwidth constraint would not be in the device under test.

netem can do all the stuff commercial gear can.
In fact, it is used by one of the commercial products!
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] high speed networking from userspace

[Stephen Hemminger]

On Tue, 13 Mar 2012 13:28:52 +0100
Hagen Paul Pfeifer ha...@jauu.net wrote:

 
 On Mon, 12 Mar 2012 17:09:44 -0700, Dave Taht dave.t...@gmail.com wrote:
 
  is the by the same guy that did QFQ, and the results are quite
  impressive. He (today) announced support for this interface for Linux.
  
  shades of VJ's 'network channels'!
 
 already implemented:
 
 see
 
 o http://www.ioremap.net/node/12
 o http://www.ioremap.net/taxonomy/term/6
 
 and all netdev discussions several years ago.
 
 HGN

User space networking works well for single application be it routing,
bridging, network trading, or single appliance. It doesn't work on a
multi-application environment (ie desktop). The gain is only because
the userspace code can choose to do less, but do it faster. So if you
want full stack, and firewall; don't bother.
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] [PATCH net-next] sch_red: Adaptative RED AQM

[Stephen Hemminger]

On Thu, 08 Dec 2011 17:06:03 +0100
Eric Dumazet eric.duma...@gmail.com wrote:

 Changes against our RED implementation are :
 
 max_p is no longer a negative power of two (1/(2^Plog)), but a Q0.32
 fixed point number, to allow full range described in Adatative paper.
 
 To deliver a random number, we now use a reciprocal divide (thats really
 a multiply), but this operation is done once per marked/droped packet
 when in RED_BETWEEN_TRESH window, so added cost (compared to previous
 AND operation) is near zero.
 
 dump operation gives current max_p value in a new TCA_RED_MAX_P
 attribute.
 
 Example on a 10Mbit link :
 
 tc qdisc add dev $DEV parent 1:1 handle 10: est 1sec 8sec red \
limit 40 min 3 max 9 avpkt 1000 \
burst 55 ecn adaptative bandwidth 10Mbit
 
 # tc -s -d qdisc show dev eth3
 ...
 qdisc red 10: parent 1:1 limit 40b min 3b max 9b ecn
 adaptative ewma 5 max_p=0.113335 Scell_log 15
  Sent 50414282 bytes 34504 pkt (dropped 35, overlimits 1392 requeues 0) 
  rate 9749Kbit 831pps backlog 72056b 16p requeues 0 
   marked 1357 early 35 pdrop 0 other 0
 
 
 Signed-off-by: Eric Dumazet eric.duma...@gmail.com

Is this backward compatible for users that don't specify
an adaptive parameter.
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] Not all the world's a WAN

[Stephen Hemminger]

On Wed, 17 Aug 2011 18:26:00 -0700
Patrick J. LoPresti lopre...@gmail.com wrote:

 Hello, BufferBloat crusaders.
 
 Permit me briefly to describe my application.  I have a rack full of
 Linux systems, all with 10GbE NICs tied together by a 10GbE switch.
 There are no routers or broader Internet connectivity.  (At least,
 none that matters here.)  Round trip ping times between systems are
 100 microseconds or so.
 
 Some of the systems are servers, some are clients.  Any single
 client may decide to slurp data from multiple servers.  For example,
 the servers could be serving up a distributed file system, so when a
 client accesses a file striped across multiple servers, it tries to
 pull data from multiple servers simultaneously.  (This is not my
 literal application, but it does represent the same access pattern.)
 
 The purpose of my cluster is to process data sets measured in hundreds
 of gigabytes, as fast as possible.  So, for my application:
 
  - Speed = Throughput (latency is irrelevant)
  - TCP retransmissions are a disaster, not least because
  - 200ms is an eternity
 
 
 The problem I have is this:  At 10 gigabits/second, it takes very
 little time to overrun even a sizable buffer in a 10GbE switch.
 Although each client does have a 10GbE connection, it is reading
 multiple sockets from multiple servers, so over short intervals the
 switch's aggregate incoming bandwidth (multiple 10GbE links from
 servers) is larger than its outgoing bandwidth (single 10GbE link to
 client).  If the servers do not throttle themselves -- say, because
 the TCP windows are large -- packets overrun the switch's buffer and
 get lost.

You need faster switches ;-)

 I have fixed this problem by using a switch with a _large_ buffer,
 plus using TCP_WINDOW_CLAMP on the clients to ensure the TCP window
 never gets very large.  This ensures that the servers never send so
 much data that they overrun the switch.  And it is actually working
 great; I am able to saturate all of my 10GbE links with zero
 retransmissions.

You just papered over the problem. If the mean queue length over
time is greater than one, you will lose packets. This maybe a case
where Ethernet flow control might help. It does have the problem
of head of line blocking when cascading switches but if the switch
is just a pass through it might help.

 I have not read all of the messages on this list, but I have read
 enough to make me a little nervous.  And thus I send this message in
 the hope that, in your quest to slay the buffer bloat dragon, you do
 not completely forget applications like mine.  I would hate to have to
 switch to Infiniband or whatever just because everyone decided that
 Web browsers are the only TCP/IP application in the world.
 

My view is this all about getting the defaults right for average
users. People with big servers will always end up tuning; thats what
they get paid for. Think of it as the difference between a Formula 1
car versus an average sedan. You want the sedan to just work, and
have all the traction control and rev limiters. For the F1 race
car, the driver knows best.

___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] Notes about hacking on AQMs

[Stephen Hemminger]

On Wed, 8 Jun 2011 10:52:07 -0600
Dave Taht dave.t...@gmail.com wrote:

 On Wed, Jun 8, 2011 at 10:41 AM, Eric Dumazet eric.duma...@gmail.com wrote:
  Le mercredi 08 juin 2011 à 09:51 -0600, Dave Taht a écrit :
 
 
  And they are *all* wrong to varying extents, which is why I like the
  'mondo classifier' idea for DSCP+firewalling mentioned earlier on this
  thread. Converging on several standards for packet marking vs the
  adhoc-ness of thousands of different partial solutions that now exist
  really makes sense to me.
 
  I can tell you there are hundred of different *valid* setups, especially
  in server farms, when you want some control of network trafic, now
  machines have Gb or 10Gb links...
 
 Well, there are hundreds of thousands of completely ad-hoc solutions
 of varying degrees of effacy.
 
 Getting it down to mere hundreds would be be a good start.
 
  Really, there is no one big thing that solves all problems,
  automatically.
 
 Oh, I agree.

It isn't just a Linux problem. Cisco and Juniper have been doing
QoS solutions for years. Like Linux there is the billions of knobs
version and the KISS version. The KISS versions are fair queueing
based. The problem is that the more complex QoS variants can't be
done in ASIC's and go down the software path.  Linux has the same
problem, the more complex QoS ends up requiring locks that embed
performance.
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] Goodput fraction w/ AQM vs bufferbloat

[Stephen Hemminger]

On Sat, 7 May 2011 19:39:22 +0300
Jonathan Morton chromati...@gmail.com wrote:

 
 On 7 May, 2011, at 1:10 am, Stephen Hemminger wrote:
 
  Rate = (MSS/RTT)*(1 / sqrt{p})
  
  where:
  Rate: is the TCP transfer rate or throughputd
  MSS: is the maximum segment size (fixed for each Internet path, typically 
  1460 bytes)
  RTT: is the round trip time (as measured by TCP)
  p: is the packet loss rate. 
 
 So if the loss rate is 1.0 (100%), the throughput is MSS/RTT.  If the loss 
 rate is 0, the throughput goes to infinity.  That doesn't seem right to me.

If loss rate is 0 there is no upper bound on TCP due to loss.
There are other limits on TCP throughput like window size but not limits
because of loss.
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

[Bloat] Fwd: Identifying TCP congestion control algorithms, and measurement results

[Stephen Hemminger]

This showed up on the end-to-end mailing list and might be of interest
to this group. It is interesting how many hosts are still using BIC
(probably RHEL/Centos 5). BIC is known to be broken and unfair.

From: Lisong Xu lisong...@gmail.com
Date: Tue, Mar 15, 2011 at 3:36 PM
Subject: [e2e] Identifying TCP congestion control algorithms, and
measurement results


Greetings,

We have recently developed a tool, called TCP Congestion Control
Avoidance Identification (CAAI), for actively identifying the TCP
congestion avoidance algorithm of a remote web server. We used CAAI to
measure the TCP algorithms of the top 5000 web sites in February 2011,
and got some preliminary results in which you might be interested.

# Only 16.85~25.58% of web servers still use the traditional AIMD.
# 14.36%, 15.82%, and 14.33% of web servers use BIC, CUBIC' (kernel
2.6.25 and before), and CUBIC (kernel 2.6.26 and after), respectively.
Total = 44.51%.
# 9.97% and 0.30~9.03% of web servers use CTCP' (Windows Server 2003
and XP Pro x64) and CTCP (Windows Server 2008, Vista, and 7),
respectively. Interestingly, CTCP' behaves very similar to HSTCP.
Total = 10.27~19%.
# Some web servers use non-default TCP algorithms (such as YEAH), some
web servers use some unknown TCP algorithms which are not available in
any major operating system family, and some web servers use abnormal
slow start algorithms.

More information is available at our project webpage
http://cse.unl.edu/~xu/research/TCPcensus.html.

Thanks
Lisong

--
Lisong Xu, Associate Professor
Computer Science  Engineering
University of Nebraska-Lincoln
http://cse.unl.edu/~xu
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat