[Bloat] hardware diversions

[Dave Taht]

Jonathan Morton  writes:

> Just to add - I think the biggest impediment to experimentation in
> asynchronous logic is the complete absence of convenient Muller
> C-element gates in the 74-series logic family.  If you want to build
> some, I recommend using NAND and OR gates as inputs to active-low SR
> flipflops.

Need millions of transistors, not dozens. :)

To me the biggest barrier is in tools. I'm still looking for the caltech
tool and language which really helped in thinking in this way, and I did
find it on github once, and it still seemed developed

And the field is not entirely dead, after all. I keep meaning to pick up
one of the new risc-v boards. Here's a async design of the risc-v... in
GO of all things. (I also really hate the universal adoption of java
amongst the circuit design folk... and I really loved the prospects of
chisel, except for the jvm dependency):

https://www.inf.pucrs.br/~calazans/publications/2017_MarcosSartori_EoTW.pdf

in the risc-v world, well, it's still trundling forward.

https://www.lowrisc.org/about/

This is pretty neat - standby is 2uA:

https://greenwaves-technologies.com/en/gap8-product/

And pulp is pretty neat.

https://pulp-platform.org//

Still, I liked xmos's stuff... rexcomputing hasn't surfaced in a while

In the last weird hardware embedded news of the day, you can get a old
intel compute stick for 34 dollars on ebay.

https://www.ebay.com/p/Intel-Compute-Stick-STCK1A8LFC-Intel-Atom-Z3735F-1-33GHz-8GB-PC-Stick-BOXSTCK1A8LFC/1102081?iid=153273128090=ps

they were painfully slow but fit on your keychain. The most modern
version of this design is
https://www.amazon.com/Intel-Compute-Computer-processor-BOXSTK2m3W64CC/dp/B01AZC4IKK/ref=sr_1_4?s=electronics=UTF8=1543389564=1-4=intel+compute+stick

2 cores, 4MB of cache, 64GB of flash... on your keychain.

I rather miss vga in that it would be better to be able to screw these in...
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

[Bloat] sparc (niagra) T1?

[Dave Taht]

anybody have an old ultrasparc T1 they are not using I could buy or
borrow (there's a few cheap ones on ebay)? or a sparc T-series in the
cloud?

I'd like to be able to check some assumptions about atomic memory models

-- 

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] when does the CoDel part of fq_codel help in the real world?

[Dave Taht]

Pete Heist  writes:

> On Nov 27, 2018, at 9:10 PM, Dave Taht 
> wrote:
> 
> EVEN with http 2.0/ I would be extremely surprised to learn that
> many
> websites fit it all into one tcp transaction.
> 
> There are very few other examples of TCP traffic requiring a low
> latency response.
> 
>
> This is the crux of what I was looking for originally- some of these
> examples along with what the impact is on TCP itself, and what that

One thing I USED to use a lot - and am trying to use again - is that
you can run emacs on multiple people's displays over X11 over TCP.

We used to have interactive apps like that for X, whiteboards, and the
like, and as best as I recall they worked *better* than doing it over
the web 

emacs over tcp works amazingly well coast to coast now, over wireguard
or ssh, with or without ecn enabled, over cubic. I've been meaning to
try bbr for a while now.

open-frame-other-display still works! if X could gain a mosh like and/or
quic-like underling transport it could make a comeback...

:crickets:

> actually means for people. I got that and then some. So for future
> readers, here’s an attempt to crudely summarize how CoDel
> (specifically as used in fq_codel) helps:
>
> For some TCP CC algorithms (probably still most, as of this writing):
> - reduces TCP RTT at saturation, improving interactivity for single
> flows with mixed bulk/interactive traffic like HTTP/2.0 or SSH
> - smooths data delivery (thus meeting user’s expectations) by avoiding
> queue overflow (and tail-drop) in larger queues
> - increases throughput efficiency with ECN (won’t tackle this further
> here!)
> - reduces memory requirements for TCP buffers
>
> Regardless of TCP CC algorithm:
> - reduces latency for “non-flow” traffic, such as that for some
> encrypted VPNs

We could use to put together a document of the size and scope of

http://www2.rdrop.com/~paulmck/scalability/paper/sfq.2002.06.04.pdf

someday.

That paper went into crc as a suitable hash function, btw... I'd like to
find a better hash... someday.

fq_codel itself is still awaiting the definitive paper with all the
plusses and minuses and measurements and "future work" - toke's recent
work taking apart the drop probability is only a piece of that elephant
- there's computational cost, of QFQ vs fq_codel (to this *day* I'd like
a QFQ + codel version), the sparse flow optimization (which could really
use a few less syllables to describe) vs straight DRR under more normal
traffic rather than traffic that attempts to highlight its usefulness,
with the common 300 byte quantum and 1514. There's the P4 work and FPGA
versions... I'd really like to revisit our LPCC paper and the fractal
self similar traffic paper. and the list goes on and on

anybody else 'roun here need a MSC or PHD?


>
>
> ___
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] when does the CoDel part of fq_codel help in the real world?

[Kathleen Nichols]

On 11/27/18 3:17 PM, Dave Taht wrote:
...
> 
> but now that we all have bedtime reading, I'm going to go back to
> hacking on libcuckoo. :)
> 
> 

Geez, louise. As if everyone doesn't have enough to do! I apologize. I
did not mean for anyone to completely read the links I sent, just look
at the relevant pictures and skim the relevant text.

___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

[Bloat] AFD

[Dave Taht]

Toke Høiland-Jørgensen  writes:

> Luca Muscariello  writes:
>
>> This procedure would allow to size FQ_codel but also SFQ.
>> It would be interesting to compare the two under this buffer sizing.
>> It would also be interesting to compare another mechanism that we have
>> mentioned during the defense
>> which is AFD + a sparse flow queue. Which is, BTW, already available in
>> Cisco nexus switches for data centres.
>
> One think I wondered about afterwards was whether or not it would be
> feasible (as in, easy to add, or maybe even supported in current
> versions) to tie in an AQM to an AFD-type virtual fairness queueing
> system? You could keep the AQM state variables along with the per-flow
> state and react appropriately. Any reason why this wouldn't work?

Just bookmarking this thought for now.

___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] when does the CoDel part of fq_codel help in the real world?

[Pete Heist]

> On Nov 27, 2018, at 9:10 PM, Dave Taht  wrote:
> 
> EVEN with http 2.0/ I would be extremely surprised to learn that many
> websites fit it all into one tcp transaction.
> 
> There are very few other examples of TCP traffic requiring a low
> latency response.

This is the crux of what I was looking for originally- some of these examples 
along with what the impact is on TCP itself, and what that actually means for 
people. I got that and then some. So for future readers, here’s an attempt to 
crudely summarize how CoDel (specifically as used in fq_codel) helps:

For some TCP CC algorithms (probably still most, as of this writing):
- reduces TCP RTT at saturation, improving interactivity for single flows with 
mixed bulk/interactive traffic like HTTP/2.0 or SSH
- smooths data delivery (thus meeting user’s expectations) by avoiding queue 
overflow (and tail-drop) in larger queues
- increases throughput efficiency with ECN (won’t tackle this further here!)
- reduces memory requirements for TCP buffers

Regardless of TCP CC algorithm:
- reduces latency for “non-flow” traffic, such as that for some encrypted VPNs

___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] when does the CoDel part of fq_codel help in the real world?

[Dave Taht]

On Tue, Nov 27, 2018 at 2:30 PM Roland Bless  wrote:
>
> Hi,
>
> On 27.11.18 at 23:19 Luca Muscariello wrote:
> > I suggest re-reading this
> >
> > https://queue.acm.org/detail.cfm?id=3022184
> Probably not without this afterwards:
> https://ieeexplore.ieee.org/document/8117540
>
> (especially sections II and III).


And this was awesome:

https://www.lk.cs.ucla.edu/data/files/Kleinrock/Internet%20congestion%20control%20using%20the%20power%20metric%20LK%20Mod%20aug%202%202018.pdf

and somewhere in that one are two points I'd like to make about BBR
with multiple flows, reverse congestion issues, and so on...

but now that we all have bedtime reading, I'm going to go back to
hacking on libcuckoo. :)





> Regards,
>  Roland
> ___
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat



-- 

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] one benefit of turning off shaping + fq_codel

[Jonathan Morton]

Just to add - I think the biggest impediment to experimentation in asynchronous 
logic is the complete absence of convenient Muller C-element gates in the 
74-series logic family.  If you want to build some, I recommend using NAND and 
OR gates as inputs to active-low SR flipflops.

 - Jonathan Morton

___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] one benefit of turning off shaping + fq_codel

[Jonathan Morton]

>> I wish I knew of a mailing list where I could get a definitive answer
>> on "modern problems with async circuits", or an update on the kind of
>> techniques the new AI chips were using to keep their power consumption
>> so low. I'll keep googling.
> 
> I’d be interested in knowing this as well. This gives some examples of async 
> circuits: 
> https://web.stanford.edu/class/archive/ee/ee371/ee371.1066/lectures/lect_12.pdf
> 
> Page 43, “Bottom Line” mentions that asynchronous design has “some delay 
> matching / overhead issues”. Apparently delay matching means getting the 
> signal outputs on two separate paths to arrive at the same time(?) Presumably 
> overhead refers to the 2x space on the die previously mentioned, for 
> completion detection. Pages 23-25 on “data-bundling constraints” might also 
> highlight some other challenges. Some more current material would be 
> interesting though...

The area overhead is at least partly mitigated by the major advantage of not 
having to distribute and gate a coherent clock signal across the entire chip.  
I half-remember seeing a quote that distributing the clock represents about 30% 
of the area and/or power consumption of a modern deep-sub-micron design.  This 
is area and power that is not directly contributing to functionality.

Generally there are two major styles of asynchronous logic:

1: Standard combinatorial logic stages accompanied by self-timing circuits with 
a matched delay, generally known as "bundled data".  This style has little 
overhead (probably less than the clock distribution it replaces) but requires 
local timing closure (the timing circuit must have strictly *more* delay than 
the logic it accompanies) to assure correct functionality.  I suspect that 
achieving local timing closure is easier than the global timing closure 
required by conventional synchronous logic.

2: Dual-rail QDI logic, in which completion is explicitly signalled by the 
arrival of a result.  This almost completely eliminates timing closure from the 
logic correctness equation, but the area overhead can be substantial.  
Achieving maximum performance in this style can also be challenging, but 
suitable approaches do exist, eg:

https://brej.org/papers/mapld.pdf

Both styles can inherently adapt timings to thermal and voltage conditions 
within a design range without much explicit provisioning, and typically have 
much cleaner power load and EMI characteristics than synchronous logic.  But as 
you can see from the above, the downsides typically associated with async logic 
tend to apply to one or the other of the styles, not to both at once.

 - Jonathan Morton

___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] when does the CoDel part of fq_codel help in the real world?

[Roland Bless]

Hi,

On 27.11.18 at 23:19 Luca Muscariello wrote:
> I suggest re-reading this 
> 
> https://queue.acm.org/detail.cfm?id=3022184
Probably not without this afterwards:
https://ieeexplore.ieee.org/document/8117540

(especially sections II and III).

Regards,
 Roland
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] when does the CoDel part of fq_codel help in the real world?

[Luca Muscariello]

I suggest re-reading this

https://queue.acm.org/detail.cfm?id=3022184


On Tue 27 Nov 2018 at 21:58, Dave Taht  wrote:

> OK, wow, this conversation got long. and I'm still 20 messages behind.
>
> Two points, and I'm going to go back to work, and maybe I'll try to
> summarize a table
> of the competing viewpoints, as there's far more than BDP of
> discussion here, and what
> we need is sqrt(bdp) to deal with all the different conversational flows.
> :)
>
> On Tue, Nov 27, 2018 at 1:24 AM Luca Muscariello
>  wrote:
> >
> > I think that this is a very good comment to the discussion at the
> defense about the comparison between
> > SFQ with longest queue drop and FQ_Codel.
> >
> > A congestion controlled protocol such as TCP or others, including QUIC,
> LEDBAT and so on
> > need at least the BDP in the transmission queue to get full link
> efficiency, i.e. the queue never empties out.
>
> no, I think it needs a BDP in flight.
>
> I think some of the confusion here is that your TCP stack needs to
> keep around a BDP in order to deal with
> retransmits, but that lives in another set of buffers entirely.
>
> > This gives rule of thumbs to size buffers which is also very practical
> and thanks to flow isolation becomes very accurate.
> >
> > Which is:
> >
> > 1) find a way to keep the number of backlogged flows at a reasonable
> value.
> > This largely depends on the minimum fair rate an application may need in
> the long term.
> > We discussed a little bit of available mechanisms to achieve that in the
> literature.
> >
> > 2) fix the largest RTT you want to serve at full utilization and size
> the buffer using BDP * N_backlogged.
> > Or the other way round: check how much memory you can use
> > in the router/line card/device and for a fixed N, compute the largest
> RTT you can serve at full utilization.
>
> My own take on the whole BDP argument is that *so long as the flows in
> that BDP are thoroughly mixed* you win.
>
> >
> > 3) there is still some memory to dimension for sparse flows in addition
> to that, but this is not based on BDP.
> > It is just enough to compute the total utilization of sparse flows and
> use the same simple model Toke has used
> > to compute the (de)prioritization probability.
> >
> > This procedure would allow to size FQ_codel but also SFQ.
> > It would be interesting to compare the two under this buffer sizing.
> > It would also be interesting to compare another mechanism that we have
> mentioned during the defense
> > which is AFD + a sparse flow queue. Which is, BTW, already available in
> Cisco nexus switches for data centres.
> >
> > I think that the the codel part would still provide the ECN feature,
> that all the others cannot have.
> > However the others, the last one especially can be implemented in
> silicon with reasonable cost.
> >
> >
> >
> >
> >
> > On Mon 26 Nov 2018 at 22:30, Jonathan Morton 
> wrote:
> >>
> >> > On 26 Nov, 2018, at 9:08 pm, Pete Heist  wrote:
> >> >
> >> > So I just thought to continue the discussion- when does the CoDel
> part of fq_codel actually help in the real world?
> >>
> >> Fundamentally, without Codel the only limits on the congestion window
> would be when the sender or receiver hit configured or calculated rwnd and
> cwnd limits (the rwnd is visible on the wire and usually chosen to be large
> enough to be a non-factor), or when the queue overflows.  Large windows
> require buffer memory in both sender and receiver, increasing costs on the
> sender in particular (who typically has many flows to manage per machine).
> >>
> >> Queue overflow tends to result in burst loss and head-of-line blocking
> in the receiver, which is visible to the user as a pause and subsequent
> jump in the progress of their download, accompanied by a major fluctuation
> in the estimated time to completion.  The lost packets also consume
> capacity upstream of the bottleneck which does not contribute to
> application throughput.  These effects are independent of whether overflow
> dropping occurs at the head or tail of the bottleneck queue, though
> recovery occurs more quickly (and fewer packets might be lost) if dropping
> occurs from the head of the queue.
> >>
> >> From a pure throughput-efficiency standpoint, Codel allows using ECN
> for congestion signalling instead of packet loss, potentially eliminating
> packet loss and associated lead-of-line blocking entirely.  Even without
> ECN, the actual cwnd is kept near the minimum necessary to satisfy the BDP
> of the path, reducing memory requirements and significantly shortening the
> recovery time of each loss cycle, to the point where the end-user may not
> notice that delivery is not perfectly smooth, and implementing accurate
> completion time estimators is considerably simplified.
> >>
> >> An important use-case is where two sequential bottlenecks exist on the
> path, the upstream one being only slightly higher capacity but lacking any
> queue management at all.  This is presently common in cases where 

Re: [Bloat] one benefit of turning off shaping + fq_codel

[Pete Heist]

> On Nov 27, 2018, at 8:09 PM, Dave Taht  wrote:
> 
> I wish I knew of a mailing list where I could get a definitive answer
> on "modern problems with async circuits", or an update on the kind of
> techniques the new AI chips were using to keep their power consumption
> so low. I'll keep googling.

I’d be interested in knowing this as well. This gives some examples of async 
circuits: 
https://web.stanford.edu/class/archive/ee/ee371/ee371.1066/lectures/lect_12.pdf 


Page 43, “Bottom Line” mentions that asynchronous design has “some delay 
matching / overhead issues”. Apparently delay matching means getting the signal 
outputs on two separate paths to arrive at the same time(?) Presumably overhead 
refers to the 2x space on the die previously mentioned, for completion 
detection. Pages 23-25 on “data-bundling constraints” might also highlight some 
other challenges. Some more current material would be interesting though...___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] when does the CoDel part of fq_codel help in the real world?

[Roland Bless]

Hi Kathie,

[long time, no see :-)]
I'm well aware of the CoDel paper and it really does a nice job
of explaining the good queue and bad queue properties. What we
found is that loss-based TCP CCs systematically build standing
queues. Their positive function is to keep up the link utilization,
their drawback is the huge queuing delay. So everyone
not aware of both papers should read them. However, if you think
something that I wrote is NOT in accordance with your findings,
please let me know.

Regards,
 Roland

On 27.11.18 at 19:44 Kathleen Nichols wrote> I have been kind of blown
away by this discussion. Jim Gettys kind of> kicked off the current wave
of dealing with full queues, dubbing it
> "bufferbloat". He wanted to write up how it happened so that people
> could start on a solution and I was enlisted to get an article written.
> We tried to draw on the accumulated knowledge of decades and use a
> context of What Jim Saw. I think the article offers some insight on
> queues (perhaps I'm biased as a co-author, but I'm not claiming any
> original insights just putting it together)
> https://queue.acm.org/detail.cfm?id=2071893
> 
> Further, in our first writing about CoDel, Van insisted on getting a
> good explanation of queues and how things go wrong. I think the figures
> and the explanation of how buffers are meant to be shock absorbers are
> very useful (yes, bias again, but I'm not saying you have to agree about
> CoDel's efficacy, just about how queues happen and why we need some
> buffer). https://queue.acm.org/detail.cfm?id=2209336
> 
> It's just kind of weird since Jim's evangelism is at the root of this
> list (and Dave's picking up the torch of course). Reading is a lost art.

___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] when does the CoDel part of fq_codel help in the real world?

[Michael Welzl]

Just a small clarification:


>> To me the switch to head dropping essentially killed the tail loss RTO
>> problem, eliminated most of the need for ecn.
> 
> I doubt that: TCP will need to retransmit that packet at the head, and that 
> takes an RTT - all the packets after it will need to wait in the receiver 
> buffer before the application gets them.
> But I don’t have measurements to prove my point, so I’m just hand-waving…

I don’t doubt that this kills the tail loss RTO problem.
I doubt that it eliminates the need for ECN.

Cheers,
Michael

___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] when does the CoDel part of fq_codel help in the real world?

[Toke Høiland-Jørgensen]

Dave Taht  writes:

> On Tue, Nov 27, 2018 at 12:54 PM Toke Høiland-Jørgensen  wrote:
>>
>> Dave Taht  writes:
>>
>> > I've done things like measure induced latency on wireguard streams of
>> > late and codel keeps it sane. still, wireguard internally is optimized
>> > for single flow "dragster" performance, and I'd like it to gain the
>> > same fq_codel optimization that did such nice things for multiple
>> > flows terminating on the router for ipsec. The before/after on that
>> > was marvelous.
>>
>> This is on my list, FWIW :)
>
> Your queue overfloweth. And as good as you are at FQ-ing yourself, I
> so dearly would like the !@#!@#@! wifi patches to land in 4.20.

Well, 4.20 is already in RC, so that is not going to happen ;)

-Toke
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] when does the CoDel part of fq_codel help in the real world?

[Dave Taht]

On Tue, Nov 27, 2018 at 12:54 PM Toke Høiland-Jørgensen  wrote:
>
> Dave Taht  writes:
>
> > I've done things like measure induced latency on wireguard streams of
> > late and codel keeps it sane. still, wireguard internally is optimized
> > for single flow "dragster" performance, and I'd like it to gain the
> > same fq_codel optimization that did such nice things for multiple
> > flows terminating on the router for ipsec. The before/after on that
> > was marvelous.
>
> This is on my list, FWIW :)

Your queue overfloweth. And as good as you are at FQ-ing yourself, I
so dearly would like the !@#!@#@! wifi patches to land in 4.20.

> -Toke



-- 

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] when does the CoDel part of fq_codel help in the real world?

[Dave Taht]

OK, wow, this conversation got long. and I'm still 20 messages behind.

Two points, and I'm going to go back to work, and maybe I'll try to
summarize a table
of the competing viewpoints, as there's far more than BDP of
discussion here, and what
we need is sqrt(bdp) to deal with all the different conversational flows. :)

On Tue, Nov 27, 2018 at 1:24 AM Luca Muscariello
 wrote:
>
> I think that this is a very good comment to the discussion at the defense 
> about the comparison between
> SFQ with longest queue drop and FQ_Codel.
>
> A congestion controlled protocol such as TCP or others, including QUIC, 
> LEDBAT and so on
> need at least the BDP in the transmission queue to get full link efficiency, 
> i.e. the queue never empties out.

no, I think it needs a BDP in flight.

I think some of the confusion here is that your TCP stack needs to
keep around a BDP in order to deal with
retransmits, but that lives in another set of buffers entirely.

> This gives rule of thumbs to size buffers which is also very practical and 
> thanks to flow isolation becomes very accurate.
>
> Which is:
>
> 1) find a way to keep the number of backlogged flows at a reasonable value.
> This largely depends on the minimum fair rate an application may need in the 
> long term.
> We discussed a little bit of available mechanisms to achieve that in the 
> literature.
>
> 2) fix the largest RTT you want to serve at full utilization and size the 
> buffer using BDP * N_backlogged.
> Or the other way round: check how much memory you can use
> in the router/line card/device and for a fixed N, compute the largest RTT you 
> can serve at full utilization.

My own take on the whole BDP argument is that *so long as the flows in
that BDP are thoroughly mixed* you win.

>
> 3) there is still some memory to dimension for sparse flows in addition to 
> that, but this is not based on BDP.
> It is just enough to compute the total utilization of sparse flows and use 
> the same simple model Toke has used
> to compute the (de)prioritization probability.
>
> This procedure would allow to size FQ_codel but also SFQ.
> It would be interesting to compare the two under this buffer sizing.
> It would also be interesting to compare another mechanism that we have 
> mentioned during the defense
> which is AFD + a sparse flow queue. Which is, BTW, already available in Cisco 
> nexus switches for data centres.
>
> I think that the the codel part would still provide the ECN feature, that all 
> the others cannot have.
> However the others, the last one especially can be implemented in silicon 
> with reasonable cost.
>
>
>
>
>
> On Mon 26 Nov 2018 at 22:30, Jonathan Morton  wrote:
>>
>> > On 26 Nov, 2018, at 9:08 pm, Pete Heist  wrote:
>> >
>> > So I just thought to continue the discussion- when does the CoDel part of 
>> > fq_codel actually help in the real world?
>>
>> Fundamentally, without Codel the only limits on the congestion window would 
>> be when the sender or receiver hit configured or calculated rwnd and cwnd 
>> limits (the rwnd is visible on the wire and usually chosen to be large 
>> enough to be a non-factor), or when the queue overflows.  Large windows 
>> require buffer memory in both sender and receiver, increasing costs on the 
>> sender in particular (who typically has many flows to manage per machine).
>>
>> Queue overflow tends to result in burst loss and head-of-line blocking in 
>> the receiver, which is visible to the user as a pause and subsequent jump in 
>> the progress of their download, accompanied by a major fluctuation in the 
>> estimated time to completion.  The lost packets also consume capacity 
>> upstream of the bottleneck which does not contribute to application 
>> throughput.  These effects are independent of whether overflow dropping 
>> occurs at the head or tail of the bottleneck queue, though recovery occurs 
>> more quickly (and fewer packets might be lost) if dropping occurs from the 
>> head of the queue.
>>
>> From a pure throughput-efficiency standpoint, Codel allows using ECN for 
>> congestion signalling instead of packet loss, potentially eliminating packet 
>> loss and associated lead-of-line blocking entirely.  Even without ECN, the 
>> actual cwnd is kept near the minimum necessary to satisfy the BDP of the 
>> path, reducing memory requirements and significantly shortening the recovery 
>> time of each loss cycle, to the point where the end-user may not notice that 
>> delivery is not perfectly smooth, and implementing accurate completion time 
>> estimators is considerably simplified.
>>
>> An important use-case is where two sequential bottlenecks exist on the path, 
>> the upstream one being only slightly higher capacity but lacking any queue 
>> management at all.  This is presently common in cases where home CPE 
>> implements inbound shaping on a generic ISP last-mile link.  In that case, 
>> without Codel running on the second bottleneck, traffic would collect in the 
>> first bottleneck's 

Re: [Bloat] when does the CoDel part of fq_codel help in the real world?

[Toke Høiland-Jørgensen]

Dave Taht  writes:

> I've done things like measure induced latency on wireguard streams of
> late and codel keeps it sane. still, wireguard internally is optimized
> for single flow "dragster" performance, and I'd like it to gain the
> same fq_codel optimization that did such nice things for multiple
> flows terminating on the router for ipsec. The before/after on that
> was marvelous.

This is on my list, FWIW :)

-Toke
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] when does the CoDel part of fq_codel help in the real world?

[Dave Taht]

On Mon, Nov 26, 2018 at 1:30 PM Jonathan Morton  wrote:
>
> > On 26 Nov, 2018, at 9:08 pm, Pete Heist  wrote:
> >
> > So I just thought to continue the discussion- when does the CoDel part of 
> > fq_codel actually help in the real world?
>
> Fundamentally, without Codel the only limits on the congestion window would 
> be when the sender or receiver hit configured or calculated rwnd and cwnd 
> limits (the rwnd is visible on the wire and usually chosen to be large enough 
> to be a non-factor), or when the queue overflows.  Large windows require 
> buffer memory in both sender and receiver, increasing costs on the sender in 
> particular (who typically has many flows to manage per machine).

You can run devices out of memory more easily with our current ecn
implementations. I am seeing folk cut memory per instance to 256k in,
for example, the gluon project.

we end up dropping (which is better than the device crashing), and in
the fq_codel case, *bulk* head dropping. I see a bifurcation on the
data when we do this, and
I have a one line patch to the fq_codel bulk dropper that appears to
make things better when extremely memory constrained like this, but
haven't got around to fully evaluating it:

https://github.com/dtaht/fq_codel_fast/commit/a524fc2e39dc291199b9b04fb890ea1548f17641

would rather like more to try the memory limits we are seeing in the
field. 32MB (fq_codel default) is wy too much. 4MB is way too much
even for gbit, I think. 256k? well, given the choice between crashing
or not...

>
> Queue overflow tends to result in burst loss and head-of-line blocking in the 
> receiver, which is visible to the user as a pause and subsequent jump in the 
> progress of their download, accompanied by a major fluctuation in the 
> estimated time to completion.  The lost packets also consume capacity 
> upstream of the bottleneck which does not contribute to application 
> throughput.  These effects are independent of whether overflow dropping 
> occurs at the head or tail of the bottleneck queue, though recovery occurs 
> more quickly (and fewer packets might be lost) if dropping occurs from the 
> head of the queue.
>
> From a pure throughput-efficiency standpoint, Codel allows using ECN for 
> congestion signalling instead of packet loss, potentially eliminating packet 
> loss and associated lead-of-line blocking entirely.  Even without ECN, the 
> actual cwnd is kept near the minimum necessary to satisfy the BDP of the 
> path, reducing memory requirements and significantly shortening the recovery 
> time of each loss cycle, to the point where the end-user may not notice that 
> delivery is not perfectly smooth, and implementing accurate completion time 
> estimators is considerably simplified.

I wish we had fractional cwnd below 1 and/or that pacing did not rely
on cwnd at all. too many flows at any rate you choose, can end up
marking 100% of packets, still not run you out of memory, and cause
delay.

>
> An important use-case is where two sequential bottlenecks exist on the path, 
> the upstream one being only slightly higher capacity but lacking any queue 
> management at all.  This is presently common in cases where home CPE 
> implements inbound shaping on a generic ISP last-mile link.  In that case, 
> without Codel running on the second bottleneck, traffic would collect in the 
> first bottleneck's queue as well, greatly reducing the beneficial effects of 
> FQ implemented on the second bottleneck.  In this topology, the overall 
> effect is inter-flow as well as intra-flow.
>
> The combination of Codel with FQ is done in such a way that a separate 
> instance of Codel is implemented for each flow.  This means that congestion 
> signals are only sent to flows that require them, and non-saturating flows 
> are unmolested.  This makes the combination synergistic, where each component 
> offers an improvement to the behaviour of the other.

also a good bullet point somewhere!

>
>  - Jonathan Morton
>
> ___
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat



-- 

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] when does the CoDel part of fq_codel help in the real world?

[Dave Taht]

On Mon, Nov 26, 2018 at 11:28 AM Neal Cardwell  wrote:
>
> I believe Dave Taht has pointed out, essentially, that the "codel" part of 
> fq_codel can be useful in cases where the definition of "flow" is not visible 
> to fq_codel, so that "fq" part is inactive. For example, if there is VPN 
> traffic, where the individual flows are not separable by fq_codel, then it 
> can help to have "codel" AQM for the aggregate of encrypted VPN traffic. I 
> imagine this rationale could apply where there is any kind of encapsulation 
> or encryption that makes the notion of "flow" opaque to fq_codel.

Yep. This should end up a bullet point somewhere. The bsd
implementation of fq_codel does not do anywhere near the dpi that
linux does to extract the relevant fields. That's a truly amazing
amount of sub-protocol hashing, sometimes I sit back and have to both
admire and shudder at this:
https://elixir.bootlin.com/linux/v4.18.6/source/net/core/flow_dissector.c#L578

and it's even more frightful if you look at the list of ethernet types
and protocols that are not dissected.

Codeling opaque flows is useful. We will always have opaque flows.
Some protocol types cannot be fq'd due to design - babel for example
mixes up hellos with route transfers.

I've done things like measure induced latency on wireguard streams of
late and codel keeps it sane. still, wireguard internally is optimized
for single flow "dragster" performance, and I'd like it to gain the
same fq_codel optimization that did such nice things for multiple
flows terminating on the router for ipsec. The before/after on that
was marvelous.

Another problem people are perpetually pointing out is hashing is
expensive, and me, saying it's "free" with hardware support for it,
with examples like most modern 10GigE ethernet cards doing it
on-board, hardware assists for sha and references to the FPGA lit,
such as: https://zistvan.github.io/doc/trets15-istvan.pdf

still... hashing is sw expensive.

Recently I went on a quest for a better hash than jenkins which is
used throughout the kernel today. cityhash and murmur for example.

>
> neal
>
>
> On Mon, Nov 26, 2018 at 2:08 PM Pete Heist  wrote:
>>
>> In Toke’s thesis defense, there was an interesting exchange with examination 
>> committee member Michael (apologies for not catching the last name) 
>> regarding how the CoDel part of fq_codel helps in the real world:
>>
>> https://www.youtube.com/watch?v=upvx6rpSLSw=2h16m20s
>>
>> My attempt at a transcript is at the end of this message. (I probably won’t 
>> attempt a full defense transcript, but if someone wants more of a particular 
>> section I can try. :)
>>
>> So I just thought to continue the discussion- when does the CoDel part of 
>> fq_codel actually help in the real world? I’ll speculate with a few 
>> possibilities:
>>
>> 1) Multiplexed HTTP/2.0 requests containing both a saturating stream and 
>> interactive traffic. For example, a game that uses HTTP/2.0 to download new 
>> map data while position updates or chat happen at the same time. Standalone 
>> programs could use HTTP/2.0 this way, or for web apps, the browser may 
>> multiplex concurrent uses of XHR over a single TCP connection. I don’t know 
>> of any examples.
>>
>> 2) SSH with port forwarding while using an interactive terminal together 
>> with a bulk transfer?
>>
>> 3) Does CoDel help the TCP protocol itself somehow? For example, does it 
>> speed up the round-trip time when acknowledging data segments, improving 
>> behavior on lossy links? Similarly, does it speed up the TCP close sequence 
>> for saturating flows?
>>
>> Pete
>>
>> ---
>>
>> M: In fq_codel what is really the point of CoDel?
>> T: Yeah, uh, a bit better intra-flow latency...
>> M: Right, who cares about that?
>> T: Apparently some people do.
>> M: No I mean specifically, what types of flows care about that?
>> T: Yeah, so, um, flows that are TCP based or have some kind of- like, 
>> elastic flows that still want low latency.
>> M: Elastic flows that are TCP based that want low latency...
>> T: Things where you want to discover the- like, you want to utilize the full 
>> link and sort of probe the bandwidth, but you still want low latency.
>> M: Can you be more concrete what kind of application is that?
>> T: I, yeah, I…
>> M: Give me any application example that’s gonna benefit from the CoDel part- 
>> CoDel bits in fq_codel? Because I have problems with this.
>> T: I, I do too... So like, you can implement things this way but 
>> equivalently if you have something like fq_codel you could, like, if you 
>> have a video streaming application that interleaves control…
>> M:  that runs on UDP often.
>> T: Yeah, but I, Netflix…
>> M: Ok that’s a long way… 
>> T: No, I tend to agree with you that, um…
>> M: Because the biggest issue in my opinion is, is web traffic- for web 
>> traffic, just giving it a huge queue makes the chance bigger that uh, 
>>  so you may end up with a (higher) 
>> faster completion time by buffering a lot. 

Re: [Bloat] when does the CoDel part of fq_codel help in the real world?

[Dave Taht]

On Mon, Nov 26, 2018 at 1:56 PM Michael Welzl  wrote:
>
> Hi folks,
>
> That “Michael” dude was me  :)
>
> About the stuff below, a few comments. First, an impressive effort to dig all 
> of this up - I also thought that this was an interesting conversation to have!
>
> However, I would like to point out that thesis defense conversations are 
> meant to be provocative, by design - when I said that CoDel doesn’t usually 
> help and long queues would be the right thing for all applications, I 
> certainly didn’t REALLY REALLY mean that.  The idea was just to be thought 
> provoking - and indeed I found this interesting: e.g., if you think about a 
> short HTTP/1 connection, a large buffer just gives it a greater chance to get 
> all packets across, and the perceived latency from the reduced round-trips 
> after not dropping anything may in fact be less than with a smaller (or 
> CoDel’ed) buffer.

I really did want Toke to have a hard time. Thanks for putting his
back against the wall!

And I'd rather this be a discussion of toke's views... I do tend to
think he thinks FQ solves more than it does and I wish we had a
sound analysis as to why 1024 queues
works so much better for us than 64 or less on the workloads we have.
I tend to think in part it's because that acts as a 1000x1
rate-shifter - but should it scale up? Or down? Is what we did with
cake (1024 setassociative) useful? or excessive? I'm regularly seeing
64,000 queues on 10Gig and up hardware due to 64 hardware queues and
fq_codel on each, on that sort of gear. I think that's too much and
renders the aqm ineffective, but lack data...

but, to rant a bit...

While I tend to believe FQ solves 97% of the problem, AQM 2.9% and ECN .09%.

BUT: Amdahls law says once you reduce one part of the problem to 0,
everything else takes 100%. :)

it often seems like me, being the sole and very lonely FQ advocate
here in 2011, have reversed the situation (in this group!), and I'm
oft the AQM advocate *here* now.

It's sort of like all the people quoting the e2e argument still, back
at me when dave reed (at least, and perhaps the other co-authors now)
have bought into this level of network interference between the
endpoints, and had no religion - or the red in a different light paper
being rejected because it attempted to overturn other religion - and
I'll be damned if I'll let fq_codel, sch_fq, pie, l4s, scream, nada,

I admit to getting kind of crusty and set in my ways, but so long as
people put code in front of me along with the paper, I still think,
when the facts change, so do my opinions.

Pacing is *really impressive* and I'd like to see that enter
everything, not just in packet processing - I've been thinking hard
about the impact of cpu bursts (like resizing a hash table), and other
forms of work that we currently do on computers that have a
"dragster-like" peak performance, and a great average, but horrible
pathologies - and I think the world would be better off if we built
more

Anyway...

Once you have FQ and a sound outer limit on buffer size (100ms),
depredations like comcast's 680ms buffers no longer matter. There's
still plenty of room to innovate. BBR works brilliantly vs fq_codel
(and you can even turn ECN on which it doesn't respect and still get a
great result). LoLa would probably work well also 'cept that the git
tree was busted when I last tried it and it hasn't been tested much in
the 1Mbit-1Gbit range.

>
> But corner cases aside, in fact I very much agree with the answers to my 
> question Pete gives below, and also with the points others have made in 
> answering this thread. Jonathan Morton even mentioned ECN - after Dave’s 
> recent over-reaction to ECN I made a point of not bringing up ECN *yet* again

Not going to go into it (much) today! We ended up starting another
project on ECN that that operates under my core ground rule - "show me
the code" - and life over there and on that mailing list has been
pleasantly quiet. https://www.bufferbloat.net/projects/ecn-sane/wiki/
.

I did get back on the tsvwg mailing list recently because of some
ludicrously inaccurate misstatements about fq_codel. I also made a
strong appeal to the l4s people, to, in general, "stop thanking me" in
their documents. To me that reads as an endorsement, where all I did
was participate in the process until I gave up and hit my "show me the
code" moment - which was about 5 years ago and hasn't moved on the
needle since except in mutating standards documents.

The other document I didn't like was an arbitary attempt to just set
the ecn backoff figure to .8 when the sanest thing, given the
deployment, and pacing... was to aim for a right number - anyway.
in that case I just wanted off the "thank you" list.

I like to think the more or less rfc3168 compliant deployment of ecn
is thus far going brilliantly, but lack data. Certainly would like a
hostile reviewers evaluation of cake's ecn method and for that matter,
pie's, honestly - from real traffic! There's 

Re: [Bloat] when does the CoDel part of fq_codel help in the real world?

[Dave Taht]

On Tue, Nov 27, 2018 at 10:44 AM Kathleen Nichols  wrote:
>
>
> I have been kind of blown away by this discussion. Jim Gettys kind of
> kicked off the current wave of dealing with full queues, dubbing it
> "bufferbloat". He wanted to write up how it happened so that people
> could start on a solution and I was enlisted to get an article written.
> We tried to draw on the accumulated knowledge of decades and use a
> context of What Jim Saw. I think the article offers some insight on
> queues (perhaps I'm biased as a co-author, but I'm not claiming any
> original insights just putting it together)
> https://queue.acm.org/detail.cfm?id=2071893
>
> Further, in our first writing about CoDel, Van insisted on getting a
> good explanation of queues and how things go wrong. I think the figures
> and the explanation of how buffers are meant to be shock absorbers are
> very useful (yes, bias again, but I'm not saying you have to agree about
> CoDel's efficacy, just about how queues happen and why we need some
> buffer). https://queue.acm.org/detail.cfm?id=2209336
>
> It's just kind of weird since Jim's evangelism is at the root of this
> list (and Dave's picking up the torch of course). Reading is a lost art.

The working title of my PHD thesis is "Sanely shedding load", which, while
our work to date has made a dent on it, is a broader question, that also applies
to people and how we manage load...

... las, for example, I'm at least 20 messages behind on these lists
this week so far,
and I figure that thesis will never complete until I learn how to
write in passive voice.

I think the two pieces above, this list, evangelism, and working
results has made an
enormous dent in how computer scientists and engineers think about
things over the
last 7 years, and it will continue to percolate through academia and
elsewhere to good effect.

And your and jim's paper has 389 cites now:

https://scholar.google.com/scholar?hl=en_sdt=0%2C5=bufferbloat==

I know, sadly, though, that my opinion about how much we've changed
the world, has more than a bit of confirmation bias in it.

I loved what uber did (https://eng.uber.com/qalm/)

I'd certainly love to see kleinrock taught not just in comp sci and
engineering, but in business and government,
as I just spent 3 hours at the DMV that I wish I didn't need to spend,
and a day where I could hit dice.com for
"queue theory" and come up with hundreds of hits across all
professions... rather than 0.

>
> Kathie
> ___
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat



-- 

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] one benefit of turning off shaping + fq_codel

[Holland, Jake]

On 2018-11-27, 10:31, "Stephen Hemminger"  wrote:
With asynchronous circuits there is too much unpredictablity and 
instability.
Seem to remember there are even cases where two inputs arrive at once and 
output is non-determistic.

IIRC they talked about that some too. I think maybe some papers were going back 
and forth. But last I heard, they proved that this is not a real objection, in 
that:
1. you can quantify the probability of failure and ensure a design keeps it 
under threshold when operating within specified conditions (e.g. normal 
temperature and voltage thresholds)
2. you can work around the issues where it's critical by adding failure 
detection and faults, and
3. you have the exact same fundamental theoretical problem with synchronous 
circuits, particularly in registers that can keep a value through a clock 
cycle, but it hasn't stopped them from being useful.

I'm not an expert and this was all a long time ago for me, but  the qdi wiki 
page doesn't disagree with what I'm remembering here, and has some good 
references on the topic:
https://en.wikipedia.org/wiki/Quasi-delay-insensitive_circuit#Stability_and_non-interference
https://en.wikipedia.org/wiki/Quasi-delay-insensitive_circuit#Timing


___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] one benefit of turning off shaping + fq_codel

[Dave Taht]

On Tue, Nov 27, 2018 at 10:31 AM Stephen Hemminger
 wrote:
>
> On Tue, 27 Nov 2018 18:14:01 +
> "Holland, Jake"  wrote:
>
> > On 2018-11-23, 08:33, "Dave Taht"  wrote:
> > Back in the day, I was a huge fan of async logic, which I first
> > encountered via caltech's cpu and later the amulet.
> >
> > https://en.wikipedia.org/wiki/Asynchronous_circuit#Asynchronous_CPU
> >
> > ...
> >
> > I've never really understood why it didn't take off, I think, in part,
> > it doesn't scale to wide busses well, and that centrally clocked designs
> > are how most engineers and fpgas and code got designed since. Anything
> > with delay built into it seems hard for EEs to grasp but I wish I
> > knew why, or had the time to go play with circuits again at a reasonable
> > scale.
> >
> > At the time, I was told the objections they got were that it uses about 2x 
> > the space for the same functionality, and space usage is approximately 
> > linear with the chip cost, and when under load you still need reasonable 
> > cooling, so it was only considered maybe worthwhile for some narrow use 
> > cases.

And the pentultimate cost here was unpredictable and many power
states, hyperthreading (which is looking to die post spectre), and
things like ddpk which spin processors madly to keep up. I always
liked things like

I wish I knew more about what fulcrum did in their switch designs...

everybody knows I'm a fan of the mill cpu which has lots of little
optimizations close to each functional unit (among many other things
using virtual memory internally for everything,
and separating out the PLB (protection level buffer) from the TLB). I
would really like to bring back an era where cpus could context or
security level switch in 5 clocks.

Someday something like that will be built. Til then, the closest chip
to something I'd like to be working on for networks is how the xmos is
designed: https://en.wikipedia.org/wiki/XMOS#xCORE_multicore_microcontrollers
- or https://www.xmos.com/developer/silicon/xcore200-ethernet which
has 1MByte of single-clock sram on it.

"The xCORE architecture delivers, in hardware, many of the elements
that are usually seen in a real-time operating system (RTOS). This
includes the task scheduler, timers, I/O operations, and channel
communication. By eliminating sources of timing uncertainty
(interrupts, caches, buses and other shared resources), xCORE can
provide deterministic and predictable performance for many
applications. A task can typically respond in nanoseconds to events
such as external I/O or timers. This makes it possible to program
xCORE devices to perform hard real-time tasks that would otherwise
require dedicated hardware."

Nobody else's ethernet controllers work this way.

> >
> > I don't really know enough to confirm or deny the claim, and the use cases 
> > may have gotten a lot closer to a good match by now, but this was the 
> > opinion of at least some of the people involved with the work, IIRC.
> >
> >
> > ___
> > Bloat mailing list
> > Bloat@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/bloat
>
> With asynchronous circuits there is too much unpredictablity and instability.
> Seem to remember there are even cases where two inputs arrive at once and 
> output is non-determistic.

Yes, that was a big problem... in the 90s... but cpus *were*
successfully designed that didn't do that.

I am the sort of character that is totally willing to toss out decades
of evolution in chip design in order to get better SNR for wireless.
:)

I wish I knew of a mailing list where I could get a definitive answer
on "modern problems with async circuits", or an update on the kind of
techniques the new AI chips were using to keep their power consumption
so low. I'll keep googling.

> ___
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat



-- 

Dave Täht
CTO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-831-205-9740
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] when does the CoDel part of fq_codel help in the real world?

[Jonathan Morton]

> On 27 Nov, 2018, at 3:19 pm, Michael Richardson  wrote:
> 
> If the drops are due to noise, then I don't think it will help.
> The congestion signals should already getting made.

If they are drops due to noise, then they are not congestion signals at all, as 
they occur independently of whether the link is saturated.  It's perfectly 
possible for these "random drops" to occur at a low enough rate as to still 
permit saturation, perhaps depending on which CC algo the sender uses.  In that 
case, you still want AQM.

 - Jonathan Morton

___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] when does the CoDel part of fq_codel help in the real world?

[Kathleen Nichols]

I have been kind of blown away by this discussion. Jim Gettys kind of
kicked off the current wave of dealing with full queues, dubbing it
"bufferbloat". He wanted to write up how it happened so that people
could start on a solution and I was enlisted to get an article written.
We tried to draw on the accumulated knowledge of decades and use a
context of What Jim Saw. I think the article offers some insight on
queues (perhaps I'm biased as a co-author, but I'm not claiming any
original insights just putting it together)
https://queue.acm.org/detail.cfm?id=2071893

Further, in our first writing about CoDel, Van insisted on getting a
good explanation of queues and how things go wrong. I think the figures
and the explanation of how buffers are meant to be shock absorbers are
very useful (yes, bias again, but I'm not saying you have to agree about
CoDel's efficacy, just about how queues happen and why we need some
buffer). https://queue.acm.org/detail.cfm?id=2209336

It's just kind of weird since Jim's evangelism is at the root of this
list (and Dave's picking up the torch of course). Reading is a lost art.

Kathie
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] one benefit of turning off shaping + fq_codel

[Stephen Hemminger]

On Tue, 27 Nov 2018 18:14:01 +
"Holland, Jake"  wrote:

> On 2018-11-23, 08:33, "Dave Taht"  wrote:
> Back in the day, I was a huge fan of async logic, which I first
> encountered via caltech's cpu and later the amulet.
> 
> https://en.wikipedia.org/wiki/Asynchronous_circuit#Asynchronous_CPU
> 
> ...
> 
> I've never really understood why it didn't take off, I think, in part,
> it doesn't scale to wide busses well, and that centrally clocked designs
> are how most engineers and fpgas and code got designed since. Anything
> with delay built into it seems hard for EEs to grasp but I wish I
> knew why, or had the time to go play with circuits again at a reasonable
> scale.
> 
> At the time, I was told the objections they got were that it uses about 2x 
> the space for the same functionality, and space usage is approximately linear 
> with the chip cost, and when under load you still need reasonable cooling, so 
> it was only considered maybe worthwhile for some narrow use cases.
> 
> I don't really know enough to confirm or deny the claim, and the use cases 
> may have gotten a lot closer to a good match by now, but this was the opinion 
> of at least some of the people involved with the work, IIRC.
> 
> 
> ___
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

With asynchronous circuits there is too much unpredictablity and instability.
Seem to remember there are even cases where two inputs arrive at once and 
output is non-determistic.
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] one benefit of turning off shaping + fq_codel

[Holland, Jake]

On 2018-11-23, 08:33, "Dave Taht"  wrote:
Back in the day, I was a huge fan of async logic, which I first
encountered via caltech's cpu and later the amulet.

https://en.wikipedia.org/wiki/Asynchronous_circuit#Asynchronous_CPU

...

I've never really understood why it didn't take off, I think, in part,
it doesn't scale to wide busses well, and that centrally clocked designs
are how most engineers and fpgas and code got designed since. Anything
with delay built into it seems hard for EEs to grasp but I wish I
knew why, or had the time to go play with circuits again at a reasonable
scale.

At the time, I was told the objections they got were that it uses about 2x the 
space for the same functionality, and space usage is approximately linear with 
the chip cost, and when under load you still need reasonable cooling, so it was 
only considered maybe worthwhile for some narrow use cases.

I don't really know enough to confirm or deny the claim, and the use cases may 
have gotten a lot closer to a good match by now, but this was the opinion of at 
least some of the people involved with the work, IIRC.


___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] when does the CoDel part of fq_codel help in the real world?

[Mikael Abrahamsson]

On Tue, 27 Nov 2018, Luca Muscariello wrote:

This is a whole different discussion but if you want to have a per-user 
context at the BNG level + TM + FQ I'm not sure that kind of beast will 
ever exist. Unless you have a very small user fan-out the hardware 
clocks could loop over several thousands of contexts. You should expect 
those kind of features to be in the CMTS or OLT.


This is per queue per customer access port (250 customers per 10GE port, 
so 250 queues). It's on an "service edge" linecard that I imagine people 
use for BNG purposes. I tend to not use words like that because to me a 
router is a router.


I do not do coax. I do not do PON. I do point to point ethernet using 
routers and switches, like god^WIEEE intended.


--
Mikael Abrahamssonemail: swm...@swm.pp.se
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] when does the CoDel part of fq_codel help in the real world?

[Luca Muscariello]

On Tue, Nov 27, 2018 at 2:49 PM Mikael Abrahamsson  wrote:

> On Tue, 27 Nov 2018, Luca Muscariello wrote:
>
> > If you, Mikael don't want more than 10ms buffer, how do you achieve that?
>
> class class-default
>random-detect 10 ms 2000 ms
>
> That's the only thing available to me on the platforms I have. If you
> would like this improved, please reach out to the Cisco ASR9k BU and tell
> them to implement ECN and PIE (or something even better). They won't do it
> because I say so, it seems. WRED is all they give me.
>

This is a whole different discussion but if you want to have a per-user
context
at the BNG level + TM + FQ I'm not sure that kind of beast will ever exist.
Unless you have a very small user fan-out the hardware clocks could loop
over
several thousands of contexts.
You should expect those kind of features to be in the CMTS or OLT.


>
> > You change the behaviour of the source and hope flow isolation is
> available.
>
> Sorry, I only transport the packets, I don't create them.
>

I'm sure you create a lot of packets. Don't be humble.


>
> > If you just cut the buffer down to 10ms and do nothing else, the only
> thing
> > you get is a short queue and may throw away half of your link capacity.
>
> If i have lots of queue I might instead get customer complaints about high
> latency for their interactive applications.
>
> --
> Mikael Abrahamssonemail: swm...@swm.pp.se
>
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] when does the CoDel part of fq_codel help in the real world?

[Mikael Abrahamsson]

On Tue, 27 Nov 2018, Luca Muscariello wrote:


If you, Mikael don't want more than 10ms buffer, how do you achieve that?


class class-default
  random-detect 10 ms 2000 ms

That's the only thing available to me on the platforms I have. If you 
would like this improved, please reach out to the Cisco ASR9k BU and tell 
them to implement ECN and PIE (or something even better). They won't do it 
because I say so, it seems. WRED is all they give me.



You change the behaviour of the source and hope flow isolation is available.


Sorry, I only transport the packets, I don't create them.


If you just cut the buffer down to 10ms and do nothing else, the only thing
you get is a short queue and may throw away half of your link capacity.


If i have lots of queue I might instead get customer complaints about high 
latency for their interactive applications.


--
Mikael Abrahamssonemail: swm...@swm.pp.se
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] when does the CoDel part of fq_codel help in the real world?

[Luca Muscariello]

Another bit to this.
A router queue is supposed to serve packets no matter what is running at
the controlled end-point, BBR, Cubic or else.
So, delay-based congestion controller still get hurt in today Internet
unless they can get their portion of buffer at the line card.
FQ creates incentives for end-points to send traffic in a smoother way
because the reward to the application is immediate and
measurable. But the end-point does not know in advance if FQ is there or
not.

So going back to sizing the link buffer, the rule I mentioned applies. And
it allows to get best completion times for a wider range of RTTs.
If you, Mikael don't want more than 10ms buffer, how do you achieve that?
You change the behaviour of the source and hope flow isolation is available.
If you just cut the buffer down to 10ms and do nothing else, the only thing
you get is a short queue and may throw away half of your link capacity.



On Tue, Nov 27, 2018 at 1:17 PM Jonathan Morton 
wrote:

> > On 27 Nov, 2018, at 1:21 pm, Mikael Abrahamsson 
> wrote:
> >
> > It's complicated. I've had people throw in my face that I need 2xBDP in
> buffer size to smoothe things out. Personally I don't want more than 10ms
> buffer (max), and I don't see why I should need more than that even if
> transfers are running over hundreds of ms of light-speed-in-medium induced
> delay between the communicating systems.
>
> I think we can agree that the ideal CC algo would pace packets out
> smoothly at exactly the path capacity, neither building a queue at the
> bottleneck nor leaving capacity on the table.
>
> Actually achieving that in practice turns out to be difficult, because
> there's no general way to discover the path capacity in advance.  AQMs like
> Codel, in combination with ECN, get us a step closer by explicitly
> informing each flow when it is exceeding that capacity while the queue is
> still reasonably short.  FQ also helps, by preventing flows from
> inadvertently interfering with each other by imperfectly managing their
> congestion windows.
>
> So with the presently deployed state of the art, we have cwnds oscillating
> around reasonably short queue lengths, backing off sharply in response to
> occasional signals, then probing back upwards when that signal goes away
> for a while.  It's a big improvement over dumb drop-tail FIFOs, but it's
> still some distance from the ideal.  That's because the information
> injected by the bottleneck AQM is a crude binary state.
>
> I do not include DCTCP in the deployed state of the art, because it is not
> deployable in the RFC-compliant Internet; it is effectively incompatible
> with Codel in particular, because it wrongly interprets CE marks and is
> thus noncompliant with the ECN RFC.
>
> However, I agree with DCTCP's goal of achieving finer-grained control of
> the cwnd, through AQMs providing more nuanced information about the state
> of the path capacity and/or bottleneck queue.  An implementation that made
> use of ECT(1) instead of changing the meaning of CE marks would remain
> RFC-compliant, and could get "sufficiently close" to the ideal described
> above.
>
>  - Jonathan Morton
>
>
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] when does the CoDel part of fq_codel help in the real world?

[Michael Richardson]

Pete Heist  wrote:
> I was asked a related question by my local WISP, who wanted to know if
> there would be any reason that fq_codel or Cake would be an improvement
> over sfq specifically for some "noisy links” (loose translation from
> Czech) in a backhaul that have some loss but also experience

If the drops are due to noise, then I don't think it will help.
The congestion signals should already getting made.
But, there are lots of cases where there is excessive buffers at the edge of
the backhaul, which is encouraging excessive traffic and thus congestion.

--
Michael Richardson , Sandelman Software Works
 -= IPv6 IoT consulting =-





signature.asc
Description: PGP signature
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] when does the CoDel part of fq_codel help in the real world?

[Bless, Roland (TM)]

Hi Michael,

Am 27.11.18 um 12:04 schrieb Michael Welzl:
> I'm lost in this conversation: I thought it started with a statement saying 
> that the queue length must be at least a BDP such that full utilization is 
> attained because the queue never drains.

I think it helps to distinguish between bottleneck buffer size (i.e.,
its capacity) and bottleneck buffer occupancy (i.e., queue length).
My point was that full bottleneck utilization doesn't require any buffer
occupancy at all. I think one should also distinguish precisely between
different viewpoints: at the sender or at the bottleneck (aggregate of
many flows from different sources with different RTTs).

> To this, I'd want to add that, in addition to the links from Roland, the 
> point of ABE is to address exactly that: 
> https://tools.ietf.org/html/draft-ietf-tcpm-alternativebackoff-ecn-12
> (in the RFC Editor queue)

Yep, because then the backoff is less drastic so the utilization is kept
at a higher level even if the queue is much smaller than a BDP (that is
concluded from the fact that when ECN is present an AQM will try to keep
the queue much smaller). Our complementary approach was the AQM Steering
approach that let's the AQM adapt.

> But now I think you're discussing a BDP worth of data *in flight*, which is 
> something else.

Yes, maybe because I think it was confused, but it's anyhow related:
having a BDP in flight will allow you fully utilize the link capacity,
having BDP+x in flight will lead to having x queued up in the bottleneck
buffer. So having 2BDP inflight will lead to 1 BDP on the wire and 1 BDP
in the buffer. That's what loss based CC variants usually have and what
BBRv1 set as limit.

Regards,
 Roland
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] when does the CoDel part of fq_codel help in the real world?

[Bless, Roland (TM)]

Hi,

Am 27.11.18 um 12:58 schrieb Luca Muscariello:
> A buffer in a router is sized once. RTT varies.
> So BDP varies. That’s as simple as that.
> So you just cannot be always at optimum because you don’t know what RTT
> you have at any time.

The endpoints can measure the RTT. Yes, it's probably a bit noisy and
there are several practical problems such as congestion on the reverse
path and multiple bottlenecks, but in general it's not impossible.

> Lola si not solving that. No protocol could BTW.

LoLa is exactly solving that. It measures RTTmin and effective RTT
(and there are lots of other delay-based CC proposals doing that)
and tries to control the overall queuing delay, even achieving
RTT-independent flow rate fairness.

> BTW I don’t see  any formal proof about queue occupancy in the paper.

It's not in the LoLa paper, it was in a different paper, but reviewers
thought it was already common knowledge.

Regards,
 Roland

> On Tue 27 Nov 2018 at 12:53, Bless, Roland (TM)  > wrote:
> 
> Hi Luca,
> 
> Am 27.11.18 um 12:01 schrieb Luca Muscariello:
> > A BDP is not a large buffer. I'm not unveiling a secret.
> 
> That depends on speed and RTT (note that typically there are
> several flows with different RTTs sharing the same buffer).
> The essential point is not how much buffer capacity is available,
> but how much is actually used, because that adds queueing delay.
> 
> > And it is just a rule of thumb to have an idea at which working point
> > the protocol is working.
> 
> No, one can actually prove that this is the best size for
> loss-based CC with backoff factor of 0.5 (assuming a single flow).
> 
> > In practice the protocol is usually working below or above that value.
> 
> That depends on the protocol.
> 
> > This is where AQM and ECN help also. So most of the time the
> protocol is
> > working at way 
> > below 100% efficiency.
> 
> > My point was that FQ_codel helps to get very close to the optimum w/o
> > adding useless queueing and latency.
> > With a single queue that's almost impossible. No, sorry. Just
> impossible.
> 
> No, it's possible. Please read the TCP LoLa paper.
> 
> Regards,
>  Roland
> 

___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] when does the CoDel part of fq_codel help in the real world?

[Jonathan Morton]

> On 27 Nov, 2018, at 1:21 pm, Mikael Abrahamsson  wrote:
> 
> It's complicated. I've had people throw in my face that I need 2xBDP in 
> buffer size to smoothe things out. Personally I don't want more than 10ms 
> buffer (max), and I don't see why I should need more than that even if 
> transfers are running over hundreds of ms of light-speed-in-medium induced 
> delay between the communicating systems.

I think we can agree that the ideal CC algo would pace packets out smoothly at 
exactly the path capacity, neither building a queue at the bottleneck nor 
leaving capacity on the table.

Actually achieving that in practice turns out to be difficult, because there's 
no general way to discover the path capacity in advance.  AQMs like Codel, in 
combination with ECN, get us a step closer by explicitly informing each flow 
when it is exceeding that capacity while the queue is still reasonably short.  
FQ also helps, by preventing flows from inadvertently interfering with each 
other by imperfectly managing their congestion windows.

So with the presently deployed state of the art, we have cwnds oscillating 
around reasonably short queue lengths, backing off sharply in response to 
occasional signals, then probing back upwards when that signal goes away for a 
while.  It's a big improvement over dumb drop-tail FIFOs, but it's still some 
distance from the ideal.  That's because the information injected by the 
bottleneck AQM is a crude binary state.

I do not include DCTCP in the deployed state of the art, because it is not 
deployable in the RFC-compliant Internet; it is effectively incompatible with 
Codel in particular, because it wrongly interprets CE marks and is thus 
noncompliant with the ECN RFC.

However, I agree with DCTCP's goal of achieving finer-grained control of the 
cwnd, through AQMs providing more nuanced information about the state of the 
path capacity and/or bottleneck queue.  An implementation that made use of 
ECT(1) instead of changing the meaning of CE marks would remain RFC-compliant, 
and could get "sufficiently close" to the ideal described above.

 - Jonathan Morton

___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] when does the CoDel part of fq_codel help in the real world?

[Luca Muscariello]

A buffer in a router is sized once. RTT varies.
So BDP varies. That’s as simple as that.
So you just cannot be always at optimum because you don’t know what RTT you
have at any time.

Lola si not solving that. No protocol could BTW.
BTW I don’t see  any formal proof about queue occupancy in the paper.



On Tue 27 Nov 2018 at 12:53, Bless, Roland (TM) 
wrote:

> Hi Luca,
>
> Am 27.11.18 um 12:01 schrieb Luca Muscariello:
> > A BDP is not a large buffer. I'm not unveiling a secret.
>
> That depends on speed and RTT (note that typically there are
> several flows with different RTTs sharing the same buffer).
> The essential point is not how much buffer capacity is available,
> but how much is actually used, because that adds queueing delay.
>
> > And it is just a rule of thumb to have an idea at which working point
> > the protocol is working.
>
> No, one can actually prove that this is the best size for
> loss-based CC with backoff factor of 0.5 (assuming a single flow).
>
> > In practice the protocol is usually working below or above that value.
>
> That depends on the protocol.
>
> > This is where AQM and ECN help also. So most of the time the protocol is
> > working at way
> > below 100% efficiency.
>
> > My point was that FQ_codel helps to get very close to the optimum w/o
> > adding useless queueing and latency.
> > With a single queue that's almost impossible. No, sorry. Just impossible.
>
> No, it's possible. Please read the TCP LoLa paper.
>
> Regards,
>  Roland
>
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] when does the CoDel part of fq_codel help in the real world?

[Bless, Roland (TM)]

Hi Luca,

Am 27.11.18 um 12:01 schrieb Luca Muscariello:
> A BDP is not a large buffer. I'm not unveiling a secret. 

That depends on speed and RTT (note that typically there are
several flows with different RTTs sharing the same buffer).
The essential point is not how much buffer capacity is available,
but how much is actually used, because that adds queueing delay.

> And it is just a rule of thumb to have an idea at which working point
> the protocol is working.

No, one can actually prove that this is the best size for
loss-based CC with backoff factor of 0.5 (assuming a single flow).

> In practice the protocol is usually working below or above that value.

That depends on the protocol.

> This is where AQM and ECN help also. So most of the time the protocol is
> working at way 
> below 100% efficiency.

> My point was that FQ_codel helps to get very close to the optimum w/o
> adding useless queueing and latency.
> With a single queue that's almost impossible. No, sorry. Just impossible.

No, it's possible. Please read the TCP LoLa paper.

Regards,
 Roland
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] when does the CoDel part of fq_codel help in the real world?

[Toke Høiland-Jørgensen]

Luca Muscariello  writes:

> This procedure would allow to size FQ_codel but also SFQ.
> It would be interesting to compare the two under this buffer sizing.
> It would also be interesting to compare another mechanism that we have
> mentioned during the defense
> which is AFD + a sparse flow queue. Which is, BTW, already available in
> Cisco nexus switches for data centres.

One think I wondered about afterwards was whether or not it would be
feasible (as in, easy to add, or maybe even supported in current
versions) to tie in an AQM to an AFD-type virtual fairness queueing
system? You could keep the AQM state variables along with the per-flow
state and react appropriately. Any reason why this wouldn't work?

-Toke
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] when does the CoDel part of fq_codel help in the real world?

[Bless, Roland (TM)]

Hi,

Am 27.11.18 um 12:40 schrieb Bless, Roland (TM):
> Hi Luca,
> 
> Am 27.11.18 um 11:40 schrieb Luca Muscariello:
>> OK. We agree.
>> That's correct, you need *at least* the BDP in flight so that the
>> bottleneck queue never empties out.
> 
> No, that's not what I meant, but it's quite simple.
> You need: data min_inflight=2 * RTTmin * bottleneck_rate to filly

Sorry, it's meant to be: min_inflight= RTTmin * bottleneck_rate

Regards,
 Roland

> utilize the bottleneck link.
> If this is true, the bottleneck queue will be empty. If your amount
> of inflight data is larger, the bottleneck queue buffer will store
> the excess packets. With just min_inflight there will be no
> bottleneck queue, the packets are "on the wire".
> 
>> This can be easily proven using fluid models for any congestion
>> controlled source no matter if it is 
>> loss-based, delay-based, rate-based, formula-based etc.
>>
>> A highly paced source gives you the ability to get as close as
>> theoretically possible to the BDP+epsilon
>> as possible.
> 
> Yep, but that BDP is "on the wire" and epsilon will be in the bottleneck
> buffer.
> 
>> link fully utilized is defined as Q>0 unless you don't include the
>> packet currently being transmitted. I do,
>> so the TXtteer is never idle. But that's a detail.
> 
> I wouldn't define link fully utilized as Q>0, but if Q>0 then
> the link is fully utilized (that's what I meant by the direction
> of implication).
> 
> Rgards,
>  Roland
> 
>>
>>
>> On Tue, Nov 27, 2018 at 11:35 AM Bless, Roland (TM)
>> mailto:roland.bl...@kit.edu>> wrote:
>>
>> Hi,
>>
>> Am 27.11.18 um 11:29 schrieb Luca Muscariello:
>> > I have never said that you need to fill the buffer to the max size to
>> > get full capacity, which is an absurdity.
>>
>> Yes, it's absurd, but that's what today's loss-based CC algorithms do.
>>
>> > I said you need at least the BDP so that the queue never empties out.
>> > The link is fully utilized IFF the queue is never emptied.
>>
>> I was also a bit imprecise: you'll need a BDP in flight, but
>> you don't need to fill the buffer at all. The latter sentence
>> is valid only in the direction: queue not empty -> link fully utilized.
>>
>> Regards,
>>  Roland
>>
>> >
>> >
>> >
>> > On Tue 27 Nov 2018 at 11:26, Bless, Roland (TM)
>> mailto:roland.bl...@kit.edu>
>> > >> wrote:
>> >
>> >     Hi Luca,
>> >
>> >     Am 27.11.18 um 10:24 schrieb Luca Muscariello:
>> >     > A congestion controlled protocol such as TCP or others,
>> including
>> >     QUIC,
>> >     > LEDBAT and so on
>> >     > need at least the BDP in the transmission queue to get full link
>> >     > efficiency, i.e. the queue never empties out.
>> >
>> >     This is not true. There are congestion control algorithms
>> >     (e.g., TCP LoLa [1] or BBRv2) that can fully utilize the
>> bottleneck link
>> >     capacity without filling the buffer to its maximum capacity.
>> The BDP
>> >     rule of thumb basically stems from the older loss-based congestion
>> >     control variants that profit from the standing queue that they
>> built
>> >     over time when they detect a loss:
>> >     while they back-off and stop sending, the queue keeps the
>> bottleneck
>> >     output busy and you'll not see underutilization of the link.
>> Moreover,
>> >     once you get good loss de-synchronization, the buffer size
>> requirement
>> >     for multiple long-lived flows decreases.
>> >
>> >     > This gives rule of thumbs to size buffers which is also very
>> practical
>> >     > and thanks to flow isolation becomes very accurate.
>> >
>> >     The positive effect of buffers is merely their role to absorb
>> >     short-term bursts (i.e., mismatch in arrival and departure rates)
>> >     instead of dropping packets. One does not need a big buffer to
>> >     fully utilize a link (with perfect knowledge you can keep the link
>> >     saturated even without a single packet waiting in the buffer).
>> >     Furthermore, large buffers (e.g., using the BDP rule of thumb)
>> >     are not useful/practical anymore at very high speed such as
>> 100 Gbit/s:
>> >     memory is also quite costly at such high speeds...
>> >
>> >     Regards,
>> >      Roland
>> >
>> >     [1] M. Hock, F. Neumeister, M. Zitterbart, R. Bless.
>> >     TCP LoLa: Congestion Control for Low Latencies and High
>> Throughput.
>> >     Local Computer Networks (LCN), 2017 IEEE 42nd Conference on, pp.
>> >     215-218, Singapore, Singapore, October 2017
>> >     http://doc.tm.kit.edu/2017-LCN-lola-paper-authors-copy.pdf
>> >
>> >     > Which is: 
>> >     >
>> >     > 1) find a way to keep the number of backlogged flows at a
>> >     

Re: [Bloat] when does the CoDel part of fq_codel help in the real world?

[Bless, Roland (TM)]

Hi Luca,

Am 27.11.18 um 11:40 schrieb Luca Muscariello:
> OK. We agree.
> That's correct, you need *at least* the BDP in flight so that the
> bottleneck queue never empties out.

No, that's not what I meant, but it's quite simple.
You need: data min_inflight=2 * RTTmin * bottleneck_rate to filly
utilize the bottleneck link.
If this is true, the bottleneck queue will be empty. If your amount
of inflight data is larger, the bottleneck queue buffer will store
the excess packets. With just min_inflight there will be no
bottleneck queue, the packets are "on the wire".

> This can be easily proven using fluid models for any congestion
> controlled source no matter if it is 
> loss-based, delay-based, rate-based, formula-based etc.
> 
> A highly paced source gives you the ability to get as close as
> theoretically possible to the BDP+epsilon
> as possible.

Yep, but that BDP is "on the wire" and epsilon will be in the bottleneck
buffer.

> link fully utilized is defined as Q>0 unless you don't include the
> packet currently being transmitted. I do,
> so the TXtteer is never idle. But that's a detail.

I wouldn't define link fully utilized as Q>0, but if Q>0 then
the link is fully utilized (that's what I meant by the direction
of implication).

Rgards,
 Roland

> 
> 
> On Tue, Nov 27, 2018 at 11:35 AM Bless, Roland (TM)
> mailto:roland.bl...@kit.edu>> wrote:
> 
> Hi,
> 
> Am 27.11.18 um 11:29 schrieb Luca Muscariello:
> > I have never said that you need to fill the buffer to the max size to
> > get full capacity, which is an absurdity.
> 
> Yes, it's absurd, but that's what today's loss-based CC algorithms do.
> 
> > I said you need at least the BDP so that the queue never empties out.
> > The link is fully utilized IFF the queue is never emptied.
> 
> I was also a bit imprecise: you'll need a BDP in flight, but
> you don't need to fill the buffer at all. The latter sentence
> is valid only in the direction: queue not empty -> link fully utilized.
> 
> Regards,
>  Roland
> 
> >
> >
> >
> > On Tue 27 Nov 2018 at 11:26, Bless, Roland (TM)
> mailto:roland.bl...@kit.edu>
> > >> wrote:
> >
> >     Hi Luca,
> >
> >     Am 27.11.18 um 10:24 schrieb Luca Muscariello:
> >     > A congestion controlled protocol such as TCP or others,
> including
> >     QUIC,
> >     > LEDBAT and so on
> >     > need at least the BDP in the transmission queue to get full link
> >     > efficiency, i.e. the queue never empties out.
> >
> >     This is not true. There are congestion control algorithms
> >     (e.g., TCP LoLa [1] or BBRv2) that can fully utilize the
> bottleneck link
> >     capacity without filling the buffer to its maximum capacity.
> The BDP
> >     rule of thumb basically stems from the older loss-based congestion
> >     control variants that profit from the standing queue that they
> built
> >     over time when they detect a loss:
> >     while they back-off and stop sending, the queue keeps the
> bottleneck
> >     output busy and you'll not see underutilization of the link.
> Moreover,
> >     once you get good loss de-synchronization, the buffer size
> requirement
> >     for multiple long-lived flows decreases.
> >
> >     > This gives rule of thumbs to size buffers which is also very
> practical
> >     > and thanks to flow isolation becomes very accurate.
> >
> >     The positive effect of buffers is merely their role to absorb
> >     short-term bursts (i.e., mismatch in arrival and departure rates)
> >     instead of dropping packets. One does not need a big buffer to
> >     fully utilize a link (with perfect knowledge you can keep the link
> >     saturated even without a single packet waiting in the buffer).
> >     Furthermore, large buffers (e.g., using the BDP rule of thumb)
> >     are not useful/practical anymore at very high speed such as
> 100 Gbit/s:
> >     memory is also quite costly at such high speeds...
> >
> >     Regards,
> >      Roland
> >
> >     [1] M. Hock, F. Neumeister, M. Zitterbart, R. Bless.
> >     TCP LoLa: Congestion Control for Low Latencies and High
> Throughput.
> >     Local Computer Networks (LCN), 2017 IEEE 42nd Conference on, pp.
> >     215-218, Singapore, Singapore, October 2017
> >     http://doc.tm.kit.edu/2017-LCN-lola-paper-authors-copy.pdf
> >
> >     > Which is: 
> >     >
> >     > 1) find a way to keep the number of backlogged flows at a
> >     reasonable value. 
> >     > This largely depends on the minimum fair rate an application may
> >     need in
> >     > the long term.
> >     > We discussed a little bit of available mechanisms to achieve
> that
> >     in the
> >     > 

Re: [Bloat] when does the CoDel part of fq_codel help in the real world?

[Mikael Abrahamsson]

On Tue, 27 Nov 2018, Luca Muscariello wrote:


A BDP is not a large buffer. I'm not unveiling a secret.


It's complicated. I've had people throw in my face that I need 2xBDP in 
buffer size to smoothe things out. Personally I don't want more than 10ms 
buffer (max), and I don't see why I should need more than that even if 
transfers are running over hundreds of ms of light-speed-in-medium induced 
delay between the communicating systems.


I have routers that are perfectly capable at buffering packets for 
hundreds of ms even at hundreds of megabits/s of access speed. I choose 
not to use them though, and configure them to drop packets much earlier.


My point was that FQ_codel helps to get very close to the optimum w/o 
adding useless queueing and latency. With a single queue that's almost 
impossible. No, sorry. Just impossible.


Right, I realise I wasn't clear I wasn't actually commenting on your 
specific text directly, my question was more generic.


--
Mikael Abrahamssonemail: swm...@swm.pp.se
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] when does the CoDel part of fq_codel help in the real world?

[Michael Welzl]

Well, I'm concerned about the delay experienced by people when they surf the 
web... flow completion time, which relates not only to the delay of packets as 
they are sent from A to B, but also the utilization.

Cheers,
Michael


> On 27 Nov 2018, at 11:50, Mikael Abrahamsson  wrote:
> 
> On Tue, 27 Nov 2018, Luca Muscariello wrote:
> 
>> link fully utilized is defined as Q>0 unless you don't include the packet 
>> currently being transmitted. I do, so the TXtteer is never idle. But that's 
>> a detail.
> 
> As someone who works with moving packets, it's perplexing to me to interact 
> with transport peeps who seem enormously focused on "goodput". My personal 
> opinion is that most people would be better off with 80% of their available 
> bandwidth being in use without any noticable buffer induced delay, as opposed 
> to the transport protocol doing its damndest to fill up the link to 100% and 
> sometimes failing and inducing delay instead.
> 
> Could someone perhaps comment on the thinking in the transport protocol 
> design "crowd" when it comes to this?
> 
> -- 
> Mikael Abrahamssonemail: swm...@swm.pp.se
> ___
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat

___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] when does the CoDel part of fq_codel help in the real world?

[Jonathan Morton]

> On 27 Nov, 2018, at 12:50 pm, Mikael Abrahamsson  wrote:
> 
> Could someone perhaps comment on the thinking in the transport protocol 
> design "crowd" when it comes to this?

BBR purports to aim for the optimum of maximum throughput at minimum latency; 
there is a sharp knee in the throughput-latency graph, at least in an idealised 
scenario.  In practice it's more complicated, hence the gradual evolution of 
BBR.

Previously, there has been a dichotomy between loss-based TCPs which aim for 
maximum throughput regardless of latency, and delay-based TCPs which aim for 
minimum latency with little regard for throughput.  Pretty much nobody uses the 
latter in the real world, because they get outcompeted by loss-based traffic 
when they meekly back down at the first sign of a queuing delay.  Everyone uses 
loss-based TCPs, generally NewReno, CUBIC, or Compound.  CUBIC is specifically 
designed to spend most of its time near queue saturation, by growing more 
slowly when near the cwnd at which loss was last experienced than when distant 
from it.

Compound is actually an interesting special case.  It's a very straightforward 
combination of a loss-based TCP and a delay-based one, such that it spends a 
lot of time at or near the optimum operating point - at least in theory.  
However, it will always eventually transition into a NewReno-like mode, and 
fill the queue until loss is experienced.

LEDBAT is a delay-based algorithm that can be applied to protocols other than 
TCP.  It's often used in BitTorrent clients as part of µTP.  However, the sheer 
weight of flows employed by BT clients tends to overwhelm the algorithm, as 
remote senders often collectively flood the queue with near-simultaneous bursts 
in response to changes in collective swarm state.  BT client authors seem to be 
ill-equipped to address this problem adequately.

 - Jonathan Morton

___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] when does the CoDel part of fq_codel help in the real world?

[Luca Muscariello]

A BDP is not a large buffer. I'm not unveiling a secret.
And it is just a rule of thumb to have an idea at which working point the
protocol is working.
In practice the protocol is usually working below or above that value.
This is where AQM and ECN help also. So most of the time the protocol is
working at way
below 100% efficiency.

My point was that FQ_codel helps to get very close to the optimum w/o
adding useless queueing and latency.
With a single queue that's almost impossible. No, sorry. Just impossible.



On Tue, Nov 27, 2018 at 11:50 AM Mikael Abrahamsson 
wrote:

> On Tue, 27 Nov 2018, Luca Muscariello wrote:
>
> > link fully utilized is defined as Q>0 unless you don't include the
> > packet currently being transmitted. I do, so the TXtteer is never idle.
> > But that's a detail.
>
> As someone who works with moving packets, it's perplexing to me to
> interact with transport peeps who seem enormously focused on "goodput". My
> personal opinion is that most people would be better off with 80% of their
> available bandwidth being in use without any noticable buffer induced
> delay, as opposed to the transport protocol doing its damndest to fill up
> the link to 100% and sometimes failing and inducing delay instead.
>
> Could someone perhaps comment on the thinking in the transport protocol
> design "crowd" when it comes to this?
>
> --
> Mikael Abrahamssonemail: swm...@swm.pp.se
>
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] when does the CoDel part of fq_codel help in the real world?

[Mikael Abrahamsson]

On Tue, 27 Nov 2018, Luca Muscariello wrote:

link fully utilized is defined as Q>0 unless you don't include the 
packet currently being transmitted. I do, so the TXtteer is never idle. 
But that's a detail.


As someone who works with moving packets, it's perplexing to me to 
interact with transport peeps who seem enormously focused on "goodput". My 
personal opinion is that most people would be better off with 80% of their 
available bandwidth being in use without any noticable buffer induced 
delay, as opposed to the transport protocol doing its damndest to fill up 
the link to 100% and sometimes failing and inducing delay instead.


Could someone perhaps comment on the thinking in the transport protocol 
design "crowd" when it comes to this?


--
Mikael Abrahamssonemail: swm...@swm.pp.se
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] when does the CoDel part of fq_codel help in the real world?

[Luca Muscariello]

OK. We agree.
That's correct, you need *at least* the BDP in flight so that the
bottleneck queue never empties out.

This can be easily proven using fluid models for any congestion controlled
source no matter if it is
loss-based, delay-based, rate-based, formula-based etc.

A highly paced source gives you the ability to get as close as
theoretically possible to the BDP+epsilon
as possible.

link fully utilized is defined as Q>0 unless you don't include the packet
currently being transmitted. I do,
so the TXtteer is never idle. But that's a detail.



On Tue, Nov 27, 2018 at 11:35 AM Bless, Roland (TM) 
wrote:

> Hi,
>
> Am 27.11.18 um 11:29 schrieb Luca Muscariello:
> > I have never said that you need to fill the buffer to the max size to
> > get full capacity, which is an absurdity.
>
> Yes, it's absurd, but that's what today's loss-based CC algorithms do.
>
> > I said you need at least the BDP so that the queue never empties out.
> > The link is fully utilized IFF the queue is never emptied.
>
> I was also a bit imprecise: you'll need a BDP in flight, but
> you don't need to fill the buffer at all. The latter sentence
> is valid only in the direction: queue not empty -> link fully utilized.
>
> Regards,
>  Roland
>
> >
> >
> >
> > On Tue 27 Nov 2018 at 11:26, Bless, Roland (TM)  > > wrote:
> >
> > Hi Luca,
> >
> > Am 27.11.18 um 10:24 schrieb Luca Muscariello:
> > > A congestion controlled protocol such as TCP or others, including
> > QUIC,
> > > LEDBAT and so on
> > > need at least the BDP in the transmission queue to get full link
> > > efficiency, i.e. the queue never empties out.
> >
> > This is not true. There are congestion control algorithms
> > (e.g., TCP LoLa [1] or BBRv2) that can fully utilize the bottleneck
> link
> > capacity without filling the buffer to its maximum capacity. The BDP
> > rule of thumb basically stems from the older loss-based congestion
> > control variants that profit from the standing queue that they built
> > over time when they detect a loss:
> > while they back-off and stop sending, the queue keeps the bottleneck
> > output busy and you'll not see underutilization of the link.
> Moreover,
> > once you get good loss de-synchronization, the buffer size
> requirement
> > for multiple long-lived flows decreases.
> >
> > > This gives rule of thumbs to size buffers which is also very
> practical
> > > and thanks to flow isolation becomes very accurate.
> >
> > The positive effect of buffers is merely their role to absorb
> > short-term bursts (i.e., mismatch in arrival and departure rates)
> > instead of dropping packets. One does not need a big buffer to
> > fully utilize a link (with perfect knowledge you can keep the link
> > saturated even without a single packet waiting in the buffer).
> > Furthermore, large buffers (e.g., using the BDP rule of thumb)
> > are not useful/practical anymore at very high speed such as 100
> Gbit/s:
> > memory is also quite costly at such high speeds...
> >
> > Regards,
> >  Roland
> >
> > [1] M. Hock, F. Neumeister, M. Zitterbart, R. Bless.
> > TCP LoLa: Congestion Control for Low Latencies and High Throughput.
> > Local Computer Networks (LCN), 2017 IEEE 42nd Conference on, pp.
> > 215-218, Singapore, Singapore, October 2017
> > http://doc.tm.kit.edu/2017-LCN-lola-paper-authors-copy.pdf
> >
> > > Which is:
> > >
> > > 1) find a way to keep the number of backlogged flows at a
> > reasonable value.
> > > This largely depends on the minimum fair rate an application may
> > need in
> > > the long term.
> > > We discussed a little bit of available mechanisms to achieve that
> > in the
> > > literature.
> > >
> > > 2) fix the largest RTT you want to serve at full utilization and
> size
> > > the buffer using BDP * N_backlogged.
> > > Or the other way round: check how much memory you can use
> > > in the router/line card/device and for a fixed N, compute the
> largest
> > > RTT you can serve at full utilization.
> > >
> > > 3) there is still some memory to dimension for sparse flows in
> > addition
> > > to that, but this is not based on BDP.
> > > It is just enough to compute the total utilization of sparse flows
> and
> > > use the same simple model Toke has used
> > > to compute the (de)prioritization probability.
> > >
> > > This procedure would allow to size FQ_codel but also SFQ.
> > > It would be interesting to compare the two under this buffer
> sizing.
> > > It would also be interesting to compare another mechanism that we
> have
> > > mentioned during the defense
> > > which is AFD + a sparse flow queue. Which is, BTW, already
> > available in
> > > Cisco nexus switches for data centres.
> > >
> > > I think that the the codel part would still 

Re: [Bloat] when does the CoDel part of fq_codel help in the real world?

[Bless, Roland (TM)]

Hi,

Am 27.11.18 um 11:29 schrieb Luca Muscariello:
> I have never said that you need to fill the buffer to the max size to
> get full capacity, which is an absurdity.

Yes, it's absurd, but that's what today's loss-based CC algorithms do.

> I said you need at least the BDP so that the queue never empties out.
> The link is fully utilized IFF the queue is never emptied.

I was also a bit imprecise: you'll need a BDP in flight, but
you don't need to fill the buffer at all. The latter sentence
is valid only in the direction: queue not empty -> link fully utilized.

Regards,
 Roland

> 
> 
> 
> On Tue 27 Nov 2018 at 11:26, Bless, Roland (TM)  > wrote:
> 
> Hi Luca,
> 
> Am 27.11.18 um 10:24 schrieb Luca Muscariello:
> > A congestion controlled protocol such as TCP or others, including
> QUIC,
> > LEDBAT and so on
> > need at least the BDP in the transmission queue to get full link
> > efficiency, i.e. the queue never empties out.
> 
> This is not true. There are congestion control algorithms
> (e.g., TCP LoLa [1] or BBRv2) that can fully utilize the bottleneck link
> capacity without filling the buffer to its maximum capacity. The BDP
> rule of thumb basically stems from the older loss-based congestion
> control variants that profit from the standing queue that they built
> over time when they detect a loss:
> while they back-off and stop sending, the queue keeps the bottleneck
> output busy and you'll not see underutilization of the link. Moreover,
> once you get good loss de-synchronization, the buffer size requirement
> for multiple long-lived flows decreases.
> 
> > This gives rule of thumbs to size buffers which is also very practical
> > and thanks to flow isolation becomes very accurate.
> 
> The positive effect of buffers is merely their role to absorb
> short-term bursts (i.e., mismatch in arrival and departure rates)
> instead of dropping packets. One does not need a big buffer to
> fully utilize a link (with perfect knowledge you can keep the link
> saturated even without a single packet waiting in the buffer).
> Furthermore, large buffers (e.g., using the BDP rule of thumb)
> are not useful/practical anymore at very high speed such as 100 Gbit/s:
> memory is also quite costly at such high speeds...
> 
> Regards,
>  Roland
> 
> [1] M. Hock, F. Neumeister, M. Zitterbart, R. Bless.
> TCP LoLa: Congestion Control for Low Latencies and High Throughput.
> Local Computer Networks (LCN), 2017 IEEE 42nd Conference on, pp.
> 215-218, Singapore, Singapore, October 2017
> http://doc.tm.kit.edu/2017-LCN-lola-paper-authors-copy.pdf
> 
> > Which is: 
> >
> > 1) find a way to keep the number of backlogged flows at a
> reasonable value. 
> > This largely depends on the minimum fair rate an application may
> need in
> > the long term.
> > We discussed a little bit of available mechanisms to achieve that
> in the
> > literature.
> >
> > 2) fix the largest RTT you want to serve at full utilization and size
> > the buffer using BDP * N_backlogged.  
> > Or the other way round: check how much memory you can use 
> > in the router/line card/device and for a fixed N, compute the largest
> > RTT you can serve at full utilization. 
> >
> > 3) there is still some memory to dimension for sparse flows in
> addition
> > to that, but this is not based on BDP. 
> > It is just enough to compute the total utilization of sparse flows and
> > use the same simple model Toke has used 
> > to compute the (de)prioritization probability.
> >
> > This procedure would allow to size FQ_codel but also SFQ.
> > It would be interesting to compare the two under this buffer sizing. 
> > It would also be interesting to compare another mechanism that we have
> > mentioned during the defense
> > which is AFD + a sparse flow queue. Which is, BTW, already
> available in
> > Cisco nexus switches for data centres.
> >
> > I think that the the codel part would still provide the ECN feature,
> > that all the others cannot have.
> > However the others, the last one especially can be implemented in
> > silicon with reasonable cost.
> 

___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] when does the CoDel part of fq_codel help in the real world?

[Luca Muscariello]

I have never said that you need to fill the buffer to the max size to get
full capacity, which is an absurdity.

I said you need at least the BDP so that the queue never empties out.
The link is fully utilized IFF the queue is never emptied.



On Tue 27 Nov 2018 at 11:26, Bless, Roland (TM) 
wrote:

> Hi Luca,
>
> Am 27.11.18 um 10:24 schrieb Luca Muscariello:
> > A congestion controlled protocol such as TCP or others, including QUIC,
> > LEDBAT and so on
> > need at least the BDP in the transmission queue to get full link
> > efficiency, i.e. the queue never empties out.
>
> This is not true. There are congestion control algorithms
> (e.g., TCP LoLa [1] or BBRv2) that can fully utilize the bottleneck link
> capacity without filling the buffer to its maximum capacity. The BDP
> rule of thumb basically stems from the older loss-based congestion
> control variants that profit from the standing queue that they built
> over time when they detect a loss:
> while they back-off and stop sending, the queue keeps the bottleneck
> output busy and you'll not see underutilization of the link. Moreover,
> once you get good loss de-synchronization, the buffer size requirement
> for multiple long-lived flows decreases.
>
> > This gives rule of thumbs to size buffers which is also very practical
> > and thanks to flow isolation becomes very accurate.
>
> The positive effect of buffers is merely their role to absorb
> short-term bursts (i.e., mismatch in arrival and departure rates)
> instead of dropping packets. One does not need a big buffer to
> fully utilize a link (with perfect knowledge you can keep the link
> saturated even without a single packet waiting in the buffer).
> Furthermore, large buffers (e.g., using the BDP rule of thumb)
> are not useful/practical anymore at very high speed such as 100 Gbit/s:
> memory is also quite costly at such high speeds...
>
> Regards,
>  Roland
>
> [1] M. Hock, F. Neumeister, M. Zitterbart, R. Bless.
> TCP LoLa: Congestion Control for Low Latencies and High Throughput.
> Local Computer Networks (LCN), 2017 IEEE 42nd Conference on, pp.
> 215-218, Singapore, Singapore, October 2017
> http://doc.tm.kit.edu/2017-LCN-lola-paper-authors-copy.pdf
>
> > Which is:
> >
> > 1) find a way to keep the number of backlogged flows at a reasonable
> value.
> > This largely depends on the minimum fair rate an application may need in
> > the long term.
> > We discussed a little bit of available mechanisms to achieve that in the
> > literature.
> >
> > 2) fix the largest RTT you want to serve at full utilization and size
> > the buffer using BDP * N_backlogged.
> > Or the other way round: check how much memory you can use
> > in the router/line card/device and for a fixed N, compute the largest
> > RTT you can serve at full utilization.
> >
> > 3) there is still some memory to dimension for sparse flows in addition
> > to that, but this is not based on BDP.
> > It is just enough to compute the total utilization of sparse flows and
> > use the same simple model Toke has used
> > to compute the (de)prioritization probability.
> >
> > This procedure would allow to size FQ_codel but also SFQ.
> > It would be interesting to compare the two under this buffer sizing.
> > It would also be interesting to compare another mechanism that we have
> > mentioned during the defense
> > which is AFD + a sparse flow queue. Which is, BTW, already available in
> > Cisco nexus switches for data centres.
> >
> > I think that the the codel part would still provide the ECN feature,
> > that all the others cannot have.
> > However the others, the last one especially can be implemented in
> > silicon with reasonable cost.
>
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] when does the CoDel part of fq_codel help in the real world?

[Bless, Roland (TM)]

Hi Luca,

Am 27.11.18 um 10:24 schrieb Luca Muscariello:
> A congestion controlled protocol such as TCP or others, including QUIC,
> LEDBAT and so on
> need at least the BDP in the transmission queue to get full link
> efficiency, i.e. the queue never empties out.

This is not true. There are congestion control algorithms
(e.g., TCP LoLa [1] or BBRv2) that can fully utilize the bottleneck link
capacity without filling the buffer to its maximum capacity. The BDP
rule of thumb basically stems from the older loss-based congestion
control variants that profit from the standing queue that they built
over time when they detect a loss:
while they back-off and stop sending, the queue keeps the bottleneck
output busy and you'll not see underutilization of the link. Moreover,
once you get good loss de-synchronization, the buffer size requirement
for multiple long-lived flows decreases.

> This gives rule of thumbs to size buffers which is also very practical
> and thanks to flow isolation becomes very accurate.

The positive effect of buffers is merely their role to absorb
short-term bursts (i.e., mismatch in arrival and departure rates)
instead of dropping packets. One does not need a big buffer to
fully utilize a link (with perfect knowledge you can keep the link
saturated even without a single packet waiting in the buffer).
Furthermore, large buffers (e.g., using the BDP rule of thumb)
are not useful/practical anymore at very high speed such as 100 Gbit/s:
memory is also quite costly at such high speeds...

Regards,
 Roland

[1] M. Hock, F. Neumeister, M. Zitterbart, R. Bless.
TCP LoLa: Congestion Control for Low Latencies and High Throughput.
Local Computer Networks (LCN), 2017 IEEE 42nd Conference on, pp.
215-218, Singapore, Singapore, October 2017
http://doc.tm.kit.edu/2017-LCN-lola-paper-authors-copy.pdf

> Which is: 
> 
> 1) find a way to keep the number of backlogged flows at a reasonable value. 
> This largely depends on the minimum fair rate an application may need in
> the long term.
> We discussed a little bit of available mechanisms to achieve that in the
> literature.
> 
> 2) fix the largest RTT you want to serve at full utilization and size
> the buffer using BDP * N_backlogged.  
> Or the other way round: check how much memory you can use 
> in the router/line card/device and for a fixed N, compute the largest
> RTT you can serve at full utilization. 
> 
> 3) there is still some memory to dimension for sparse flows in addition
> to that, but this is not based on BDP. 
> It is just enough to compute the total utilization of sparse flows and
> use the same simple model Toke has used 
> to compute the (de)prioritization probability.
> 
> This procedure would allow to size FQ_codel but also SFQ.
> It would be interesting to compare the two under this buffer sizing. 
> It would also be interesting to compare another mechanism that we have
> mentioned during the defense
> which is AFD + a sparse flow queue. Which is, BTW, already available in
> Cisco nexus switches for data centres.
> 
> I think that the the codel part would still provide the ECN feature,
> that all the others cannot have.
> However the others, the last one especially can be implemented in
> silicon with reasonable cost.
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] when does the CoDel part of fq_codel help in the real world?

[Jonathan Morton]

> On 27 Nov, 2018, at 10:54 am, Pete Heist  wrote:
> 
> …any reason that fq_codel or Cake would be an improvement over sfq 
> specifically for some "noisy links” (loose translation from Czech) in a 
> backhaul that have some loss but also experience saturation.

If the random loss is low enough that saturation can be achieved, then adding 
AQM will be beneficial in the usual way.  Also, it will not interfere in those 
cases where saturation is not achieved (for whatever reason).

 - Jonathan Morton

___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] when does the CoDel part of fq_codel help in the real world?

[Luca Muscariello]

I think that this is a very good comment to the discussion at the defense
about the comparison between
SFQ with longest queue drop and FQ_Codel.

A congestion controlled protocol such as TCP or others, including QUIC,
LEDBAT and so on
need at least the BDP in the transmission queue to get full link
efficiency, i.e. the queue never empties out.
This gives rule of thumbs to size buffers which is also very practical and
thanks to flow isolation becomes very accurate.

Which is:

1) find a way to keep the number of backlogged flows at a reasonable value.
This largely depends on the minimum fair rate an application may need in
the long term.
We discussed a little bit of available mechanisms to achieve that in the
literature.

2) fix the largest RTT you want to serve at full utilization and size the
buffer using BDP * N_backlogged.
Or the other way round: check how much memory you can use
in the router/line card/device and for a fixed N, compute the largest RTT
you can serve at full utilization.

3) there is still some memory to dimension for sparse flows in addition to
that, but this is not based on BDP.
It is just enough to compute the total utilization of sparse flows and use
the same simple model Toke has used
to compute the (de)prioritization probability.

This procedure would allow to size FQ_codel but also SFQ.
It would be interesting to compare the two under this buffer sizing.
It would also be interesting to compare another mechanism that we have
mentioned during the defense
which is AFD + a sparse flow queue. Which is, BTW, already available in
Cisco nexus switches for data centres.

I think that the the codel part would still provide the ECN feature, that
all the others cannot have.
However the others, the last one especially can be implemented in silicon
with reasonable cost.





On Mon 26 Nov 2018 at 22:30, Jonathan Morton  wrote:

> > On 26 Nov, 2018, at 9:08 pm, Pete Heist  wrote:
> >
> > So I just thought to continue the discussion- when does the CoDel part
> of fq_codel actually help in the real world?
>
> Fundamentally, without Codel the only limits on the congestion window
> would be when the sender or receiver hit configured or calculated rwnd and
> cwnd limits (the rwnd is visible on the wire and usually chosen to be large
> enough to be a non-factor), or when the queue overflows.  Large windows
> require buffer memory in both sender and receiver, increasing costs on the
> sender in particular (who typically has many flows to manage per machine).
>
> Queue overflow tends to result in burst loss and head-of-line blocking in
> the receiver, which is visible to the user as a pause and subsequent jump
> in the progress of their download, accompanied by a major fluctuation in
> the estimated time to completion.  The lost packets also consume capacity
> upstream of the bottleneck which does not contribute to application
> throughput.  These effects are independent of whether overflow dropping
> occurs at the head or tail of the bottleneck queue, though recovery occurs
> more quickly (and fewer packets might be lost) if dropping occurs from the
> head of the queue.
>
> From a pure throughput-efficiency standpoint, Codel allows using ECN for
> congestion signalling instead of packet loss, potentially eliminating
> packet loss and associated lead-of-line blocking entirely.  Even without
> ECN, the actual cwnd is kept near the minimum necessary to satisfy the BDP
> of the path, reducing memory requirements and significantly shortening the
> recovery time of each loss cycle, to the point where the end-user may not
> notice that delivery is not perfectly smooth, and implementing accurate
> completion time estimators is considerably simplified.
>
> An important use-case is where two sequential bottlenecks exist on the
> path, the upstream one being only slightly higher capacity but lacking any
> queue management at all.  This is presently common in cases where home CPE
> implements inbound shaping on a generic ISP last-mile link.  In that case,
> without Codel running on the second bottleneck, traffic would collect in
> the first bottleneck's queue as well, greatly reducing the beneficial
> effects of FQ implemented on the second bottleneck.  In this topology, the
> overall effect is inter-flow as well as intra-flow.
>
> The combination of Codel with FQ is done in such a way that a separate
> instance of Codel is implemented for each flow.  This means that congestion
> signals are only sent to flows that require them, and non-saturating flows
> are unmolested.  This makes the combination synergistic, where each
> component offers an improvement to the behaviour of the other.
>
>  - Jonathan Morton
>
> ___
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] when does the CoDel part of fq_codel help in the real world?

[Pete Heist]

Thank you all for the responses!

I was asked a related question by my local WISP, who wanted to know if there 
would be any reason that fq_codel or Cake would be an improvement over sfq 
specifically for some "noisy links” (loose translation from Czech) in a 
backhaul that have some loss but also experience saturation. I conservatively 
answered no, but that may not be correct, in case the reduced TCP RTT could 
help with loss recovery or other behaviors, as Jon pointed out. I suspect more 
research would be needed to quantify this. Neal’s/Dave's point about “non-flow" 
traffic is well taken also.

> On Nov 26, 2018, at 11:13 PM, Toke Høiland-Jørgensen  wrote:
> 
> Michael Welzl  writes:
> 
>> However, I would like to point out that thesis defense conversations
>> are meant to be provocative, by design - when I said that CoDel
>> doesn’t usually help and long queues would be the right thing for all
>> applications, I certainly didn’t REALLY REALLY mean that.
> 
> Just as I don't REALLY REALLY mean that bigger buffers are always better
> as you so sneakily tricked me into blurting out ;)

I think most of us knew that “yeah” wasn’t a confirmation. Yeah can be used in 
a dozen different ways depending on context and intonation, but it did give 
some comic relief. :)

>> BTW, Anna Brunstrom was also very quick to also give me the HTTP/2.0
>> example in the break after the defense.
> 
> Yup, was thinking of HTTP/2 when I said "control data on the same
> connection as the payload". Can see from Pete's transcript that it
> didn't come across terribly clearly, though :P

Ah, sorry for missing that part! I thought there was more there but didn’t want 
to write something unless I was sure I heard it.

>> I’ll use the opportunity to tell folks that I was also pretty
>> impressed with Toke’s thesis as well as his performance at the
>> defense.

I’ll second that. I’m enjoying digesting the thesis, as well as the results of 
airtime fairness in a real world deployment. :)
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat