Re: [Cake] [Rpm] [Make-wifi-fast] The most wonderful video ever about bufferbloat

[Stuart Cheshire via Cake]

On 20 Oct 2022, at 02:36, Sebastian Moeller  wrote:

> Hi Stuart,
> 
> [SM] That seems to be somewhat optimistic. We have been there before, short 
> of mandating actually-working oracle schedulers on all end-points, 
> intermediate hops will see queues some more and some less transient. So we 
> can strive to minimize queue build-up sure, but can not avoid queues and long 
> queues completely so we need methods to deal with them gracefully.
> Also not many applications are actually helped all that much by letting 
> information get stale in their own buffers as compared to an on-path queue. 
> Think an on-line reaction-time gated game, the need is to distribute current 
> world state to all participating clients ASAP.

I’m afraid you are wrong about this. If an on-line game wants low delay, the 
only answer is for it to avoid generating position updates faster than the 
network carry them. One packet giving the current game player position is 
better than a backlog of ten previous stale ones waiting to go out. Sending 
packets faster than the network can carry them does not get them to the 
destination faster; it gets them there slower. The same applies to frames in a 
screen sharing application. Sending the current state of the screen *now* is 
better than having a backlog of ten previous stale frames sitting in buffers 
somewhere on their way to the destination. Stale data is not inevitable. 
Applications don’t need to have stale data if they avoid generating stale data 
in the first place.

Please watch this video, which explains it better than I can in a written email:



Stuart Cheshire

___
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] [Rpm] [Make-wifi-fast] The most wonderful video ever about bufferbloat

[Sebastian Moeller via Cake]

Hi Dave,


> On Oct 26, 2022, at 22:42, Dave Taht  wrote:
> 
> I loved paced chirping.

[SM] Yes it sounded like a clever idea (however I would prefer a 
clearer signal from the network about queue-filling). However I have heard 
precious little about parced-chirping actually working in the real internet, 
which IMHO means the following questions are still open:
a) does it actually work under any realistic conditions?
b) under which condition will it fail?
c) how likely are these conditions over the existing internet?

IIRC it uses packet spacing to deduce whether capacity has been reached, and 
since packet spacing is a known unreliable source of information it needs to 
average and aggregate to make up for operating based on questionable data. IMHO 
it would be preferable to solve the "questionable data" problem as my gut 
feeling is with better data would come simpler solutions to the same challenge.
Or I am simply misremembering the whole thing and be barking up the wrong tree 
;)

> I also loved packet subwindows.

That was allowing "rates" below one packet per RTT?

> I wish we could all agree to get
> cracking on working on those two things for cubic and reno rather than
> whinging all the time about the stuff we will never agree on.

;). You might have seen some of my code, which should indicate that maybe I am 
not going to be all that helpful in the "get cracking" department ;)


Regards
Sebastian
___
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] [Rpm] [Make-wifi-fast] The most wonderful video ever about bufferbloat

[Dave Taht via Cake]

I loved paced chirping.

I also loved packet subwindows. I wish we could all agree to get
cracking on working on those two things for cubic and reno rather than
whinging all the time about the stuff we will never agree on.
___
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] [Rpm] [Make-wifi-fast] The most wonderful video ever about bufferbloat

[Sebastian Moeller via Cake]

Hi Stuart,


> On Oct 20, 2022, at 20:32, Stuart Cheshire  wrote:
> 
> On 20 Oct 2022, at 02:36, Sebastian Moeller  wrote:
> 
>> Hi Stuart,
>> 
>> [SM] That seems to be somewhat optimistic. We have been there before, short 
>> of mandating actually-working oracle schedulers on all end-points, 
>> intermediate hops will see queues some more and some less transient. So we 
>> can strive to minimize queue build-up sure, but can not avoid queues and 
>> long queues completely so we need methods to deal with them gracefully.
>> Also not many applications are actually helped all that much by letting 
>> information get stale in their own buffers as compared to an on-path queue. 
>> Think an on-line reaction-time gated game, the need is to distribute current 
>> world state to all participating clients ASAP.
> 
> I’m afraid you are wrong about this.

[SM] Well possible, would not be a first. ;)


> If an on-line game wants low delay, the only answer is for it to avoid 
> generating position updates faster than the network carry them.

[SM] +1; it seems I misconstrued the argument I wanted to make when 
bringing up gaming though. If you allow I will try to lay out why I believe 
that for some applications like some forms of gaming a competent scheduler can 
be leaps and bounds more helpful than the best AQM.
Let's start with me conceding that when the required average rate of an 
application exceeds the networks capacity (for too much of the time) that 
application and that network path are not going to become/stay friends.
That out of the way, the application profile I wanted to describe with 
the "gaming" tag is an application that on average sends relatively little, 
albeit in a clocked/bursty way, where every X milliseconds it wants to send a 
bunch of packets to each client; and where the fidelity of the predictive 
"simulation" performed by the clients critically depends on not deviating from 
the server-managed "word-state" for too long. (The longer the simulation runs 
without server updates the larger the expected deviation becomes and the more 
noticeable any actions that need to be taken later once world-updates arrive, 
so the goal is to send world-state relevant updates as soon as possible after 
the server calculated the authoritative state).
These burtst will likely be sent close to the server's line rate and 
hence will create a (hopefully) transient queue at all places where the 
capacity gets smaller along the path. However the end result is that these 
packets arrive at the client as fast as possible.


> One packet giving the current game player position is better than a backlog 
> of ten previous stale ones waiting to go out.

[SM] Yes! In a multiplayer game each client really needs to be informed 
about all other players'/entites' actions. If this information is often? send 
in multiple packets (either because aggregate size exceeds a packet's "MTU/MSS" 
or because implementation-wise sending one packet per individual entity 
(players, NPCs, "bullets", ...) is preferable. Then all packets need to be 
received to appropriately update world-state... the faster this goes the less 
do clients go out of "sync".


> Sending packets faster than the network can carry them does not get them to 
> the destination faster; it gets them there slower.

[SM] Again I fully agree. Although in the limit case on an otherwise 
idle network sending our hypothetical bunch of packets from the server either 
at line rate or paced out to the bottleneck-rate of the path should deliver the 
bunch equally fast. That is sending the bunch as bunch is IMHO a rationale and 
defensible strategy for the server relieving it from having to keep state for 
the capacity for each client.

> The same applies to frames in a screen sharing application. Sending the 
> current state of the screen *now* is better than having a backlog of ten 
> previous stale frames sitting in buffers somewhere on their way to the 
> destination.

[SM] I respectfully argue that a screen sharing application that will 
send for prolonged durations well above a path's capacity is either not 
optimally designed or mis-configured. As I wrote before, I used (the free 
version of nomachine's) NX remote control across the Atlantic to southern 
California, and while not all that enjoyable it was leaps and bounds more 
usable than what you demonstrated in the video below. (I did however make 
concessions, like configuring NX to expect a slow WAN link manually, and did 
not configure full window dragging on the remote host).


> Stale data is not inevitable. Applications don’t need to have stale data if 
> they avoid generating stale data in the first place.

[SM] Alas no application using an internet path is in full control of 
avoiding queueing. Queues have a reason to exist (I personally like 
Nichols/Jacobsen description of queues acting as shock absorbers), especially 
over shared path wi

Re: [Cake] [Rpm] [Make-wifi-fast] The most wonderful video ever about bufferbloat

[Bob McMahon via Cake]

Hi All,

I just wanted to say thanks for this discussion. I always learn from each
and all of you.

Bob

On Thu, Oct 20, 2022 at 12:40 PM Sebastian Moeller  wrote:

> Hi Dave,
>
>
> > On Oct 20, 2022, at 21:12, Dave Taht via Rpm 
> wrote:
> >
> > On Thu, Oct 20, 2022 at 12:04 PM Bob McMahon via Make-wifi-fast
> >  wrote:
> >>
> >> Intel has a good analogous video on this with their CPU video here
> going over branches and failed predictions. And to Stuart's point, the
> longer pipelines make the forks worse in the amount of in-process bytes
> that need to be thrown away. Interactivity, in my opinion, suggests
> shrinking the pipeline because, with networks, there is no quick way to
> throw away stale data rather every forwarding device continues forward with
> invalid data. That's bad for the network too, spending resources on
> something that's no longer valid. We in the test & measurement community
> never measure this.
> >
> > One of my all time favorite demos was of stuart's remote desktop
> > scenario, where he moved the mouse and the window moved with it.
>
> [SM] Fair enough. However in 2015 I had been using NX's remote X11
> desktop solution which even from Central Europe to California allowed me to
> remote control graphical applications way better than the first demo with
> the multi-second delay between mouse movement and resulting screen updates.
> (This was over a 6/1 Mbps ADSL link, admittedly using HTB-fq_codel, but
> since it did not saturate the link I assign the usability to NX's better
> design). I will make an impolite suggestion here, that the demonstrated
> screen sharing program simply had not yet been optimized/designed for
> longer slower paths...
>
> Regards
> Sebastian
>
>
> >
> >> There have been a few requests that iperf 2 measure the "bytes thrown
> away" per a fork (user moves a video pointer, etc.) I haven't come up with
> a good test yet. I'm still trying to get basic awareness about existing
> latency, OWD and responsiveness metrics. I do think measuring the amount of
> resources spent on stale data is sorta like food waste, few really pay
> attention to it.
> >>
> >> Bob
> >>
> >> FYI, iperf 2 supports TCP_NOTSENT_LOWAT for those interested.
> >>
> >> --tcp-write-prefetch n[kmKM]
> >> Set TCP_NOTSENT_LOWAT on the socket and use event based writes per
> select() on the socket.
> >>
> >>
> >> On Thu, Oct 20, 2022 at 11:32 AM Stuart Cheshire via Make-wifi-fast <
> make-wifi-f...@lists.bufferbloat.net> wrote:
> >>>
> >>> On 20 Oct 2022, at 02:36, Sebastian Moeller  wrote:
> >>>
>  Hi Stuart,
> 
>  [SM] That seems to be somewhat optimistic. We have been there before,
> short of mandating actually-working oracle schedulers on all end-points,
> intermediate hops will see queues some more and some less transient. So we
> can strive to minimize queue build-up sure, but can not avoid queues and
> long queues completely so we need methods to deal with them gracefully.
>  Also not many applications are actually helped all that much by
> letting information get stale in their own buffers as compared to an
> on-path queue. Think an on-line reaction-time gated game, the need is to
> distribute current world state to all participating clients ASAP.
> >>>
> >>> I’m afraid you are wrong about this. If an on-line game wants low
> delay, the only answer is for it to avoid generating position updates
> faster than the network carry them. One packet giving the current game
> player position is better than a backlog of ten previous stale ones waiting
> to go out. Sending packets faster than the network can carry them does not
> get them to the destination faster; it gets them there slower. The same
> applies to frames in a screen sharing application. Sending the current
> state of the screen *now* is better than having a backlog of ten previous
> stale frames sitting in buffers somewhere on their way to the destination.
> Stale data is not inevitable. Applications don’t need to have stale data if
> they avoid generating stale data in the first place.
> >>>
> >>> Please watch this video, which explains it better than I can in a
> written email:
> >>>
> >>> 
> >>>
> >>> Stuart Cheshire
> >>>
> >>> ___
> >>> Make-wifi-fast mailing list
> >>> make-wifi-f...@lists.bufferbloat.net
> >>> https://lists.bufferbloat.net/listinfo/make-wifi-fast
> >>
> >>
> >> This electronic communication and the information and any files
> transmitted with it, or attached to it, are confidential and are intended
> solely for the use of the individual or entity to whom it is addressed and
> may contain information that is confidential, legally privileged, protected
> by privacy laws, or otherwise restricted from disclosure to anyone else. If
> you are not the intended recipient or the person responsible for delivering
> the e-mail to the intended recipient, you are hereby notified tha

Re: [Cake] [Rpm] [Make-wifi-fast] The most wonderful video ever about bufferbloat

[Sebastian Moeller via Cake]

Hi Dave,


> On Oct 20, 2022, at 21:12, Dave Taht via Rpm  
> wrote:
> 
> On Thu, Oct 20, 2022 at 12:04 PM Bob McMahon via Make-wifi-fast
>  wrote:
>> 
>> Intel has a good analogous video on this with their CPU video here going 
>> over branches and failed predictions. And to Stuart's point, the longer 
>> pipelines make the forks worse in the amount of in-process bytes that need 
>> to be thrown away. Interactivity, in my opinion, suggests shrinking the 
>> pipeline because, with networks, there is no quick way to throw away stale 
>> data rather every forwarding device continues forward with invalid data. 
>> That's bad for the network too, spending resources on something that's no 
>> longer valid. We in the test & measurement community never measure this.
> 
> One of my all time favorite demos was of stuart's remote desktop
> scenario, where he moved the mouse and the window moved with it.

[SM] Fair enough. However in 2015 I had been using NX's remote X11 
desktop solution which even from Central Europe to California allowed me to 
remote control graphical applications way better than the first demo with the 
multi-second delay between mouse movement and resulting screen updates. (This 
was over a 6/1 Mbps ADSL link, admittedly using HTB-fq_codel, but since it did 
not saturate the link I assign the usability to NX's better design). I will 
make an impolite suggestion here, that the demonstrated screen sharing program 
simply had not yet been optimized/designed for longer slower paths... 

Regards
Sebastian


> 
>> There have been a few requests that iperf 2 measure the "bytes thrown away" 
>> per a fork (user moves a video pointer, etc.) I haven't come up with a good 
>> test yet. I'm still trying to get basic awareness about existing latency, 
>> OWD and responsiveness metrics. I do think measuring the amount of resources 
>> spent on stale data is sorta like food waste, few really pay attention to it.
>> 
>> Bob
>> 
>> FYI, iperf 2 supports TCP_NOTSENT_LOWAT for those interested.
>> 
>> --tcp-write-prefetch n[kmKM]
>> Set TCP_NOTSENT_LOWAT on the socket and use event based writes per select() 
>> on the socket.
>> 
>> 
>> On Thu, Oct 20, 2022 at 11:32 AM Stuart Cheshire via Make-wifi-fast 
>>  wrote:
>>> 
>>> On 20 Oct 2022, at 02:36, Sebastian Moeller  wrote:
>>> 
 Hi Stuart,
 
 [SM] That seems to be somewhat optimistic. We have been there before, 
 short of mandating actually-working oracle schedulers on all end-points, 
 intermediate hops will see queues some more and some less transient. So we 
 can strive to minimize queue build-up sure, but can not avoid queues and 
 long queues completely so we need methods to deal with them gracefully.
 Also not many applications are actually helped all that much by letting 
 information get stale in their own buffers as compared to an on-path 
 queue. Think an on-line reaction-time gated game, the need is to 
 distribute current world state to all participating clients ASAP.
>>> 
>>> I’m afraid you are wrong about this. If an on-line game wants low delay, 
>>> the only answer is for it to avoid generating position updates faster than 
>>> the network carry them. One packet giving the current game player position 
>>> is better than a backlog of ten previous stale ones waiting to go out. 
>>> Sending packets faster than the network can carry them does not get them to 
>>> the destination faster; it gets them there slower. The same applies to 
>>> frames in a screen sharing application. Sending the current state of the 
>>> screen *now* is better than having a backlog of ten previous stale frames 
>>> sitting in buffers somewhere on their way to the destination. Stale data is 
>>> not inevitable. Applications don’t need to have stale data if they avoid 
>>> generating stale data in the first place.
>>> 
>>> Please watch this video, which explains it better than I can in a written 
>>> email:
>>> 
>>> 
>>> 
>>> Stuart Cheshire
>>> 
>>> ___
>>> Make-wifi-fast mailing list
>>> make-wifi-f...@lists.bufferbloat.net
>>> https://lists.bufferbloat.net/listinfo/make-wifi-fast
>> 
>> 
>> This electronic communication and the information and any files transmitted 
>> with it, or attached to it, are confidential and are intended solely for the 
>> use of the individual or entity to whom it is addressed and may contain 
>> information that is confidential, legally privileged, protected by privacy 
>> laws, or otherwise restricted from disclosure to anyone else. If you are not 
>> the intended recipient or the person responsible for delivering the e-mail 
>> to the intended recipient, you are hereby notified that any use, copying, 
>> distributing, dissemination, forwarding, printing, or copying of this e-mail 
>> is strictly prohibited. If you received this e-mail in error, please return 
>> the e-mail to 

Re: [Cake] [Rpm] [Make-wifi-fast] The most wonderful video ever about bufferbloat

[Dave Taht via Cake]

On Thu, Oct 20, 2022 at 11:32 AM Stuart Cheshire  wrote:
>
> On 20 Oct 2022, at 02:36, Sebastian Moeller  wrote:
>
> > Hi Stuart,
> >
> > [SM] That seems to be somewhat optimistic. We have been there before, short 
> > of mandating actually-working oracle schedulers on all end-points, 
> > intermediate hops will see queues some more and some less transient. So we 
> > can strive to minimize queue build-up sure, but can not avoid queues and 
> > long queues completely so we need methods to deal with them gracefully.
> > Also not many applications are actually helped all that much by letting 
> > information get stale in their own buffers as compared to an on-path queue. 
> > Think an on-line reaction-time gated game, the need is to distribute 
> > current world state to all participating clients ASAP.
>
> I’m afraid you are wrong about this. If an on-line game wants low delay, the 
> only answer is for it to avoid generating position updates faster than the 
> network carry them. One packet giving the current game player position is 
> better than a backlog of ten previous stale ones waiting to go out. Sending 
> packets faster than the network can carry them does not get them to the 
> destination faster; it gets them there slower. The same applies to frames in 
> a screen sharing application. Sending the current state of the screen *now* 
> is better than having a backlog of ten previous stale frames sitting in 
> buffers somewhere on their way to the destination. Stale data is not 
> inevitable. Applications don’t need to have stale data if they avoid 
> generating stale data in the first place.

The core  of what you describe is that transports and applications are
evolving towards being delay aware, which is the primary outcome you
get from  FQ'd environment, be the FQs are physical (VoQs, LAGs,
multiple channels or subcarriers in wireless technologies) or virtual
(fq-codel, cake, fq-pie), so that the only source of congestion is
self-harm.

Everything from BBR to googles' gcc for videoconferencing, to recent
work on swift ( https://research.google/pubs/pub49448/ ) seems to be
pointing this way.

I'm also loving the work on reliable FQ detection for QUIC.
> Please watch this video, which explains it better than I can in a written 
> email:
>
> 
>
> Stuart Cheshire
>


-- 
This song goes out to all the folk that thought Stadia would work:
https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-698135607352320-FXtz
Dave Täht CEO, TekLibre, LLC
___
Cake mailing list
Cake@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cake

Re: [Cake] [Rpm] [Make-wifi-fast] The most wonderful video ever about bufferbloat

[Sebastian Moeller via Cake]

Hi Stuart,


> On Oct 19, 2022, at 22:44, Stuart Cheshire via Rpm 
>  wrote:
> 
> On Mon, Oct 17, 2022 at 5:02 PM Stuart Cheshire  wrote:
> 
>> Accuracy be damned. The analogy to common experience resonates more.
> 
> I feel it is not an especially profound insight to observe that, “people 
> don’t like waiting in line.” The conclusion, “therefore privileged people 
> should get to go to the front,” describes an airport first class checkin 
> counter, Disney Fastpass, and countless other analogies from everyday life, 
> all of which are the wrong solution for packets in a network.
> 
>> I think the person with the cheetos pulling out a gun and shooting everyone 
>> in front of him (AQM) would not go down well.
> 
> Which is why starting with a bad analogy (people waiting in a grocery store) 
> inevitably leads to bad conclusions.
> 
> If we want to struggle to make the grocery store analogy work, perhaps we 
> show people checking some grocery store app on their smartphone before they 
> leave home, and if they see that a long line is beginning to form they wait 
> until later, when the line is shorter. The challenge is not how to deal with 
> a long queue when it’s there, it is how to avoid a long queue in the first 
> place.

[SM] That seems to be somewhat optimistic. We have been there before, short of 
mandating actually-working oracle schedulers on all end-points, intermediate 
hops will see queues some more and some less transient. So we can strive to 
minimize queue build-up sure, but can not avoid queues and long queues 
completely so we need methods to deal with them gracefully.
Also not many applications are actually helped all that much by letting 
information get stale in their own buffers as compared to an on-path queue. 
Think an on-line reaction-time gated game, the need is to distribute current 
world state to all participating clients ASAP. That often means a bunch of 
packets that can not reasonably be held back by the server to pace them out as 
world state IIUC needs to be transmitted completely for clients to be able to 
actually do the right thing. Such an application will continue to dump its 
world state burtst per client into the network as that is the required mode of 
operation. I think that there are other applications with similar requirements 
which will make sure that traffic stays burtsy and that IMHO will cause 
transient queues to build up. (Probably short duration ones, but still).



> 
>> Actually that analogy is fairly close to fair queuing. The multiple checker 
>> analogy is one of the most common analogies in queue theory itself.
> 
> I disagree. You are describing the “FQ” part of FQ_CoDel. It’s the “CoDel” 
> part of FQ_CoDel that solves bufferbloat. FQ has been around for a long time, 
> and at best it partially masked the effects of bufferbloat. Having more 
> queues does not solve bufferbloat. Managing the queue(s) better solves 
> bufferbloat.

[SM] Yes and no. IMHO it is the FQ part that gets greedy traffic off 
the back of those flows that stay below their capacity share, as it (unless 
overloaded) will isolate the consequence of exceeding one's capacity share to 
the flow(s) doing so. The AQM part then helps for greedy traffic not to congest 
itself unduly.
So for quite a lot of application classes (e.g. my world-state 
distribution example above) FQ (or any other type of competent scheduling) will 
already solve most of the problem, heck if ubiquitious it would even allow 
greedy traffic to switch to delay based CC methods that can help keeping queues 
small even without competent AQM at the bottlenecks (not that I 
recommend/endorse that, I am all for competent AQM/scheduling at the 
bottlenecks*).



> 
>> I like the idea of a guru floating above a grocery cart with a better string 
>> of explanations, explaining
>> 
>>  - "no, grasshopper, the solution to bufferbloat is no line... at all".
> 
> That is the kind of thing I had in mind. Or a similar quote from The Matrix. 
> While everyone is debating ways to live with long queues, the guru asks, 
> “What if there were no queues?” That is the “mind blown” realization.

[SM] However the "no queues" state is generally not achievable nor 
would it be desirable; queues have utility as "shock absorbers" and to help 
keeping a link busy***. I admit though that "no oversized queues" is far less 
snappy.


Regards
Sebastian


*) Which is why I am vehemently opposed to L4S, it offers neither competent 
scheduling nor competent AQM, in both regimes it is admittedly better then the 
current status quo of having neither but it falls short of the state of the art 
in both so much that deploying L4S today seems indefensible on technical 
grounds. And lo and behold one of L4S biggest proponents does so mainly on 
ideological grounds (just read "Flow rate fairness dismantling a religion" 
https://dl.acm.org/doi/10.1145/1232919.1232926 and then ask yourself whether 
you should trust such an