Re: [Cerowrt-devel] [Bloat] Little's Law mea culpa, but not invalidating my main point

[David P. Reed]

I use the example all the time, but not for interviewing. What's sad is that 
the answers seem to be quoting from some set of textbooks or popular 
explanations of the Internet that really have got it all wrong, but which many 
professionals seem to believe is true.
 
The same phenomenon appears in the various subfields of the design of radio 
communications at the physical and front end electronics level. The examples of 
mental models that are truly broken that are repeated by "experts" are truly 
incredible, and cover all fields. Two or three:
 
1. why do the AM commercial broadcast band (540-1600 kHz) signals you receive 
in your home travel farther than VHF band TV signals and UHF band TV signals?  
How does this explanation relate to the fact that we can see stars a million 
light-years away using receivers that respond to 500 Terahertz radio (visible 
light antennas)?
 
2. What is the "aperture" of an antenna system? Does it depend on frequency of 
the radiation? How does this relate to the idea of the size of an RF photon, 
and the mass of an RF photon? How big must a cellphone be to contain the 
antenna needed to receive and transmit signals in the 3G phone frequencies?
 
3. We can digitize the entire FM broadcast frequency band into a sequence of 
14-bit digital samples at the Nyquist sampling rate of about 40 Mega-samples 
per second, which covers the 20 Mhz bandwidth of the FM band. Does this allow a 
receiver to use a digital receiver to tune into any FM station that can be 
received with an "analog FM radio" using the same antenna? Why or why not?
 
I'm sure Dick Roy understands all three of these questions, and what is going 
on. But I'm equally sure that the designers of WiFi radios or broadcast radios 
or even the base stations of cellular data systems include few who understand.
 
And literally no one at the FCC or CTIA understand how to answer these 
questions.  But the problem is that they are *confident* that they know the 
answers, and that they are right.
 
The same is true about the packet layers and routing layers of the Internet. 
Very few engineers, much less lay people realize that what they have been told 
by "experts" is like how Einstein explained how radio works to a teenaged kid:
 
  "Imagine a cat whose tail is in New York and his head is in Los Angeles. If 
you pinch his tail in NY, he howls in Los Angeles. Except there is no cat."
 
Though others have missed it, Einstein was not making a joke. The non-cat is 
the laws of quantum electrodynamics (or classically, the laws of Maxwell's 
Equations). The "cat" would be all the stories people talk about how radio 
works - beams of energy (or puffs of energy), modulated by some analog 
waveform, bouncing off of hard materials, going through less dense materials, 
"hugging the ground", "far field" and "near field" effects, etc.
 
Einstein's point was that there is no cat - that is, all the metaphors and 
models aren't accurate or equivalent to how radio actually works. But the 
underlying physical phenomenon supporting radio is real, and scientists do 
understand it pretty deeply.
 
Same with how packet networks work. There are no "streams" that behave like 
water in pipes, the connection you have to a shared network has no "speed" in 
megabits per second built in to it, A "website" isn't coming from one place in 
the world, and bits don't have inherent meaning.
 
There is NO CAT (not even a metaphorical one that behaves like the Internet 
actually works).
 
But in the case of the Internet, unlike radio communications, there is no deep 
mystery that requires new discoveries to understand it, because it's been built 
by humans. We don't need metaphors like "streams of water" or "sites in a 
place". We do it a disservice by making up these metaphors, which are only apt 
in a narrow context.
 
For example, congestion in a shared network is just unnecessary queuing delay 
caused by multiplexing the capacity of a particular link among different users. 
It can be cured by slowing down all the different packet sources in some more 
or less fair way. The simplest approach is just to discard from the queue 
excess packets that make that queue longer than can fit through the link Then 
there can't be any congestion. However, telling the sources to slow down 
somehow would be an improvement, hopefully before any discards are needed.
 
There is no "back pressure", because there is no "pressure" at all in a packet 
network. There are just queues and links that empty queues of packets at a 
certain rate. Thinking about back pressure comes from thinking about sessions 
and pipes. But 90% of the Internet has no sessions and no pipes. Just as there 
is "no cat" in real radio systems.
 
On Monday, September 20, 2021 12:09am, "David Lang"  said:



> On Mon, 20 Sep 2021, Valdis Klētnieks wrote:
> 
> > On Sun, 19 Sep 2021 18:21:56 -0700, Dave Taht said:
> >> what actually happens during a web page load,
> >
> > I'm pretty sure that nobody 

Re: [Cerowrt-devel] [Bloat] Little's Law mea culpa, but not invalidating my main point

[David Lang]

On Mon, 20 Sep 2021, Valdis Klētnieks wrote:


On Sun, 19 Sep 2021 18:21:56 -0700, Dave Taht said:

what actually happens during a web page load,


I'm pretty sure that nobody actually understands that anymore, in any
more than handwaving levels.


This is my favorite interview question, it's amazing and saddning at the answers 
that I get, even from supposedly senior security and networking people.


David Lang___
Bloat mailing list
bl...@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat
___
Cerowrt-devel mailing list
Cerowrt-devel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel

Re: [Cerowrt-devel] [Bloat] Little's Law mea culpa, but not invalidating my main point

[Valdis Klētnieks]

On Sun, 19 Sep 2021 18:21:56 -0700, Dave Taht said:
> what actually happens during a web page load,

I'm pretty sure that nobody actually understands that anymore, in any
more than handwaving levels.

I have a nice Chrome extension called IPvFoo that actually tracks the IP
addresses contacted during the load of the displayed page. I'll let you make
a guess as to how many unique IP addresses were contacted during a load
of https://www.cnn.com

...


...


...


145, at least half of which appeared to be analytics.  And that's only the
hosts that were contacted by my laptop for HTTP, and doesn't count DNS, or
load-balancing front ends, or all the back-end boxes.  As I commented over on
NANOG, we've gotten to a point similar to that of AT long distance, where 60%
of the effort of connecting a long distance phone call was the cost of
accounting and billing for the call.










pgpQTVS9pAMEx.pgp
Description: PGP signature
___
Cerowrt-devel mailing list
Cerowrt-devel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel

Re: [Cerowrt-devel] [Bloat] Little's Law mea culpa, but not invalidating my main point

[Dave Taht]

I just wanted to comment on how awesome this thread was, and how few
people outside this group deeply grok what was discussed here. I would
so like
to somehow construct an educational TV series explaining "How the
Internet really works" to a wider, and new audience, consisting of
animations, anecdotes,
and interviews with the key figures of its evolution.

While I deeply understood Len Kleinrock's work in the period
2011-2015, and tried to pass on analogies and intuition without using
the math since, inspired by van jacobson's
analogies and Radia Perlman's poetry, it's hard for me now to follow
the argument. Queue theory in particular, is not well known or taught
anymore, despite its obvious applications to things like the Covid
crisis.

But that would be just one thing! The end to end argument, the side
effects of spitting postscript into a lego robot, what actually
happens during a web page load, how a cpu actually works, are all
things that are increasingly lost in multiple mental models, and in my
mind many could be taught in kindergarden, if we worked at explaining
it hard enough.
___
Cerowrt-devel mailing list
Cerowrt-devel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel

Re: [Cerowrt-devel] [Bloat] Little's Law mea culpa, but not invalidating my main point

[Bob McMahon via Cerowrt-devel]

--- Begin Message ---
Thanks for this. I plan to purchase the second volume to go with my copy of
volume 1. There is (always) more to learn and your expertise is very
helpful.

Bob

PS.  As a side note, I've added support for TCP_NOTSENT_LOWAT in iperf 2.1.4
 and it's proving
interesting per WiFi/BT latency testing including helping to mitigate
sender side bloat.
*--tcp-write-prefetch **n*[kmKM]Set TCP_NOTSENT_LOWAT on the socket and use
event based writes per select() on the socket.I'll probably add measuring
the select() delays to see if that correlates to things like RF
arbitrations, etc.


On Wed, Jul 21, 2021 at 4:20 PM Leonard Kleinrock  wrote:

> Just a few comments following David Reed's insightful comments re the
> history of the ARPANET and its approach to flow control.  I have attached
> some pages from my Volume II which provide an understanding of how we
> addressed flow control and its implementation in the ARPANET.
>
> The early days of the ARPANET design and evaluation involved detailed
> design of what we did call “Flow Control”.  In my "Queueing Systems, Volume
> II: Computer Applications”, John Wiley, 1976, I documented much of what we
> designed and evaluated for the ARPANET, and focused on performance,
> deadlocks, lockups and degradations due to flow control design.  Aspects of
> congestion control were considered, but this 2-volume book was mostly about
> understanding congestion.Of interest are the many deadlocks that we
> discovered in those early days as we evaluated and measured the network
> behavior.  Flow control was designed into that early network, but it had a
> certain ad-hoc flavor and I point out the danger of requiring flows to
> depend upon the acquisition of multiple tokens that were allocated from
> different portions of the network at the same time in a distributed
> fashion.  The attached relevant sections of the book address these issues;
>  I thought it would be of value to see what we were looking at back then.
>
> On a related topic regarding flow and congestion control (as triggered by
> David’s comment* "**at most one packet waiting for each egress link in
> the bottleneck path.”*), in 1978, I published a paper
> 
>  in
> which I extended the notion of Power (the ratio of throughput to response
> time) that had been introduced by Giessler, et a
> l
> and I pointed out the amazing properties that emerged when Power is
> optimized, e.g., that one should keep each hop in the pipe “just full”,
> i.e., one message per hop.  As it turns out, and as has been discussed in
> this email chain, Jaffe
>  showed
> in 1981 that this optimization was not decentralizable and so no one
> pursued this optimal operating point (notwithstanding the fact that I
> published other papers on this issue, for example in 1979
> 
>  and
> in 1981 ).  So this
> issue of Power lay dormant for decades until Van Jacobsen, et al,
> resurrected the idea with their BBR flow control design in 2016
>  when they showed that
> indeed one could decentralize power.  Considerable research has since
> followed their paper including another by me in 2018
> .
> (This was not the first time that a publication challenging the merits of a
> new idea negatively impacted that idea for decades - for example, the 1988
> book “Perceptrons”
> 
>  by
> Minsky and Papert discouraged research into neural networks for many years
> until that idea was proven to have merit.)  But the story is not over as
> much  work has yet to be done to develop the algorithms that can properly
> deal with congestion in the sense that this email chain continues to
> discuss it.
>
> Best,
> Len
>
>
>
>
>
>
>
> On Jul 13, 2021, at 10:49 AM, David P. Reed  wrote:
>
> Bob -
>
> On Tuesday, July 13, 2021 1:07pm, "Bob McMahon" 
> said:
>
> "Control at endpoints benefits greatly from even small amounts of
> information supplied by the network about the degree of congestion present
> on the path."
>
> Agreed. The ECN mechanism seems like a shared thermostat in a building.
> It's basically an on/off where everyone is trying to set the temperature.
> It does affect, in a non-linear manner, but still an effect. Better than a
> thermostat set at infinity or 0 Kelvin 

Re: [Cerowrt-devel] [Bloat] Little's Law mea culpa, but not invalidating my main point

[Bob McMahon via Cerowrt-devel]

--- Begin Message ---
Just an FYI,

iperf 2 uses a 4 usec delay for TCP and 100 usec delay for UDP to fill the
token bucket. We thought about providing a knob for this but decided not
to. We figured a busy wait CPU thread wasn't a big deal because of the
trend of many CPU cores. The threaded design works well for this. We also
support fq-pacing and isochronous traffic using clock_nanosleep() to
schedule the writes. We'll probably add Markov chain support but that's not
critical and may not affect actionable engineering. We found isoch as a
useful traffic profile, at least for our WiFi testing. I'm going to add
support for TCP_NOTSENT_LOWAT for select()/write() based transmissions. I'm
doubtful this is very useful as event based scheduling based on times seems
better. We'll probably use it for unit testing WiFi aggregation and see if
it helps there or not. I'll see if it aligns with the OWD measurements.

On queue depth, we use two techniques. The most obvious is to measure the
end to end delay and use rx histograms, getting all the samples without
averaging. The second, internal for us only, is using network telemetry and
mapping all the clock domains to the GPS domain. Any moment in time the
end/end path can be inspected to where every packet is.

Our automated testing is focused around unit tests and used to
statistically monitor code changes (which come at a high rate and apply to
a broad range of chips) - so the requirements can be very different from a
network or service provider.

Agreed that the amount of knobs and reactive components are a challenge.
And one must assume non-linearity which becomes obvious after a few direct
measurements (i.e. no averaging.) The challenge of statistical;y
reproducible is always there. We find Montec Carlo techniques can be useful
only when they are proven to be statistically reproducible.

Bob


On Sat, Jul 17, 2021 at 4:29 PM Aaron Wood  wrote:

> On Mon, Jul 12, 2021 at 1:32 PM Ben Greear 
> wrote:
>
>> UDP is better for getting actual packet latency, for sure.  TCP is
>> typical-user-experience-latency though,
>> so it is also useful.
>>
>> I'm interested in the test and visualization side of this.  If there were
>> a way to give engineers
>> a good real-time look at a complex real-world network, then they have
>> something to go on while trying
>> to tune various knobs in their network to improve it.
>>
>
> I've always liked the smoke-ping visualization, although a single graph is
> only really useful for a single pair of endpoints (or a single segment,
> maybe).  But I can see using a repeated set of graphs (Tufte has some
> examples), that can represent an overview of pairwise collections of
> latency+loss:
> https://www.edwardtufte.com/bboard/images/0003Cs-8047.GIF
> https://www.edwardtufte.com/tufte/psysvcs_p2
>
> These work for understanding because the tiled graphs are all identically
> constructed, and the reader first learns how to read a single tile, and
> then learns the pattern of which tiles represent which measurements.
>
> Further, they are opinionated.  In the second link above, the y axis is
> not based on the measured data, but standardized expected values, which (I
> think) is key to quick readability.  You never need to read the axes.  Much
> like setting up gauges such that "nominal" is always at the same indicator
> position for all graphs (e.g. straight up).  At a glance, you can see if
> things are "correct" or not.
>
> That tiling arrangement wouldn't be great for showing interrelationships
> (although it may give you a good historical view of correlated behavior).
> One thought is to overlay a network graph diagram (graph of all network
> links) with small "sparkline" type graphs.
>
> For a more physical-based network graph, I could see visualizing the queue
> depth for each egress port (max value over a time of X, or percentage of
> time at max depth).
>
> Taken together, the timewise correlation could be useful (which peers are
> having problems communicating, and which ports between them are impacted?).
>
> I think getting good data about queue depth may be the hard part,
> especially catching transients and the duty cycle / pulse-width of the load
> (and then converting that to a number).  Back when I uncovered the iperf
> application-level pacing granularity was too high 5 years ago, I called it
> them "millibursts", and maybe dtaht pointed out that link utilization is
> always 0% or 100%, and it's just a matter of the PWM of the packet rate
> that makes it look like something in between.
> https://burntchrome.blogspot.com/2016/09/iperf3-and-microbursts.html
>
>
>
> I'll let others try to figure out how build and tune the knobs, but the
>> data acquisition and
>> visualization is something we might try to accomplish.  I have a feeling
>> I'm not the
>> first person to think of this, howeverprobably someone already has
>> done such
>> a thing.
>>
>> Thanks,
>> Ben
>>
>> On 7/12/21 1:04 PM, Bob McMahon wrote:
>> > I believe end 

Re: [Cerowrt-devel] [Bloat] Little's Law mea culpa, but not invalidating my main point

[Aaron Wood]

On Mon, Jul 12, 2021 at 1:32 PM Ben Greear  wrote:

> UDP is better for getting actual packet latency, for sure.  TCP is
> typical-user-experience-latency though,
> so it is also useful.
>
> I'm interested in the test and visualization side of this.  If there were
> a way to give engineers
> a good real-time look at a complex real-world network, then they have
> something to go on while trying
> to tune various knobs in their network to improve it.
>

I've always liked the smoke-ping visualization, although a single graph is
only really useful for a single pair of endpoints (or a single segment,
maybe).  But I can see using a repeated set of graphs (Tufte has some
examples), that can represent an overview of pairwise collections of
latency+loss:
https://www.edwardtufte.com/bboard/images/0003Cs-8047.GIF
https://www.edwardtufte.com/tufte/psysvcs_p2

These work for understanding because the tiled graphs are all identically
constructed, and the reader first learns how to read a single tile, and
then learns the pattern of which tiles represent which measurements.

Further, they are opinionated.  In the second link above, the y axis is not
based on the measured data, but standardized expected values, which (I
think) is key to quick readability.  You never need to read the axes.  Much
like setting up gauges such that "nominal" is always at the same indicator
position for all graphs (e.g. straight up).  At a glance, you can see if
things are "correct" or not.

That tiling arrangement wouldn't be great for showing interrelationships
(although it may give you a good historical view of correlated behavior).
One thought is to overlay a network graph diagram (graph of all network
links) with small "sparkline" type graphs.

For a more physical-based network graph, I could see visualizing the queue
depth for each egress port (max value over a time of X, or percentage of
time at max depth).

Taken together, the timewise correlation could be useful (which peers are
having problems communicating, and which ports between them are impacted?).

I think getting good data about queue depth may be the hard part,
especially catching transients and the duty cycle / pulse-width of the load
(and then converting that to a number).  Back when I uncovered the iperf
application-level pacing granularity was too high 5 years ago, I called it
them "millibursts", and maybe dtaht pointed out that link utilization is
always 0% or 100%, and it's just a matter of the PWM of the packet rate
that makes it look like something in between.
https://burntchrome.blogspot.com/2016/09/iperf3-and-microbursts.html



I'll let others try to figure out how build and tune the knobs, but the
> data acquisition and
> visualization is something we might try to accomplish.  I have a feeling
> I'm not the
> first person to think of this, howeverprobably someone already has
> done such
> a thing.
>
> Thanks,
> Ben
>
> On 7/12/21 1:04 PM, Bob McMahon wrote:
> > I believe end host's TCP stats are insufficient as seen per the "failed"
> congested control mechanisms over the last decades. I think Jaffe pointed
> this out in
> > 1979 though he was using what's been deemed on this thread as "spherical
> cow queueing theory."
> >
> > "Flow control in store-and-forward computer networks is appropriate for
> decentralized execution. A formal description of a class of "decentralized
> flow control
> > algorithms" is given. The feasibility of maximizing power with such
> algorithms is investigated. On the assumption that communication links
> behave like M/M/1
> > servers it is shown that no "decentralized flow control algorithm" can
> maximize network power. Power has been suggested in the literature as a
> network
> > performance objective. It is also shown that no objective based only on
> the users' throughputs and average delay is decentralizable. Finally, a
> restricted class
> > of algorithms cannot even approximate power."
> >
> > https://ieeexplore.ieee.org/document/1095152
> >
> > Did Jaffe make a mistake?
> >
> > Also, it's been observed that latency is non-parametric in it's
> distributions and computing gaussians per the central limit theorem for OWD
> feedback loops
> > aren't effective. How does one design a control loop around things that
> are non-parametric? It also begs the question, what are the feed forward
> knobs that can
> > actually help?
> >
> > Bob
> >
> > On Mon, Jul 12, 2021 at 12:07 PM Ben Greear  > wrote:
> >
> > Measuring one or a few links provides a bit of data, but seems like
> if someone is trying to understand
> > a large and real network, then the OWD between point A and B needs
> to just be input into something much
> > more grand.  Assuming real-time OWD data exists between 100 to 1000
> endpoint pairs, has anyone found a way
> > to visualize this in a useful manner?
> >
> > Also, considering something better than ntp may not really scale to
> 1000+ endpoints, maybe round-trip
> >  

Re: [Cerowrt-devel] [Bloat] Little's Law mea culpa, but not invalidating my main point

[Holland, Jake via Cerowrt-devel]

--- Begin Message ---
From: Bob McMahon via Bloat 
> Date: Wed,2021-07-14 at 11:38 AM
> One challenge I faced with iperf 2 was around flow control's effects on
> latency. I find if iperf 2 rate limits on writes then the end/end
> latencies, RTT look good because the pipe is basically empty, while rate
> limiting reads to the same value fills the window and drives the RTT up.
> One might conclude, from a network perspective, the write side is
> better.  But in reality, the write rate limiting is just pushing the
> delay into the application's logic, i.e. the relevant bytes may not be
> in the pipe but they aren't at the receiver either, they're stuck
> somewhere in the "tx application space."
>
> It wasn't obvious to me how to address this. We added burst measurements
> (burst xfer time, and bursts/sec) which, I think, helps.
...
>>> I find the assumption that congestion occurs "in network" as not always
>>> true. Taking OWD measurements with read side rate limiting suggests that
>>> equally important to mitigating bufferbloat driven latency using congestion
>>> signals is to make sure apps read "fast enough" whatever that means. I
>>> rarely hear about how important it is for apps to prioritize reads over
>>> open sockets. Not sure why that's overlooked and bufferbloat gets all the
>>> attention. I'm probably missing something.

Hi Bob,

You're right that the sender generally also has to avoid sending
more than the receiver can handle to avoid delays in a message-
reply cycle on the same TCP flow.

In general, I think of failures here as application faults rather
than network faults.  While important for user experience, it's
something that an app developer can solve.  That's importantly
different from network buffering.

It's also somewhat possible to avoid getting excessively backed up
in the network because of your own traffic.  Here bbr usually does
a decent job of keeping the queues decently low.  (And you'll maybe
find that some of the bufferbloat measurement efforts are relying
on the self-congestion you get out of cubic, so if you switch them
to bbr you might not get a good answer on how big the network buffers
are.)

In general, anything along these lines has to give backpressure to
the sender somehow.  What I'm guessing you saw when you did receiver-
side rate limiting was that the backpressure had to fill bytes all
the way back to a full receive kernel buffer (making a 0 rwnd for
TCP) and a full send kernel buffer before the send writes start
failing (I think with ENOBUFS iirc?), and that's the first hint the
sender has that it can't send more data right now.  The assumption
that the receiver can receive as fast as the sender can send is so
common that it often goes unstated.

(If you love to suffer, you can maybe get the backpressure to start
earlier, and with maybe a lower impact to your app-level RTT, if
you try hard enough from the receive side with TCP_WINDOW_CLAMP:
https://man7.org/linux/man-pages/man7/tcp.7.html#:~:text=tcp_window_clamp
But you'll still be living with a full send buffer ahead of the
message-response.)

But usually the right thing to do if you want receiver-driven rate
control is to send back some kind of explicit "slow down, it's too
fast for me" feedback at the app layer that will make the sender send
slower.  For instance most ABR players will shift down their bitrate
if they're failing to render video fast enough just as well as if the
network isn't feeding the video segments fast enough, like if they're
CPU-bound from something else churning on the machine.  (RTP-based
video players are supposed to send feedback with this same kind of
"slow down" capability, and sometimes they do.)

But what you can't fix from the endpoints no matter how hard you
try is the buffers in the network that get filled by other people's
traffic.

Getting other people's traffic to avoid breaking my latency when
we're sharing a bottleneck requires deploying something in the network
and it's not something I can fix myself except inside my own network.

While the app-specific fixes would make for very fine blog posts or
stack overflow questions that could help someone who managed to search
the right terms, there's a lot of different approaches for different
apps that can solve it more or less, and anyone who tries hard enough
will land on something that works well enough for them, and you don't
need a whole movement to get people to make it so their own app works
ok for them and their users.  The problems can be subtle and maybe
there will be some late and frustrating nights involved, but anyone
who gets it reproducible and keeps digging will solve it eventually.

But getting stuff deployed in networks to stop people's traffic
breaking each other's latency is harder, especially when it's a
major challenge for people to even grasp the problem and understand
its causes.  The only possible paths to getting a solution widely
deployed (assuming you have one that works) start with things like
"start 

Re: [Cerowrt-devel] [Bloat] Little's Law mea culpa, but not invalidating my main point

[Bob McMahon via Cerowrt-devel]

--- Begin Message ---
Ok, adding support for TCP_WINDOW_CLAMP and TCP_NOTSENT_LOWAT into iperf 2
seems useful for TCP WiFi latency related testing.  These option names are
quite obfuscated. I can't see many but some ultimate networking geeks
knowing what these actually do.

Here are some proposed command/option names which wouldn't pass any "parser
police" (Parser police was the internal discussion list we used at Cisco to
review router commands. The Cisco cli is a disaster even with pp, but less
so than what could have been.)

{"*tcp-rx-window-clamp*", required_argument, , 1},
{"*tcp-not-sent-low-watermark*", required_argument, , 1},

I'd for sure like to rename "tcp-not-sent-low-watermark" to something more
intuitive. (My daughter, trained in linguistics, is having a field day
laughing at this "nerd language" that is beyond human comprehension.) This
cli option sets the socket option and causes the use of select for writes
(vs a write spin loop.)

Thanks in advance for any suggestions here,
Bob

On Wed, Jul 14, 2021 at 6:27 PM Holland, Jake  wrote:

> From: Bob McMahon via Bloat 
> > Date: Wed,2021-07-14 at 11:38 AM
> > One challenge I faced with iperf 2 was around flow control's effects on
> > latency. I find if iperf 2 rate limits on writes then the end/end
> > latencies, RTT look good because the pipe is basically empty, while rate
> > limiting reads to the same value fills the window and drives the RTT up.
> > One might conclude, from a network perspective, the write side is
> > better.  But in reality, the write rate limiting is just pushing the
> > delay into the application's logic, i.e. the relevant bytes may not be
> > in the pipe but they aren't at the receiver either, they're stuck
> > somewhere in the "tx application space."
> >
> > It wasn't obvious to me how to address this. We added burst measurements
> > (burst xfer time, and bursts/sec) which, I think, helps.
> ...
> >>> I find the assumption that congestion occurs "in network" as not always
> >>> true. Taking OWD measurements with read side rate limiting suggests
> that
> >>> equally important to mitigating bufferbloat driven latency using
> congestion
> >>> signals is to make sure apps read "fast enough" whatever that means. I
> >>> rarely hear about how important it is for apps to prioritize reads over
> >>> open sockets. Not sure why that's overlooked and bufferbloat gets all
> the
> >>> attention. I'm probably missing something.
>
> Hi Bob,
>
> You're right that the sender generally also has to avoid sending
> more than the receiver can handle to avoid delays in a message-
> reply cycle on the same TCP flow.
>
> In general, I think of failures here as application faults rather
> than network faults.  While important for user experience, it's
> something that an app developer can solve.  That's importantly
> different from network buffering.
>
> It's also somewhat possible to avoid getting excessively backed up
> in the network because of your own traffic.  Here bbr usually does
> a decent job of keeping the queues decently low.  (And you'll maybe
> find that some of the bufferbloat measurement efforts are relying
> on the self-congestion you get out of cubic, so if you switch them
> to bbr you might not get a good answer on how big the network buffers
> are.)
>
> In general, anything along these lines has to give backpressure to
> the sender somehow.  What I'm guessing you saw when you did receiver-
> side rate limiting was that the backpressure had to fill bytes all
> the way back to a full receive kernel buffer (making a 0 rwnd for
> TCP) and a full send kernel buffer before the send writes start
> failing (I think with ENOBUFS iirc?), and that's the first hint the
> sender has that it can't send more data right now.  The assumption
> that the receiver can receive as fast as the sender can send is so
> common that it often goes unstated.
>
> (If you love to suffer, you can maybe get the backpressure to start
> earlier, and with maybe a lower impact to your app-level RTT, if
> you try hard enough from the receive side with TCP_WINDOW_CLAMP:
> https://man7.org/linux/man-pages/man7/tcp.7.html#:~:text=tcp_window_clamp
> But you'll still be living with a full send buffer ahead of the
> message-response.)
>
> But usually the right thing to do if you want receiver-driven rate
> control is to send back some kind of explicit "slow down, it's too
> fast for me" feedback at the app layer that will make the sender send
> slower.  For instance most ABR players will shift down their bitrate
> if they're failing to render video fast enough just as well as if the
> network isn't feeding the video segments fast enough, like if they're
> CPU-bound from something else churning on the machine.  (RTP-based
> video players are supposed to send feedback with this same kind of
> "slow down" capability, and sometimes they do.)
>
> But what you can't fix from the endpoints no matter how hard you
> try is the buffers in the network that get filled by other 

Re: [Cerowrt-devel] [Bloat] Little's Law mea culpa, but not invalidating my main point

[Bob McMahon via Cerowrt-devel]

--- Begin Message ---
Thanks for this. I find it both interesting and useful. Learning from those
who came before me reminds me of "standing on the shoulders of giants." I
try to teach my kids that it's not so much us as the giants we choose - so
choose judiciously and, more importantly, be grateful when they provide
their shoulders from which to see.

One challenge I faced with iperf 2 was around flow control's effects on
latency. I find if iperf 2 rate limits on writes then the end/end
latencies, RTT look good because the pipe is basically empty, while rate
limiting reads to the same value fills the window and drives the RTT up.
One might conclude, from a network perspective, the write side is better.
But in reality, the write rate limiting is just pushing the delay into the
application's logic, i.e. the relevant bytes may not be in the pipe but
they aren't at the receiver either, they're stuck somewhere in the "tx
application space."

It wasn't obvious to me how to address this. We added burst measurements
(burst xfer time, and bursts/sec) which, I think, helps.

Bob

On Tue, Jul 13, 2021 at 10:49 AM David P. Reed  wrote:

> Bob -
>
> On Tuesday, July 13, 2021 1:07pm, "Bob McMahon" 
> said:
>
> > "Control at endpoints benefits greatly from even small amounts of
> > information supplied by the network about the degree of congestion
> present
> > on the path."
> >
> > Agreed. The ECN mechanism seems like a shared thermostat in a building.
> > It's basically an on/off where everyone is trying to set the temperature.
> > It does affect, in a non-linear manner, but still an effect. Better than
> a
> > thermostat set at infinity or 0 Kelvin for sure.
> >
> > I find the assumption that congestion occurs "in network" as not always
> > true. Taking OWD measurements with read side rate limiting suggests that
> > equally important to mitigating bufferbloat driven latency using
> congestion
> > signals is to make sure apps read "fast enough" whatever that means. I
> > rarely hear about how important it is for apps to prioritize reads over
> > open sockets. Not sure why that's overlooked and bufferbloat gets all the
> > attention. I'm probably missing something.
>
> In the early days of the Internet protocol and also even ARPANET Host-Host
> protocol there were those who conflated host-level "flow control" (matching
> production rate of data into the network to the destination *process*
> consumption rate of data on a virtual circuit with a source capable of
> variable and unbounded bit rate) with "congestion control" in the network.
> The term "congestion control" wasn't even used in the Internetworking
> project when it was discussing design in the late 1970's. I tried to use it
> in our working group meetings, and every time I said "congestion" the
> response would be phrased as "flow".
>
> The classic example was printing a file's contents from disk to an ASR33
> terminal on an TIP (Terminal IMP). There was flow control in the end-to-end
> protocol to avoid overflowing the TTY's limited buffer. But those who grew
> up with ARPANET knew that thare was no way to accumulate queueing in the
> IMP network, because of RFNM's that required permission for each new packet
> to be sent. RFNM's implicitly prevented congestion from being caused by a
> virtual circuit. But a flow control problem remained, because at the higher
> level protocol, buffering would overflow at the TIP.
>
> TCP adopted a different end-to-end *flow* control, so it solved the flow
> control problem by creating a Windowing mechanism. But it did not by itself
> solve the *congestion* control problem, even congestion built up inside the
> network by a wide-open window and a lazy operating system at the receiving
> end that just said, I've got a lot of virtual memory so I'll open the
> window to maximum size.
>
> There was a lot of confusion, because the guys who came from the ARPANET
> environment, with all links being the same speed and RFNM limits on rate,
> couldn't see why the Internet stack was so collapse-prone. I think Multics,
> for example, as a giant virtual memory system caused congestion by opening
> up its window too much.
>
> This is where Van Jacobson discovered that dropped packets were a "good
> enough" congestion signal because of "fate sharing" among the packets that
> flowed on a bottleneck path, and that windowing (invented for flow control
> by the receiver to protect itself from overflow if the receiver couldn't
> receive fast enough) could be used to slow down the sender to match the
> rate of senders to the capacity of the internal bottleneck link. An elegant
> "hack" that actually worked really well in practice.
>
> Now we view it as a bug if the receiver opens its window too much, or
> otherwise doesn't translate dropped packets (or other incipient-congestion
> signals) to shut down the source transmission rate as quickly as possible.
> Fortunately, the proper state of the internet - the one it should seek as
> its ideal 

Re: [Cerowrt-devel] [Bloat] Little's Law mea culpa, but not invalidating my main point

[David P. Reed]

Bob -

On Tuesday, July 13, 2021 1:07pm, "Bob McMahon"  said:

> "Control at endpoints benefits greatly from even small amounts of
> information supplied by the network about the degree of congestion present
> on the path."
> 
> Agreed. The ECN mechanism seems like a shared thermostat in a building.
> It's basically an on/off where everyone is trying to set the temperature.
> It does affect, in a non-linear manner, but still an effect. Better than a
> thermostat set at infinity or 0 Kelvin for sure.
> 
> I find the assumption that congestion occurs "in network" as not always
> true. Taking OWD measurements with read side rate limiting suggests that
> equally important to mitigating bufferbloat driven latency using congestion
> signals is to make sure apps read "fast enough" whatever that means. I
> rarely hear about how important it is for apps to prioritize reads over
> open sockets. Not sure why that's overlooked and bufferbloat gets all the
> attention. I'm probably missing something.

In the early days of the Internet protocol and also even ARPANET Host-Host 
protocol there were those who conflated host-level "flow control" (matching 
production rate of data into the network to the destination *process* 
consumption rate of data on a virtual circuit with a source capable of variable 
and unbounded bit rate) with "congestion control" in the network. The term 
"congestion control" wasn't even used in the Internetworking project when it 
was discussing design in the late 1970's. I tried to use it in our working 
group meetings, and every time I said "congestion" the response would be 
phrased as "flow".

The classic example was printing a file's contents from disk to an ASR33 
terminal on an TIP (Terminal IMP). There was flow control in the end-to-end 
protocol to avoid overflowing the TTY's limited buffer. But those who grew up 
with ARPANET knew that thare was no way to accumulate queueing in the IMP 
network, because of RFNM's that required permission for each new packet to be 
sent. RFNM's implicitly prevented congestion from being caused by a virtual 
circuit. But a flow control problem remained, because at the higher level 
protocol, buffering would overflow at the TIP.

TCP adopted a different end-to-end *flow* control, so it solved the flow 
control problem by creating a Windowing mechanism. But it did not by itself 
solve the *congestion* control problem, even congestion built up inside the 
network by a wide-open window and a lazy operating system at the receiving end 
that just said, I've got a lot of virtual memory so I'll open the window to 
maximum size.

There was a lot of confusion, because the guys who came from the ARPANET 
environment, with all links being the same speed and RFNM limits on rate, 
couldn't see why the Internet stack was so collapse-prone. I think Multics, for 
example, as a giant virtual memory system caused congestion by opening up its 
window too much.

This is where Van Jacobson discovered that dropped packets were a "good enough" 
congestion signal because of "fate sharing" among the packets that flowed on a 
bottleneck path, and that windowing (invented for flow control by the receiver 
to protect itself from overflow if the receiver couldn't receive fast enough) 
could be used to slow down the sender to match the rate of senders to the 
capacity of the internal bottleneck link. An elegant "hack" that actually 
worked really well in practice.

Now we view it as a bug if the receiver opens its window too much, or otherwise 
doesn't translate dropped packets (or other incipient-congestion signals) to 
shut down the source transmission rate as quickly as possible. Fortunately, the 
proper state of the internet - the one it should seek as its ideal state - is 
that there is at most one packet waiting for each egress link in the bottleneck 
path. This stable state ensures that the window-reduction or slow-down signal 
encounters no congestion, with high probability. [Excursions from one-packet 
queue occur, but since only one-packet waiting is sufficient to fill the 
bottleneck link to capacity, they can't achieve higher throughput in steady 
state. In practice, noisy arrival distributions can reduce throughput, so 
allowing a small number of packets to be waiting on a bottleneck link's queue 
can slightly increase throughput. That's not asymptotically relevant, but as 
mentioned, the Internet is never near asymptotic behavior.]


> 
> Bob
> 
> On Tue, Jul 13, 2021 at 12:15 AM Amr Rizk  wrote:
> 
>> Ben,
>>
>> it depends on what one tries to measure. Doing a rate scan using UDP (to
>> measure latency distributions under load) is the best thing that we have
>> but without actually knowing how resources are shared (fair share as in
>> WiFi, FIFO as nearly everywhere else) it becomes very difficult to
>> interpret the results or provide a proper argument on latency. You are
>> right - TCP stats are a proxy for user experience but I believe they are
>> difficult to 

Re: [Cerowrt-devel] [Bloat] Little's Law mea culpa, but not invalidating my main point

[Bob McMahon via Cerowrt-devel]

--- Begin Message ---
"the infinite TCP flow that converges to a steady behavior is purely
academic"

We find this to be mostly true. Sadly, the tools such as iperf drove to
this condition. While still useful, not realistic.

We added, in iperf 2, the ability to test TCP bursts (--burst-size and
--burst-period) over low duty cycles and get completely different sets of
phenomena with TCP.

It seems (past) time that peak average throughput driving the control loop
needs reconsideration, particularly if TCP_NODELAY is set on a socket. This
is particularly challenging for WiFi because "requests for low latency"
usually triggers no aggregation which doesn't always reduce tail latencies.

Bob

On Tue, Jul 13, 2021 at 12:15 AM Amr Rizk  wrote:

> Ben,
>
> it depends on what one tries to measure. Doing a rate scan using UDP (to
> measure latency distributions under load) is the best thing that we have
> but without actually knowing how resources are shared (fair share as in
> WiFi, FIFO as nearly everywhere else) it becomes very difficult to
> interpret the results or provide a proper argument on latency. You are
> right - TCP stats are a proxy for user experience but I believe they are
> difficult to reproduce (we are always talking about very short TCP flows -
> the infinite TCP flow that converges to a steady behavior is purely
> academic).
>
> By the way, Little's law is a strong tool when it comes to averages. To be
> able to say more (e.g. 1% of the delays is larger than x) one requires more
> information (e.g. the traffic - On-OFF pattern) see [1].  I am not sure
> when does such information readily exist.
>
> Best
> Amr
>
> [1] https://dl.acm.org/doi/10.1145/3341617.3326146 or if behind a paywall
> https://www.dcs.warwick.ac.uk/~florin/lib/sigmet19b.pdf
>
> 
> Amr Rizk (amr.r...@uni-due.de)
> University of Duisburg-Essen
>
> -Ursprüngliche Nachricht-
> Von: Bloat  Im Auftrag von Ben Greear
> Gesendet: Montag, 12. Juli 2021 22:32
> An: Bob McMahon 
> Cc: starl...@lists.bufferbloat.net; Make-Wifi-fast <
> make-wifi-f...@lists.bufferbloat.net>; Leonard Kleinrock ;
> David P. Reed ; Cake List ;
> co...@lists.bufferbloat.net; cerowrt-devel <
> cerowrt-devel@lists.bufferbloat.net>; bloat 
> Betreff: Re: [Bloat] Little's Law mea culpa, but not invalidating my main
> point
>
> UDP is better for getting actual packet latency, for sure.  TCP is
> typical-user-experience-latency though, so it is also useful.
>
> I'm interested in the test and visualization side of this.  If there were
> a way to give engineers a good real-time look at a complex real-world
> network, then they have something to go on while trying to tune various
> knobs in their network to improve it.
>
> I'll let others try to figure out how build and tune the knobs, but the
> data acquisition and visualization is something we might try to
> accomplish.  I have a feeling I'm not the first person to think of this,
> howeverprobably someone already has done such a thing.
>
> Thanks,
> Ben
>
> On 7/12/21 1:04 PM, Bob McMahon wrote:
> > I believe end host's TCP stats are insufficient as seen per the
> > "failed" congested control mechanisms over the last decades. I think
> > Jaffe pointed this out in
> > 1979 though he was using what's been deemed on this thread as "spherical
> cow queueing theory."
> >
> > "Flow control in store-and-forward computer networks is appropriate
> > for decentralized execution. A formal description of a class of
> > "decentralized flow control algorithms" is given. The feasibility of
> > maximizing power with such algorithms is investigated. On the
> > assumption that communication links behave like M/M/1 servers it is
> shown that no "decentralized flow control algorithm" can maximize network
> power. Power has been suggested in the literature as a network performance
> objective. It is also shown that no objective based only on the users'
> throughputs and average delay is decentralizable. Finally, a restricted
> class of algorithms cannot even approximate power."
> >
> > https://ieeexplore.ieee.org/document/1095152
> >
> > Did Jaffe make a mistake?
> >
> > Also, it's been observed that latency is non-parametric in it's
> > distributions and computing gaussians per the central limit theorem
> > for OWD feedback loops aren't effective. How does one design a control
> loop around things that are non-parametric? It also begs the question, what
> are the feed forward knobs that can actually help?
> >
> > Bob
> >
> > On Mon, Jul 12, 2021 at 12:07 PM Ben Greear  > wrote:
> >
> > Measuring one or a few links provides a bit of data, but seems like
> if someone is trying to understand
> > a large and real network, then the OWD between point A and B needs
> to just be input into something much
> > more grand.  Assuming real-time OWD data exists between 100 to 1000
> endpoint pairs, has anyone found a way
> > to visualize this in a useful 

Re: [Cerowrt-devel] [Bloat] Little's Law mea culpa, but not invalidating my main point

[Bob McMahon via Cerowrt-devel]

--- Begin Message ---
"Control at endpoints benefits greatly from even small amounts of
information supplied by the network about the degree of congestion present
on the path."

Agreed. The ECN mechanism seems like a shared thermostat in a building.
It's basically an on/off where everyone is trying to set the temperature.
It does affect, in a non-linear manner, but still an effect. Better than a
thermostat set at infinity or 0 Kelvin for sure.

I find the assumption that congestion occurs "in network" as not always
true. Taking OWD measurements with read side rate limiting suggests that
equally important to mitigating bufferbloat driven latency using congestion
signals is to make sure apps read "fast enough" whatever that means. I
rarely hear about how important it is for apps to prioritize reads over
open sockets. Not sure why that's overlooked and bufferbloat gets all the
attention. I'm probably missing something.

Bob

On Tue, Jul 13, 2021 at 12:15 AM Amr Rizk  wrote:

> Ben,
>
> it depends on what one tries to measure. Doing a rate scan using UDP (to
> measure latency distributions under load) is the best thing that we have
> but without actually knowing how resources are shared (fair share as in
> WiFi, FIFO as nearly everywhere else) it becomes very difficult to
> interpret the results or provide a proper argument on latency. You are
> right - TCP stats are a proxy for user experience but I believe they are
> difficult to reproduce (we are always talking about very short TCP flows -
> the infinite TCP flow that converges to a steady behavior is purely
> academic).
>
> By the way, Little's law is a strong tool when it comes to averages. To be
> able to say more (e.g. 1% of the delays is larger than x) one requires more
> information (e.g. the traffic - On-OFF pattern) see [1].  I am not sure
> when does such information readily exist.
>
> Best
> Amr
>
> [1] https://dl.acm.org/doi/10.1145/3341617.3326146 or if behind a paywall
> https://www.dcs.warwick.ac.uk/~florin/lib/sigmet19b.pdf
>
> 
> Amr Rizk (amr.r...@uni-due.de)
> University of Duisburg-Essen
>
> -Ursprüngliche Nachricht-
> Von: Bloat  Im Auftrag von Ben Greear
> Gesendet: Montag, 12. Juli 2021 22:32
> An: Bob McMahon 
> Cc: starl...@lists.bufferbloat.net; Make-Wifi-fast <
> make-wifi-f...@lists.bufferbloat.net>; Leonard Kleinrock ;
> David P. Reed ; Cake List ;
> co...@lists.bufferbloat.net; cerowrt-devel <
> cerowrt-devel@lists.bufferbloat.net>; bloat 
> Betreff: Re: [Bloat] Little's Law mea culpa, but not invalidating my main
> point
>
> UDP is better for getting actual packet latency, for sure.  TCP is
> typical-user-experience-latency though, so it is also useful.
>
> I'm interested in the test and visualization side of this.  If there were
> a way to give engineers a good real-time look at a complex real-world
> network, then they have something to go on while trying to tune various
> knobs in their network to improve it.
>
> I'll let others try to figure out how build and tune the knobs, but the
> data acquisition and visualization is something we might try to
> accomplish.  I have a feeling I'm not the first person to think of this,
> howeverprobably someone already has done such a thing.
>
> Thanks,
> Ben
>
> On 7/12/21 1:04 PM, Bob McMahon wrote:
> > I believe end host's TCP stats are insufficient as seen per the
> > "failed" congested control mechanisms over the last decades. I think
> > Jaffe pointed this out in
> > 1979 though he was using what's been deemed on this thread as "spherical
> cow queueing theory."
> >
> > "Flow control in store-and-forward computer networks is appropriate
> > for decentralized execution. A formal description of a class of
> > "decentralized flow control algorithms" is given. The feasibility of
> > maximizing power with such algorithms is investigated. On the
> > assumption that communication links behave like M/M/1 servers it is
> shown that no "decentralized flow control algorithm" can maximize network
> power. Power has been suggested in the literature as a network performance
> objective. It is also shown that no objective based only on the users'
> throughputs and average delay is decentralizable. Finally, a restricted
> class of algorithms cannot even approximate power."
> >
> > https://ieeexplore.ieee.org/document/1095152
> >
> > Did Jaffe make a mistake?
> >
> > Also, it's been observed that latency is non-parametric in it's
> > distributions and computing gaussians per the central limit theorem
> > for OWD feedback loops aren't effective. How does one design a control
> loop around things that are non-parametric? It also begs the question, what
> are the feed forward knobs that can actually help?
> >
> > Bob
> >
> > On Mon, Jul 12, 2021 at 12:07 PM Ben Greear  > wrote:
> >
> > Measuring one or a few links provides a bit of data, but seems like
> if someone is trying to understand
> > a large and real network, 

Re: [Cerowrt-devel] [Bloat] Little's Law mea culpa, but not invalidating my main point

[Amr Rizk]

Ben, 

it depends on what one tries to measure. Doing a rate scan using UDP (to 
measure latency distributions under load) is the best thing that we have but 
without actually knowing how resources are shared (fair share as in WiFi, FIFO 
as nearly everywhere else) it becomes very difficult to interpret the results 
or provide a proper argument on latency. You are right - TCP stats are a proxy 
for user experience but I believe they are difficult to reproduce (we are 
always talking about very short TCP flows - the infinite TCP flow that 
converges to a steady behavior is purely academic).

By the way, Little's law is a strong tool when it comes to averages. To be able 
to say more (e.g. 1% of the delays is larger than x) one requires more 
information (e.g. the traffic - On-OFF pattern) see [1].  I am not sure when 
does such information readily exist. 

Best
Amr 

[1] https://dl.acm.org/doi/10.1145/3341617.3326146 or if behind a paywall 
https://www.dcs.warwick.ac.uk/~florin/lib/sigmet19b.pdf


Amr Rizk (amr.r...@uni-due.de)
University of Duisburg-Essen

-Ursprüngliche Nachricht-
Von: Bloat  Im Auftrag von Ben Greear
Gesendet: Montag, 12. Juli 2021 22:32
An: Bob McMahon 
Cc: starl...@lists.bufferbloat.net; Make-Wifi-fast 
; Leonard Kleinrock ; 
David P. Reed ; Cake List ; 
co...@lists.bufferbloat.net; cerowrt-devel 
; bloat 
Betreff: Re: [Bloat] Little's Law mea culpa, but not invalidating my main point

UDP is better for getting actual packet latency, for sure.  TCP is 
typical-user-experience-latency though, so it is also useful.

I'm interested in the test and visualization side of this.  If there were a way 
to give engineers a good real-time look at a complex real-world network, then 
they have something to go on while trying to tune various knobs in their 
network to improve it.

I'll let others try to figure out how build and tune the knobs, but the data 
acquisition and visualization is something we might try to accomplish.  I have 
a feeling I'm not the first person to think of this, howeverprobably 
someone already has done such a thing.

Thanks,
Ben

On 7/12/21 1:04 PM, Bob McMahon wrote:
> I believe end host's TCP stats are insufficient as seen per the 
> "failed" congested control mechanisms over the last decades. I think 
> Jaffe pointed this out in
> 1979 though he was using what's been deemed on this thread as "spherical cow 
> queueing theory."
> 
> "Flow control in store-and-forward computer networks is appropriate 
> for decentralized execution. A formal description of a class of 
> "decentralized flow control algorithms" is given. The feasibility of 
> maximizing power with such algorithms is investigated. On the 
> assumption that communication links behave like M/M/1 servers it is shown 
> that no "decentralized flow control algorithm" can maximize network power. 
> Power has been suggested in the literature as a network performance 
> objective. It is also shown that no objective based only on the users' 
> throughputs and average delay is decentralizable. Finally, a restricted class 
> of algorithms cannot even approximate power."
> 
> https://ieeexplore.ieee.org/document/1095152
> 
> Did Jaffe make a mistake?
> 
> Also, it's been observed that latency is non-parametric in it's 
> distributions and computing gaussians per the central limit theorem 
> for OWD feedback loops aren't effective. How does one design a control loop 
> around things that are non-parametric? It also begs the question, what are 
> the feed forward knobs that can actually help?
> 
> Bob
> 
> On Mon, Jul 12, 2021 at 12:07 PM Ben Greear  > wrote:
> 
> Measuring one or a few links provides a bit of data, but seems like if 
> someone is trying to understand
> a large and real network, then the OWD between point A and B needs to 
> just be input into something much
> more grand.  Assuming real-time OWD data exists between 100 to 1000 
> endpoint pairs, has anyone found a way
> to visualize this in a useful manner?
> 
> Also, considering something better than ntp may not really scale to 1000+ 
> endpoints, maybe round-trip
> time is only viable way to get this type of data.  In that case, maybe 
> clever logic could use things
> like trace-route to get some idea of how long it takes to get 'onto' the 
> internet proper, and so estimate
> the last-mile latency.  My assumption is that the last-mile latency is 
> where most of the pervasive
> assymetric network latencies would exist (or just ping 8.8.8.8 which is 
> 20ms from everywhere due to
> $magic).
> 
> Endpoints could also triangulate a bit if needed, using some anchor 
> points in the network
> under test.
> 
> Thanks,
> Ben
> 
> On 7/12/21 11:21 AM, Bob McMahon wrote:
>  > iperf 2 supports OWD and gives full histograms for TCP write to read, 
> TCP connect times, latency of packets (with UDP), latency of "frames" with
> 

Re: [Cerowrt-devel] [Bloat] Little's Law mea culpa, but not invalidating my main point

[Bob McMahon via Cerowrt-devel]

--- Begin Message ---
We in WiFi find UDP, while useful, also has severe limitations. The impact
to the TCP control loop matters a lot for things like aggregation.

Visualizations can be useful but also a bit limiting. We use stats
techniques such as PCA which is more mathematical and less visual.

We find syscall connect() times as a bit more relevant to user experience
than ICMP pings which are typically originated and terminated in kernel
space.

Bob

On Mon, Jul 12, 2021 at 1:32 PM Ben Greear  wrote:

> UDP is better for getting actual packet latency, for sure.  TCP is
> typical-user-experience-latency though,
> so it is also useful.
>
> I'm interested in the test and visualization side of this.  If there were
> a way to give engineers
> a good real-time look at a complex real-world network, then they have
> something to go on while trying
> to tune various knobs in their network to improve it.
>
> I'll let others try to figure out how build and tune the knobs, but the
> data acquisition and
> visualization is something we might try to accomplish.  I have a feeling
> I'm not the
> first person to think of this, howeverprobably someone already has
> done such
> a thing.
>
> Thanks,
> Ben
>
> On 7/12/21 1:04 PM, Bob McMahon wrote:
> > I believe end host's TCP stats are insufficient as seen per the "failed"
> congested control mechanisms over the last decades. I think Jaffe pointed
> this out in
> > 1979 though he was using what's been deemed on this thread as "spherical
> cow queueing theory."
> >
> > "Flow control in store-and-forward computer networks is appropriate for
> decentralized execution. A formal description of a class of "decentralized
> flow control
> > algorithms" is given. The feasibility of maximizing power with such
> algorithms is investigated. On the assumption that communication links
> behave like M/M/1
> > servers it is shown that no "decentralized flow control algorithm" can
> maximize network power. Power has been suggested in the literature as a
> network
> > performance objective. It is also shown that no objective based only on
> the users' throughputs and average delay is decentralizable. Finally, a
> restricted class
> > of algorithms cannot even approximate power."
> >
> > https://ieeexplore.ieee.org/document/1095152
> >
> > Did Jaffe make a mistake?
> >
> > Also, it's been observed that latency is non-parametric in it's
> distributions and computing gaussians per the central limit theorem for OWD
> feedback loops
> > aren't effective. How does one design a control loop around things that
> are non-parametric? It also begs the question, what are the feed forward
> knobs that can
> > actually help?
> >
> > Bob
> >
> > On Mon, Jul 12, 2021 at 12:07 PM Ben Greear  > wrote:
> >
> > Measuring one or a few links provides a bit of data, but seems like
> if someone is trying to understand
> > a large and real network, then the OWD between point A and B needs
> to just be input into something much
> > more grand.  Assuming real-time OWD data exists between 100 to 1000
> endpoint pairs, has anyone found a way
> > to visualize this in a useful manner?
> >
> > Also, considering something better than ntp may not really scale to
> 1000+ endpoints, maybe round-trip
> > time is only viable way to get this type of data.  In that case,
> maybe clever logic could use things
> > like trace-route to get some idea of how long it takes to get 'onto'
> the internet proper, and so estimate
> > the last-mile latency.  My assumption is that the last-mile latency
> is where most of the pervasive
> > assymetric network latencies would exist (or just ping 8.8.8.8 which
> is 20ms from everywhere due to
> > $magic).
> >
> > Endpoints could also triangulate a bit if needed, using some anchor
> points in the network
> > under test.
> >
> > Thanks,
> > Ben
> >
> > On 7/12/21 11:21 AM, Bob McMahon wrote:
> >  > iperf 2 supports OWD and gives full histograms for TCP write to
> read, TCP connect times, latency of packets (with UDP), latency of "frames"
> with
> >  > simulated video traffic (TCP and UDP), xfer times of bursts with
> low duty cycle traffic, and TCP RTT (sampling based.) It also has support
> for sampling (per
> >  > interval reports) down to 100 usecs if configured with
> --enable-fastsampling, otherwise the fastest sampling is 5 ms. We've
> released all this as open source.
> >  >
> >  > OWD only works if the end realtime clocks are synchronized using
> a "machine level" protocol such as IEEE 1588 or PTP. Sadly, *most data
> centers don't
> > provide
> >  > sufficient level of clock accuracy and the GPS pulse per second *
> to colo and vm customers.
> >  >
> >  > https://iperf2.sourceforge.io/iperf-manpage.html
> >  >
> >  > Bob
> >  >
> >  > On Mon, Jul 12, 2021 at 10:40 AM David P. Reed <
> dpr...@deepplum.com   

Re: [Cerowrt-devel] [Bloat] Little's Law mea culpa, but not invalidating my main point

[Ben Greear]

UDP is better for getting actual packet latency, for sure.  TCP is 
typical-user-experience-latency though,
so it is also useful.

I'm interested in the test and visualization side of this.  If there were a way 
to give engineers
a good real-time look at a complex real-world network, then they have something 
to go on while trying
to tune various knobs in their network to improve it.

I'll let others try to figure out how build and tune the knobs, but the data 
acquisition and
visualization is something we might try to accomplish.  I have a feeling I'm 
not the
first person to think of this, howeverprobably someone already has done such
a thing.

Thanks,
Ben

On 7/12/21 1:04 PM, Bob McMahon wrote:
I believe end host's TCP stats are insufficient as seen per the "failed" congested control mechanisms over the last decades. I think Jaffe pointed this out in 
1979 though he was using what's been deemed on this thread as "spherical cow queueing theory."


"Flow control in store-and-forward computer networks is appropriate for decentralized execution. A formal description of a class of "decentralized flow control 
algorithms" is given. The feasibility of maximizing power with such algorithms is investigated. On the assumption that communication links behave like M/M/1 
servers it is shown that no "decentralized flow control algorithm" can maximize network power. Power has been suggested in the literature as a network 
performance objective. It is also shown that no objective based only on the users' throughputs and average delay is decentralizable. Finally, a restricted class 
of algorithms cannot even approximate power."


https://ieeexplore.ieee.org/document/1095152

Did Jaffe make a mistake?

Also, it's been observed that latency is non-parametric in it's distributions and computing gaussians per the central limit theorem for OWD feedback loops 
aren't effective. How does one design a control loop around things that are non-parametric? It also begs the question, what are the feed forward knobs that can 
actually help?


Bob

On Mon, Jul 12, 2021 at 12:07 PM Ben Greear mailto:gree...@candelatech.com>> wrote:

Measuring one or a few links provides a bit of data, but seems like if 
someone is trying to understand
a large and real network, then the OWD between point A and B needs to just 
be input into something much
more grand.  Assuming real-time OWD data exists between 100 to 1000 
endpoint pairs, has anyone found a way
to visualize this in a useful manner?

Also, considering something better than ntp may not really scale to 1000+ 
endpoints, maybe round-trip
time is only viable way to get this type of data.  In that case, maybe 
clever logic could use things
like trace-route to get some idea of how long it takes to get 'onto' the 
internet proper, and so estimate
the last-mile latency.  My assumption is that the last-mile latency is 
where most of the pervasive
assymetric network latencies would exist (or just ping 8.8.8.8 which is 
20ms from everywhere due to
$magic).

Endpoints could also triangulate a bit if needed, using some anchor points 
in the network
under test.

Thanks,
Ben

On 7/12/21 11:21 AM, Bob McMahon wrote:
 > iperf 2 supports OWD and gives full histograms for TCP write to read, TCP connect 
times, latency of packets (with UDP), latency of "frames" with
 > simulated video traffic (TCP and UDP), xfer times of bursts with low 
duty cycle traffic, and TCP RTT (sampling based.) It also has support for sampling 
(per
 > interval reports) down to 100 usecs if configured with 
--enable-fastsampling, otherwise the fastest sampling is 5 ms. We've released all 
this as open source.
 >
 > OWD only works if the end realtime clocks are synchronized using a "machine 
level" protocol such as IEEE 1588 or PTP. Sadly, *most data centers don't
provide
 > sufficient level of clock accuracy and the GPS pulse per second * to 
colo and vm customers.
 >
 > https://iperf2.sourceforge.io/iperf-manpage.html
 >
 > Bob
 >
 > On Mon, Jul 12, 2021 at 10:40 AM David P. Reed mailto:dpr...@deepplum.com> >> wrote:
 >
 >
 >     On Monday, July 12, 2021 9:46am, "Livingood, Jason" 
mailto:jason_living...@comcast.com>
>> 
said:
 >
 >      > I think latency/delay is becoming seen to be as important 
certainly, if not a more direct proxy for end user QoE. This is all still evolving 
and I
have
 >     to say is a super interesting & fun thing to work on. :-)
 >
 >     If I could manage to sell one idea to the management hierarchy of 
communications industry CEOs (operators, vendors, ...) it is this one:
 >
 >     "It's the end-to-end latency, stupid!"
 >
 >     And I mean, by end-to-end, latency to complete a task at a relevant 

Re: [Cerowrt-devel] [Bloat] Little's Law mea culpa, but not invalidating my main point

[Bob McMahon via Cerowrt-devel]

--- Begin Message ---
I believe end host's TCP stats are insufficient as seen per the "failed"
congested control mechanisms over the last decades. I think Jaffe pointed
this out in 1979 though he was using what's been deemed on this thread as
"spherical cow queueing theory."

"Flow control in store-and-forward computer networks is appropriate for
decentralized execution. A formal description of a class of "decentralized
flow control algorithms" is given. The feasibility of maximizing power with
such algorithms is investigated. On the assumption that communication links
behave like M/M/1 servers it is shown that no "decentralized flow control
algorithm" can maximize network power. Power has been suggested in the
literature as a network performance objective. It is also shown that no
objective based only on the users' throughputs and average delay is
decentralizable. Finally, a restricted class of algorithms cannot even
approximate power."

https://ieeexplore.ieee.org/document/1095152

Did Jaffe make a mistake?

Also, it's been observed that latency is non-parametric in it's
distributions and computing gaussians per the central limit theorem for OWD
feedback loops aren't effective. How does one design a control loop around
things that are non-parametric? It also begs the question, what are the
feed forward knobs that can actually help?

Bob

On Mon, Jul 12, 2021 at 12:07 PM Ben Greear  wrote:

> Measuring one or a few links provides a bit of data, but seems like if
> someone is trying to understand
> a large and real network, then the OWD between point A and B needs to just
> be input into something much
> more grand.  Assuming real-time OWD data exists between 100 to 1000
> endpoint pairs, has anyone found a way
> to visualize this in a useful manner?
>
> Also, considering something better than ntp may not really scale to 1000+
> endpoints, maybe round-trip
> time is only viable way to get this type of data.  In that case, maybe
> clever logic could use things
> like trace-route to get some idea of how long it takes to get 'onto' the
> internet proper, and so estimate
> the last-mile latency.  My assumption is that the last-mile latency is
> where most of the pervasive
> assymetric network latencies would exist (or just ping 8.8.8.8 which is
> 20ms from everywhere due to
> $magic).
>
> Endpoints could also triangulate a bit if needed, using some anchor points
> in the network
> under test.
>
> Thanks,
> Ben
>
> On 7/12/21 11:21 AM, Bob McMahon wrote:
> > iperf 2 supports OWD and gives full histograms for TCP write to read,
> TCP connect times, latency of packets (with UDP), latency of "frames" with
> > simulated video traffic (TCP and UDP), xfer times of bursts with low
> duty cycle traffic, and TCP RTT (sampling based.) It also has support for
> sampling (per
> > interval reports) down to 100 usecs if configured with
> --enable-fastsampling, otherwise the fastest sampling is 5 ms. We've
> released all this as open source.
> >
> > OWD only works if the end realtime clocks are synchronized using a
> "machine level" protocol such as IEEE 1588 or PTP. Sadly, *most data
> centers don't provide
> > sufficient level of clock accuracy and the GPS pulse per second * to
> colo and vm customers.
> >
> > https://iperf2.sourceforge.io/iperf-manpage.html
> >
> > Bob
> >
> > On Mon, Jul 12, 2021 at 10:40 AM David P. Reed  > wrote:
> >
> >
> > On Monday, July 12, 2021 9:46am, "Livingood, Jason" <
> jason_living...@comcast.com > said:
> >
> >  > I think latency/delay is becoming seen to be as important
> certainly, if not a more direct proxy for end user QoE. This is all still
> evolving and I have
> > to say is a super interesting & fun thing to work on. :-)
> >
> > If I could manage to sell one idea to the management hierarchy of
> communications industry CEOs (operators, vendors, ...) it is this one:
> >
> > "It's the end-to-end latency, stupid!"
> >
> > And I mean, by end-to-end, latency to complete a task at a relevant
> layer of abstraction.
> >
> > At the link level, it's packet send to packet receive completion.
> >
> > But at the transport level including retransmission buffers, it's
> datagram (or message) origination until the acknowledgement arrives for
> that message being
> > delivered after whatever number of retransmissions, freeing the
> retransmission buffer.
> >
> > At the WWW level, it's mouse click to display update corresponding
> to completion of the request.
> >
> > What should be noted is that lower level latencies don't directly
> predict the magnitude of higher-level latencies. But longer lower level
> latencies almost
> > always amplfify higher level latencies. Often non-linearly.
> >
> > Throughput is very, very weakly related to these latencies, in
> contrast.
> >
> > The amplification process has to do with the presence of queueing.
> Queueing is ALWAYS bad for latency, and 

Re: [Cerowrt-devel] [Bloat] Little's Law mea culpa, but not invalidating my main point

[Ben Greear]

Measuring one or a few links provides a bit of data, but seems like if someone 
is trying to understand
a large and real network, then the OWD between point A and B needs to just be 
input into something much
more grand.  Assuming real-time OWD data exists between 100 to 1000 endpoint 
pairs, has anyone found a way
to visualize this in a useful manner?

Also, considering something better than ntp may not really scale to 1000+ 
endpoints, maybe round-trip
time is only viable way to get this type of data.  In that case, maybe clever 
logic could use things
like trace-route to get some idea of how long it takes to get 'onto' the 
internet proper, and so estimate
the last-mile latency.  My assumption is that the last-mile latency is where 
most of the pervasive
assymetric network latencies would exist (or just ping 8.8.8.8 which is 20ms 
from everywhere due to
$magic).

Endpoints could also triangulate a bit if needed, using some anchor points in 
the network
under test.

Thanks,
Ben

On 7/12/21 11:21 AM, Bob McMahon wrote:
iperf 2 supports OWD and gives full histograms for TCP write to read, TCP connect times, latency of packets (with UDP), latency of "frames" with 
simulated video traffic (TCP and UDP), xfer times of bursts with low duty cycle traffic, and TCP RTT (sampling based.) It also has support for sampling (per 
interval reports) down to 100 usecs if configured with --enable-fastsampling, otherwise the fastest sampling is 5 ms. We've released all this as open source.


OWD only works if the end realtime clocks are synchronized using a "machine level" protocol such as IEEE 1588 or PTP. Sadly, *most data centers don't provide 
sufficient level of clock accuracy and the GPS pulse per second * to colo and vm customers.


https://iperf2.sourceforge.io/iperf-manpage.html

Bob

On Mon, Jul 12, 2021 at 10:40 AM David P. Reed mailto:dpr...@deepplum.com>> wrote:


On Monday, July 12, 2021 9:46am, "Livingood, Jason" mailto:jason_living...@comcast.com>> said:

 > I think latency/delay is becoming seen to be as important certainly, if 
not a more direct proxy for end user QoE. This is all still evolving and I have
to say is a super interesting & fun thing to work on. :-)

If I could manage to sell one idea to the management hierarchy of 
communications industry CEOs (operators, vendors, ...) it is this one:

"It's the end-to-end latency, stupid!"

And I mean, by end-to-end, latency to complete a task at a relevant layer 
of abstraction.

At the link level, it's packet send to packet receive completion.

But at the transport level including retransmission buffers, it's datagram 
(or message) origination until the acknowledgement arrives for that message 
being
delivered after whatever number of retransmissions, freeing the 
retransmission buffer.

At the WWW level, it's mouse click to display update corresponding to 
completion of the request.

What should be noted is that lower level latencies don't directly predict 
the magnitude of higher-level latencies. But longer lower level latencies almost
always amplfify higher level latencies. Often non-linearly.

Throughput is very, very weakly related to these latencies, in contrast.

The amplification process has to do with the presence of queueing. Queueing 
is ALWAYS bad for latency, and throughput only helps if it is in exactly the
right place (the so-called input queue of the bottleneck process, which is 
often a link, but not always).

Can we get that slogan into Harvard Business Review? Can we get it taught 
in Managerial Accounting at HBS? (which does address logistics/supply chain 
queueing).







This electronic communication and the information and any files transmitted with it, or attached to it, are confidential and are intended solely for the use of 
the individual or entity to whom it is addressed and may contain information that is confidential, legally privileged, protected by privacy laws, or otherwise 
restricted from disclosure to anyone else. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, 
you are hereby notified that any use, copying, distributing, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited. If you 
received this e-mail in error, please return the e-mail to the sender, delete it from your computer, and destroy any printed copy of it.



--
Ben Greear 
Candela Technologies Inc  http://www.candelatech.com

___
Cerowrt-devel mailing list
Cerowrt-devel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel

Re: [Cerowrt-devel] [Bloat] Little's Law mea culpa, but not invalidating my main point

[Bob McMahon via Cerowrt-devel]

--- Begin Message ---
To be clear, it's a OS write() using a socket opened with TCP and the final
OS read() of that write. The write size is set using -l or --length. OWD
requires --trip-times option.

Bob

On Mon, Jul 12, 2021 at 11:21 AM Bob McMahon 
wrote:

> iperf 2 supports OWD and gives full histograms for TCP write to read, TCP
> connect times, latency of packets (with UDP), latency of "frames" with
> simulated video traffic (TCP and UDP), xfer times of bursts with low duty
> cycle traffic, and TCP RTT (sampling based.) It also has support for
> sampling (per interval reports) down to 100 usecs if configured with
> --enable-fastsampling, otherwise the fastest sampling is 5 ms. We've
> released all this as open source.
>
> OWD only works if the end realtime clocks are synchronized using a
> "machine level" protocol such as IEEE 1588 or PTP. Sadly, *most data
> centers don't provide sufficient level of clock accuracy and the GPS pulse
> per second * to colo and vm customers.
>
> https://iperf2.sourceforge.io/iperf-manpage.html
>
> Bob
>
> On Mon, Jul 12, 2021 at 10:40 AM David P. Reed 
> wrote:
>
>>
>> On Monday, July 12, 2021 9:46am, "Livingood, Jason" <
>> jason_living...@comcast.com> said:
>>
>> > I think latency/delay is becoming seen to be as important certainly, if
>> not a more direct proxy for end user QoE. This is all still evolving and I
>> have to say is a super interesting & fun thing to work on. :-)
>>
>> If I could manage to sell one idea to the management hierarchy of
>> communications industry CEOs (operators, vendors, ...) it is this one:
>>
>> "It's the end-to-end latency, stupid!"
>>
>> And I mean, by end-to-end, latency to complete a task at a relevant layer
>> of abstraction.
>>
>> At the link level, it's packet send to packet receive completion.
>>
>> But at the transport level including retransmission buffers, it's
>> datagram (or message) origination until the acknowledgement arrives for
>> that message being delivered after whatever number of retransmissions,
>> freeing the retransmission buffer.
>>
>> At the WWW level, it's mouse click to display update corresponding to
>> completion of the request.
>>
>> What should be noted is that lower level latencies don't directly predict
>> the magnitude of higher-level latencies. But longer lower level latencies
>> almost always amplfify higher level latencies. Often non-linearly.
>>
>> Throughput is very, very weakly related to these latencies, in contrast.
>>
>> The amplification process has to do with the presence of queueing.
>> Queueing is ALWAYS bad for latency, and throughput only helps if it is in
>> exactly the right place (the so-called input queue of the bottleneck
>> process, which is often a link, but not always).
>>
>> Can we get that slogan into Harvard Business Review? Can we get it taught
>> in Managerial Accounting at HBS? (which does address logistics/supply chain
>> queueing).
>>
>>
>>
>>
>>
>>
>>

-- 
This electronic communication and the information and any files transmitted 
with it, or attached to it, are confidential and are intended solely for 
the use of the individual or entity to whom it is addressed and may contain 
information that is confidential, legally privileged, protected by privacy 
laws, or otherwise restricted from disclosure to anyone else. If you are 
not the intended recipient or the person responsible for delivering the 
e-mail to the intended recipient, you are hereby notified that any use, 
copying, distributing, dissemination, forwarding, printing, or copying of 
this e-mail is strictly prohibited. If you received this e-mail in error, 
please return the e-mail to the sender, delete it from your computer, and 
destroy any printed copy of it.


smime.p7s
Description: S/MIME Cryptographic Signature
--- End Message ---
___
Cerowrt-devel mailing list
Cerowrt-devel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel

Re: [Cerowrt-devel] [Bloat] Little's Law mea culpa, but not invalidating my main point

[Bob McMahon via Cerowrt-devel]

--- Begin Message ---
iperf 2 supports OWD and gives full histograms for TCP write to read, TCP
connect times, latency of packets (with UDP), latency of "frames" with
simulated video traffic (TCP and UDP), xfer times of bursts with low duty
cycle traffic, and TCP RTT (sampling based.) It also has support for
sampling (per interval reports) down to 100 usecs if configured with
--enable-fastsampling, otherwise the fastest sampling is 5 ms. We've
released all this as open source.

OWD only works if the end realtime clocks are synchronized using a "machine
level" protocol such as IEEE 1588 or PTP. Sadly, *most data centers don't
provide sufficient level of clock accuracy and the GPS pulse per second *
to colo and vm customers.

https://iperf2.sourceforge.io/iperf-manpage.html

Bob

On Mon, Jul 12, 2021 at 10:40 AM David P. Reed  wrote:

>
> On Monday, July 12, 2021 9:46am, "Livingood, Jason" <
> jason_living...@comcast.com> said:
>
> > I think latency/delay is becoming seen to be as important certainly, if
> not a more direct proxy for end user QoE. This is all still evolving and I
> have to say is a super interesting & fun thing to work on. :-)
>
> If I could manage to sell one idea to the management hierarchy of
> communications industry CEOs (operators, vendors, ...) it is this one:
>
> "It's the end-to-end latency, stupid!"
>
> And I mean, by end-to-end, latency to complete a task at a relevant layer
> of abstraction.
>
> At the link level, it's packet send to packet receive completion.
>
> But at the transport level including retransmission buffers, it's datagram
> (or message) origination until the acknowledgement arrives for that message
> being delivered after whatever number of retransmissions, freeing the
> retransmission buffer.
>
> At the WWW level, it's mouse click to display update corresponding to
> completion of the request.
>
> What should be noted is that lower level latencies don't directly predict
> the magnitude of higher-level latencies. But longer lower level latencies
> almost always amplfify higher level latencies. Often non-linearly.
>
> Throughput is very, very weakly related to these latencies, in contrast.
>
> The amplification process has to do with the presence of queueing.
> Queueing is ALWAYS bad for latency, and throughput only helps if it is in
> exactly the right place (the so-called input queue of the bottleneck
> process, which is often a link, but not always).
>
> Can we get that slogan into Harvard Business Review? Can we get it taught
> in Managerial Accounting at HBS? (which does address logistics/supply chain
> queueing).
>
>
>
>
>
>
>

-- 
This electronic communication and the information and any files transmitted 
with it, or attached to it, are confidential and are intended solely for 
the use of the individual or entity to whom it is addressed and may contain 
information that is confidential, legally privileged, protected by privacy 
laws, or otherwise restricted from disclosure to anyone else. If you are 
not the intended recipient or the person responsible for delivering the 
e-mail to the intended recipient, you are hereby notified that any use, 
copying, distributing, dissemination, forwarding, printing, or copying of 
this e-mail is strictly prohibited. If you received this e-mail in error, 
please return the e-mail to the sender, delete it from your computer, and 
destroy any printed copy of it.


smime.p7s
Description: S/MIME Cryptographic Signature
--- End Message ---
___
Cerowrt-devel mailing list
Cerowrt-devel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel

Re: [Cerowrt-devel] [Bloat] Little's Law mea culpa, but not invalidating my main point

[David P. Reed]

 
On Monday, July 12, 2021 9:46am, "Livingood, Jason" 
 said:

> I think latency/delay is becoming seen to be as important certainly, if not a 
> more direct proxy for end user QoE. This is all still evolving and I have to 
> say is a super interesting & fun thing to work on. :-)
 
If I could manage to sell one idea to the management hierarchy of 
communications industry CEOs (operators, vendors, ...) it is this one:

"It's the end-to-end latency, stupid!"

And I mean, by end-to-end, latency to complete a task at a relevant layer of 
abstraction.

At the link level, it's packet send to packet receive completion.

But at the transport level including retransmission buffers, it's datagram (or 
message) origination until the acknowledgement arrives for that message being 
delivered after whatever number of retransmissions, freeing the retransmission 
buffer.

At the WWW level, it's mouse click to display update corresponding to 
completion of the request.

What should be noted is that lower level latencies don't directly predict the 
magnitude of higher-level latencies. But longer lower level latencies almost 
always amplfify higher level latencies. Often non-linearly.

Throughput is very, very weakly related to these latencies, in contrast.

The amplification process has to do with the presence of queueing. Queueing is 
ALWAYS bad for latency, and throughput only helps if it is in exactly the right 
place (the so-called input queue of the bottleneck process, which is often a 
link, but not always).

Can we get that slogan into Harvard Business Review? Can we get it taught in 
Managerial Accounting at HBS? (which does address logistics/supply chain 
queueing).
 
 
 
 
 

___
Cerowrt-devel mailing list
Cerowrt-devel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel

Re: [Cerowrt-devel] [Bloat] Little's Law mea culpa, but not invalidating my main point

[Livingood, Jason via Cerowrt-devel]

--- Begin Message ---
> 2) Users are pissed off, because they clicked on a web page, and got nothing 
> back. They retry on their screen, or they try another site. Meanwhile, the 
> underlying TCP connection remains there, pumping the network full of more 
> packets on that old path, which is still backed up with packets that haven't 
> been delivered that are sitting in queues.



Agree. I’ve experienced that as utilization of a network segment or supporting 
network systems (e.g. DNS) increases, you may see very small delay creep in - 
but not much as things are stable until they are *quite suddenly* not so. At 
that stability inflection point you immediately & dramatically fall off a 
cliff, which is then exacerbated by what you note here – user and machine-based 
retries/retransmissions that drives a huge increase in traffic. The solution 
has typically been throwing massive new capacity at it until the storm recedes.



> I should say that most operators, and especially ATT in this case, do not 
> measure end-to-end latency. Instead they use Little's Lemma to query routers 
> for their current throughput in bits per second, and calculate latency as if 
> Little's Lemma applied.



IMO network operators views/practices vary widely & have been evolving quite a 
bit in recent years. Yes, it used to be all about capacity utilization metrics 
but I think that is changing. In my day job, we run E2E latency tests (among 
others) to CPE and the distribution is a lot more instructive than the 
mean/median to continuously improving the network experience.



> And management responds, Hooray! Because utilization of 100% of their 
> hardware is their investors' metric of maximizing profits. The hardware they 
> are operating is fully utilized. No waste! And users are happy because no 
> packets have been dropped!



Well, I hope it wasn’t 100% utilization meant they were ‘green’ on their 
network KPIs. ;-) Ha. But I think you are correct that a network engineering 
team would have been measured by how well they kept ahead of utilization/demand 
& network capacity decisions driven in large part by utilization trends. In a 
lot of networks I suspect an informal rule of thumb arose that things got a 
little funny once p98 utilization got to around 94-95% of link capacity – so 
backup from there to figure out when you need to trigger automatic capacity 
augments to avoid that. While I do not think managing via utilization in that 
way is incorrect, ISTM it’s mostly being used as the measure is an indirect 
proxy for end user QoE. I think latency/delay is becoming seen to be as 
important certainly, if not a more direct proxy for end user QoE. This is all 
still evolving and I have to say is a super interesting & fun thing to work on. 
:-)



Jason












--- End Message ---
___
Cerowrt-devel mailing list
Cerowrt-devel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel

Re: [Cerowrt-devel] [Bloat] Little's Law mea culpa, but not invalidating my main point

[Jonathan Morton]

> On 10 Jul, 2021, at 2:01 am, Leonard Kleinrock  wrote:
> 
> No question that non-stationarity and instability are what we often see in 
> networks.  And, non-stationarity and instability are both topics that lead to 
> very complex analytical problems in queueing theory.  You can find some 
> results on the transient analysis in the queueing theory literature 
> (including the second volume of my Queueing Systems book), but they are 
> limited and hard. Nevertheless, the literature does contain some works on 
> transient analysis of queueing systems as applied to network congestion 
> control - again limited. On the other hand, as you said, control theory 
> addresses stability head on and does offer some tools as well, but again, it 
> is hairy. 

I was just about to mention control theory.

One basic characteristic of Poisson traffic is that it is inelastic, and 
assumes there is no control feedback whatsoever.  This means it can only be a 
valid model when the following are both true:

1: The offered load is *below* the link capacity, for all links, averaged over 
time.

2: A high degree of statistical multiplexing exists.

If 1: is not true and the traffic is truly inelastic, then the queues will 
inevitably fill up and congestion collapse will result, as shown from ARPANET 
experience in the 1980s; the solution was to introduce control feedback to the 
traffic, initially in the form of TCP Reno.  If 2: is not true then the traffic 
cannot be approximated as Poisson arrivals, regardless of load relative to 
capacity, because the degree of correlation is too high.

Taking the iPhone introduction anecdote as an illustrative example, measuring 
utilisation as very close to 100% is a clear warning sign that the Poisson 
model was inappropriate, and a control-theory approach was needed instead, to 
capture the feedback effects of congestion control.  The high degree of 
statistical multiplexing inherent to a major ISP backhaul is irrelevant to that 
determination.

Such a model would have found that the primary source of control feedback was 
human users giving up in disgust.  However, different humans have different 
levels of tolerance and persistence, so this feedback was not sufficient to 
reduce the load sufficiently to give the majority of users a good service; 
instead, *all* users received a poor service and many users received no usable 
service.  Introducing a technological control feedback, in the form of packet 
loss upon overflow of correctly-sized queues, improved service for everyone.

(BTW, DNS becomes significantly unreliable around 1-2 seconds RTT, due to 
protocol timeouts, which is inherited by all applications that rely on DNS 
lookups.  Merely reducing the delays consistently below that threshold would 
have improved perceived reliability markedly.)

Conversely, when talking about the traffic on a single ISP subscriber's 
last-mile link, the Poisson model has to be discarded due to criterion 2 being 
false.  The number of flows going to even a family household is probably in the 
low dozens at best.  A control-theory approach can also work here.

 - Jonathan Morton
___
Cerowrt-devel mailing list
Cerowrt-devel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel

Re: [Cerowrt-devel] [Bloat] Little's Law mea culpa, but not invalidating my main point

[Toke Høiland-Jørgensen via Cerowrt-devel]

--- Begin Message ---
"Holland, Jake via Bloat"  writes:

> Hi David,
>
> That’s an interesting point, and I think you’re right that packet
> arrival is poorly modeled as a Poisson process, because in practice
> packet transmissions are very rarely unrelated to other packet
> transmissions.
>
> But now you’ve got me wondering what the right approach is. Do you
> have any advice for how to improve this kind of modeling?

I actually tried my hand at finding something better for my master's
thesis and came across something called a Markov-Modulated Poisson
Process (MMPP/D/1 queue)[0]. It looked promising, but unfortunately I
failed to make it produce any useful predictions. Most likely this was
as much a result of my own failings as a queueing theorist as it was the
fault of the model (I was in way over my head by the time I got to that
model); so I figured I'd mention it here in case anyone more qualified
would have any opinion on it.

I did manage to get the Linux kernel to produce queueing behaviour that
resembled that of a standard M/M/1 queue (if you squint a bit); all you
have to do is to use a traffic generator that emits packets with the
distribution the model assumes... :)

The full thesis is still available[1] for the perusal of morbidly curious.

-Toke

[0] https://www.sciencedirect.com/science/article/abs/pii/016653169390035S
[1] https://rucforsk.ruc.dk/ws/files/57613884/thesis-final.pdf
--- End Message ---
___
Cerowrt-devel mailing list
Cerowrt-devel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel

Re: [Cerowrt-devel] [Bloat] Little's Law mea culpa, but not invalidating my main point

[Holland, Jake via Cerowrt-devel]

--- Begin Message ---
Hi David,

That’s an interesting point, and I think you’re right that packet arrival is 
poorly modeled as a Poisson process, because in practice packet transmissions 
are very rarely unrelated to other packet transmissions.

But now you’ve got me wondering what the right approach is.  Do you have any 
advice for how to improve this kind of modeling?

I’m thinking maybe a useful adjustment is to use Poisson start times on packet 
bursts, with a distribution on some burst characteristics? (Maybe like 
duration, rate, choppiness?)  Part of the point being that burst parameters 
then have a chance to become significant, as well as the load from aggregate 
user behavior.

And although I think user behavior is probably often ok to model as independent 
(outside of a background average that changes by time of day), in some contexts 
maybe it needs a 2nd overlay for bursts in user activity to address 
user-synchronizing events...  But for some problems I expect this kind of 
approach might still miss important feedback loop effects, and maybe for some 
problems it needs a more generalized suite of patterns besides a “burst”.  But 
maybe it would still be a step in the right direction for examining network 
loading problems in the abstract?

Or maybe it’s better to ask a different question:
Are there any good exemplars to follow here?  Any network traffic analysis (or 
related) work you’d recommend as having useful results that apply more broadly 
than a specific set of simulation/testing parameters, and that you wish more 
people would follow their example?

Also related: any particular papers come to mind that you wish someone would 
re-do with a better model?


Anyway, coming back to where that can of worms opened, I gotta say I like the 
“language of math” idea as a goal to aim for, and it would be surprising to me 
if no such useful information could be extracted from iperf runs.

A Little’s Law-based average queue estimate sounds possibly useful to me 
(especially compared across different runs or against external stats on 
background cross-traffic activity), and some kind of tail analysis on latency 
samples also sounds relevant to user experience. Maybe there’s some other 
things that would be better to include?

Best regards,
Jake


From: "David P. Reed" 
Date: Fri,2021-07-09 at 12:31 PM
To: Luca Muscariello 
Cc: Cake List , Make-Wifi-fast 
, Leonard Kleinrock , 
Bob McMahon , "starl...@lists.bufferbloat.net" 
, "co...@lists.bufferbloat.net" 
, cerowrt-devel 
, bloat , Ben 
Greear 
Subject: Re: [Bloat] Little's Law mea culpa, but not invalidating my main point


Len - I admit I made a mistake in challenging Little's Law as being based on 
Poisson processes. It is more general. But it tells you an "average" in its 
base form, and latency averages are not useful for end user applications.



However, Little's Law does assume something that is not actually valid about 
the kind of distributions seen in the network, and in fact, it is NOT true that 
networks converge on Poisson arrival times.



The key issue is well-described in the sandard analysis of the M/M/1 queue 
(e.g. 
https://en.wikipedia.org/wiki/M/M/1_queue)
 , which is done only for Poisson processes, and is also limited to "stable" 
systems. But networks are never stable when fully loaded. They get unstable and 
those instabilities persist for a long time in the network. Instability is at 
core the underlying *requirement* of the Internet's usage.



So specifically: real networks, even large ones, and certainly the Internet 
today, are not asymptotic limits of sums of stationary stochastic arrival 
processes. Each esternal terminal of any real network has a real user there, 
running a real application, and the network is a complex graph. This makes it 
completely unlike a single queue. Even the links within a network carry a 
relatively small number of application flows. There's no ability to apply the 
Law of Large Numbers to the distributions, because any particular path contains 
only a small number of serialized flows with hightly variable rates.



Here's an example of what really happens in a real network (I've observed this 
in 5 different cities on ATT's cellular network, back when it was running 
Alcatel Lucent HSPA+ gear in those cities).

But you can see this on any network where transient overload occurs, creating 
instability.





At 7 AM, the data transmission of the network is roughty stable. That's because 
no links are overloaded within the network. Little's Law can tell you by 
observing the delay and throughput on any path that the average delay in the 
network is X.



Continue sampling delay in the network as the day wears on. At about 10 AM, 
ping delay starts to soar into the multiple second range. No packers are lost. 
The peak ping time is about 4000