Re: [Cerowrt-devel] [Cake] [Bloat] Little's Law mea culpa, but not invalidating my main point

[Bob McMahon via Cerowrt-devel]

--- Begin Message ---
Agreed that UDP is important but it's also much easier to test and debug
for WiFi coders. We find it's the connect() and TCP control loop that
challenges WiFi logic systems on end hosts. APs are a whole different story
per things like OFDMA.

Nothing is simple anymore it seems. Reminds me of the standard model
developed over time at CERN. It ain't E=MC**2

Bob

On Mon, Jul 12, 2021 at 1:36 PM David Lang  wrote:

> I have seen some performance tests that do explicit DNS timing tests
> separate
> from other throughput/latency tests.
>
> Since DNS uses UDP (even if it then falls back to TCP in some cases), UDP
> performance (and especially probability of loss at congested links) is
> very
> important.
>
> David Lang
>
> On Mon, 12 Jul 2021, Ben Greear wrote:
>
> > UDP is better for getting actual packet latency, for sure.  TCP is
> > typical-user-experience-latency though,
> > so it is also useful.
> >
> > I'm interested in the test and visualization side of this.  If there
> were a
> > way to give engineers
> > a good real-time look at a complex real-world network, then they have
> > something to go on while trying
> > to tune various knobs in their network to improve it.
> >
> > I'll let others try to figure out how build and tune the knobs, but the
> data
> > acquisition and
> > visualization is something we might try to accomplish.  I have a feeling
> I'm
> > not the
> > first person to think of this, howeverprobably someone already has
> done
> > such
> > a thing.
> >
> > Thanks,
> > Ben
> >
> > On 7/12/21 1:04 PM, Bob McMahon wrote:
> >> I believe end host's TCP stats are insufficient as seen per the
> "failed"
> > congested control mechanisms over the last decades. I think Jaffe
> pointed
> > this out in
> >> 1979 though he was using what's been deemed on this thread as
> "spherical
> > cow queueing theory."
> >>
> >> "Flow control in store-and-forward computer networks is appropriate for
> > decentralized execution. A formal description of a class of
> "decentralized
> > flow control
> >> algorithms" is given. The feasibility of maximizing power with such
> > algorithms is investigated. On the assumption that communication links
> behave
> > like M/M/1
> >> servers it is shown that no "decentralized flow control algorithm" can
> > maximize network power. Power has been suggested in the literature as a
> > network
> >> performance objective. It is also shown that no objective based only on
> the
> > users' throughputs and average delay is decentralizable. Finally, a
> > restricted class
> >> of algorithms cannot even approximate power."
> >>
> >> https://ieeexplore.ieee.org/document/1095152
> >>
> >> Did Jaffe make a mistake?
> >>
> >> Also, it's been observed that latency is non-parametric in it's
> > distributions and computing gaussians per the central limit theorem for
> OWD
> > feedback loops
> >> aren't effective. How does one design a control loop around things that
> are
> > non-parametric? It also begs the question, what are the feed forward
> knobs
> > that can
> >> actually help?
> >>
> >> Bob
> >>
> >> On Mon, Jul 12, 2021 at 12:07 PM Ben Greear  > > wrote:
> >>
> >> Measuring one or a few links provides a bit of data, but seems like
> if
> > someone is trying to understand
> >> a large and real network, then the OWD between point A and B needs
> to
> > just be input into something much
> >> more grand.  Assuming real-time OWD data exists between 100 to 1000
> > endpoint pairs, has anyone found a way
> >> to visualize this in a useful manner?
> >>
> >> Also, considering something better than ntp may not really scale to
> > 1000+ endpoints, maybe round-trip
> >> time is only viable way to get this type of data.  In that case,
> maybe
> > clever logic could use things
> >> like trace-route to get some idea of how long it takes to get
> 'onto'
> > the internet proper, and so estimate
> >> the last-mile latency.  My assumption is that the last-mile latency
> is
> > where most of the pervasive
> >> assymetric network latencies would exist (or just ping 8.8.8.8
> which is
> > 20ms from everywhere due to
> >> $magic).
> >>
> >> Endpoints could also triangulate a bit if needed, using some anchor
> > points in the network
> >> under test.
> >>
> >> Thanks,
> >> Ben
> >>
> >> On 7/12/21 11:21 AM, Bob McMahon wrote:
> >>  > iperf 2 supports OWD and gives full histograms for TCP write to
> > read, TCP connect times, latency of packets (with UDP), latency of
> "frames"
> > with
> >>  > simulated video traffic (TCP and UDP), xfer times of bursts with
> low
> > duty cycle traffic, and TCP RTT (sampling based.) It also has support
> for
> > sampling (per
> >>  > interval reports) down to 100 usecs if configured with
> > --enable-fastsampling, otherwise the fastest sampling is 5 ms. We've
> released
> > all this as open source.
> >>  >
> >>  > OWD only works if the end 

Re: [Cerowrt-devel] [Bloat] Little's Law mea culpa, but not invalidating my main point

[Bob McMahon via Cerowrt-devel]

--- Begin Message ---
We in WiFi find UDP, while useful, also has severe limitations. The impact
to the TCP control loop matters a lot for things like aggregation.

Visualizations can be useful but also a bit limiting. We use stats
techniques such as PCA which is more mathematical and less visual.

We find syscall connect() times as a bit more relevant to user experience
than ICMP pings which are typically originated and terminated in kernel
space.

Bob

On Mon, Jul 12, 2021 at 1:32 PM Ben Greear  wrote:

> UDP is better for getting actual packet latency, for sure.  TCP is
> typical-user-experience-latency though,
> so it is also useful.
>
> I'm interested in the test and visualization side of this.  If there were
> a way to give engineers
> a good real-time look at a complex real-world network, then they have
> something to go on while trying
> to tune various knobs in their network to improve it.
>
> I'll let others try to figure out how build and tune the knobs, but the
> data acquisition and
> visualization is something we might try to accomplish.  I have a feeling
> I'm not the
> first person to think of this, howeverprobably someone already has
> done such
> a thing.
>
> Thanks,
> Ben
>
> On 7/12/21 1:04 PM, Bob McMahon wrote:
> > I believe end host's TCP stats are insufficient as seen per the "failed"
> congested control mechanisms over the last decades. I think Jaffe pointed
> this out in
> > 1979 though he was using what's been deemed on this thread as "spherical
> cow queueing theory."
> >
> > "Flow control in store-and-forward computer networks is appropriate for
> decentralized execution. A formal description of a class of "decentralized
> flow control
> > algorithms" is given. The feasibility of maximizing power with such
> algorithms is investigated. On the assumption that communication links
> behave like M/M/1
> > servers it is shown that no "decentralized flow control algorithm" can
> maximize network power. Power has been suggested in the literature as a
> network
> > performance objective. It is also shown that no objective based only on
> the users' throughputs and average delay is decentralizable. Finally, a
> restricted class
> > of algorithms cannot even approximate power."
> >
> > https://ieeexplore.ieee.org/document/1095152
> >
> > Did Jaffe make a mistake?
> >
> > Also, it's been observed that latency is non-parametric in it's
> distributions and computing gaussians per the central limit theorem for OWD
> feedback loops
> > aren't effective. How does one design a control loop around things that
> are non-parametric? It also begs the question, what are the feed forward
> knobs that can
> > actually help?
> >
> > Bob
> >
> > On Mon, Jul 12, 2021 at 12:07 PM Ben Greear  > wrote:
> >
> > Measuring one or a few links provides a bit of data, but seems like
> if someone is trying to understand
> > a large and real network, then the OWD between point A and B needs
> to just be input into something much
> > more grand.  Assuming real-time OWD data exists between 100 to 1000
> endpoint pairs, has anyone found a way
> > to visualize this in a useful manner?
> >
> > Also, considering something better than ntp may not really scale to
> 1000+ endpoints, maybe round-trip
> > time is only viable way to get this type of data.  In that case,
> maybe clever logic could use things
> > like trace-route to get some idea of how long it takes to get 'onto'
> the internet proper, and so estimate
> > the last-mile latency.  My assumption is that the last-mile latency
> is where most of the pervasive
> > assymetric network latencies would exist (or just ping 8.8.8.8 which
> is 20ms from everywhere due to
> > $magic).
> >
> > Endpoints could also triangulate a bit if needed, using some anchor
> points in the network
> > under test.
> >
> > Thanks,
> > Ben
> >
> > On 7/12/21 11:21 AM, Bob McMahon wrote:
> >  > iperf 2 supports OWD and gives full histograms for TCP write to
> read, TCP connect times, latency of packets (with UDP), latency of "frames"
> with
> >  > simulated video traffic (TCP and UDP), xfer times of bursts with
> low duty cycle traffic, and TCP RTT (sampling based.) It also has support
> for sampling (per
> >  > interval reports) down to 100 usecs if configured with
> --enable-fastsampling, otherwise the fastest sampling is 5 ms. We've
> released all this as open source.
> >  >
> >  > OWD only works if the end realtime clocks are synchronized using
> a "machine level" protocol such as IEEE 1588 or PTP. Sadly, *most data
> centers don't
> > provide
> >  > sufficient level of clock accuracy and the GPS pulse per second *
> to colo and vm customers.
> >  >
> >  > https://iperf2.sourceforge.io/iperf-manpage.html
> >  >
> >  > Bob
> >  >
> >  > On Mon, Jul 12, 2021 at 10:40 AM David P. Reed <
> dpr...@deepplum.com   

Re: [Cerowrt-devel] [Cake] [Bloat] Little's Law mea culpa, but not invalidating my main point

[David Lang]

I have seen some performance tests that do explicit DNS timing tests separate 
from other throughput/latency tests.


Since DNS uses UDP (even if it then falls back to TCP in some cases), UDP 
performance (and especially probability of loss at congested links) is very 
important.


David Lang

On Mon, 12 Jul 2021, Ben Greear wrote:

UDP is better for getting actual packet latency, for sure.  TCP is 
typical-user-experience-latency though,

so it is also useful.

I'm interested in the test and visualization side of this.  If there were a 
way to give engineers
a good real-time look at a complex real-world network, then they have 
something to go on while trying

to tune various knobs in their network to improve it.

I'll let others try to figure out how build and tune the knobs, but the data 
acquisition and
visualization is something we might try to accomplish.  I have a feeling I'm 
not the
first person to think of this, howeverprobably someone already has done 
such

a thing.

Thanks,
Ben

On 7/12/21 1:04 PM, Bob McMahon wrote:
I believe end host's TCP stats are insufficient as seen per the "failed" 
congested control mechanisms over the last decades. I think Jaffe pointed 
this out in 
1979 though he was using what's been deemed on this thread as "spherical 

cow queueing theory."


"Flow control in store-and-forward computer networks is appropriate for 
decentralized execution. A formal description of a class of "decentralized 
flow control 
algorithms" is given. The feasibility of maximizing power with such 
algorithms is investigated. On the assumption that communication links behave 
like M/M/1 
servers it is shown that no "decentralized flow control algorithm" can 
maximize network power. Power has been suggested in the literature as a 
network 
performance objective. It is also shown that no objective based only on the 
users' throughputs and average delay is decentralizable. Finally, a 
restricted class 

of algorithms cannot even approximate power."

https://ieeexplore.ieee.org/document/1095152

Did Jaffe make a mistake?

Also, it's been observed that latency is non-parametric in it's 
distributions and computing gaussians per the central limit theorem for OWD 
feedback loops 
aren't effective. How does one design a control loop around things that are 
non-parametric? It also begs the question, what are the feed forward knobs 
that can 

actually help?

Bob

On Mon, Jul 12, 2021 at 12:07 PM Ben Greear 
> wrote:


Measuring one or a few links provides a bit of data, but seems like if 

someone is trying to understand
a large and real network, then the OWD between point A and B needs to 

just be input into something much
more grand.  Assuming real-time OWD data exists between 100 to 1000 

endpoint pairs, has anyone found a way

to visualize this in a useful manner?

Also, considering something better than ntp may not really scale to 

1000+ endpoints, maybe round-trip
time is only viable way to get this type of data.  In that case, maybe 

clever logic could use things
like trace-route to get some idea of how long it takes to get 'onto' 

the internet proper, and so estimate
the last-mile latency.  My assumption is that the last-mile latency is 

where most of the pervasive
assymetric network latencies would exist (or just ping 8.8.8.8 which is 

20ms from everywhere due to

$magic).

Endpoints could also triangulate a bit if needed, using some anchor 

points in the network

under test.

Thanks,
Ben

On 7/12/21 11:21 AM, Bob McMahon wrote:
 > iperf 2 supports OWD and gives full histograms for TCP write to 
read, TCP connect times, latency of packets (with UDP), latency of "frames" 
with
 > simulated video traffic (TCP and UDP), xfer times of bursts with low 
duty cycle traffic, and TCP RTT (sampling based.) It also has support for 
sampling (per
 > interval reports) down to 100 usecs if configured with 
--enable-fastsampling, otherwise the fastest sampling is 5 ms. We've released 
all this as open source.

 >
 > OWD only works if the end realtime clocks are synchronized using a 
"machine level" protocol such as IEEE 1588 or PTP. Sadly, *most data centers 
don't

provide
 > sufficient level of clock accuracy and the GPS pulse per second * to 

colo and vm customers.

 >
 > https://iperf2.sourceforge.io/iperf-manpage.html
 >
 > Bob
 >
 > On Mon, Jul 12, 2021 at 10:40 AM David P. Reed 
 >> wrote:
 >
 >
 >     On Monday, July 12, 2021 9:46am, "Livingood, Jason" 

mailto:jason_living...@comcast.com>
>> said:

 >
 >      > I think latency/delay is becoming seen to be as important 
certainly, if not a more direct proxy for end user QoE. This is all still 
evolving and I

 

Re: [Cerowrt-devel] [Bloat] Little's Law mea culpa, but not invalidating my main point

[Ben Greear]

UDP is better for getting actual packet latency, for sure.  TCP is 
typical-user-experience-latency though,
so it is also useful.

I'm interested in the test and visualization side of this.  If there were a way 
to give engineers
a good real-time look at a complex real-world network, then they have something 
to go on while trying
to tune various knobs in their network to improve it.

I'll let others try to figure out how build and tune the knobs, but the data 
acquisition and
visualization is something we might try to accomplish.  I have a feeling I'm 
not the
first person to think of this, howeverprobably someone already has done such
a thing.

Thanks,
Ben

On 7/12/21 1:04 PM, Bob McMahon wrote:
I believe end host's TCP stats are insufficient as seen per the "failed" congested control mechanisms over the last decades. I think Jaffe pointed this out in 
1979 though he was using what's been deemed on this thread as "spherical cow queueing theory."


"Flow control in store-and-forward computer networks is appropriate for decentralized execution. A formal description of a class of "decentralized flow control 
algorithms" is given. The feasibility of maximizing power with such algorithms is investigated. On the assumption that communication links behave like M/M/1 
servers it is shown that no "decentralized flow control algorithm" can maximize network power. Power has been suggested in the literature as a network 
performance objective. It is also shown that no objective based only on the users' throughputs and average delay is decentralizable. Finally, a restricted class 
of algorithms cannot even approximate power."


https://ieeexplore.ieee.org/document/1095152

Did Jaffe make a mistake?

Also, it's been observed that latency is non-parametric in it's distributions and computing gaussians per the central limit theorem for OWD feedback loops 
aren't effective. How does one design a control loop around things that are non-parametric? It also begs the question, what are the feed forward knobs that can 
actually help?


Bob

On Mon, Jul 12, 2021 at 12:07 PM Ben Greear mailto:gree...@candelatech.com>> wrote:

Measuring one or a few links provides a bit of data, but seems like if 
someone is trying to understand
a large and real network, then the OWD between point A and B needs to just 
be input into something much
more grand.  Assuming real-time OWD data exists between 100 to 1000 
endpoint pairs, has anyone found a way
to visualize this in a useful manner?

Also, considering something better than ntp may not really scale to 1000+ 
endpoints, maybe round-trip
time is only viable way to get this type of data.  In that case, maybe 
clever logic could use things
like trace-route to get some idea of how long it takes to get 'onto' the 
internet proper, and so estimate
the last-mile latency.  My assumption is that the last-mile latency is 
where most of the pervasive
assymetric network latencies would exist (or just ping 8.8.8.8 which is 
20ms from everywhere due to
$magic).

Endpoints could also triangulate a bit if needed, using some anchor points 
in the network
under test.

Thanks,
Ben

On 7/12/21 11:21 AM, Bob McMahon wrote:
 > iperf 2 supports OWD and gives full histograms for TCP write to read, TCP connect 
times, latency of packets (with UDP), latency of "frames" with
 > simulated video traffic (TCP and UDP), xfer times of bursts with low 
duty cycle traffic, and TCP RTT (sampling based.) It also has support for sampling 
(per
 > interval reports) down to 100 usecs if configured with 
--enable-fastsampling, otherwise the fastest sampling is 5 ms. We've released all 
this as open source.
 >
 > OWD only works if the end realtime clocks are synchronized using a "machine 
level" protocol such as IEEE 1588 or PTP. Sadly, *most data centers don't
provide
 > sufficient level of clock accuracy and the GPS pulse per second * to 
colo and vm customers.
 >
 > https://iperf2.sourceforge.io/iperf-manpage.html
 >
 > Bob
 >
 > On Mon, Jul 12, 2021 at 10:40 AM David P. Reed mailto:dpr...@deepplum.com> >> wrote:
 >
 >
 >     On Monday, July 12, 2021 9:46am, "Livingood, Jason" 
mailto:jason_living...@comcast.com>
>> 
said:
 >
 >      > I think latency/delay is becoming seen to be as important 
certainly, if not a more direct proxy for end user QoE. This is all still evolving 
and I
have
 >     to say is a super interesting & fun thing to work on. :-)
 >
 >     If I could manage to sell one idea to the management hierarchy of 
communications industry CEOs (operators, vendors, ...) it is this one:
 >
 >     "It's the end-to-end latency, stupid!"
 >
 >     And I mean, by end-to-end, latency to complete a task at a relevant 

Re: [Cerowrt-devel] [Bloat] Little's Law mea culpa, but not invalidating my main point

[Bob McMahon via Cerowrt-devel]

--- Begin Message ---
I believe end host's TCP stats are insufficient as seen per the "failed"
congested control mechanisms over the last decades. I think Jaffe pointed
this out in 1979 though he was using what's been deemed on this thread as
"spherical cow queueing theory."

"Flow control in store-and-forward computer networks is appropriate for
decentralized execution. A formal description of a class of "decentralized
flow control algorithms" is given. The feasibility of maximizing power with
such algorithms is investigated. On the assumption that communication links
behave like M/M/1 servers it is shown that no "decentralized flow control
algorithm" can maximize network power. Power has been suggested in the
literature as a network performance objective. It is also shown that no
objective based only on the users' throughputs and average delay is
decentralizable. Finally, a restricted class of algorithms cannot even
approximate power."

https://ieeexplore.ieee.org/document/1095152

Did Jaffe make a mistake?

Also, it's been observed that latency is non-parametric in it's
distributions and computing gaussians per the central limit theorem for OWD
feedback loops aren't effective. How does one design a control loop around
things that are non-parametric? It also begs the question, what are the
feed forward knobs that can actually help?

Bob

On Mon, Jul 12, 2021 at 12:07 PM Ben Greear  wrote:

> Measuring one or a few links provides a bit of data, but seems like if
> someone is trying to understand
> a large and real network, then the OWD between point A and B needs to just
> be input into something much
> more grand.  Assuming real-time OWD data exists between 100 to 1000
> endpoint pairs, has anyone found a way
> to visualize this in a useful manner?
>
> Also, considering something better than ntp may not really scale to 1000+
> endpoints, maybe round-trip
> time is only viable way to get this type of data.  In that case, maybe
> clever logic could use things
> like trace-route to get some idea of how long it takes to get 'onto' the
> internet proper, and so estimate
> the last-mile latency.  My assumption is that the last-mile latency is
> where most of the pervasive
> assymetric network latencies would exist (or just ping 8.8.8.8 which is
> 20ms from everywhere due to
> $magic).
>
> Endpoints could also triangulate a bit if needed, using some anchor points
> in the network
> under test.
>
> Thanks,
> Ben
>
> On 7/12/21 11:21 AM, Bob McMahon wrote:
> > iperf 2 supports OWD and gives full histograms for TCP write to read,
> TCP connect times, latency of packets (with UDP), latency of "frames" with
> > simulated video traffic (TCP and UDP), xfer times of bursts with low
> duty cycle traffic, and TCP RTT (sampling based.) It also has support for
> sampling (per
> > interval reports) down to 100 usecs if configured with
> --enable-fastsampling, otherwise the fastest sampling is 5 ms. We've
> released all this as open source.
> >
> > OWD only works if the end realtime clocks are synchronized using a
> "machine level" protocol such as IEEE 1588 or PTP. Sadly, *most data
> centers don't provide
> > sufficient level of clock accuracy and the GPS pulse per second * to
> colo and vm customers.
> >
> > https://iperf2.sourceforge.io/iperf-manpage.html
> >
> > Bob
> >
> > On Mon, Jul 12, 2021 at 10:40 AM David P. Reed  > wrote:
> >
> >
> > On Monday, July 12, 2021 9:46am, "Livingood, Jason" <
> jason_living...@comcast.com > said:
> >
> >  > I think latency/delay is becoming seen to be as important
> certainly, if not a more direct proxy for end user QoE. This is all still
> evolving and I have
> > to say is a super interesting & fun thing to work on. :-)
> >
> > If I could manage to sell one idea to the management hierarchy of
> communications industry CEOs (operators, vendors, ...) it is this one:
> >
> > "It's the end-to-end latency, stupid!"
> >
> > And I mean, by end-to-end, latency to complete a task at a relevant
> layer of abstraction.
> >
> > At the link level, it's packet send to packet receive completion.
> >
> > But at the transport level including retransmission buffers, it's
> datagram (or message) origination until the acknowledgement arrives for
> that message being
> > delivered after whatever number of retransmissions, freeing the
> retransmission buffer.
> >
> > At the WWW level, it's mouse click to display update corresponding
> to completion of the request.
> >
> > What should be noted is that lower level latencies don't directly
> predict the magnitude of higher-level latencies. But longer lower level
> latencies almost
> > always amplfify higher level latencies. Often non-linearly.
> >
> > Throughput is very, very weakly related to these latencies, in
> contrast.
> >
> > The amplification process has to do with the presence of queueing.
> Queueing is ALWAYS bad for latency, and 

Re: [Cerowrt-devel] [Bloat] Little's Law mea culpa, but not invalidating my main point

[Ben Greear]

Measuring one or a few links provides a bit of data, but seems like if someone 
is trying to understand
a large and real network, then the OWD between point A and B needs to just be 
input into something much
more grand.  Assuming real-time OWD data exists between 100 to 1000 endpoint 
pairs, has anyone found a way
to visualize this in a useful manner?

Also, considering something better than ntp may not really scale to 1000+ 
endpoints, maybe round-trip
time is only viable way to get this type of data.  In that case, maybe clever 
logic could use things
like trace-route to get some idea of how long it takes to get 'onto' the 
internet proper, and so estimate
the last-mile latency.  My assumption is that the last-mile latency is where 
most of the pervasive
assymetric network latencies would exist (or just ping 8.8.8.8 which is 20ms 
from everywhere due to
$magic).

Endpoints could also triangulate a bit if needed, using some anchor points in 
the network
under test.

Thanks,
Ben

On 7/12/21 11:21 AM, Bob McMahon wrote:
iperf 2 supports OWD and gives full histograms for TCP write to read, TCP connect times, latency of packets (with UDP), latency of "frames" with 
simulated video traffic (TCP and UDP), xfer times of bursts with low duty cycle traffic, and TCP RTT (sampling based.) It also has support for sampling (per 
interval reports) down to 100 usecs if configured with --enable-fastsampling, otherwise the fastest sampling is 5 ms. We've released all this as open source.


OWD only works if the end realtime clocks are synchronized using a "machine level" protocol such as IEEE 1588 or PTP. Sadly, *most data centers don't provide 
sufficient level of clock accuracy and the GPS pulse per second * to colo and vm customers.


https://iperf2.sourceforge.io/iperf-manpage.html

Bob

On Mon, Jul 12, 2021 at 10:40 AM David P. Reed mailto:dpr...@deepplum.com>> wrote:


On Monday, July 12, 2021 9:46am, "Livingood, Jason" mailto:jason_living...@comcast.com>> said:

 > I think latency/delay is becoming seen to be as important certainly, if 
not a more direct proxy for end user QoE. This is all still evolving and I have
to say is a super interesting & fun thing to work on. :-)

If I could manage to sell one idea to the management hierarchy of 
communications industry CEOs (operators, vendors, ...) it is this one:

"It's the end-to-end latency, stupid!"

And I mean, by end-to-end, latency to complete a task at a relevant layer 
of abstraction.

At the link level, it's packet send to packet receive completion.

But at the transport level including retransmission buffers, it's datagram 
(or message) origination until the acknowledgement arrives for that message 
being
delivered after whatever number of retransmissions, freeing the 
retransmission buffer.

At the WWW level, it's mouse click to display update corresponding to 
completion of the request.

What should be noted is that lower level latencies don't directly predict 
the magnitude of higher-level latencies. But longer lower level latencies almost
always amplfify higher level latencies. Often non-linearly.

Throughput is very, very weakly related to these latencies, in contrast.

The amplification process has to do with the presence of queueing. Queueing 
is ALWAYS bad for latency, and throughput only helps if it is in exactly the
right place (the so-called input queue of the bottleneck process, which is 
often a link, but not always).

Can we get that slogan into Harvard Business Review? Can we get it taught 
in Managerial Accounting at HBS? (which does address logistics/supply chain 
queueing).







This electronic communication and the information and any files transmitted with it, or attached to it, are confidential and are intended solely for the use of 
the individual or entity to whom it is addressed and may contain information that is confidential, legally privileged, protected by privacy laws, or otherwise 
restricted from disclosure to anyone else. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, 
you are hereby notified that any use, copying, distributing, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited. If you 
received this e-mail in error, please return the e-mail to the sender, delete it from your computer, and destroy any printed copy of it.



--
Ben Greear 
Candela Technologies Inc  http://www.candelatech.com

___
Cerowrt-devel mailing list
Cerowrt-devel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel

Re: [Cerowrt-devel] [Bloat] Little's Law mea culpa, but not invalidating my main point

[Bob McMahon via Cerowrt-devel]

--- Begin Message ---
To be clear, it's a OS write() using a socket opened with TCP and the final
OS read() of that write. The write size is set using -l or --length. OWD
requires --trip-times option.

Bob

On Mon, Jul 12, 2021 at 11:21 AM Bob McMahon 
wrote:

> iperf 2 supports OWD and gives full histograms for TCP write to read, TCP
> connect times, latency of packets (with UDP), latency of "frames" with
> simulated video traffic (TCP and UDP), xfer times of bursts with low duty
> cycle traffic, and TCP RTT (sampling based.) It also has support for
> sampling (per interval reports) down to 100 usecs if configured with
> --enable-fastsampling, otherwise the fastest sampling is 5 ms. We've
> released all this as open source.
>
> OWD only works if the end realtime clocks are synchronized using a
> "machine level" protocol such as IEEE 1588 or PTP. Sadly, *most data
> centers don't provide sufficient level of clock accuracy and the GPS pulse
> per second * to colo and vm customers.
>
> https://iperf2.sourceforge.io/iperf-manpage.html
>
> Bob
>
> On Mon, Jul 12, 2021 at 10:40 AM David P. Reed 
> wrote:
>
>>
>> On Monday, July 12, 2021 9:46am, "Livingood, Jason" <
>> jason_living...@comcast.com> said:
>>
>> > I think latency/delay is becoming seen to be as important certainly, if
>> not a more direct proxy for end user QoE. This is all still evolving and I
>> have to say is a super interesting & fun thing to work on. :-)
>>
>> If I could manage to sell one idea to the management hierarchy of
>> communications industry CEOs (operators, vendors, ...) it is this one:
>>
>> "It's the end-to-end latency, stupid!"
>>
>> And I mean, by end-to-end, latency to complete a task at a relevant layer
>> of abstraction.
>>
>> At the link level, it's packet send to packet receive completion.
>>
>> But at the transport level including retransmission buffers, it's
>> datagram (or message) origination until the acknowledgement arrives for
>> that message being delivered after whatever number of retransmissions,
>> freeing the retransmission buffer.
>>
>> At the WWW level, it's mouse click to display update corresponding to
>> completion of the request.
>>
>> What should be noted is that lower level latencies don't directly predict
>> the magnitude of higher-level latencies. But longer lower level latencies
>> almost always amplfify higher level latencies. Often non-linearly.
>>
>> Throughput is very, very weakly related to these latencies, in contrast.
>>
>> The amplification process has to do with the presence of queueing.
>> Queueing is ALWAYS bad for latency, and throughput only helps if it is in
>> exactly the right place (the so-called input queue of the bottleneck
>> process, which is often a link, but not always).
>>
>> Can we get that slogan into Harvard Business Review? Can we get it taught
>> in Managerial Accounting at HBS? (which does address logistics/supply chain
>> queueing).
>>
>>
>>
>>
>>
>>
>>

-- 
This electronic communication and the information and any files transmitted 
with it, or attached to it, are confidential and are intended solely for 
the use of the individual or entity to whom it is addressed and may contain 
information that is confidential, legally privileged, protected by privacy 
laws, or otherwise restricted from disclosure to anyone else. If you are 
not the intended recipient or the person responsible for delivering the 
e-mail to the intended recipient, you are hereby notified that any use, 
copying, distributing, dissemination, forwarding, printing, or copying of 
this e-mail is strictly prohibited. If you received this e-mail in error, 
please return the e-mail to the sender, delete it from your computer, and 
destroy any printed copy of it.


smime.p7s
Description: S/MIME Cryptographic Signature
--- End Message ---
___
Cerowrt-devel mailing list
Cerowrt-devel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel

Re: [Cerowrt-devel] [Bloat] Little's Law mea culpa, but not invalidating my main point

[Bob McMahon via Cerowrt-devel]

--- Begin Message ---
iperf 2 supports OWD and gives full histograms for TCP write to read, TCP
connect times, latency of packets (with UDP), latency of "frames" with
simulated video traffic (TCP and UDP), xfer times of bursts with low duty
cycle traffic, and TCP RTT (sampling based.) It also has support for
sampling (per interval reports) down to 100 usecs if configured with
--enable-fastsampling, otherwise the fastest sampling is 5 ms. We've
released all this as open source.

OWD only works if the end realtime clocks are synchronized using a "machine
level" protocol such as IEEE 1588 or PTP. Sadly, *most data centers don't
provide sufficient level of clock accuracy and the GPS pulse per second *
to colo and vm customers.

https://iperf2.sourceforge.io/iperf-manpage.html

Bob

On Mon, Jul 12, 2021 at 10:40 AM David P. Reed  wrote:

>
> On Monday, July 12, 2021 9:46am, "Livingood, Jason" <
> jason_living...@comcast.com> said:
>
> > I think latency/delay is becoming seen to be as important certainly, if
> not a more direct proxy for end user QoE. This is all still evolving and I
> have to say is a super interesting & fun thing to work on. :-)
>
> If I could manage to sell one idea to the management hierarchy of
> communications industry CEOs (operators, vendors, ...) it is this one:
>
> "It's the end-to-end latency, stupid!"
>
> And I mean, by end-to-end, latency to complete a task at a relevant layer
> of abstraction.
>
> At the link level, it's packet send to packet receive completion.
>
> But at the transport level including retransmission buffers, it's datagram
> (or message) origination until the acknowledgement arrives for that message
> being delivered after whatever number of retransmissions, freeing the
> retransmission buffer.
>
> At the WWW level, it's mouse click to display update corresponding to
> completion of the request.
>
> What should be noted is that lower level latencies don't directly predict
> the magnitude of higher-level latencies. But longer lower level latencies
> almost always amplfify higher level latencies. Often non-linearly.
>
> Throughput is very, very weakly related to these latencies, in contrast.
>
> The amplification process has to do with the presence of queueing.
> Queueing is ALWAYS bad for latency, and throughput only helps if it is in
> exactly the right place (the so-called input queue of the bottleneck
> process, which is often a link, but not always).
>
> Can we get that slogan into Harvard Business Review? Can we get it taught
> in Managerial Accounting at HBS? (which does address logistics/supply chain
> queueing).
>
>
>
>
>
>
>

-- 
This electronic communication and the information and any files transmitted 
with it, or attached to it, are confidential and are intended solely for 
the use of the individual or entity to whom it is addressed and may contain 
information that is confidential, legally privileged, protected by privacy 
laws, or otherwise restricted from disclosure to anyone else. If you are 
not the intended recipient or the person responsible for delivering the 
e-mail to the intended recipient, you are hereby notified that any use, 
copying, distributing, dissemination, forwarding, printing, or copying of 
this e-mail is strictly prohibited. If you received this e-mail in error, 
please return the e-mail to the sender, delete it from your computer, and 
destroy any printed copy of it.


smime.p7s
Description: S/MIME Cryptographic Signature
--- End Message ---
___
Cerowrt-devel mailing list
Cerowrt-devel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel

Re: [Cerowrt-devel] [Bloat] Little's Law mea culpa, but not invalidating my main point

[David P. Reed]

 
On Monday, July 12, 2021 9:46am, "Livingood, Jason" 
 said:

> I think latency/delay is becoming seen to be as important certainly, if not a 
> more direct proxy for end user QoE. This is all still evolving and I have to 
> say is a super interesting & fun thing to work on. :-)
 
If I could manage to sell one idea to the management hierarchy of 
communications industry CEOs (operators, vendors, ...) it is this one:

"It's the end-to-end latency, stupid!"

And I mean, by end-to-end, latency to complete a task at a relevant layer of 
abstraction.

At the link level, it's packet send to packet receive completion.

But at the transport level including retransmission buffers, it's datagram (or 
message) origination until the acknowledgement arrives for that message being 
delivered after whatever number of retransmissions, freeing the retransmission 
buffer.

At the WWW level, it's mouse click to display update corresponding to 
completion of the request.

What should be noted is that lower level latencies don't directly predict the 
magnitude of higher-level latencies. But longer lower level latencies almost 
always amplfify higher level latencies. Often non-linearly.

Throughput is very, very weakly related to these latencies, in contrast.

The amplification process has to do with the presence of queueing. Queueing is 
ALWAYS bad for latency, and throughput only helps if it is in exactly the right 
place (the so-called input queue of the bottleneck process, which is often a 
link, but not always).

Can we get that slogan into Harvard Business Review? Can we get it taught in 
Managerial Accounting at HBS? (which does address logistics/supply chain 
queueing).
 
 
 
 
 

___
Cerowrt-devel mailing list
Cerowrt-devel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel

Re: [Cerowrt-devel] [Bloat] Little's Law mea culpa, but not invalidating my main point

[Livingood, Jason via Cerowrt-devel]

--- Begin Message ---
> 2) Users are pissed off, because they clicked on a web page, and got nothing 
> back. They retry on their screen, or they try another site. Meanwhile, the 
> underlying TCP connection remains there, pumping the network full of more 
> packets on that old path, which is still backed up with packets that haven't 
> been delivered that are sitting in queues.



Agree. I’ve experienced that as utilization of a network segment or supporting 
network systems (e.g. DNS) increases, you may see very small delay creep in - 
but not much as things are stable until they are *quite suddenly* not so. At 
that stability inflection point you immediately & dramatically fall off a 
cliff, which is then exacerbated by what you note here – user and machine-based 
retries/retransmissions that drives a huge increase in traffic. The solution 
has typically been throwing massive new capacity at it until the storm recedes.



> I should say that most operators, and especially ATT in this case, do not 
> measure end-to-end latency. Instead they use Little's Lemma to query routers 
> for their current throughput in bits per second, and calculate latency as if 
> Little's Lemma applied.



IMO network operators views/practices vary widely & have been evolving quite a 
bit in recent years. Yes, it used to be all about capacity utilization metrics 
but I think that is changing. In my day job, we run E2E latency tests (among 
others) to CPE and the distribution is a lot more instructive than the 
mean/median to continuously improving the network experience.



> And management responds, Hooray! Because utilization of 100% of their 
> hardware is their investors' metric of maximizing profits. The hardware they 
> are operating is fully utilized. No waste! And users are happy because no 
> packets have been dropped!



Well, I hope it wasn’t 100% utilization meant they were ‘green’ on their 
network KPIs. ;-) Ha. But I think you are correct that a network engineering 
team would have been measured by how well they kept ahead of utilization/demand 
& network capacity decisions driven in large part by utilization trends. In a 
lot of networks I suspect an informal rule of thumb arose that things got a 
little funny once p98 utilization got to around 94-95% of link capacity – so 
backup from there to figure out when you need to trigger automatic capacity 
augments to avoid that. While I do not think managing via utilization in that 
way is incorrect, ISTM it’s mostly being used as the measure is an indirect 
proxy for end user QoE. I think latency/delay is becoming seen to be as 
important certainly, if not a more direct proxy for end user QoE. This is all 
still evolving and I have to say is a super interesting & fun thing to work on. 
:-)



Jason












--- End Message ---
___
Cerowrt-devel mailing list
Cerowrt-devel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel