Re: [Bloat] [Make-wifi-fast] [Cerowrt-devel] Due Aug 2: Internet Quality workshop CFP for the internet architecture board

[Bob McMahon via Bloat]

I think we need the language of math here. It seems like the network power
metric, introduced by Kleinrock and Jaffe in the late 70s, is something
useful. Effective end/end queue depths per Little's law also seems useful.
Both are available in iperf 2 from a test perspective. Repurposing test
techniques to actual traffic could be useful. Hence the question around
what exact telemetry is useful to apps making socket write() and read()
calls.

Bob

On Fri, Jul 2, 2021 at 10:07 AM Dave Taht  wrote:

> In terms of trying to find "Quality" I have tried to encourage folk to
> both read "zen and the art of motorcycle maintenance"[0], and Deming's
> work on "total quality management".
>
> My own slice at this network, computer and lifestyle "issue" is aiming
> for "imperceptible latency" in all things. [1]. There's a lot of
> fallout from that in terms of not just addressing queuing delay, but
> caching, prefetching, and learning more about what a user really needs
> (as opposed to wants) to know via intelligent agents.
>
> [0] If you want to get depressed, read Pirsig's successor to "zen...",
> lila, which is in part about what happens when an engineer hits an
> insoluble problem.
> [1] https://www.internetsociety.org/events/latency2013/
>
>
>
> On Thu, Jul 1, 2021 at 6:16 PM David P. Reed  wrote:
> >
> > Well, nice that the folks doing the conference  are willing to consider
> that quality of user experience has little to do with signalling rate at
> the physical layer or throughput of FTP transfers.
> >
> >
> >
> > But honestly, the fact that they call the problem "network quality"
> suggests that they REALLY, REALLY don't understand the Internet isn't the
> hardware or the routers or even the routing algorithms *to its users*.
> >
> >
> >
> > By ignoring the diversity of applications now and in the future, and the
> fact that we DON'T KNOW what will be coming up, this conference will likely
> fall into the usual trap that net-heads fall into - optimizing for some
> imaginary reality that doesn't exist, and in fact will probably never be
> what users actually will do given the chance.
> >
> >
> >
> > I saw this issue in 1976 in the group developing the original Internet
> protocols - a desire to put *into the network* special tricks to optimize
> ASR33 logins to remote computers from terminal concentrators (aka remote
> login), bulk file transfers between file systems on different time-sharing
> systems, and "sessions" (virtual circuits) that required logins. And then
> trying to exploit underlying "multicast" by building it into the IP layer,
> because someone thought that TV broadcast would be the dominant application.
> >
> >
> >
> > Frankly, to think of "quality" as something that can be "provided" by
> "the network" misses the entire point of "end-to-end argument in system
> design". Quality is not a property defined or created by The Network. If
> you want to talk about Quality, you need to talk about users - all the
> users at all times, now and into the future, and that's something you can't
> do if you don't bother to include current and future users talking about
> what they might expect to experience that they don't experience.
> >
> >
> >
> > There was much fighting back in 1976 that basically involved "network
> experts" saying that the network was the place to "solve" such issues as
> quality, so applications could avoid having to solve such issues.
> >
> >
> >
> > What some of us managed to do was to argue that you can't "solve" such
> issues. All you can do is provide a framework that enables different uses
> to *cooperate* in some way.
> >
> >
> >
> > Which is why the Internet drops packets rather than queueing them, and
> why diffserv cannot work.
> >
> > (I know the latter is conftroversial, but at the moment, ALL of diffserv
> attempts to talk about end-to-end applicaiton specific metrics, but never,
> ever explains what the diffserv control points actually do w.r.t. what the
> IP layer can actually control. So it is meaningless - another violation of
> the so-called end-to-end principle).
> >
> >
> >
> > Networks are about getting packets from here to there, multiplexing the
> underlying resources. That's it. Quality is a whole different thing.
> Quality can be improved by end-to-end approaches, if the underlying network
> provides some kind of thing that actually creates a way for end-to-end
> applications to affect queueing and routing decisions, and more importantly
> getting "telemetry" from the network regarding what is actually going on
> with the other end-to-end users sharing the infrastructure.
> >
> >
> >
> > This conference won't talk about it this way. So don't waste your time.
> >
> >
> >
> >
> >
> >
> >
> > On Wednesday, June 30, 2021 8:12pm, "Dave Taht" 
> said:
> >
> > > The program committee members are *amazing*. Perhaps, finally, we can
> > > move the bar for the internet's quality metrics past endless, blind
> > > repetitions of speedtest.
> > >
> > > For complete 

Re: [Bloat] Bechtolschiem

[Jonathan Morton]

> On 2 Jul, 2021, at 7:59 pm, Stephen Hemminger  
> wrote:
> 
> In real world tests, TCP Cubic will consume any buffer it sees at a
> congested link. Maybe that is what they mean by capture effect.

First, I'll note that what they call "small buffer" corresponds to about a 
tenth of a millisecond at the port's link rate.  This would be ludicrously 
small at Internet scale, but is actually reasonable for datacentre conditions 
where RTTs are often in the microseconds.

Assuming the effect as described is real, it ultimately stems from a burst of 
traffic from a particular flow arriving at a queue that is *already* full.  
Such bursts are expected from ack-clocked flows coming out of 
application-limited mode (ie. on completion of a disk read), in slow-start, or 
recovering from earlier losses.  It is also possible for a heavily coalesced 
ack to abruptly open the receive and congestion windows and trigger a send 
burst.  These bursts occur much less in paced flows, because the object of 
pacing is to avoid bursts.

The queue is full because tail drop upon queue overflow is the only congestion 
signal provided by the switch, and ack-clocked capacity-seeking transports 
naturally keep the queue as full as they can - especially under high 
statistical multiplexing conditions where a single multiplicative decrease 
event does not greatly reduce the total traffic demand. CUBIC arguably spends 
more time with the queue very close to full than Reno does, due to the plateau 
designed into it, but at these very short RTTs I would not be surprised if 
CUBIC is equivalent to Reno in practice.

The solution is to keep some normally-unused space in the queue for bursts of 
traffic to use occasionally.  This is most naturally done using ECN applied by 
some AQM algorithm, or the AQM can pre-emptively and selectively drop packets 
in Not-ECT flows.  And because the AQM is more likely to mark or drop packets 
from flows that occupy more link time or queue capacity, it has a natural 
equalising effect between flows.

Applying ECN requires some Layer 3 awareness in the switch, which might not be 
practical.  A simple alternative it to drop packets instead.  Single packet 
losses are easily recovered from by retransmission after approximately one RTT. 
 There are also emerging techniques for applying congestion signals at Layer 2, 
which can be converted into ECN signals at some convenient point downstream.

However it is achieved, the point is that keeping the *standing* queue down to 
some fraction of the total queue depth reserves space for accommodating those 
bursts which are expected occasionally in normal traffic.  Because those bursts 
are not lost, the flows experiencing them are not disadvantaged and the 
so-called "capture effect" will not occur.

 - Jonathan Morton
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] Bechtolschiem

[Matt Mathis via Bloat]

The argument is absolutely correct for Reno, CUBIC and all
other self-clocked protocols.  One of the core assumptions in Jacobson88,
was that the clock for the entire system comes from packets draining
through the bottleneck queue.  In this world, the clock is intrinsically
brittle if the buffers are too small.  The drain time needs to be a
substantial fraction of the RTT.

However, we have reached the point where we need to discard that
requirement.  One of the side points of BBR is that in many environments it
is cheaper to burn serving CPU to pace into short queue networks than it is
to "right size" the network queues.

The fundamental problem with the old way is that in some contexts the
buffer memory has to beat Moore's law, because to maintain constant drain
time the memory size and BW both have to scale with the link (laser) BW.

See the slides I gave at the Stanford Buffer Sizing workshop december
2019: Buffer
Sizing: Position Paper



Note that we are talking about DC and Internet core.  At the edge, BW is
low enough where memory is relatively cheap.   In some sense BB came about
because memory is too cheap in these environments.

Thanks,
--MM--
The best way to predict the future is to create it.  - Alan Kay

We must not tolerate intolerance;
   however our response must be carefully measured:
too strong would be hypocritical and risks spiraling out of
control;
too weak risks being mistaken for tacit approval.


On Fri, Jul 2, 2021 at 9:59 AM Stephen Hemminger 
wrote:

> On Fri, 2 Jul 2021 09:42:24 -0700
> Dave Taht  wrote:
>
> > "Debunking Bechtolsheim credibly would get a lot of attention to the
> > bufferbloat cause, I suspect." - dpreed
> >
> > "Why Big Data Needs Big Buffer Switches" -
> >
> http://www.arista.com/assets/data/pdf/Whitepapers/BigDataBigBuffers-WP.pdf
> >
>
> Also, a lot depends on the TCP congestion control algorithm being used.
> They are using NewReno which only researchers use in real life.
>
> Even TCP Cubic has gone through several revisions. In my experience, the
> NS-2 models don't correlate well to real world behavior.
>
> In real world tests, TCP Cubic will consume any buffer it sees at a
> congested link. Maybe that is what they mean by capture effect.
>
> There is also a weird oscillation effect with multiple streams, where one
> flow will take the buffer, then see a packet loss and back off, the
> other flow will take over the buffer until it sees loss.
>
> ___
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] Bechtolschiem

[Dave Collier-Brown]

It's written to look like an academic paper, but it's pure marketing.  
"Memory is cheap, we used a lot, so let's select some evidence that 
argues this is a good thing."


As always with the coin-operated, the way to get them to change is to 
offer additional information which


 * captures their attention,

and, more importantly

 * offers them a cheap way to /make more money/.

For example, a software change that make their big buffers not fill up 
with elephants...


--dave

On 2021-07-02 12:59 p.m., Stephen Hemminger wrote:

On Fri, 2 Jul 2021 09:42:24 -0700
Dave Taht  wrote:


"Debunking Bechtolsheim credibly would get a lot of attention to the
bufferbloat cause, I suspect." - dpreed

"Why Big Data Needs Big Buffer Switches" -
http://www.arista.com/assets/data/pdf/Whitepapers/BigDataBigBuffers-WP.pdf


Also, a lot depends on the TCP congestion control algorithm being used.
They are using NewReno which only researchers use in real life.

Even TCP Cubic has gone through several revisions. In my experience, the
NS-2 models don't correlate well to real world behavior.

In real world tests, TCP Cubic will consume any buffer it sees at a
congested link. Maybe that is what they mean by capture effect.

There is also a weird oscillation effect with multiple streams, where one
flow will take the buffer, then see a packet loss and back off, the
other flow will take over the buffer until it sees loss.

___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat


--
David Collier-Brown, | Always do right. This will gratify
System Programmer and Author | some people and astonish the rest
dave.collier-br...@indexexchange.com |  -- Mark Twain

___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

Re: [Bloat] [Cerowrt-devel] Due Aug 2: Internet Quality workshop CFP for the internet architecture board

[Dave Taht]

In terms of trying to find "Quality" I have tried to encourage folk to
both read "zen and the art of motorcycle maintenance"[0], and Deming's
work on "total quality management".

My own slice at this network, computer and lifestyle "issue" is aiming
for "imperceptible latency" in all things. [1]. There's a lot of
fallout from that in terms of not just addressing queuing delay, but
caching, prefetching, and learning more about what a user really needs
(as opposed to wants) to know via intelligent agents.

[0] If you want to get depressed, read Pirsig's successor to "zen...",
lila, which is in part about what happens when an engineer hits an
insoluble problem.
[1] https://www.internetsociety.org/events/latency2013/



On Thu, Jul 1, 2021 at 6:16 PM David P. Reed  wrote:
>
> Well, nice that the folks doing the conference  are willing to consider that 
> quality of user experience has little to do with signalling rate at the 
> physical layer or throughput of FTP transfers.
>
>
>
> But honestly, the fact that they call the problem "network quality" suggests 
> that they REALLY, REALLY don't understand the Internet isn't the hardware or 
> the routers or even the routing algorithms *to its users*.
>
>
>
> By ignoring the diversity of applications now and in the future, and the fact 
> that we DON'T KNOW what will be coming up, this conference will likely fall 
> into the usual trap that net-heads fall into - optimizing for some imaginary 
> reality that doesn't exist, and in fact will probably never be what users 
> actually will do given the chance.
>
>
>
> I saw this issue in 1976 in the group developing the original Internet 
> protocols - a desire to put *into the network* special tricks to optimize 
> ASR33 logins to remote computers from terminal concentrators (aka remote 
> login), bulk file transfers between file systems on different time-sharing 
> systems, and "sessions" (virtual circuits) that required logins. And then 
> trying to exploit underlying "multicast" by building it into the IP layer, 
> because someone thought that TV broadcast would be the dominant application.
>
>
>
> Frankly, to think of "quality" as something that can be "provided" by "the 
> network" misses the entire point of "end-to-end argument in system design". 
> Quality is not a property defined or created by The Network. If you want to 
> talk about Quality, you need to talk about users - all the users at all 
> times, now and into the future, and that's something you can't do if you 
> don't bother to include current and future users talking about what they 
> might expect to experience that they don't experience.
>
>
>
> There was much fighting back in 1976 that basically involved "network 
> experts" saying that the network was the place to "solve" such issues as 
> quality, so applications could avoid having to solve such issues.
>
>
>
> What some of us managed to do was to argue that you can't "solve" such 
> issues. All you can do is provide a framework that enables different uses to 
> *cooperate* in some way.
>
>
>
> Which is why the Internet drops packets rather than queueing them, and why 
> diffserv cannot work.
>
> (I know the latter is conftroversial, but at the moment, ALL of diffserv 
> attempts to talk about end-to-end applicaiton specific metrics, but never, 
> ever explains what the diffserv control points actually do w.r.t. what the IP 
> layer can actually control. So it is meaningless - another violation of the 
> so-called end-to-end principle).
>
>
>
> Networks are about getting packets from here to there, multiplexing the 
> underlying resources. That's it. Quality is a whole different thing. Quality 
> can be improved by end-to-end approaches, if the underlying network provides 
> some kind of thing that actually creates a way for end-to-end applications to 
> affect queueing and routing decisions, and more importantly getting 
> "telemetry" from the network regarding what is actually going on with the 
> other end-to-end users sharing the infrastructure.
>
>
>
> This conference won't talk about it this way. So don't waste your time.
>
>
>
>
>
>
>
> On Wednesday, June 30, 2021 8:12pm, "Dave Taht"  said:
>
> > The program committee members are *amazing*. Perhaps, finally, we can
> > move the bar for the internet's quality metrics past endless, blind
> > repetitions of speedtest.
> >
> > For complete details, please see:
> > https://www.iab.org/activities/workshops/network-quality/
> >
> > Submissions Due: Monday 2nd August 2021, midnight AOE (Anywhere On Earth)
> > Invitations Issued by: Monday 16th August 2021
> >
> > Workshop Date: This will be a virtual workshop, spread over three days:
> >
> > 1400-1800 UTC Tue 14th September 2021
> > 1400-1800 UTC Wed 15th September 2021
> > 1400-1800 UTC Thu 16th September 2021
> >
> > Workshop co-chairs: Wes Hardaker, Evgeny Khorov, Omer Shapira
> >
> > The Program Committee members:
> >
> > Jari Arkko, Olivier Bonaventure, Vint Cerf, Stuart Cheshire, Sam
> > 

Re: [Bloat] Bechtolschiem

[Stephen Hemminger]

On Fri, 2 Jul 2021 09:42:24 -0700
Dave Taht  wrote:

> "Debunking Bechtolsheim credibly would get a lot of attention to the
> bufferbloat cause, I suspect." - dpreed
> 
> "Why Big Data Needs Big Buffer Switches" -
> http://www.arista.com/assets/data/pdf/Whitepapers/BigDataBigBuffers-WP.pdf
> 

Also, a lot depends on the TCP congestion control algorithm being used.
They are using NewReno which only researchers use in real life.

Even TCP Cubic has gone through several revisions. In my experience, the
NS-2 models don't correlate well to real world behavior.

In real world tests, TCP Cubic will consume any buffer it sees at a
congested link. Maybe that is what they mean by capture effect.

There is also a weird oscillation effect with multiple streams, where one
flow will take the buffer, then see a packet loss and back off, the
other flow will take over the buffer until it sees loss.

___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

[Bloat] Bechtolschiem

[Dave Taht]

"Debunking Bechtolsheim credibly would get a lot of attention to the
bufferbloat cause, I suspect." - dpreed

"Why Big Data Needs Big Buffer Switches" -
http://www.arista.com/assets/data/pdf/Whitepapers/BigDataBigBuffers-WP.pdf

..

i think i've just gained access to a few networks with arista gear in
the bottleneck path.

On Mon, Jun 6, 2016 at 7:52 PM  wrote:
>
> So did anyone write a response debunking their paper?   Their NS-2 simulation 
> is most likely the erroneous part of their analysis - the white paper would 
> not pass a review by qualified referees because there is no way to check 
> their results and some of what they say beggars belief.
>
>
>
> Bechtolsheim is one of those guys who can write any damn thing and it becomes 
> "truth" - mostly because he co-founded Sun. But that doesn't mean that he 
> can't make huge errors - any of us can.
>
>
>
> The so-called TCP/IP Bandwidth Capture effect that he refers to doesn't sound 
> like any capture effect I've ever heard of.  There is an "Ethernet Capture 
> Effect" (which is cited), which is due to properties of CSMA/CD binary 
> exponential backoff, not anything to do with TCP's flow/congestion control.  
> So it has that "truthiness" that makes glib people sound like they know what 
> they are talking about, but I'd like to see a reference that says this is a 
> property of TCP!
>
>
>
> What's interesting is that the reference to the Ethernet Capture Effect in 
> that white paper proposes a solution that involves changing the backoff 
> algorithm slightly at the Ethernet level - NOT increasing buffer size!
>
>
>
> Another thing that would probably improve matters a great deal would be to 
> drop/ECN-mark packets when a contended output port on an Arista switch 
> develops a backlog.  This will throttle TCP sources sharing the path.
>
>
>
> The comments in the white paper that say that ACK contention in TCP in the 
> reverse direction are the problem that causes the "so-called TCP/IP Bandwidth 
> Capture effect" that is invented by the authors appears to be hogwash of the 
> first order.
>
>
>
> Debunking Bechtolsheim credibly would get a lot of attention to the 
> bufferbloat cause, I suspect.
>
>
>
>
>
> On Monday, June 6, 2016 5:16pm, "Ketan Kulkarni"  said:
>
> some time back they had this whitepaper -
> "Why Big Data Needs Big Buffer Switches"
> http://www.arista.com/assets/data/pdf/Whitepapers/BigDataBigBuffers-WP.pdf
> the type of apps they talk about is big data, hadoop etc
>
> On Mon, Jun 6, 2016 at 11:37 AM, Mikael Abrahamsson  wrote:
>>
>> On Mon, 6 Jun 2016, Jonathan Morton wrote:
>>
>>> At 100ms buffering, their 10Gbps switch is effectively turning any DC it’s 
>>> installed in into a transcontinental Internet path, as far as peak latency 
>>> is concerned.  Just because RAM is cheap these days…
>>
>> Nono, nononononono. I can tell you they're spending serious money on 
>> inserting this kind of buffering memory into these kinds of devices. Buying 
>> these devices without deep buffers is a lot lower cost.
>>
>> These types of switch chips either have on-die memory (usually 16MB or 
>> less), or they have very expensive (a direct cost of lowered port density) 
>> off-chip buffering memory.
>>
>> Typically you do this:
>>
>> ports ---|---
>> ports ---|  |
>> ports ---| chip |
>> ports ---|---
>>
>> Or you do this
>>
>> ports ---|--|---buffer
>> ports ---| chip |---TCAM
>>  
>>
>> or if you do a multi-linecard-device
>>
>> ports ---|--|---buffer
>>  | chip |---TCAM
>>  
>> |
>> switch fabric
>>
>> (or any variant of them)
>>
>> So basically if you want to buffer and if you want large L2-L4 lookup 
>> tables, you have to sacrifice ports. Sacrifice lots of ports.
>>
>> So never say these kinds of devices add buffering because RAM is cheap. This 
>> is most definitely not why they're doing it. Buffer memory for them is 
>> EXTREMELY EXPENSIVE.
>>
>> --
>> Mikael Abrahamssonemail: swm...@swm.pp.se
>> ___
>> Cerowrt-devel mailing list
>> cerowrt-de...@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>>
> ___
> Cerowrt-devel mailing list
> cerowrt-de...@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel



-- 
Latest Podcast:
https://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/

Dave Täht CTO, TekLibre, LLC
___
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat