Re: [Cerowrt-devel] [Bloat] DC behaviors today

[Pedro Tumusok]

For in home or even SMB, I doubt that 10G to the user PC is the main use
case.
Its having the uplink capable of support of more than1G, that 1G does not
necessarily need to be generated by only one host on the LAN.



Pedro

On Mon, Dec 4, 2017 at 11:27 AM, Joel Wirāmu Pauling 
wrote:

> How to deliver a switch, when the wiring and port standard isn't
> actually workable?
>
> 10GBase-T is out of Voltage Spec with SFP+ ; you can get copper SFP+
> but they are out of spec... 10GbaseT doesn't really work over Cat5e
> more than a couple of meters (if you are lucky) and even Cat6 is only
> rated at 30M... there is a reason no-one is producing Home Copper
> switches and it's not just the NIC Silicon cost (that was a factor
> until Recently obviously, but only part of the equation).
>
> On the flip side:
> Right now I am typing this via a 40gbit network, comprised of the
> cheap and readily available Tb3 port - it's daisy chained and limited
> to 6 ports, but right now it's easily the cheapest and most effective
> port. Pitty that the fabled optical tb3 cables are damn expensive...
> so you're limited to daisy-chains of 2m. They seem to have screwed the
> pooch on the USB-C network standard quite badly - which looked so
> promising, so for the moment Tb3 it is for me at least.
>
> On 4 December 2017 at 23:18, Mikael Abrahamsson  wrote:
> > On Mon, 4 Dec 2017, Joel Wirāmu Pauling wrote:
> >
> >> I'm not going to pretend that 1Gig isn't enough for most people. But I
> >> refuse to believe it's the networks equivalent of a 10A power (20A
> >> depending on where you live in the world) AC residential phase
> >> distribution circuit.
> >
> >
> > That's a good analogy. I actually believe it is, at least for the near
> 5-10
> > years.
> >
> >> This isn't a question about what people need, it's more about what the
> >> market can deliver. 10GPON (GPON-X) and others now make it a viable
> >> service that can and is being deployed in residential and commercial
> >> access networks.
> >
> >
> > Well, you're sharing that bw with everybody else on that splitter.
> Sounds to
> > me that the service being delivered over that would instead be in the 2-3
> > gigabit/s range for the individual subscriber (this is what I typically
> see
> > on equivalent shared mediums, that the top speed individual subscriptions
> > are will be in the 20-40% of max theoretical speed the entire solution
> can
> > deliver).
> >
> >> The problem is now that Retail Servicer Provider X can deliver a post
> >> Gigabit service... what is capable of taking it off the ONU/CMNT point
> in
> >> the home? As usual it's a follow the money question, once RSP's can
> deliver
> >> Gbit+ they will need an ecosystem in the home to feed into it, and
> right now
> >> there isn't a good technology platform that supports it;
> 10GBase-X/10GBaseT
> >> is a non-starter due to the variability in home wiring - arguably the 7
> year
> >> leap from 100-1000mbit was easy It's mean a gap of 12 years and
> counting for
> >> the same.. it's not just the NIC's and CPU's in the gateways it's the
> >> connector and in-home wiring problems as well.
> >
> >
> > As soon as one goes above 1GE, prices increases A LOT on everything
> > involved. I doubt we'll see any 2.5G or higher speed equipment in wide
> use
> > in home/SME in the next 5 years.
> >
> >> Blatant Plug - request :
> >> I'm interested to hear opinions on this as I have a talk on this very
> >> topic 'The long and Winding Road to 10Gbit+ in the home'
> >> https://linux.conf.au/ at Linuxconf in January. In particular if you
> >> have any home network gore/horror stories and photos you would be
> >> happy for me to include in my talk, please include.
> >
> >
> > I am still waiting for a decently priced 10GE switch. I can get 1GE
> 24port
> > managed ones, fanless, for 100-200USD. As soon as I go 10GE, price jumps
> up
> > a lot, and I get fans. The NICs aren't widely available, even though
> they're
> > not the biggest problem. My in-house cabling can do 10GE, but I guess
> I'm an
> > outlier.
> >
> >
> > --
> > Mikael Abrahamssonemail: swm...@swm.pp.se
> ___
> Bloat mailing list
> bl...@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>



-- 
Best regards / Mvh
Jan Pedro Tumusok
___
Cerowrt-devel mailing list
Cerowrt-devel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel

Re: [Cerowrt-devel] [Bloat] DC behaviors today

[Joel Wirāmu Pauling]

Bingo; that's definitely step one - gateways capable of 10gbit
becoming the norm.

On 4 December 2017 at 23:43, Pedro Tumusok  wrote:
> For in home or even SMB, I doubt that 10G to the user PC is the main use
> case.
> Its having the uplink capable of support of more than1G, that 1G does not
> necessarily need to be generated by only one host on the LAN.
>
>
>
> Pedro
>
> On Mon, Dec 4, 2017 at 11:27 AM, Joel Wirāmu Pauling 
> wrote:
>>
>> How to deliver a switch, when the wiring and port standard isn't
>> actually workable?
>>
>> 10GBase-T is out of Voltage Spec with SFP+ ; you can get copper SFP+
>> but they are out of spec... 10GbaseT doesn't really work over Cat5e
>> more than a couple of meters (if you are lucky) and even Cat6 is only
>> rated at 30M... there is a reason no-one is producing Home Copper
>> switches and it's not just the NIC Silicon cost (that was a factor
>> until Recently obviously, but only part of the equation).
>>
>> On the flip side:
>> Right now I am typing this via a 40gbit network, comprised of the
>> cheap and readily available Tb3 port - it's daisy chained and limited
>> to 6 ports, but right now it's easily the cheapest and most effective
>> port. Pitty that the fabled optical tb3 cables are damn expensive...
>> so you're limited to daisy-chains of 2m. They seem to have screwed the
>> pooch on the USB-C network standard quite badly - which looked so
>> promising, so for the moment Tb3 it is for me at least.
>>
>> On 4 December 2017 at 23:18, Mikael Abrahamsson  wrote:
>> > On Mon, 4 Dec 2017, Joel Wirāmu Pauling wrote:
>> >
>> >> I'm not going to pretend that 1Gig isn't enough for most people. But I
>> >> refuse to believe it's the networks equivalent of a 10A power (20A
>> >> depending on where you live in the world) AC residential phase
>> >> distribution circuit.
>> >
>> >
>> > That's a good analogy. I actually believe it is, at least for the near
>> > 5-10
>> > years.
>> >
>> >> This isn't a question about what people need, it's more about what the
>> >> market can deliver. 10GPON (GPON-X) and others now make it a viable
>> >> service that can and is being deployed in residential and commercial
>> >> access networks.
>> >
>> >
>> > Well, you're sharing that bw with everybody else on that splitter.
>> > Sounds to
>> > me that the service being delivered over that would instead be in the
>> > 2-3
>> > gigabit/s range for the individual subscriber (this is what I typically
>> > see
>> > on equivalent shared mediums, that the top speed individual
>> > subscriptions
>> > are will be in the 20-40% of max theoretical speed the entire solution
>> > can
>> > deliver).
>> >
>> >> The problem is now that Retail Servicer Provider X can deliver a post
>> >> Gigabit service... what is capable of taking it off the ONU/CMNT point
>> >> in
>> >> the home? As usual it's a follow the money question, once RSP's can
>> >> deliver
>> >> Gbit+ they will need an ecosystem in the home to feed into it, and
>> >> right now
>> >> there isn't a good technology platform that supports it;
>> >> 10GBase-X/10GBaseT
>> >> is a non-starter due to the variability in home wiring - arguably the 7
>> >> year
>> >> leap from 100-1000mbit was easy It's mean a gap of 12 years and
>> >> counting for
>> >> the same.. it's not just the NIC's and CPU's in the gateways it's the
>> >> connector and in-home wiring problems as well.
>> >
>> >
>> > As soon as one goes above 1GE, prices increases A LOT on everything
>> > involved. I doubt we'll see any 2.5G or higher speed equipment in wide
>> > use
>> > in home/SME in the next 5 years.
>> >
>> >> Blatant Plug - request :
>> >> I'm interested to hear opinions on this as I have a talk on this very
>> >> topic 'The long and Winding Road to 10Gbit+ in the home'
>> >> https://linux.conf.au/ at Linuxconf in January. In particular if you
>> >> have any home network gore/horror stories and photos you would be
>> >> happy for me to include in my talk, please include.
>> >
>> >
>> > I am still waiting for a decently priced 10GE switch. I can get 1GE
>> > 24port
>> > managed ones, fanless, for 100-200USD. As soon as I go 10GE, price jumps
>> > up
>> > a lot, and I get fans. The NICs aren't widely available, even though
>> > they're
>> > not the biggest problem. My in-house cabling can do 10GE, but I guess
>> > I'm an
>> > outlier.
>> >
>> >
>> > --
>> > Mikael Abrahamssonemail: swm...@swm.pp.se
>> ___
>> Bloat mailing list
>> bl...@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/bloat
>
>
>
>
> --
> Best regards / Mvh
> Jan Pedro Tumusok
>
>
> ___
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>
___
Cerowrt-devel mailing list
Cerowrt-devel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel

Re: [Cerowrt-devel] [Bloat] DC behaviors today

[Pedro Tumusok]

Looking at chipsets coming/just arrived from the chipset vendors, I think
we will see CPE with 10G SFP+ and 802.11ax Q3/Q4 this year.
Price is of course a bit steeper than the 15USD USB DSL modem :P, but
probably fits nicely for the SMB segment.

Pedro

On Mon, Dec 4, 2017 at 11:47 AM, Joel Wirāmu Pauling 
wrote:

> Bingo; that's definitely step one - gateways capable of 10gbit
> becoming the norm.
>
> On 4 December 2017 at 23:43, Pedro Tumusok 
> wrote:
> > For in home or even SMB, I doubt that 10G to the user PC is the main use
> > case.
> > Its having the uplink capable of support of more than1G, that 1G does not
> > necessarily need to be generated by only one host on the LAN.
> >
> >
> >
> > Pedro
> >
> > On Mon, Dec 4, 2017 at 11:27 AM, Joel Wirāmu Pauling 
> > wrote:
> >>
> >> How to deliver a switch, when the wiring and port standard isn't
> >> actually workable?
> >>
> >> 10GBase-T is out of Voltage Spec with SFP+ ; you can get copper SFP+
> >> but they are out of spec... 10GbaseT doesn't really work over Cat5e
> >> more than a couple of meters (if you are lucky) and even Cat6 is only
> >> rated at 30M... there is a reason no-one is producing Home Copper
> >> switches and it's not just the NIC Silicon cost (that was a factor
> >> until Recently obviously, but only part of the equation).
> >>
> >> On the flip side:
> >> Right now I am typing this via a 40gbit network, comprised of the
> >> cheap and readily available Tb3 port - it's daisy chained and limited
> >> to 6 ports, but right now it's easily the cheapest and most effective
> >> port. Pitty that the fabled optical tb3 cables are damn expensive...
> >> so you're limited to daisy-chains of 2m. They seem to have screwed the
> >> pooch on the USB-C network standard quite badly - which looked so
> >> promising, so for the moment Tb3 it is for me at least.
> >>
> >> On 4 December 2017 at 23:18, Mikael Abrahamsson 
> wrote:
> >> > On Mon, 4 Dec 2017, Joel Wirāmu Pauling wrote:
> >> >
> >> >> I'm not going to pretend that 1Gig isn't enough for most people. But
> I
> >> >> refuse to believe it's the networks equivalent of a 10A power (20A
> >> >> depending on where you live in the world) AC residential phase
> >> >> distribution circuit.
> >> >
> >> >
> >> > That's a good analogy. I actually believe it is, at least for the near
> >> > 5-10
> >> > years.
> >> >
> >> >> This isn't a question about what people need, it's more about what
> the
> >> >> market can deliver. 10GPON (GPON-X) and others now make it a viable
> >> >> service that can and is being deployed in residential and commercial
> >> >> access networks.
> >> >
> >> >
> >> > Well, you're sharing that bw with everybody else on that splitter.
> >> > Sounds to
> >> > me that the service being delivered over that would instead be in the
> >> > 2-3
> >> > gigabit/s range for the individual subscriber (this is what I
> typically
> >> > see
> >> > on equivalent shared mediums, that the top speed individual
> >> > subscriptions
> >> > are will be in the 20-40% of max theoretical speed the entire solution
> >> > can
> >> > deliver).
> >> >
> >> >> The problem is now that Retail Servicer Provider X can deliver a post
> >> >> Gigabit service... what is capable of taking it off the ONU/CMNT
> point
> >> >> in
> >> >> the home? As usual it's a follow the money question, once RSP's can
> >> >> deliver
> >> >> Gbit+ they will need an ecosystem in the home to feed into it, and
> >> >> right now
> >> >> there isn't a good technology platform that supports it;
> >> >> 10GBase-X/10GBaseT
> >> >> is a non-starter due to the variability in home wiring - arguably
> the 7
> >> >> year
> >> >> leap from 100-1000mbit was easy It's mean a gap of 12 years and
> >> >> counting for
> >> >> the same.. it's not just the NIC's and CPU's in the gateways it's the
> >> >> connector and in-home wiring problems as well.
> >> >
> >> >
> >> > As soon as one goes above 1GE, prices increases A LOT on everything
> >> > involved. I doubt we'll see any 2.5G or higher speed equipment in wide
> >> > use
> >> > in home/SME in the next 5 years.
> >> >
> >> >> Blatant Plug - request :
> >> >> I'm interested to hear opinions on this as I have a talk on this very
> >> >> topic 'The long and Winding Road to 10Gbit+ in the home'
> >> >> https://linux.conf.au/ at Linuxconf in January. In particular if you
> >> >> have any home network gore/horror stories and photos you would be
> >> >> happy for me to include in my talk, please include.
> >> >
> >> >
> >> > I am still waiting for a decently priced 10GE switch. I can get 1GE
> >> > 24port
> >> > managed ones, fanless, for 100-200USD. As soon as I go 10GE, price
> jumps
> >> > up
> >> > a lot, and I get fans. The NICs aren't widely available, even though
> >> > they're
> >> > not the biggest problem. My in-house cabling can do 10GE, but I guess
> >> > I'm an
> >> > outlier.
> >> >
> >> >
> >> > --
> >> > Mikael Abrahamssonemail: swm...@swm.pp.se
> >> 

Re: [Cerowrt-devel] [Bloat] DC behaviors today

[Joel Wirāmu Pauling]

Oh we have these in the Enterprise segment already. The main use case
is VNF on edge device for SDN applications right now. But even so the
range of vendors/devices is pretty limited.

On 4 December 2017 at 23:57, Pedro Tumusok  wrote:
> Looking at chipsets coming/just arrived from the chipset vendors, I think we
> will see CPE with 10G SFP+ and 802.11ax Q3/Q4 this year.
> Price is of course a bit steeper than the 15USD USB DSL modem :P, but
> probably fits nicely for the SMB segment.
>
> Pedro
>
> On Mon, Dec 4, 2017 at 11:47 AM, Joel Wirāmu Pauling 
> wrote:
>>
>> Bingo; that's definitely step one - gateways capable of 10gbit
>> becoming the norm.
>>
>> On 4 December 2017 at 23:43, Pedro Tumusok 
>> wrote:
>> > For in home or even SMB, I doubt that 10G to the user PC is the main use
>> > case.
>> > Its having the uplink capable of support of more than1G, that 1G does
>> > not
>> > necessarily need to be generated by only one host on the LAN.
>> >
>> >
>> >
>> > Pedro
>> >
>> > On Mon, Dec 4, 2017 at 11:27 AM, Joel Wirāmu Pauling 
>> > wrote:
>> >>
>> >> How to deliver a switch, when the wiring and port standard isn't
>> >> actually workable?
>> >>
>> >> 10GBase-T is out of Voltage Spec with SFP+ ; you can get copper SFP+
>> >> but they are out of spec... 10GbaseT doesn't really work over Cat5e
>> >> more than a couple of meters (if you are lucky) and even Cat6 is only
>> >> rated at 30M... there is a reason no-one is producing Home Copper
>> >> switches and it's not just the NIC Silicon cost (that was a factor
>> >> until Recently obviously, but only part of the equation).
>> >>
>> >> On the flip side:
>> >> Right now I am typing this via a 40gbit network, comprised of the
>> >> cheap and readily available Tb3 port - it's daisy chained and limited
>> >> to 6 ports, but right now it's easily the cheapest and most effective
>> >> port. Pitty that the fabled optical tb3 cables are damn expensive...
>> >> so you're limited to daisy-chains of 2m. They seem to have screwed the
>> >> pooch on the USB-C network standard quite badly - which looked so
>> >> promising, so for the moment Tb3 it is for me at least.
>> >>
>> >> On 4 December 2017 at 23:18, Mikael Abrahamsson 
>> >> wrote:
>> >> > On Mon, 4 Dec 2017, Joel Wirāmu Pauling wrote:
>> >> >
>> >> >> I'm not going to pretend that 1Gig isn't enough for most people. But
>> >> >> I
>> >> >> refuse to believe it's the networks equivalent of a 10A power (20A
>> >> >> depending on where you live in the world) AC residential phase
>> >> >> distribution circuit.
>> >> >
>> >> >
>> >> > That's a good analogy. I actually believe it is, at least for the
>> >> > near
>> >> > 5-10
>> >> > years.
>> >> >
>> >> >> This isn't a question about what people need, it's more about what
>> >> >> the
>> >> >> market can deliver. 10GPON (GPON-X) and others now make it a viable
>> >> >> service that can and is being deployed in residential and commercial
>> >> >> access networks.
>> >> >
>> >> >
>> >> > Well, you're sharing that bw with everybody else on that splitter.
>> >> > Sounds to
>> >> > me that the service being delivered over that would instead be in the
>> >> > 2-3
>> >> > gigabit/s range for the individual subscriber (this is what I
>> >> > typically
>> >> > see
>> >> > on equivalent shared mediums, that the top speed individual
>> >> > subscriptions
>> >> > are will be in the 20-40% of max theoretical speed the entire
>> >> > solution
>> >> > can
>> >> > deliver).
>> >> >
>> >> >> The problem is now that Retail Servicer Provider X can deliver a
>> >> >> post
>> >> >> Gigabit service... what is capable of taking it off the ONU/CMNT
>> >> >> point
>> >> >> in
>> >> >> the home? As usual it's a follow the money question, once RSP's can
>> >> >> deliver
>> >> >> Gbit+ they will need an ecosystem in the home to feed into it, and
>> >> >> right now
>> >> >> there isn't a good technology platform that supports it;
>> >> >> 10GBase-X/10GBaseT
>> >> >> is a non-starter due to the variability in home wiring - arguably
>> >> >> the 7
>> >> >> year
>> >> >> leap from 100-1000mbit was easy It's mean a gap of 12 years and
>> >> >> counting for
>> >> >> the same.. it's not just the NIC's and CPU's in the gateways it's
>> >> >> the
>> >> >> connector and in-home wiring problems as well.
>> >> >
>> >> >
>> >> > As soon as one goes above 1GE, prices increases A LOT on everything
>> >> > involved. I doubt we'll see any 2.5G or higher speed equipment in
>> >> > wide
>> >> > use
>> >> > in home/SME in the next 5 years.
>> >> >
>> >> >> Blatant Plug - request :
>> >> >> I'm interested to hear opinions on this as I have a talk on this
>> >> >> very
>> >> >> topic 'The long and Winding Road to 10Gbit+ in the home'
>> >> >> https://linux.conf.au/ at Linuxconf in January. In particular if you
>> >> >> have any home network gore/horror stories and photos you would be
>> >> >> happy for me to include in my talk, please include.
>> >> >
>> >> >
>> >> > I am still waiting for a decently priced 10GE sw

Re: [Cerowrt-devel] [Bloat] DC behaviors today

[Mikael Abrahamsson]

On Mon, 4 Dec 2017, Pedro Tumusok wrote:


Looking at chipsets coming/just arrived from the chipset vendors, I think
we will see CPE with 10G SFP+ and 802.11ax Q3/Q4 this year.
Price is of course a bit steeper than the 15USD USB DSL modem :P, but
probably fits nicely for the SMB segment.


https://kb.netgear.com/31408/What-SFP-modules-are-compatible-with-my-Nighthawk-X10-R9000-router

This has been available for a while now. Only use-case I see for it is 
Comcast 2 gigabit/s service, that's the only one I know of that would fit 
this product (since it has no downlink 10GE ports).


--
Mikael Abrahamssonemail: swm...@swm.pp.se
___
Cerowrt-devel mailing list
Cerowrt-devel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel

Re: [Cerowrt-devel] [Bloat] DC behaviors today

[dpreed]

I suggest we stop talking about throughput, which has been the mistaken idea 
about networking for 30-40 years.
 
Almost all networking ends up being about end-to-end response time in a 
multiplexed system.
 
Or put another way: "It's the Latency, Stupid".
 
I get (and have come to expect) 27 msec. RTT's under significant load, from 
Boston suburb to Sunnyvale, CA.
 
I get 2 microsecond RTT's within my house (using 10 GigE).
 
What will we expect tomorrow?
 
This is related to Bufferbloat, because queueing delay is just not a good thing 
in these contexts - contexts where Latency Matters. We provision multiplexed 
networks based on "peak capacity" never being reached.
 
Consequently, 1 Gig to the home is "table stakes". And in DOCSIS 3.1 
deployments that is what is being delivered, cheap, today.
 
And 10 Gig within the home is becoming "table stakes", especially for 
applications that need quick response to human interaction.
 
1 NvME drive already delivers around 11 Gb/sec at its interface. That's what is 
needed in the network to "impedance match".
 
802.11ax already gives around 10 Gb/sec. wireless (and will be on the market 
soon).
 
The folks who think that having 1 Gb/sec to the home would only be important if 
you had to transfer at that rate 8 hours a day are just not thinking clearly 
about what "responsiveness" means.
 
For a different angle on this, think about what the desirable "channel change 
time" is if a company like Netflix were covering all the football (I mean US's 
soccer) games in the world. You'd like to fill the "buffer" in 100 msec. so 
channel change to some new channel is responsive. 100 msec. of 4K sports, which 
you are watching in "real time" needs to be buffered, and you want no more than 
a second or two of delay from camera to your screen. So buffering up 1 second 
of a newly selected 4 K video stream in 100 msec. on demand is why you need 
such speeds. Do the math.
 
VR sports coverage - even moreso.
 
 


On Monday, December 4, 2017 7:44am, "Mikael Abrahamsson"  
said:



> On Mon, 4 Dec 2017, Pedro Tumusok wrote:
> 
> > Looking at chipsets coming/just arrived from the chipset vendors, I think
> > we will see CPE with 10G SFP+ and 802.11ax Q3/Q4 this year.
> > Price is of course a bit steeper than the 15USD USB DSL modem :P, but
> > probably fits nicely for the SMB segment.
> 
> https://kb.netgear.com/31408/What-SFP-modules-are-compatible-with-my-Nighthawk-X10-R9000-router
> 
> This has been available for a while now. Only use-case I see for it is
> Comcast 2 gigabit/s service, that's the only one I know of that would fit
> this product (since it has no downlink 10GE ports).
> 
> --
> Mikael Abrahamsson email: swm...@swm.pp.se
> ___
> Cerowrt-devel mailing list
> Cerowrt-devel@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/cerowrt-devel
> ___
Cerowrt-devel mailing list
Cerowrt-devel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel

Re: [Cerowrt-devel] [Bloat] DC behaviors today

[Mikael Abrahamsson]

On Mon, 4 Dec 2017, dpr...@reed.com wrote:

I suggest we stop talking about throughput, which has been the mistaken 
idea about networking for 30-40 years.


We need to talk both about latency and speed. Yes, speed is talked about 
too much (relative to RTT), but it's not irrelevant.


Speed of light in fiber means RTT is approx 1ms per 100km, so from 
Stockholm to SFO my RTT is never going to be significantly below 85ms 
(8625km great circle). It's current twice that.


So we just have to accept that some services will never be deliverable 
across the wider Internet, but have to be deployed closer to the customer 
(as per your examples, some need 1ms RTT to work well), and we need lower 
access latency and lower queuing delay. So yes, agreed.


However, I am not going to concede that speed is "mistaken idea about 
networking". No amount of smarter queuing is going to fix the problem if I 
don't have enough throughput available to me that I need for my 
application.


--
Mikael Abrahamssonemail: swm...@swm.pp.se
___
Cerowrt-devel mailing list
Cerowrt-devel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel

Re: [Cerowrt-devel] [Bloat] DC behaviors today

[Dave Taht]

Luca Muscariello  writes:

> I think everything is about response time, even throughput. 
>
> If we compare the time to transmit a single packet from A to B, including
> propagation delay, transmission delay and queuing delay,
> to the time to move a much larger amount of data from A to B we use throughput
> in this second case because it is a normalized
> quantity w.r.t. response time (bytes over delivery time). For a single
> transmission we tend to use latency.
> But in the end response time is what matters.
>
> Also, even instantaneous throughput is well defined only for a time scale 
> which
> has to be much larger than the min RTT (propagation + transmission delays)
> Agree also that looking at video, latency and latency budgets are better
> quantities than throughput. At least more accurate.
>
> On Fri, Dec 8, 2017 at 8:05 AM, Mikael Abrahamsson  wrote:
>
> On Mon, 4 Dec 2017, dpr...@reed.com wrote:
> 
> I suggest we stop talking about throughput, which has been the 
> mistaken
> idea about networking for 30-40 years.
> 
>
> We need to talk both about latency and speed. Yes, speed is talked about 
> too
> much (relative to RTT), but it's not irrelevant.
> 
> Speed of light in fiber means RTT is approx 1ms per 100km, so from 
> Stockholm
> to SFO my RTT is never going to be significantly below 85ms (8625km great
> circle). It's current twice that.
> 
> So we just have to accept that some services will never be deliverable
> across the wider Internet, but have to be deployed closer to the customer
> (as per your examples, some need 1ms RTT to work well), and we need lower
> access latency and lower queuing delay. So yes, agreed.
> 
> However, I am not going to concede that speed is "mistaken idea about
> networking". No amount of smarter queuing is going to fix the problem if I
> don't have enough throughput available to me that I need for my 
> application.

In terms of the bellcurve here, throughput has increased much more
rapidly than than latency has decreased, for most, and in an increasing
majority of human-interactive cases (like video streaming), we often
have enough throughput.

And the age old argument regarding "just have overcapacity, always"
tends to work in these cases.

I tend not to care as much about how long it takes for things that do
not need R/T deadlines as humans and as steering wheels do.

Propigation delay, while ultimately bound by the speed of light, is also
affected by the wires wrapping indirectly around the earth - much slower
than would be possible if we worked at it:

https://arxiv.org/pdf/1505.03449.pdf

Then there's inside the boxes themselves:

A lot of my struggles of late has been to get latencies and adaquate
sampling techniques down below 3ms (my previous value for starting to
reject things due to having too much noise) - and despite trying fairly
hard, well... a process can't even sleep accurately much below 1ms, on
bare metal linux. A dream of mine has been 8 channel high quality audio,
with a video delay of not much more than 2.7ms for AR applications.

For comparison, an idle quad core aarch64 and dual core x86_64:

root@nanopineo2:~# irtt sleep

Testing sleep accuracy...

Sleep DurationMean Error   % Error

   1ns  13.353µs 1335336.9

  10ns   14.34µs  143409.5

 100ns  13.343µs   13343.9

   1µs  12.791µs1279.2

  10µs 148.661µs1486.6

 100µs 150.907µs 150.9

   1ms 168.001µs  16.8

  10ms 131.235µs   1.3

 100ms 145.611µs   0.1

 200ms 162.917µs   0.1

 500ms 169.885µs   0.0


d@nemesis:~$ irtt sleep

Testing sleep accuracy...


Sleep DurationMean Error   % Error

   1ns 668ns   66831.9

  10ns 672ns6723.7

 100ns 557ns 557.6

   1µs  57.749µs5774.9

  10µs  63.063µs 630.6

 100µs  67.737µs  67.7

   1ms 153.978µs  15.4

  10ms 169.709µs   1.7

 100ms 186.685µs   0.2

 200ms 176.859µs   0.1

 500ms 177.271µs   0.0

> 
> -- 
> Mikael Abrahamsson email: swm...@swm.pp.se
> ___
> 
> 
> Bloat mailing list
> bl...@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
> 
>
>
> ___
> Bloat mailing list
> bl...@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
___
Cerowrt-devel mailing list
Cerowrt-devel@lists.bufferbloat.net
https://

Re: [Cerowrt-devel] [Bloat] DC behaviors today

[dpreed]

Luca's point tends to be correct - variable latency destroys the stability of 
flow control loops, which destroys throughput, even when there is sufficient 
capacity to handle the load.
 
This is an indirect result of Little's Lemma (which is strictly true only for 
Poisson arrival, but almost any arrival process will have a similar interaction 
between latency and throughput).
 
However, the other reason I say what I say so strongly is this:
 
Rant on.
 
Peak/avg. load ratio always exceeds a factor of 10 or more, IRL. Only 
"benchmark setups" (or hot-rod races done for academic reasons or marketing 
reasons to claim some sort of "title") operate at peak supportable load any 
significant part of the time.
 
The reason for this is not just "fat pipes are better", but because bitrate of 
the underlying medium is an insignificant fraction of systems operational and 
capital expense.
 
SLA's are specified in "uptime" not "bits transported", and a clogged pipe is 
defined as down when latency exceeds a small number.
 
Typical operating points of corporate networks where the users are happy are 
single-digit percentage of max load.
 
This is also true of computer buses and memory controllers and storage 
interfaces IRL. Again, latency is the primary measure, and the system never 
focuses on operating points anywhere near max throughput.
 
Rant off.

On Tuesday, December 12, 2017 1:36pm, "Dave Taht"  said:



> 
> Luca Muscariello  writes:
> 
> > I think everything is about response time, even throughput.
> >
> > If we compare the time to transmit a single packet from A to B, including
> > propagation delay, transmission delay and queuing delay,
> > to the time to move a much larger amount of data from A to B we use
> throughput
> > in this second case because it is a normalized
> > quantity w.r.t. response time (bytes over delivery time). For a single
> > transmission we tend to use latency.
> > But in the end response time is what matters.
> >
> > Also, even instantaneous throughput is well defined only for a time scale
> which
> > has to be much larger than the min RTT (propagation + transmission delays)
> > Agree also that looking at video, latency and latency budgets are better
> > quantities than throughput. At least more accurate.
> >
> > On Fri, Dec 8, 2017 at 8:05 AM, Mikael Abrahamsson 
> wrote:
> >
> > On Mon, 4 Dec 2017, dpr...@reed.com wrote:
> >
> > I suggest we stop talking about throughput, which has been the
> mistaken
> > idea about networking for 30-40 years.
> >
> >
> > We need to talk both about latency and speed. Yes, speed is talked about
> too
> > much (relative to RTT), but it's not irrelevant.
> >
> > Speed of light in fiber means RTT is approx 1ms per 100km, so from
> Stockholm
> > to SFO my RTT is never going to be significantly below 85ms (8625km
> great
> > circle). It's current twice that.
> >
> > So we just have to accept that some services will never be deliverable
> > across the wider Internet, but have to be deployed closer to the
> customer
> > (as per your examples, some need 1ms RTT to work well), and we need
> lower
> > access latency and lower queuing delay. So yes, agreed.
> >
> > However, I am not going to concede that speed is "mistaken idea about
> > networking". No amount of smarter queuing is going to fix the problem if
> I
> > don't have enough throughput available to me that I need for my
> application.
> 
> In terms of the bellcurve here, throughput has increased much more
> rapidly than than latency has decreased, for most, and in an increasing
> majority of human-interactive cases (like video streaming), we often
> have enough throughput.
> 
> And the age old argument regarding "just have overcapacity, always"
> tends to work in these cases.
> 
> I tend not to care as much about how long it takes for things that do
> not need R/T deadlines as humans and as steering wheels do.
> 
> Propigation delay, while ultimately bound by the speed of light, is also
> affected by the wires wrapping indirectly around the earth - much slower
> than would be possible if we worked at it:
> 
> https://arxiv.org/pdf/1505.03449.pdf
> 
> Then there's inside the boxes themselves:
> 
> A lot of my struggles of late has been to get latencies and adaquate
> sampling techniques down below 3ms (my previous value for starting to
> reject things due to having too much noise) - and despite trying fairly
> hard, well... a process can't even sleep accurately much below 1ms, on
> bare metal linux. A dream of mine has been 8 channel high quality audio,
> with a video delay of not much more than 2.7ms for AR applications.
> 
> For comparison, an idle quad core aarch64 and dual core x86_64:
> 
> root@nanopineo2:~# irtt sleep
> 
> Testing sleep accuracy...
> 
> Sleep Duration Mean Error % Error
> 
> 1ns 13.353µs 1335336.9
> 
> 10ns 14.34µs 143409.5
> 
> 100ns 13.343µs 13343.9
> 
> 1µs 12.791µs 1279.2
> 
> 10µs 148.661µs 1486.6
> 
> 100µs 150.907µs 150.9
> 
> 1ms 168.001µs 16.8
> 
> 10ms 131.235µs 1.

Re: [Cerowrt-devel] [Bloat] DC behaviors today

[Jonathan Morton]

This is also true in the consumer space, and is the reason why ISPs can
save money by taking advantage of statistical multiplexing.  On average, I
personally could be satisfied with a megabit, but it's a real pain to
download gigabyte-class software updates at that speed.

If it takes me literally days to download a game's beta versions, any
comments I might have will be stale by the time they can be heard, and in
the meantime my other uses of the internet have been seriously impaired.
It's much more useful if I can download the same data in an hour, and spend
the remaining time evaluating.  So throughput is indeed a factor in
response time, once the size of the response is sufficiently large.

Occasionally, of course, practically everyone in the country wants to tune
into coverage of some event at the same time.  More commonly, they simply
get home from work and school at the same time every day.  That breaks the
assumptions behind pure statistical multiplexing, and requires a greater
provisioning factor.

- Jonathan Morton
___
Cerowrt-devel mailing list
Cerowrt-devel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel

Re: [Cerowrt-devel] [Bloat] DC behaviors today

[Mikael Abrahamsson]

On Wed, 13 Dec 2017, Jonathan Morton wrote:

Occasionally, of course, practically everyone in the country wants to 
tune into coverage of some event at the same time.  More commonly, they 
simply get home from work and school at the same time every day.  That 
breaks the assumptions behind pure statistical multiplexing, and 
requires a greater provisioning factor.


Reasonable operators have provisioning guidelines that look at actual 
usage, although they probably look at it in 5 minute averages and not 
millisecond as done here in this context.


So they might say "if busy hour average is over 50% 3 days in a week" this 
will trigger a provisioning alarm for that link, and the person (or 
system) will take a more detailed look and look at 5minute average graph 
and decide if this needs to be upgraded or not.


For me the interesting point is always "what's going on in busy hour of 
the day" and never "what's the monthly average transferred amount of 
data".


Of course, this can hide subsecond bufferbloat extremely well (and has), 
but at least this is typically how statistical overprovisioning is done. 
You look at actual usage and make sure your network is never full for any 
sustained amount of time, in normal operation, and make sure you perform 
upgrades well before the growth has resulted in network being full.


--
Mikael Abrahamssonemail: swm...@swm.pp.se
___
Cerowrt-devel mailing list
Cerowrt-devel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel

Re: [Cerowrt-devel] [Bloat] DC behaviors today

[Neil Davies]

> On 12 Dec 2017, at 22:53, dpr...@reed.com wrote:
> 
> Luca's point tends to be correct - variable latency destroys the stability of 
> flow control loops, which destroys throughput, even when there is sufficient 
> capacity to handle the load.
> 
> This is an indirect result of Little's Lemma (which is strictly true only for 
> Poisson arrival, but almost any arrival process will have a similar 
> interaction between latency and throughput).

Actually it is true for general arrival patterns (can’t lay my hands on the 
reference for the moment - but it was a while back that was shown) - what this 
points to is an underlying conservation law - that “delay and loss” are 
conserved in a scheduling process. This comes out of the M/M/1/K/K queueing 
system and associated analysis.

There is  conservation law (and Klienrock refers to this - at least in terms of 
delay - in 1965 - 
http://onlinelibrary.wiley.com/doi/10.1002/nav.3800120206/abstract 
) at work 
here.

All scheduling systems can do is “distribute” the resulting “delay and loss” 
differentially amongst the (instantaneous set of) competing streams.

Let me just repeat that - The “delay and loss” are a conserved quantity - 
scheduling can’t “destroy” it (they can influence higher level protocol 
behaviour) but not reduce the total amount of “delay and loss” that is being 
induced into the collective set of streams...

> 
> However, the other reason I say what I say so strongly is this:
> 
> Rant on.
> 
> Peak/avg. load ratio always exceeds a factor of 10 or more, IRL. Only 
> "benchmark setups" (or hot-rod races done for academic reasons or marketing 
> reasons to claim some sort of "title") operate at peak supportable load any 
> significant part of the time.

Have you considered what this means for the economics of the operation of 
networks? What other industry that “moves things around” (i.e logistical or 
similar) system creates a solution in which they have 10x as much 
infrastructure than their peak requirement?

> 
> The reason for this is not just "fat pipes are better", but because bitrate 
> of the underlying medium is an insignificant fraction of systems operational 
> and capital expense.

Agree that (if you are the incumbent that ‘owns’ the low level transmission 
medium) that this is true (though the costs of lighting a new lambda are not 
trivial) - but that is not the experience of anyone else in the digital supply 
time

> 
> SLA's are specified in "uptime" not "bits transported", and a clogged pipe is 
> defined as down when latency exceeds a small number.

Do you have any evidence you can reference for an SLA that treats a few ms as 
“down”? Most of the SLAs I’ve had dealings with use averages over fairly long 
time periods (e.g. a month) - and there is no quality in averages.

> 
> Typical operating points of corporate networks where the users are happy are 
> single-digit percentage of max load.

Or less - they also detest the costs that they have to pay the network 
providers to try and de-risk their applications. There is also the issue that 
they measure averages (over 5min to 15min) they completely fail to capture (for 
example) the 15seconds when delay and jitter was high so the CEO’s video 
conference broke up.

> 
> This is also true of computer buses and memory controllers and storage 
> interfaces IRL. Again, latency is the primary measure, and the system never 
> focuses on operating points anywhere near max throughput.

Agreed - but wouldn’t it be nice if they could? I’ve worked on h/w systems 
where we have designed system to run near limits (the set-top box market is 
pretty cut-throat and the closer to saturation you can run and still deliver 
the acceptable outcome the cheaper the box the greater the profit margin for 
the set-top box provider)

> 
> Rant off.
> 

Cheers

Neil


> On Tuesday, December 12, 2017 1:36pm, "Dave Taht"  > said:
> 
> >
> > Luca Muscariello  writes:
> >
> > > I think everything is about response time, even throughput.
> > >
> > > If we compare the time to transmit a single packet from A to B, including
> > > propagation delay, transmission delay and queuing delay,
> > > to the time to move a much larger amount of data from A to B we use
> > throughput
> > > in this second case because it is a normalized
> > > quantity w.r.t. response time (bytes over delivery time). For a single
> > > transmission we tend to use latency.
> > > But in the end response time is what matters.
> > >
> > > Also, even instantaneous throughput is well defined only for a time scale
> > which
> > > has to be much larger than the min RTT (propagation + transmission delays)
> > > Agree also that looking at video, latency and latency budgets are better
> > > quantities than throughput. At least more accurate.
> > >
> > > On Fri, Dec 8, 2017 at 8:05 AM, Mikael Abrahamsson 
> > wrote:
> > >
> > > On Mon, 4 Dec 2017, dpr...@reed.com

Re: [Cerowrt-devel] [Bloat] DC behaviors today

[Jonathan Morton]

> Have you considered what this means for the economics of the operation of
networks? What other industry that “moves things around” (i.e logistical or
similar) system creates a solution in which they have 10x as much
infrastructure than their peak requirement?

Ten times peak demand?  No.

Ten times average demand estimated at time of deployment, and struggling
badly with peak demand a decade later, yes.  And this is the transportation
industry, where a decade is a *short* time - like less than a year in
telecoms.

- Jonathan Morton

On 13 Dec 2017 17:27, "Neil Davies"  wrote:

>
> On 12 Dec 2017, at 22:53, dpr...@reed.com wrote:
>
> Luca's point tends to be correct - variable latency destroys the stability
> of flow control loops, which destroys throughput, even when there is
> sufficient capacity to handle the load.
>
>
> This is an indirect result of Little's Lemma (which is strictly true only
> for Poisson arrival, but almost any arrival process will have a similar
> interaction between latency and throughput).
>
>
> Actually it is true for general arrival patterns (can’t lay my hands on
> the reference for the moment - but it was a while back that was shown) -
> what this points to is an underlying conservation law - that “delay and
> loss” are conserved in a scheduling process. This comes out of the
> M/M/1/K/K queueing system and associated analysis.
>
> There is  conservation law (and Klienrock refers to this - at least in
> terms of delay - in 1965 - http://onlinelibrary.wiley.com/doi/10.1002/nav.
> 3800120206/abstract) at work here.
>
> All scheduling systems can do is “distribute” the resulting “delay and
> loss” differentially amongst the (instantaneous set of) competing streams.
>
> Let me just repeat that - The “delay and loss” are a conserved quantity -
> scheduling can’t “destroy” it (they can influence higher level protocol
> behaviour) but not reduce the total amount of “delay and loss” that is
> being induced into the collective set of streams...
>
>
> However, the other reason I say what I say so strongly is this:
>
>
> Rant on.
>
>
> Peak/avg. load ratio always exceeds a factor of 10 or more, IRL. Only
> "benchmark setups" (or hot-rod races done for academic reasons or marketing
> reasons to claim some sort of "title") operate at peak supportable load any
> significant part of the time.
>
>
> Have you considered what this means for the economics of the operation of
> networks? What other industry that “moves things around” (i.e logistical or
> similar) system creates a solution in which they have 10x as much
> infrastructure than their peak requirement?
>
>
> The reason for this is not just "fat pipes are better", but because
> bitrate of the underlying medium is an insignificant fraction of systems
> operational and capital expense.
>
>
> Agree that (if you are the incumbent that ‘owns’ the low level
> transmission medium) that this is true (though the costs of lighting a new
> lambda are not trivial) - but that is not the experience of anyone else in
> the digital supply time
>
>
> SLA's are specified in "uptime" not "bits transported", and a clogged pipe
> is defined as down when latency exceeds a small number.
>
>
> Do you have any evidence you can reference for an SLA that treats a few ms
> as “down”? Most of the SLAs I’ve had dealings with use averages over fairly
> long time periods (e.g. a month) - and there is no quality in averages.
>
>
> Typical operating points of corporate networks where the users are happy
> are single-digit percentage of max load.
>
>
> Or less - they also detest the costs that they have to pay the network
> providers to try and de-risk their applications. There is also the issue
> that they measure averages (over 5min to 15min) they completely fail to
> capture (for example) the 15seconds when delay and jitter was high so the
> CEO’s video conference broke up.
>
>
> This is also true of computer buses and memory controllers and storage
> interfaces IRL. Again, latency is the primary measure, and the system never
> focuses on operating points anywhere near max throughput.
>
>
> Agreed - but wouldn’t it be nice if they could? I’ve worked on h/w systems
> where we have designed system to run near limits (the set-top box market is
> pretty cut-throat and the closer to saturation you can run and still
> deliver the acceptable outcome the cheaper the box the greater the profit
> margin for the set-top box provider)
>
>
> Rant off.
>
>
> Cheers
>
> Neil
>
>
> On Tuesday, December 12, 2017 1:36pm, "Dave Taht"  said:
>
> >
> > Luca Muscariello  writes:
> >
> > > I think everything is about response time, even throughput.
> > >
> > > If we compare the time to transmit a single packet from A to B,
> including
> > > propagation delay, transmission delay and queuing delay,
> > > to the time to move a much larger amount of data from A to B we use
> > throughput
> > > in this second case because it is a normalized
> > > quantity w.r.t. response time (bytes over deliv

Re: [Cerowrt-devel] [Bloat] DC behaviors today

[dpreed]

Just to be clear, I have built and operated a whole range of network platforms, 
as well as diagnosing problems and planning deployments of systems that include 
digital packet delivery in real contexts where cost and performance matter, for 
nearly 40 years now. So this isn't only some kind of radical opinion, but 
hard-won knowledge across my entire career. I also havea very strong 
theoretical background in queueing theory and control theory -- enough to teach 
a graduate seminar, anyway.
That said, there are lots of folks out there who have opinions different than 
mine. But far too many (such as those who think big buffers are "good", who 
brought us bufferbloat) are not aware of how networks are really used or the 
practical effects of their poor models of usage.
 
If it comforts you to think that I am just stating an "opinion", which must be 
wrong because it is not the "conventional wisdom" in the circles where you 
travel, fine. You are entitled to dismiss any ideas you don't like. But I would 
suggest you get data about your assumptions.
 
I don't know if I'm being trolled, but a couple of comments on the recent 
comments:
 
1. Statistical multiplexing viewed as an averaging/smoothing as an idea is, in 
my personal opinion and experience measuring real network behavior, a 
description of a theoretical phenomenon that is not real (e.g. "consider a 
spherical cow") that is amenable to theoretical analysis. Such theoretical 
analysis can make some gross estimates, but it breaks down quickly. The same 
thing is true of common economic theory that models practical markets by linear 
models (linear systems of differential equations are common) and gaussian 
probability distributions (gaussians are easily analyzed, but wrong. You can 
read the popular books by Nassim Taleb for an entertaining and enlightening 
deeper understanding of the economic problems with such modeling).
 
One of the features well observed in real measurements of real systems is that 
packet flows are "fractal", which means that there is a self-similarity of rate 
variability all time scales from micro to macro. As you look at smaller and 
smaller time scales, or larger and larger time scales, the packet request 
density per unit time never smooths out due to "averaging over sources". That 
is, there's no practical "statistical multiplexing" effect. There's also 
significant correlation among many packet arrivals - assuming they are 
statistically independent (which is required for the "law of large numbers" to 
apply) is often far from the real situation - flows that are assumed to be 
independent are usually strongly coupled.
 
The one exception where flows average out at a constant rate is when there is a 
"bottleneck". Then, there being no more capacity, the constant rate is forced, 
not by statistical averaging but by a very different process. One that is 
almost never desirable.
 
This is just what is observed in case after case.  Designers may imagine that 
their networks have "smooth averaging" properties. There's a strong thread in 
networking literature that makes this pretty-much-always-false assumption the 
basis of protocol designs, thinking about "Quality of Service" and other sorts 
of things. You can teach graduate students about a reality that does not exist, 
and get papers accepted in conferences where the reviewers have been trained in 
the same tradition of unreal assumptions.
 
2. I work every day with "datacenter" networking and distributed systems on 10 
GigE and faster Ethernet fabrics with switches and trunking. I see the packet 
flows driven by distributed computing in real systems. Whenever the sustained 
peak load on a switch path reaches 100%, that's not "good", that's not 
"efficient" resource usage. That is a situation where computing is experiencing 
huge wasted capacity due to network congestion that is dramatically slowing 
down the desired workload.
 
Again this is because *real workloads* in distributed computation don't have 
smooth or averagable rates over interconnects. Latency is everything in that 
application too!
 
Yes, because one buys switches from vendors who don't know how to build or 
operate a server or a database at all, you see vendors trying to demonstrate 
their amazing throughput, but the people who build these systems (me, for 
example) are not looking at throughput or statistical multiplexing at all! We 
use "throughput" as a proxy for "latency under load". (and it is a poor proxy! 
Because vendors throw in big buffers, causing bufferbloat. See Arista Networks' 
attempts to justify their huge buffers as a "good thing" -- when it is just a 
case of something you have to design around by clocking the packets so they 
never accumulate in a buffer).
 
So, yes, the peak transfer rate matters, of course. And sometimes it is 
utilized for very good reason (when the latency of a file transfer as a whole 
is the latency that matters). But to be clear, just because as a user I want to

Re: [Cerowrt-devel] [Bloat] DC behaviors today

[Neil Davies]

Please - my email was not an intention to troll - I wanted to establish a 
dialogue, I am sorry if I’ve offended.
> On 13 Dec 2017, at 18:08, dpr...@reed.com wrote:
> 
> Just to be clear, I have built and operated a whole range of network 
> platforms, as well as diagnosing problems and planning deployments of systems 
> that include digital packet delivery in real contexts where cost and 
> performance matter, for nearly 40 years now. So this isn't only some kind of 
> radical opinion, but hard-won knowledge across my entire career. I also havea 
> very strong theoretical background in queueing theory and control theory -- 
> enough to teach a graduate seminar, anyway.

I accept that - if we are laying out bona fides, I have acted as thesis advisor 
to people working in this area over 20 years, and I continue to work with 
network operators, system designers and research organisations (mainly in the 
EU) in this area.

> That said, there are lots of folks out there who have opinions different than 
> mine. But far too many (such as those who think big buffers are "good", who 
> brought us bufferbloat) are not aware of how networks are really used or the 
> practical effects of their poor models of usage.
> 
> If it comforts you to think that I am just stating an "opinion", which must 
> be wrong because it is not the "conventional wisdom" in the circles where you 
> travel, fine. You are entitled to dismiss any ideas you don't like. But I 
> would suggest you get data about your assumptions.
> 
> I don't know if I'm being trolled, but a couple of comments on the recent 
> comments:
> 
> 1. Statistical multiplexing viewed as an averaging/smoothing as an idea is, 
> in my personal opinion and experience measuring real network behavior, a 
> description of a theoretical phenomenon that is not real (e.g. "consider a 
> spherical cow") that is amenable to theoretical analysis. Such theoretical 
> analysis can make some gross estimates, but it breaks down quickly. The same 
> thing is true of common economic theory that models practical markets by 
> linear models (linear systems of differential equations are common) and 
> gaussian probability distributions (gaussians are easily analyzed, but wrong. 
> You can read the popular books by Nassim Taleb for an entertaining and 
> enlightening deeper understanding of the economic problems with such 
> modeling).

I would fully accept that seeing statistical (or perhaps, better named, 
stochastic) multiplexing as an averaging process is vast over simplification of 
the complexity. However, I see the underlying mathematics as capturing a much 
richer description(s), for example of the transient behaviour - queuing theory 
(in its usual undergraduate formulation) tends to gloss over the edge / extreme 
conditions  as well as dealing with non-stationary arrival phenomena (such as 
can occur in the presence of adaptive protocols).

For example - one approach to solve the underlying Markov Chain systems (as the 
operational semantic representation of a queueing system)  is to represent them 
as transition matrices and then “solve” those matrices for steady state [as you 
probably know - think of that as backstory for the interested reader].

We’ve used such transition matrices to examine “relaxation times” of queueing / 
scheduling algorithms - i.e given that a buffer has filled, how quickly will 
the system relax back towards “steady state”. There are assumptions behind 
this, of course, but viewing the buffer state as a probability distribution and 
seeing how that distribution evolves after, say, an impulse change in load 
helps at lot to generate new approaches.

Cards on the table - I don’t see networks as a (purely) natural phenomena (as, 
say, chemistry or physics) - but as a more mathematical one. Queuing systems 
are (relatively) simple automata being pushed through their states by 
(non-stationary but broadly characterisable in stochastic terms) arrivals and 
departures (which are not so stochastically varied as they are related to the 
actual packet sizes.). There are rules to that mathematical game imposed by 
real-world physics, but there are other ways of constructing (and configuring) 
the actions of those automata to create “better” solutions (for various types 
of “better”).

> 
> One of the features well observed in real measurements of real systems is 
> that packet flows are "fractal", which means that there is a self-similarity 
> of rate variability all time scales from micro to macro. As you look at 
> smaller and smaller time scales, or larger and larger time scales, the packet 
> request density per unit time never smooths out due to "averaging over 
> sources". That is, there's no practical "statistical multiplexing" effect. 
> There's also significant correlation among many packet arrivals - assuming 
> they are statistically independent (which is required for the "law of large 
> numbers" to apply) is often far from the real situation - flows t

Re: [Cerowrt-devel] [Bloat] DC behaviors today

[Jonathan Morton]

(Awesome development - I have a computer with a sane e-mail client again.  One 
that doesn’t assume I want to top-post if I quote anything at all, *and* lets 
me type with an actual keyboard.  Luxury!)

>> One of the features well observed in real measurements of real systems is 
>> that packet flows are "fractal", which means that there is a self-similarity 
>> of rate variability all time scales from micro to macro.
> 
> I remember this debate and its evolution, Hurst parameters and all that. I 
> also understand that a collection of on/off Poisson sources looks fractal - I 
> found that “the universe if fractal - live with it” ethos of limited 
> practical use (except to help people say it was not solvable).

>> Designers may imagine that their networks have "smooth averaging" 
>> properties. There's a strong thread in networking literature that makes this 
>> pretty-much-always-false assumption the basis of protocol designs, thinking 
>> about "Quality of Service" and other sorts of things. You can teach graduate 
>> students about a reality that does not exist, and get papers accepted in 
>> conferences where the reviewers have been trained in the same tradition of 
>> unreal assumptions.
> 
> Agreed - there is a massive disconnect between a lot of the literature (and 
> the people who make their living generating it - [to those people, please 
> don’t take offence, queueing theory is really useful it is just the real 
> world is a lot more non-stationary than you model]) and reality.

Probably a lot of theoreticians would be horrified at the extent to which I 
ignored mathematics and relied on intuition (and observations of real traffic, 
ie. eating my own dogfood) while building Cake.

That approach, however, led me to some novel algorithms and combinations 
thereof which seem to work well in practice, as well as to some practical 
observations about the present state of the Internet.  I’ve also used some 
contributions from others, but only where they made sense at an intuitive level.

However, Cake isn’t designed to work in the datacentre.  Nor is it likely to 
work optimally in an ISP’s core networks.  The combination of features in Cake 
is not optimised for those environments, rather for last-mile links which are 
typically the bottlenecks experienced by ordinary Internet users.  Some of 
Cake's algorithms could reasonably be reused in a different combination for a 
different environment.

> I see large scale (i.e. public internets) not as a mono-service but as a 
> “poly service” - there are multiple demands for timeliness etc that exist out 
> there for “real services”.

This is definitely true.  However, the biggest problem I’ve noticed is with 
distinguishing these traffic types from each other.  In some cases there are 
heuristics which are accurate enough to be useful.  In others, there are not.  
Rarely is the distinction *explicitly* marked in any way, and some common 
protocols explicitly obscure themselves due to historical mistreatment.

Diffserv is very hard to use in practice.  There’s a controversial fact for you 
to chew on.

> We’ve worked with people who have created risks for Netflix delivery 
> (accidentally I might add - they though they were doing “the right thing”) by 
> increasing their network infrastructure to 100G delivery everywhere. That 
> change (combined with others made by CDN people - TCP offload engines) 
> created so much non-stationarity in the load so as to cause delay and loss 
> spikes that *did* cause VoD playout buffers to empty.  This is an example of 
> where “more capacity” produced worse outcomes.

That’s an interesting and counter-intuitive result.  I’ll hazard a guess that 
it had something to do with burst loss in dumb tail-drop FIFOs?  Offload 
engines tend to produce extremely bursty traffic which - with a nod to another 
thread presently ongoing - makes a mockery of any ack-clocking or pacing which 
TCP designers normally assume is in effect.

One of the things that fq_codel and Cake can do really well is to take a deep 
queue full of consecutive line-rate bursts and turn them into interleaved 
packet streams, which are at least slightly better “paced” than the originals. 
They also specifically try to avoid burst loss and (at least in Cake’s case) 
tail loss.

It is of course regrettable that this behaviour conflicts with the assumptions 
of most network acceleration hardware, and that maximum throughput might 
therefore be compromised.  The *qualitative* behaviour is however improved.

> I would suggest that there are other ways of dealing with the impact of 
> “peak” (i.e where instantaneous demand exceeds supply over a long enough 
> timescale to start effecting the most delay/loss sensitive application in the 
> collective multiplexed stream).

Such as signalling to the traffic that congestion exists, and to please slow 
down a bit to make room?  ECN and AQM are great ways of doing that, especially 
in combination with flow isolation - the

Re: [Cerowrt-devel] [Bloat] DC behaviors today

[Mikael Abrahamsson]

On Wed, 13 Dec 2017, Jonathan Morton wrote:

Ten times average demand estimated at time of deployment, and struggling 
badly with peak demand a decade later, yes.  And this is the 
transportation industry, where a decade is a *short* time - like less 
than a year in telecoms.


I've worked in ISPs since 1999 or so. I've been at startups and I've been 
at established ISPs.


It's kind of an S curve when it comes to traffic growth, when you're 
adding customers you can easily see 100%-300% growth per year (or more). 
Then after market becomes saturated growth comes from per-customer 
increased usage, and for the past 20 years or so, this has been in the 
neighbourhood of 20-30% per year.


Running a network that congests parts of the day, it's hard to tell what 
"Quality of Experience" your customers will have. I've heard of horror 
stories from the 90ties where a then large US ISP was running an OC3 (155 
megabit/s) full most of the day. So someone said "oh, we need to upgrade 
this", and after a while, they did, to 2xOC3. Great, right? No, after that 
upgrade both OC3:s were completely congested. Ok, then upgrade to OC12 
(622 megabit/s). After that upgrade, evidently that link was not congested 
a few hours of the day, and of course needed more upgrades.


So at the places I've been, I've advocated for planning rules that say 
that when the link is peaking at 5 minute averages of more than 50% of 
link capacity, then upgrade needs to be ordered. This 50% number can be 
larger if the link aggregates larger number of customers, because 
typically your "statistical overbooking" varies less the more customers 
participates.


These devices do not do per-flow anything. They might have 10G or 100G 
link to/from it with many many millions of flows, and it's all NPU 
forwarding. Typically they might do DIFFserv-based queueing and WRED to 
mitigate excessive buffering. Today, they typically don't even do ECN 
marking (which I have advocated for, but there is not much support from 
other ISPs in this mission).


Now, on the customer access line it's a completely different matter. 
Typically people build with BRAS or similar, where (tens of) thousands of 
customers might sit on a (very expensive) access card with hundreds of 
thousands of queues per NPU. This still leaves just a few queues per 
customer, unfortunately. So these do not do per-flow anything either. This 
is where PIE comes in, because these devices like these can do PIE in the 
NPU fairly easily because it's kind of like WRED.


So back to the capacity issue. Since these devices typically aren't good 
at assuring per-customer access to the shared medium (backbone links), 
it's easier to just make sure the backbone links are not regularily full. 
This doesn't mean you're going to have 10x capacity all the time, it 
probably means you're going to be bouncing between 25-70% utilization of 
your links (for the normal case, because you need spare capacity to handle 
events that increase traffic temporarily, plus handle loss of capacity in 
case of a link fault). The upgrade might be to add another link, or a 
higher tier speed interface, bringing down the utilization to typically 
half or quarter of what you had before.


--
Mikael Abrahamssonemail: swm...@swm.pp.se
___
Cerowrt-devel mailing list
Cerowrt-devel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel

Re: [Cerowrt-devel] [Bloat] DC behaviors today

[Benjamin Cronce]

This is an interesting topic to me. Over the past 5+ years, I've been
reading about GPON fiber aggregators(GPON chassis for lack of a proper
term) with 400Gb-1Tb/s of uplink, 1-2Tb/s line-cards, and enough GPON ports
for several thousand customers.

When my current ISP started rolling out fiber(all of it underground, no
above-ground fiber), I called support during a graveyard hour on the
weekend, and I got a senior network admin answering the phone instead of
normal tech support. When talking to him, I asked him what they claimed by
"guaranteed" bandwidth. I guess I should mention that my ISP claims
dedicated bandwidth for everyone. He told me that they played with
over-subscription for a while, but it just resulted in complex situations
that caused customers to complain. Complaining customers are expensive
because they eat up support phone time. They eventually went to a
non-oversubscribed flat model. He told me that the GPON chassis plugs
strait into the core router. I asked him about GPON port shared bandwidth
and the GPON uplink. He said they will not over-subscribe a GPON port, so
all ONTs on the port can use 100% of their provisioned rate, and they will
not place more provision bandwidth on a single GPON chassis than what they
uplink can support.

For the longest time, their max sold bandwidth was 50Mb/s. After some time,
they were having some issues resulting in packet-loss during peak hours.
Turned out their old core router could not support all of the new customers
in the ARP cache and was causing massive amounts of broadcasted packets. I
actually helped them solve this issue. They had me work with a hired
consulting service that was having issues diagnosing the problem, much
because of the older hardware not supporting modern diagnostic features.
They fixed the problem by upgrading the core router. Because I was already
in contact with them during this issue, I was made privy that their new
core router could handle about 10Tb/s with a lot of room for 100Gb+ ports.
No exact details, but told their slowest internal link was now 100Gb.

Their core router actually had traffic shaping and an AQM built in. They
switched from using ONT rate limiting for provisioning to letting the core
router handle provisioning. I can actually see 1Gb bursts as their shaping
seems to be like a sliding window over a few tens of ms. I have actually
tested their AQM a bit via a DOS testing service. At the time, I had a
100Mb/100Mb service, and externally flooding my connection with 110Mb/s
resulted in about 10% packetloss, but my ping stayed under 20ms. I tried
200Mb/s for about 20 seconds, which resulted in about 50% loss and still
~20ms pings. For about 10 seconds I tested 1Gb/s DOS and had about 90%
loss(not a long time to sample, but was sampled at a rate of 10pps against
their speed-test server), but 20-40ms pings. I tested this during off
hours, like 1am.

A few months after the upgrade, I got upgraded to a 100Mb connection with
no change in price and several new higher tiers were added, all the way up
to 1Gb/s. I asked them about this. Yes, the 1Gb tier was also not
over-subscribed. I'm not sure if some lone customer pretty much got their
own GPON port or they had some WDM-PON linecards.

I'm currently paying about $40/m for 150/150 for a "dedicated" connection.
I'm currently getting about 1ms+-0.1ms pings to my ISP's speedtest server
24/7.  If I do a ping flood, I can get my avg ping down near 0.12ms. I
assume this is because of GPON scheduling. Of course I only test this
against their speedtest server and during off hours.

As for the trunk, I've also talked to them about that, at least in the past
and I can't speak for more current times. They had 3 trunks, 2 to Level 3
Chicago and one to Global Crossing Minnesota. I was told each link was a
paired link for immediate fail-over. I was told that in some cases, they've
bonded the links, primarily due to DDOS attacks, to quickly double their
bandwidth. Their GX link was their fail-over and the two Chicago Level 3
links were the load balanced primaries. Based on trace-routes, they seemed
to be load-balanced by some lower-bits in the IP address. This gave a total
of 6 links. The network admin told me that any given link had enough
bandwidth provisioned, that if all 5 other links were down, that one link
would have a 95th percentile below 80% during peak hours, and customers
should be completely unaffected.

They've been advertising guaranteed dedicated bandwidth for over 15 years
now. They recently had a marketing campaign against the local incumbent
where they poked fun at them for only selling "up to" bandwidth. This went
on for at least a year. They openly advertised that their bandwidth was not
"up to", but that customers will always get all of their bandwidth all the
time. In the small print it said "to their transit provider". In short, my
ISP is claiming I should always get my provisioned bandwidth to Level 3
24/7. As far as I have cared to measure, this is tru

Re: [Cerowrt-devel] [Bloat] DC behaviors today

[Mikael Abrahamsson]

On Sun, 17 Dec 2017, Matthias Tafelmeier wrote:

What I actually wanted to posit in relation to that is that one could 
get sooner a c-cabable backbone sibling by marrying two ideas: the 
airborne concept ongoing as outlined plus what NASA is planning to bring 
about for the space backbone, e.g [1][2]. It's laser based instead of 
directed radio-wave only. Sure, both is in the speed range of c, 
apparantely, laser transmission has in addition a significantly higher 
bandwidth to offer. "10 to 100 times as much data at a time as 
radio-frequency systems"[3]. Attenuations to photons in clean 
atmospheric air are neglible (few mps - refractive index of about 
1.0003), so actually a neglible slowdown - easily competing with top 
notch fibres (99.7% the vacuum speed of light). Sure, that's the ideal 
case, though, if cleverly done from the procurement of platforms and 
overall system steering perspective, might feasible.


Todays laser links are in the few km per hop range, with is easily at 
least one magnitude shorter than radio based equivalents.


I don't know the physics behind it, but people who have better insight 
than I do tell me "it's hard" to run longer hops (if one wants any kind of 
high bitrate).


--
Mikael Abrahamssonemail: swm...@swm.pp.se
___
Cerowrt-devel mailing list
Cerowrt-devel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel

Re: [Cerowrt-devel] [Bloat] DC behaviors today

[Mikael Abrahamsson]

On Sun, 17 Dec 2017, Benjamin Cronce wrote:

This is an interesting topic to me. Over the past 5+ years, I've been 
reading about GPON fiber aggregators(GPON chassis for lack of a proper 
term) with 400Gb-1Tb/s of uplink, 1-2Tb/s line-cards, and enough GPON 
ports for several thousand customers.


Yep, they're available if people want to pay for them.


because they eat up support phone time. They eventually went to a
non-oversubscribed flat model. He told me that the GPON chassis plugs
strait into the core router. I asked him about GPON port shared bandwidth
and the GPON uplink. He said they will not over-subscribe a GPON port, so
all ONTs on the port can use 100% of their provisioned rate, and they will
not place more provision bandwidth on a single GPON chassis than what they
uplink can support.


Yes, it makes sense to have no or very small oversubscription of the GPON 
port. The aggregation uplink is another beast. In order to do statistical 
oversubscription, you need lots of customers so the statitics turn into an 
average that doesn't fluctuate too much.



loss(not a long time to sample, but was sampled at a rate of 10pps against
their speed-test server), but 20-40ms pings. I tested this during off
hours, like 1am.


Yep, sounds like they configured their queues correctly then.

Historically we had 600ms of buffering in backbone platforms in the end of 
the 90ties and beginning of 2000ds, then this has slowly eroded over time 
so now the "golden standard" is down to 30-50ms of buffer. There are 
platforms that have basically no buffer at all, such as 64 port 100G 
equipment with 12-24 megabyte of buffer. These are typically for DC 
deployment where RTTs are extremely low and special congestion algorithms 
are used. They don't work very well for ISP deployments.



to 1Gb/s. I asked them about this. Yes, the 1Gb tier was also not
over-subscribed. I'm not sure if some lone customer pretty much got their
own GPON port or they had some WDM-PON linecards.


Sounds like it, non-oversubscribing gig customers on GPON sounds 
expensive.


I'm currently paying about $40/m for 150/150 for a "dedicated" 
connection. I'm currently getting about 1ms+-0.1ms pings to my ISP's 
speedtest server 24/7.  If I do a ping flood, I can get my avg ping down 
near 0.12ms. I assume this is because of GPON scheduling. Of course I 
only test this against their speedtest server and during off hours.


Yes, sounds like GPON scheduler indeed.


to be load-balanced by some lower-bits in the IP address. This gave a total


Load-balancing can be done a lot of different ways. Typically for 
ISP-speak this is called "hashing", some do it on L3, some on L3/L4 
information, some do it even deeper into the packet.



of 6 links. The network admin told me that any given link had enough
bandwidth provisioned, that if all 5 other links were down, that one link
would have a 95th percentile below 80% during peak hours, and customers
should be completely unaffected.


That's a hefty margin. I'd say prudent is to make sure you can handle 
single fault without customer degradation. However, transit isn't that 
expensive so it might make sense.


what... 300Gb/s or so split among 32 customers?. As far as I can tell, 
last mile bandwidth is a solved problem short of incompetence, greed, or 
extreme circumstances.


Sure, it's not a technical problem. We have the technology. It's a money, 
politics, law, regulation and will problem. So yes, what you said.



Ahh yes. Statistical over-subscription was the topic. This works well for
backbone providers where they have many peering links with a heavy mix of
flows. Level 3 has a blog where they were showing off a 10Gb link where
below the 95th percentile, the link had zero total packets lost and a
queuing delay of less than 0.1ms. But above 80%, suddenly loss and jitter
went up with a hockey-stick curve. Then they showed a 400Gb link. It was at
98% utilization for the 95th percentile and it had zero total packets lost
and a max queuing delay of 0.01ms with an average of 0.00ms.


Yes, this is still true today as it was 2002 when this presentation was 
done:


https://internetdagarna.se/arkiv/2002/15-bygganat/id2002-peter-lothberg.pdf

(slide 44-47). The speeds have only increased, but this premise still is 
the same.



There was a major European IX that had a blog about bandwidth planning and
over-provisioning. They had a 95th percentile in the many-terabits, and
they said they said they could always predict peak bandwidth to within 1%
for any given day. Given a large mix of flow types, statistics is very good.


Indeed, the bigger the aggreation, the bigger the statistics show same 
behaviour every day.



On a slightly different topic, I wonder what trunk providers are using for
AQMs. My ISP was under a massive DDOS some time in the past year and I use
a Level 3 looking glass from Chicago, which showed only a 40ms delta
between the pre-hop and hitting my ISP, where it was normally about 11ms
for th

Re: [Cerowrt-devel] [Bloat] DC behaviors today

[Joel Wirāmu Pauling]

As I am writing up my slide-pack for LCA2018 this reminded me to test out
irtt sleep bench against my running system.

Seems either the Skylake Parts are much better in Combination with current
kernels at this than what you were running on - what is the kernel of the
x86 result?
---
aenertia@kiorewha:~/go/bin$ inxi -C
CPU:   Quad core Intel Core i7-7700K (-HT-MCP-) cache: 8192 KB
   clock speeds: max: 4700 MHz 1: 4603 MHz 2: 4616 MHz 3: 4600 MHz
4: 4602 MHz 5: 4601 MHz 6: 4612 MHz
   7: 4601 MHz 8: 4600 MHz
aenertia@kiorewha:~/go/bin$ uname -a
Linux kiorewha 4.15.0-rc7+ #12 SMP Tue Jan 16 20:16:35 NZDT 2018 x86_64
x86_64 x86_64 GNU/Linux
aenertia@kiorewha:~/go/bin$ irtt sleep
Testing sleep accuracy...

Sleep DurationMean Error   % Error
   1ns 245ns   24586.2
  10ns 234ns2345.1
 100ns  10.272µs   10272.9
   1µs  52.995µs5299.6
  10µs  53.189µs 531.9
 100µs  53.926µs  53.9
   1ms  80.973µs   8.1
  10ms  86.933µs   0.9
 100ms  86.563µs   0.1
 200ms  66.967µs   0.0
 500ms  64.883µs   0.0


On 13 December 2017 at 07:36, Dave Taht  wrote:

>
> Luca Muscariello  writes:
>
> > I think everything is about response time, even throughput.
> >
> > If we compare the time to transmit a single packet from A to B, including
> > propagation delay, transmission delay and queuing delay,
> > to the time to move a much larger amount of data from A to B we use
> throughput
> > in this second case because it is a normalized
> > quantity w.r.t. response time (bytes over delivery time). For a single
> > transmission we tend to use latency.
> > But in the end response time is what matters.
> >
> > Also, even instantaneous throughput is well defined only for a time
> scale which
> > has to be much larger than the min RTT (propagation + transmission
> delays)
> > Agree also that looking at video, latency and latency budgets are better
> > quantities than throughput. At least more accurate.
> >
> > On Fri, Dec 8, 2017 at 8:05 AM, Mikael Abrahamsson 
> wrote:
> >
> > On Mon, 4 Dec 2017, dpr...@reed.com wrote:
> >
> > I suggest we stop talking about throughput, which has been the
> mistaken
> > idea about networking for 30-40 years.
> >
> >
> > We need to talk both about latency and speed. Yes, speed is talked
> about too
> > much (relative to RTT), but it's not irrelevant.
> >
> > Speed of light in fiber means RTT is approx 1ms per 100km, so from
> Stockholm
> > to SFO my RTT is never going to be significantly below 85ms (8625km
> great
> > circle). It's current twice that.
> >
> > So we just have to accept that some services will never be
> deliverable
> > across the wider Internet, but have to be deployed closer to the
> customer
> > (as per your examples, some need 1ms RTT to work well), and we need
> lower
> > access latency and lower queuing delay. So yes, agreed.
> >
> > However, I am not going to concede that speed is "mistaken idea about
> > networking". No amount of smarter queuing is going to fix the
> problem if I
> > don't have enough throughput available to me that I need for my
> application.
>
> In terms of the bellcurve here, throughput has increased much more
> rapidly than than latency has decreased, for most, and in an increasing
> majority of human-interactive cases (like video streaming), we often
> have enough throughput.
>
> And the age old argument regarding "just have overcapacity, always"
> tends to work in these cases.
>
> I tend not to care as much about how long it takes for things that do
> not need R/T deadlines as humans and as steering wheels do.
>
> Propigation delay, while ultimately bound by the speed of light, is also
> affected by the wires wrapping indirectly around the earth - much slower
> than would be possible if we worked at it:
>
> https://arxiv.org/pdf/1505.03449.pdf
>
> Then there's inside the boxes themselves:
>
> A lot of my struggles of late has been to get latencies and adaquate
> sampling techniques down below 3ms (my previous value for starting to
> reject things due to having too much noise) - and despite trying fairly
> hard, well... a process can't even sleep accurately much below 1ms, on
> bare metal linux. A dream of mine has been 8 channel high quality audio,
> with a video delay of not much more than 2.7ms for AR applications.
>
> For comparison, an idle quad core aarch64 and dual core x86_64:
>
> root@nanopineo2:~# irtt sleep
>
> Testing sleep accuracy...
>
> Sleep DurationMean Error   % Error
>
>1ns  13.353µs 1335336.9
>
>   10ns   14.34µs  143409.5
>
>  100ns  13.343µs   13343.9
>
>1µs  12.791µs1279.2
>
> 

Re: [Cerowrt-devel] [Bloat] DC behaviors today

[Luca Muscariello]

I think everything is about response time, even throughput.

If we compare the time to transmit a single packet from A to B, including
propagation delay, transmission delay and queuing delay,
to the time to move a much larger amount of data from A to B we use
throughput in this second case because it is a normalized
quantity w.r.t. response time (bytes over delivery time). For a single
transmission we tend to use latency.
But in the end response time is what matters.

Also, even instantaneous throughput is well defined only for a time scale
which has to be much larger than the min RTT (propagation +  transmission
delays)
Agree also that looking at video, latency and latency budgets are better
quantities than throughput. At least more accurate.










On Fri, Dec 8, 2017 at 8:05 AM, Mikael Abrahamsson  wrote:

> On Mon, 4 Dec 2017, dpr...@reed.com wrote:
>
> I suggest we stop talking about throughput, which has been the mistaken
>> idea about networking for 30-40 years.
>>
>
> We need to talk both about latency and speed. Yes, speed is talked about
> too much (relative to RTT), but it's not irrelevant.
>
> Speed of light in fiber means RTT is approx 1ms per 100km, so from
> Stockholm to SFO my RTT is never going to be significantly below 85ms
> (8625km great circle). It's current twice that.
>
> So we just have to accept that some services will never be deliverable
> across the wider Internet, but have to be deployed closer to the customer
> (as per your examples, some need 1ms RTT to work well), and we need lower
> access latency and lower queuing delay. So yes, agreed.
>
> However, I am not going to concede that speed is "mistaken idea about
> networking". No amount of smarter queuing is going to fix the problem if I
> don't have enough throughput available to me that I need for my application.
>
> --
> Mikael Abrahamssonemail: swm...@swm.pp.se
> ___
> Bloat mailing list
> bl...@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>
___
Cerowrt-devel mailing list
Cerowrt-devel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel

Re: [Cerowrt-devel] [Bloat] DC behaviors today

[Matthias Tafelmeier]

>
> Probably, me forming some papers wrapping this up would be worthwile.
>
> [1]https://phys.org/news/2017-08-high-bandwidth-capability-ships.html
>
> [2]https://arxiv.org/pdf/1705.10630.pdf
>
I couldn't refrain. Currently, I've no affiliation rights, therefore I'm
not eligible to push up to arxiv.org or equivalent: this I why I went
for a blog article here [1] in favour of a paer. It's mediocre, thin,
yes ... nevertheless, was astounded during recherche that the idea is
actually already ~ 15 y. old [2]. Back then, this was really an
'intellectual step'.


[1]
https://matthias0tafelmeier.wordpress.com/2017/12/26/speed-of-light-in-the-air-towards-an-airborne-internet-backbone

[2] 
https://www.researchgate.net/publication/224793937_Stratospheric_Optical_Inter-Platform_Links_for_High_Altitude_Platforms
 

-- 
Besten Gruß

Matthias Tafelmeier



0x8ADF343B.asc
Description: application/pgp-keys


signature.asc
Description: OpenPGP digital signature
___
Cerowrt-devel mailing list
Cerowrt-devel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel

Re: [Cerowrt-devel] [Bloat] DC behaviors today

[Matthias Tafelmeier]

> I tend not to care as much about how long it takes for things that do
> not need R/T deadlines as humans and as steering wheels do.
>
> Propigation delay, while ultimately bound by the speed of light, is also
> affected by the wires wrapping indirectly around the earth - much slower
> than would be possible if we worked at it:
>
> https://arxiv.org/pdf/1505.03449.pdf

Enchanting that somebody actually quantified this intricacy!

*My* Addendum/Errata:

The alternative to a 'fast lane' backbone is not necessarily a mast
based microwave link as stated, which is probably infeasible/inflexible etc.

They were mentioning 'weather-balloons' as well- which actually boils
down to - I'm presuming this - the probably ongoing airborne platform
internet extension attempts by Goggle/Lockheed Martin etc., you call
them ...

These attempts are not based on balloons, but airships
('dirigible'-balloons so to speak), being able to stay aloft for
potentially years. That's widely known and so let's get them away, not
out for that.

NB. Airships are quite impressive for its own and therefore worth bein
recherched for!!!

What I actually wanted to posit in relation to that is that one could
get sooner a c-cabable backbone sibling by marrying two ideas: the
airborne concept ongoing as outlined plus what NASA is planning to bring
about for the space backbone, e.g [1][2]. It's laser based instead of
directed radio-wave only. Sure, both is in the speed range of c,
apparantely, laser transmission has in addition a significantly higher
bandwidth to offer. "10 to 100 times as much data at a time as
radio-frequency systems"[3]. Attenuations to photons in clean
atmospheric air are neglible (few mps - refractive index of about
1.0003), so actually a neglible slowdown - easily competing with top
notch fibres (99.7% the vacuum speed of light). Sure, that's the ideal
case, though, if cleverly done from the procurement of platforms and
overall system steering perspective, might feasible.


[1]
https://www.nasa.gov/feature/goddard/2017/nasa-taking-first-steps-toward-high-speed-space-internet

[2]
https://www.nasa.gov/feature/new-solar-system-internet-technology-debuts-on-the-international-space-station

[3]
https://www.nasa.gov/feature/goddard/2017/tdrs-an-era-of-continuous-space-communications

-- 
Besten Gruß

Matthias Tafelmeier



0x8ADF343B.asc
Description: application/pgp-keys


signature.asc
Description: OpenPGP digital signature
___
Cerowrt-devel mailing list
Cerowrt-devel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel

Re: [Cerowrt-devel] [Bloat] DC behaviors today

[Matthias Tafelmeier]

>
>> What I actually wanted to posit in relation to that is that one could
>> get sooner a c-cabable backbone sibling by marrying two ideas: the
>> airborne concept ongoing as outlined plus what NASA is planning to
>> bring about for the space backbone, e.g [1][2]. It's laser based
>> instead of directed radio-wave only. Sure, both is in the speed range
>> of c, apparantely, laser transmission has in addition a significantly
>> higher bandwidth to offer. "10 to 100 times as much data at a time as
>> radio-frequency systems"[3]. Attenuations to photons in clean
>> atmospheric air are neglible (few mps - refractive index of about
>> 1.0003), so actually a neglible slowdown - easily competing with top
>> notch fibres (99.7% the vacuum speed of light). Sure, that's the
>> ideal case, though, if cleverly done from the procurement of
>> platforms and overall system steering perspective, might feasible.
>
> Todays laser links are in the few km per hop range, with is easily at
> least one magnitude shorter than radio based equivalents.
>
Hold on! This is a severe oversimplifcation, isn' it. The devices you're
probably referring to are in the low-end segment, dillentically and
maybe terrestrially operated only - to mention a few limiting factor
conceivable possibly being perceived.

Certainly, there are range limiting factors when fully submerged in the
near-ground atmospheric ranges. E.g. in the darkest snow storm, one
cannot expect optics to be reliablly working - admitting that.
Nothwithstanding, recent research[1] showed astounding achievements of
FSOs even in harsh atmospheric conditions - "up to 10 gigabits per
second" while in vivid movement, in heavy fog ... for a single pathed laser.

90% mass of the atmosphere  is below 16 km (52,000 ft), therefore also
most of it's randomness[2]. Meaning, one only had to surpass this
distance to more decently unfold the capabilities of an airborne
backbone. Therefore, a hierarchy of airborne vessels might be necessary.
Might smaller, more numerous ones gatewaying the optics out of the dense
parts of the atmosphere to the actual backbone-net borne lasers, might
by doing this relaying not laser beam based. Far more mitigation
techniques are conceivable. From there on, the shortcomings appear
controllable.

> I don't know the physics behind it, but people who have better insight
> than I do tell me "it's hard" to run longer hops (if one wants any
> kind of high bitrate).
If one looks up what is achievable in space, where the conditions
shouldn't be too different from earth atmosphere over 16 km. Thousands
of kilometres for a single hop, single path. Now imagine a decent degree
of multipathing.

Physical intricacies are certainly a headache in this topic, though
shouldn't be decisive, I'd dare to categorize the largest complexity
compartment of such a system into the algorithmics for steering,
converging or stabilizing the airborne components, directing the optics
problerly and in time. The overall automatic or even autonomic
operations to abstract it.

Probably, me forming some papers wrapping this up would be worthwile.

[1]https://phys.org/news/2017-08-high-bandwidth-capability-ships.html

[2]https://arxiv.org/pdf/1705.10630.pdf

-- 
Besten Gruß

Matthias Tafelmeier



0x8ADF343B.asc
Description: application/pgp-keys


signature.asc
Description: OpenPGP digital signature
___
Cerowrt-devel mailing list
Cerowrt-devel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel

Re: [Cerowrt-devel] [Bloat] DC behaviors today

[Luca Muscariello]

+1 on all.
Except that Little's Law is very general as it applies to any ergodic
process.
It just derives from the law of large numbers. And BTW, Little's law is a
very powerful law.
We use it unconsciously all the time.



On Tue, Dec 12, 2017 at 11:53 PM,  wrote:

> Luca's point tends to be correct - variable latency destroys the stability
> of flow control loops, which destroys throughput, even when there is
> sufficient capacity to handle the load.
>
>
>
> This is an indirect result of Little's Lemma (which is strictly true only
> for Poisson arrival, but almost any arrival process will have a similar
> interaction between latency and throughput).
>
>
>
> However, the other reason I say what I say so strongly is this:
>
>
>
> Rant on.
>
>
>
> Peak/avg. load ratio always exceeds a factor of 10 or more, IRL. Only
> "benchmark setups" (or hot-rod races done for academic reasons or marketing
> reasons to claim some sort of "title") operate at peak supportable load any
> significant part of the time.
>
>
>
> The reason for this is not just "fat pipes are better", but because
> bitrate of the underlying medium is an insignificant fraction of systems
> operational and capital expense.
>
>
>
> SLA's are specified in "uptime" not "bits transported", and a clogged pipe
> is defined as down when latency exceeds a small number.
>
>
>
> Typical operating points of corporate networks where the users are happy
> are single-digit percentage of max load.
>
>
>
> This is also true of computer buses and memory controllers and storage
> interfaces IRL. Again, latency is the primary measure, and the system never
> focuses on operating points anywhere near max throughput.
>
>
>
> Rant off.
>
>
> On Tuesday, December 12, 2017 1:36pm, "Dave Taht"  said:
>
> >
> > Luca Muscariello  writes:
> >
> > > I think everything is about response time, even throughput.
> > >
> > > If we compare the time to transmit a single packet from A to B,
> including
> > > propagation delay, transmission delay and queuing delay,
> > > to the time to move a much larger amount of data from A to B we use
> > throughput
> > > in this second case because it is a normalized
> > > quantity w.r.t. response time (bytes over delivery time). For a single
> > > transmission we tend to use latency.
> > > But in the end response time is what matters.
> > >
> > > Also, even instantaneous throughput is well defined only for a time
> scale
> > which
> > > has to be much larger than the min RTT (propagation + transmission
> delays)
> > > Agree also that looking at video, latency and latency budgets are
> better
> > > quantities than throughput. At least more accurate.
> > >
> > > On Fri, Dec 8, 2017 at 8:05 AM, Mikael Abrahamsson 
> > wrote:
> > >
> > > On Mon, 4 Dec 2017, dpr...@reed.com wrote:
> > >
> > > I suggest we stop talking about throughput, which has been the
> > mistaken
> > > idea about networking for 30-40 years.
> > >
> > >
> > > We need to talk both about latency and speed. Yes, speed is talked
> about
> > too
> > > much (relative to RTT), but it's not irrelevant.
> > >
> > > Speed of light in fiber means RTT is approx 1ms per 100km, so from
> > Stockholm
> > > to SFO my RTT is never going to be significantly below 85ms (8625km
> > great
> > > circle). It's current twice that.
> > >
> > > So we just have to accept that some services will never be deliverable
> > > across the wider Internet, but have to be deployed closer to the
> > customer
> > > (as per your examples, some need 1ms RTT to work well), and we need
> > lower
> > > access latency and lower queuing delay. So yes, agreed.
> > >
> > > However, I am not going to concede that speed is "mistaken idea about
> > > networking". No amount of smarter queuing is going to fix the problem
> if
> > I
> > > don't have enough throughput available to me that I need for my
> > application.
> >
> > In terms of the bellcurve here, throughput has increased much more
> > rapidly than than latency has decreased, for most, and in an increasing
> > majority of human-interactive cases (like video streaming), we often
> > have enough throughput.
> >
> > And the age old argument regarding "just have overcapacity, always"
> > tends to work in these cases.
> >
> > I tend not to care as much about how long it takes for things that do
> > not need R/T deadlines as humans and as steering wheels do.
> >
> > Propigation delay, while ultimately bound by the speed of light, is also
> > affected by the wires wrapping indirectly around the earth - much slower
> > than would be possible if we worked at it:
> >
> > https://arxiv.org/pdf/1505.03449.pdf
> >
> > Then there's inside the boxes themselves:
> >
> > A lot of my struggles of late has been to get latencies and adaquate
> > sampling techniques down below 3ms (my previous value for starting to
> > reject things due to having too much noise) - and despite trying fairly
> > hard, well... a process can't even sleep accurately much below 1ms, on
> > bare metal linux. A dream of