Jake,
On 20/03/2019 19:04, Holland, Jake wrote:
Hi Bob & Greg,
I agree there has been a reasonably open conversation about the L4S
work, and thanks for all your efforts to make it so.
However, I think there's 2 senses in which "private" might be fair that
I didn't see covered in your rebuttals (merging forks and including
Greg's rebuttal by reference from here:
https://lists.bufferbloat.net/pipermail/bloat/2019-March/009038.html )
Please note:
I don't consider these senses of "private" a disqualifying argument
against the use of L4S, though I do consider them costs that should be
taken into account (and of course opinions may differ here).
Thanks for engaging in this.
I do also intend to address one or two other recurring questions on the
bloat list, but I have a lot on at the mo.
With that said, I wondered whether either of you have any responses that
speak to these points:
1. the L4S use of the ECT(1) codepoint can't be marked CE except by a
patent-protected AQM scheduler.
I understand that l4s-id suggests the possibility of an alternate
scheme. However, comparing the MUSTs of the section 5 requirements
with the claims made by the patent seems to leave no room for an
alternate that would not infringe the patent claims, unless I'm missing
something?
https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-06#section-5
https://patents.google.com/patent/US20170019343A1/en
1/ In 2016, I arranged for the hire of a patent attorney to undertake
the unusual step of filing a third party observation with the European
Patent Office. This went through Al-Lu's patent application claim by
claim pointing to prior art and giving the patent examiner all the
arguments to reject each claim. However, the examiner chose to take very
little note of it, which was disappointing and costly for us. The main
prior art is:
Gibbens, R.J. & Kelly, F.P., "On Packet Marking at Priority
Queues," IEEE Transactions on Automatic Control 47(6):1016--1020 (June 2002)
The guys named as inventors in AL-Lu's filing published a paper on PI2
with me, in which we included a citation of this Gibbens paper as
inspiration for the coupling. The Gibbens paper was already cited as
background by other patents, so the EPO has it in their citation index.
The coupling was also based on my prior research with Mirja before I
started working with the guys from Al-Lu in the RITE European
Collaborative project. we had to go through a few rejections, but Mirja
and I finally got this work published in 2014 - still before the
priority date of the Al-Lu patent application:
Kühlewind, M., Wagner, D.P., Espinosa, J.M.R. & Briscoe, B., "Using
Data Center TCP (DCTCP) in the Internet," In: Proc. Third IEEE Globecom
Workshop on Telecommunications Standards: From Research to Standards
pp.583-588 (December 2014)
2/ The only claim that I could not find prior art for (in the original
EU filing) was a very specific claim about using a square root for the
coupling. The Linux implementation runs this the other way round so that
it only has to do a squaring. So I figured we were safe from that.
However, until just now, I had not noticed that Al-Lu has
retrospectively re-written the claims in the US patent and in the EU
patent application to claim this the other way round - as a squaring.
And to claim the two random number trick. Both restructuring to use a
squaring and the two random number trick were definitely my ideas (while
working for BT in a collaboration with Al-Lu). I have emails to prove
this (from memory they were actually both in the same email). This is
important, because a patent has to be about mechanism, not algorithm.
3/ This is a positive development. It means this patent is on very shaky
legal ground. I have been trying to put pressure on Nokia to license
this royalty free. But now I see what they have done, I am going to have
to get a different type of legal advice.
2. the L4S use of the ECT(1) codepoint privileges the low latency use
case.
As Jonathan Morton pointed out, low latency is only one of several
known use cases that would be worthwhile to identify to an AQM
scheduler, which the L4S approach cannot be extended to handle:
- Minimize Latency
- Minimize Loss
- Minimize Cost
- Maximize Throughput
https://lists.bufferbloat.net/pipermail/ecn-sane/2019-March/000066.html
I understand that "Minimize Cost" is perhaps covered by LEPHB, and that
operator tuning parameters for a dualq node can possibly allow for
tuning between minimizing loss and latency, as mentioned in section
4.1 of aqm-dualq, but since the signaling is bundled together, it can
only happen for one at a time, if I'm reading it right.
Not at all. There is a reason why it's called Low Latency, Low Loss,
Scalable throughput (L4S).
The L4S definition of ECN marking addresses all four of these at once.
Strictly it directly addresses all except minimize cost, but minimize
cost can be built around the system. This was the subject of my PhD. I
haven't described L4S in these terms, because most people are only
interested in the latency. But this is the underlying reason for my
obsession with ECN.
Frank Kelly predicted that queuing delay would be removed from the
optimization as it was minimized. With L4S we've got very close to that.
ECN removes all congestion loss.
And the use of a inverse linear congestion controller gives the scalable
throughput. I shall be touching on this in my talk for netdev tomorrow,
but it's not really a subject for an implementation conference.
Minimize cost is something you do by combining the congestion signals
across a network. So any AQM is part of that. And congestion controllers
are the other part - they implicitly optimize cost, using the congestion
signals as shadow prices. The square root in classic TCP distorts this,
but DCTCP's inverse linear controller gives proportional fairness
directly. Without a weight term in the congestion controller, there is
not really an economic optimization, but that can be built onto a
proportionally fair system and competition will gradually cause that to
happen (or regulation as a proxy for competition). These are very long
term processes though.
But more importantly, the L4S usage couples the minimized latency use
case to any possibility of getting a high fidelity explicit congestion
signal, so the "maximize throughput" use case can't ever get it.
Eh? There's definitely a misunderstanding or a difference in terminology
between us here. The whole point of using a congestion controller like
DCTCP is so that flow rate can scale indefinitely with capacity. Van
Jacobson actually noted that the original TCP was unscalable in a
footnote to the tech report version of the SIGCOMM paper.
The high fidelity congestion signal of what we call scalable congestion
controllers (like DCTCP) is inversely proportional to the window. So as
window scales up, the congestion signal scales down, so that their
product remains constant. That means the number of ECN marks per RTT is
scale-invariant. So the control signal remains just as tight at any scale.
Cheers
Bob
Regards,
Jake
PS:
If you get a chance, I'm still interested in seeing answers to my
questions about deployment mitigations on the tsvwg list:
https://mailarchive.ietf.org/arch/msg/tsvwg/TWOVpI-SvVsYVy0_U6K8R04eq3A
I'm not surprised if it slipped by unnoticed, there have been a lot of
emails on this. But good answers to those questions would go a long way
toward easing my concerns about the urgency on this discussion.
PPS:
These issues are a bit sideways to my technical reasons for preferring
the SCE formulation of ECT(1), which have more to do with the confusing
semantics and proliferation of corner cases it creates for CE and ECE.
But I thought I’d ask about them since it seemed like maybe there was a
disconnect here.
*From: *Bob Briscoe <i...@bobbriscoe.net>
*Date: *2019-03-18 at 18:07
*To: *"David P. Reed" <dpr...@deepplum.com>, Vint Cerf <v...@google.com>
*Cc: *tsvwg IETF list <ts...@ietf.org>, bloat
<bloat@lists.bufferbloat.net>
*Subject: *Re: [Bloat] [Ecn-sane] [iccrg] Fwd: [tcpPrague]
Implementation and experimentation of TCP Prague/L4S hackaton at IETF104
David,
On 17/03/2019 18:07, David P. Reed wrote:
Vint -
BBR is the end-to-end control logic that adjusts the source rate
to match the share of the bolttleneck link it should use.
It depends on getting reliable current congestion information via
packet drops and/or ECN.
So the proposal by these guys (not the cable guys) is an attempt
to improve the quality of the congestion signal inserted by the
router with the bottleneck outbound link.
What do you mean 'not the cable guys'?
This thread was reasonably civil until this intervention.
THe cable guys are trying to get a "private" field in the IP
header for their own use.
There is nothing private about this codepoint, and there never has
been. Here's some data points:
* The IP header codepoint in question (ECT(1) in the ECN field) was
proposed for use as an alternative ECN behaviour in July 2105 in the
IETF AQM WG and the IETF's transport area WG (which handles all ECN
matters).
* A year later there followed a packed IETF BoF on the subject (after
2 open Bar BoFs).
* Long discussion ensued on the merits of different IP header field
combinations, on both these IETF lists, involving people active on
this list (bloat), including Dave Taht, who is acknowledged for his
contributions in the IETF draft.
* That was when it was decided that ECT(1) was most appropriate.
* The logic of the decision is written up in an appendix of
draft-ietf-ecn-l4s-id.
* David Black, one of the co-chairs of the IETF's transport area WG
and co-author of both the original ECN and Diffserv RFCs, wrote
RFC8311 to lay out the process for reclaiming and reusing the
necessary codepoints.
* This work and the process of freeing up codepoints has been very
visible at every IETF ever since, with multiple drafts to fix other
aspects of the protocols working their way through the IETF process in
multiple WGs (tsvwg, tcpm, trill, etc).
* L4S has also been mentioned in IETF liaisons with the IEEE and 3GPP.
Some history:
* I had been researching the idea since 2012.
* In fact my first presentation on it was scheduled directly after Van
Jacobson's first presentation of CoDel at the IETF in July 2012. VJ
overran by nearly 20mins leaving just 3 mins for my presentation.
* Mirja Kuehlewind and I did early groundwork in 2013 and published a
paper
* Then I (working for BT) brought the work into the EU RITE project
(Reducing Internet Transport Latency) with Simula, Alcatel-Lucent, etc.
* By 2015 the two main L4S proponents were Koen De Schepper from
Alcatel Lucent and myself (I had just switched from BT to Simula),
along with Olga Bondarenko (now Albisser), a PhD student at Simula who
now works for Microsoft (on something else) and is still doing her PhD
part-time with Simula
o By that time, Al-Lu and Simula had a cool prototype running.
o This was demonstrated publicly for the first time in the IETF AQM
WG over DC+core+backhaul+DSL+home networks.
https://riteproject.eu/dctth/#1511dispatchwg
<https://urldefense.proofpoint.com/v2/url?u=https-3A__riteproject.eu_dctth_-231511dispatchwg&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=W5ZSTVXb4iSChTS8-sSOHWDszX3AitVf8Qwh-dXbqCY&e=>
* In May 2016, L4S was demonstrated at MultiMediaSystems'16 with
/every/ packet from all the following simultaneous applications
achieving ~1ms queuing delay or less over a 40Mb/s Internet access
link (7ms base RTT):
o cloud-rendered remote presence in a racing car within a VR headset
o the interactive cloud-rendered video already linked above
o an online gaming benchmark
o HAS video streaming
o a number of bulk file downloads
o a heavy synthetic load of web browsing
L4S has never been access-technology-specific. Indeed the cable
industry has been funding my work at the IETF to help encourage a
wider L4S ecosystem. There is nothing private to the cable industry in
this:
* Al-Lu used DSL as a use-case, but L4S was relevant to all the access
technologies they supplied.
* BT was obviously interested in DSL,
* but BT's initial motivating use-case was to incrementally deploy the
low queuing delay of DCTCP over BT's data centre interconnect networks.
* In Jul 2016 the open-source Linux repo for the Coupled AQM was
published, with a fully working version to be used and abused.
Now at: https://github.com/L4STeam/sch_dualpi2_upstream
<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_L4STeam_sch-5Fdualpi2-5Fupstream&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=IrFWAYZca2EEiXNTyliUfh3DgYYEyvabNTq8xYIQjBQ&e=>
* Of course, DCTCP was already open-sourced in Linux and FreeBSD, as
well as available in Windows
* In Jul 2016, the main IETF BoF on L4S was held:
o Ingemar Johansson from Ericsson was one of the proponents, focused
on using L4S in LTE
o along with Kevin Smith from Vodafone and
o Praveen Balasubramanian from Microsoft (who maintains the Windows
TCP stack, including DCTCP).
o Ingemar has since written an open-source L4S variant of the SCReAM
congestion controller for real-time media:
https://github.com/EricssonResearch/scream/
<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_EricssonResearch_scream_&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=KudwyCeLp1jJbSm0Qv-Rm45UKacU0Q0rtT_Kca9Z2uA&e=>
o Mirja Kuehlewind of ETHZ (and now Ericsson) implemented the
necessary feedback in TCP https://github.com/mirjak/linux-accecn
<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_mirjak_linux-2Daccecn&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=8xmJipLHdxCtcbf-ZSYOZUWjzgNd8p0dF4XTOe-Lwxo&e=>
* In summer 2017 CableLabs started work on Low Latency DOCSIS, and
hired me later in the year to help develop and specify it, along with
support for L4S
o In Jan 2019 the Low Latency DOCSIS spec was published and is now
being implemented.
o You can find the primary companies involved at the end of the
White Paper.
https://cablela.bs/low-latency-docsis-technology-overview-february-2019
<https://urldefense.proofpoint.com/v2/url?u=https-3A__cablela.bs_low-2Dlatency-2Ddocsis-2Dtechnology-2Doverview-2Dfebruary-2D2019&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=rAKo34ElWnLOIk827MWT75KG3rrRmc6dM3UaTtC9VBc&e=>
o Operators:
Liberty Global
Charter
Rogers
Comcast
Shaw
Cox Communications
o Equipment vendors
ARRIS
Huawei
Broadcom
Intel
Casa
Nokia
Cisco
Videotron
* Nicolas Kuhn of CNES has been assessing the use of L4S for satellite.
* Magnus Westerlund of Ericsson with a team of others has written the
necessary ECN feedback into QUIC
* L4S hardware is also being implemented for hi-speed switches at the
moment
(the developer wants to announce it themselves, so I have been
asked not to identify them).
There's a lot more stuff been going on, but I've tried to pick out
highlights.
All this is good healthy development of much lower latency for
Internet technology.
I find it extremely disappointing that some people on this list are
taking such a negative attitude to the major development in their own
field that they seem not to have noticed since it first hit the
limelight in 2015.
L4S has been open-sourced since 2016 so that everyone can develop it
and make it better...
If I was in this position, having overlooked something important for
multiple years, I would certainly not condemn it while I was trying to
understand what it was and how it worked. Can I suggest everyone takes
a step back, and suspends judgement until they have understood the
potential, the goals and the depth of what they have missed. People
who know me, know that I am very careful with Internet architecture,
and particularly with balancing public policy against commercial
issues. Please presume respect unless proven otherwise.
Best Regards
Bob
PS. Oh and BBR would be welcome to use the ECT(1) codepoint to get
into the L4S queue. But only if it can keep latency down below around
1ms. Currently those ~15-25ms delay spikes would not pass muster.
Indeed, its delay is not consistently low enough between the spikes
either.
-----Original Message-----
From: "Vint Cerf" <v...@google.com> <mailto:v...@google.com>
Sent: Saturday, March 16, 2019 5:57pm
To: "Holland, Jake" <jholl...@akamai.com> <mailto:jholl...@akamai.com>
Cc: "Mikael Abrahamsson" <swm...@swm.pp.se>
<mailto:swm...@swm.pp.se>, "David P. Reed" <dpr...@deepplum.com>
<mailto:dpr...@deepplum.com>, "ecn-s...@lists.bufferbloat.net"
<mailto:ecn-s...@lists.bufferbloat.net>
<ecn-s...@lists.bufferbloat.net>
<mailto:ecn-s...@lists.bufferbloat.net>, "bloat"
<bloat@lists.bufferbloat.net> <mailto:bloat@lists.bufferbloat.net>
Subject: Re: [Ecn-sane] [Bloat] [iccrg] Fwd: [tcpPrague]
Implementation and experimentation of TCP Prague/L4S hackaton at
IETF104
where does BBR fit into all this?
v
On Sat, Mar 16, 2019 at 5:39 PM Holland, Jake <jholl...@akamai.com
<mailto:jholl...@akamai.com>> wrote:
On 2019-03-15, 11:37, "Mikael Abrahamsson" <swm...@swm.pp.se
<mailto:swm...@swm.pp.se>> wrote:
L4S has a much better possibility of actually getting
deployment into the
wider Internet packet-moving equipment than anything being
talked about
here. Same with PIE as opposed to FQ_CODEL. I know it's
might not be as
good, but it fits better into actual silicon and it's
being proposed by
people who actually have better channels into the people
setting hard
requirements.
I suggest you consider joining them instead of opposing them.
Hi Mikael,
I agree it makes sense that fq_anything has issues when you're
talking
about the OLT/CMTS/BNG/etc., and I believe it when you tell me PIE
makes better sense there.
But fq_x makes great sense and provides real value for the
uplink in a
home, small office, coffee shop, etc. (if you run the final
rate limit
on the home side of the access link.) I'm thinking maybe
there's a
disconnect here driven by the different use cases for where
AQMs can go.
The thing is, each of these is the most likely congestion point at
different times, and it's worthwhile for each of them to be
able to
AQM (and mark packets) under congestion.
One of the several things that bothers me with L4S is that
I've seen
precious little concern over interfering with the ability for
another
different AQM in-path to mark packets, and because it changes the
semantics of CE, you can't have both working at the same time
unless
they both do L4S.
SCE needs a lot of details filled in, but it's so much cleaner
that it
seems to me there's reasonably obvious answers to all (or
almost all) of
those detail questions, and because the semantics are so much
cleaner,
it's much easier to tell it's non-harmful.
<aside regarding="non-harmful">
The point you raised in another thread about reordering is mostly
well-taken, and a good counterpoint to the claim "non-harmful
relative
to L4S".
To me it seems sad and dumb that switches ended up trying to make
ordering guarantees at cost of switching performance, because
if it's
useful to put ordering in the switch, then it must be equally
useful to
put it in the receiver's NIC or OS.
So why isn't it in all the receivers' NIC or OS (where it
would render
the switch's ordering efforts moot) instead of in all the
switches?
I'm guessing the answer is a competition trap for the switch
vendors,
plus "with ordering goes faster than without, when you
benchmark the
switch with typical load and current (non-RACK) receivers".
If that's the case, it seems like the drive for a competitive
advantage
caused deployment of a packet ordering workaround in the wrong
network
location(s), out of a pure misalignment of incentives.
RACK rates to fix that in the end, but a lot of damage is
already done,
and the L4S approach gives switches a flag that can double as
proof that
RACK is there on the receiver, so they can stop trying to
order those
packets.
So point granted, I understand and agree there's a cost to
abandoning
that advantage.
</aside>
But as you also said so well in another thread, this is
important. ("The
last unicorn", IIRC.) How much does it matter if there's a
feature that
has value today, but only until RACK is widely deployed? If
you were
convinced RACK would roll out everywhere within 3 years and
SCE would
produce better results than L4S over the following 15 years,
would that
change your mind?
It would for me, and that's why I'd like to see SCE explored
before
making a call. I think at its core, it provides the same
thing L4S does
(a high-fidelity explicit congestion signal for the sender),
but with
much cleaner semantics that can be incrementally added to
congestion
controls that people are already using.
Granted, it still remains to be seen whether SCE in practice
can match
the results of L4S, and L4S was here first. But it seems to
me L4S comes
with some problems that have not yet been examined, and that
are nicely
dodged by a SCE-based approach.
If L4S really is as good as they seem to think, I could
imagine getting
behind it, but I don't think that's proven yet. I'm not
certain, but
all the comparative analyses I remember seeing have been from
more or
less the same team, and I'm not convinced they don't have some
misaligned incentives of their own.
I understand a lot of work has gone into L4S, but this move to
jump it
from interesting experiment to de-facto standard without a
more critical
review that digs deeper into some of the potential deployment
problems
has me concerned.
If it really does turn out to be good enough to be permanent,
I'm not
opposed to it, but I'm just not convinced that it's
non-harmful, and my
default position is that the cleaner solution is going to be
better in
the long run, if they can do the same job.
It's not that I want it to be a fight, but I do want to end up
with the
best solution we can get. We only have the one internet.
Just my 2c.
-Jake
_______________________________________________
Ecn-sane mailing list
ecn-s...@lists.bufferbloat.net
<mailto:ecn-s...@lists.bufferbloat.net>
https://lists.bufferbloat.net/listinfo/ecn-sane
<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.bufferbloat.net_listinfo_ecn-2Dsane&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=6aOGTXQW4Yh0LX-6RMhcK4d8BixblVH6c-yIGT7IVS8&e=>
--
New postal address:
Google
1875 Explorer Street, 10th Floor
Reston, VA 20190
_______________________________________________
Bloat mailing list
Bloat@lists.bufferbloat.net <mailto:Bloat@lists.bufferbloat.net>
https://lists.bufferbloat.net/listinfo/bloat
<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.bufferbloat.net_listinfo_bloat&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=SQx_1nK8EWZnWRYRJfdA_I-eLl0XlKNoC6YRxjfVdkw&e=>
--
________________________________________________________________
Bob Briscoehttp://bobbriscoe.net/
<https://urldefense.proofpoint.com/v2/url?u=http-3A__bobbriscoe.net_&d=DwMDaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=bqnFROivDo_4iF8Z3R4DyNWKbbMeXr0LOgLnElT1Ook&m=zfPW2a1vuvsWS3Jyy_VK9_HR7vCG_9ICWuN2-7yuuPU&s=wzv0H2d7H27kBtLtx2XWzBZkzJ_s0BmWpPnMn9B7Pc8&e=>
--
________________________________________________________________
Bob Briscoe http://bobbriscoe.net/
_______________________________________________
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat
