On Mon, 13 Apr 2015, Bob Briscoe wrote:
David,
Returning from a fortnight offlist...
I think your conception of how ECN works is incorrect. You describe ECN as if
the AQM marks one packet when it drops another packet. You say that the
ECN-mark speeds up the retransmission of the dropped packet. On the contrary,
the idea of classic ECN [RFC3168] is that the ECN marks replace the drops. In
all known testing (except pathological cases), classic ECN effectively
eliminates drops for all ECN-capable packets.
That's what I thought, and if that was the case, then marking packets as
ECN-capable would mean that they would have an advantage over non-ECN packets
(by not getting dropped, so getting a higher share of bandwidth)
that's what the gaming ECN thread was about, and if I understood the responses,
I was being told that marking packets as ECN-capable, but not slowing down
(actually responding to ECN) would not let an application get any advantage
because the packets would just end up getting dropped anyway, since marking and
dropping happen at the same level, even on ECN-capable flows.
If the packets are just marked, but not dropped, then the ECN-capable flows will
occupy a disproportinate share of the available buffer space, since they just
get marked instead of dropped.
David Lang
Nonetheless, I do agree with your sentiment that the perfect is the enemy of
the good. We can remove most of the really bloat-induced latency without ECN.
So the message must be clear: Deploy AQM now. No need to wait for ECN. But
implementations SHOULD allow ECN packets to be classifed into a separately
configurable instance of the AQM algo. {Note 1}
Similarly, this WG made sure we did not deprecate RED in the AQM
recommendations. Because, in existing equipment, even a poorly tuned RED is
usually much better than a bloated buffer with no AQM.
Just as the WG mustn't confuse messages, you mustn't get confused by a
discussion about the potential for more ambitious reductions in latency. Koen
started the thread with reference to our presentation in the ICCRG in the
IRTF, where 'R' in both cases stands for Research.
And I believe it is valid for the ECN benefits draft (in the IETF AQM WG) to
point to the potential of ECN, by using insights from research in progress.
There are a number of different contributions to unnecessary latency, not
just the one popularised as bufferbloat:
[In the following, I will use the term 'short message(s)' as shorthand for
either a short interactive flow or a long flow consisting of short
interactive messages (like in a game), or video frames or voice datagrams or
anything where the perceived latency depends on the latency of each
'message', especially a string of messages with serial dependency as is
typical in Web.]
#1 The well-known bufferbloat problem, where a long-running flow fills a
bloated buffer, delays short messages.
- AQM and/or flow queuing can remove this delay, without needing ECN.
#2 A loss causes head-of-line blocking for short message(s) while waiting for
the retransmission.
- AQM without ECN cannot remove this delay.
- Flow queuing cannot remove this delay, if the losses are self-induced.
- FEC can remove this delay without needing ECN, but the increased
redundancy is equivalent to poorer utilisation (altho only the short flows
need the redundancy).
- ECN can remove this delay.
#3 A loss near the end of a short flow can lead to multi-RTT delay.
- AQM without ECN cannot remove this delay.
- Techniques like tail-loss probe and RTO-restart can mitigate this delay,
without ECN, but not remove it.
- ECN can remove this delay.
#4 The Reno/Cubic sawtooth causes variation in delay between 1 and 2 base
RTTs. This can affect short messages.
- AQM without ECN cannot remove this delay unless configured to sacrifice
utilisation.
- Flow queuing removes this delay variation if caused by a separate
long-running flow.{Note 2}
- A change to the TCP algo (e.g. DCTCP) can remove this delay variation.
Smaller sawteeth imply a much higher signalling rate, which in turn requires
ECN, otherwise drop probability would be excessive. (This was the main point
Koen was making.)
- Therefore, ECN can remove this delay.
#5 Slow-starts{Note 3} cause spikes in delay.
- AQM without ECN cannot remove this delay, and typically AQM is designed
to allow such bursts of delay in the hope they will disappear of their own
accord.
- Flow queuing can remove the effect of these delay bursts on other flows,
but only if it gives all flows a separate queue from the start.{Note 2}
- Delay-based softening of slow-start, such as Hybrid Slow-Start in Linux,
can mitigate these variations, but with increased risk of coming out of SS
early, causing significantly longer completion time.
- ECN with AQM based on the instantaneous queue limits this delay, without
the risk of longer completion time.
#6 Slow-starts{Note 3} can cause runs of losses, which in turn cause delays.
- AQM without ECN cannot remove these delays.
- Flow queuing cannot remove these losses, if self-induced.
- Delay-based SS like HSS can mitigate these losses, with increased risk of
longer completion time.
- ECN can remove these losses, and the consequent delays.
Summary:
* AQM alone solves the main problem
* Flow queuing solves or mitigates most of the remaining secondary problems.
* ECN has the potential to solve all the remaining secondary problems
(pending further research to prove some of them).
Whether flow queuing is applicable depends on the scale. The work I'm doing
with Koen is to reduce the cost of the queuing mechanisms on our BNGs
(broadband network gateways). We're trying to reduce the cost of per-customer
queuing at scale, so per-flow queuing is simply out of the question. Whereas
ECN requires no more processing than drop.
ECN has potential cheating problems, but we have per-customer queues anyway.
Using flow as the unit of allocation also has its own problems, with no
proposed solutions.
Bob
{Note 1}: Your general point is that the perfect can be the enemy of the
good. Here's the presentation I gave in TSVAREA straight after VJ presented
CoDel in 2012 entitled "DCTCP & CoDel; the Best is the Friend of the Good."
<http://www.bobbriscoe.net/presents/1207ietf/1207-tsvarea-dctcp.pdf>
{Note 2}: A lone flow can cause this delay variation to itself, but that's
irrelevant because, if the delay were not in the network it would be at the
sender.
{Note 3}: Delay and loss spikes can equivalently be caused when Cubic's
window rises to seek out newly available capacity after another flow finishes
or the link rate varies.
At 05:16 30/03/2015, David Lang wrote:
On Sat, 28 Mar 2015, Scheffenegger, Richard wrote:
David,
Perhaps you would care to provide some text to address the misconception
that you pointed out? (To wait for a 100% fix as a 90% fix appears much
less appealing, while the current state of art is at 0%)
Ok, you put me on the spot :-) Here goes.
If you think that aqm-recommendations is not strogly enough worded. I
think this particular discussion (to aqm or not) really belongs there. The
other document (ecn benefits) has a different target in arguing for going
those last 10%...
so here is my "elevator pitch" on the problem. Feel free to take anything I
say here for any purpose, and I'm sure I'll get corrected for anything I am
wong on
Problem statement: Transmit buffers are needed to keep the network layer
fully utilized, but excessive buffers result in poor latency for all
traffic. This latency is frequently bad enough to cause some types of
traffic to fail entirely.
<link to more background goes here, including how separate benchmarks for
throughput and latency have mislead people, "packet loss considered evil",
cheaper memory encouraging larger buffers, etc. Include tests like
netperf-wrapper and ping latency while under load, etc. Include examples
where buffers have resulted in latencies so long that packets are
retransmitted before the first copy gets to the destination>
Traditionally, transmit buffers have been sized to handle a fixed number of
packets. Due to teh variation in packet sizes, it is impossible to tune
this value to both keep the link fully utilized when small packets dominate
the trafific without having the queue size be large enough to cause latency
problems when large packets dominate the traffic.
Shifting to Byte Queue Lengths where queues are allowed to hold a variable
number of packets depending on how large they are makes it possible to
manually tune the transmit buffer size to get good latency under all
traffic conditions at a given speed. However, this step forward revealed
two additional problems.
1. whenever the data rate changes, this value needs to be manually changed
(multi-link paths loose a link, noise degrades max throughput on a link,
etc)
2. high volume flows (i.e. bulk downloads) can starve other flows (DNS
lookups, VoIP, Gaming, etc). this happens because space in tue queue is on
a first-com-first-served basis, so the high-volume traffic fills the queue
(at which point it starts to be dropped), but all other traffic that tries
to arrive is also dropped. It turns out that these light flows tend to have
a larger effect on the user experience than heavier flows, because things
tend to be serialized behind the lighter flows (DNS lookup before doing a
large download, retrieving a small HTML page to find what additional
resources need to be fetched to display a page), or the user experience is
directly effected by light flows (gaming lag, VoIP drops, etc)
Active Queue Management addresses these problems by adapting the amount of
data that is buffered to match the data transmission capacity, and prevents
high volume flows from starving low-volume flows without the need to
implement QoS classifications.
<insert link about how you can't trust QoS tags that are made by other
organizations, ways that it can be abused, etc>
This is possible because AQM algoithms don't have to drop the new packet
that arrives, the algorithm can decide to drop the packet for one of the
heavy flows rather than for one of the lightweight flows.
<insert references to currently favored AQM options here, PIE, fq_codel,
cake, ???. Also links to failed approaches>
Turning on aqm on every bottleneck link makes the Internet usable for
everyone, no matter what sort of application they are using.
<insert link on how to deal with equipment you can't configure by
throttling bandwidth before the bottleneck oand/or doing ingress shaping of
traffic>
While AQM makes the network usable, there is still additional room for
improvement. While dropping packets does result in the TCP senders slowing
down,and eventually stabilizing at around the right speed to keep the link
fully utilized, the only way that senders have been able to detect problems
is to discover that they have not received an ack for the traffic within
the allowed time. This causes a 'bubble' in the flow as teh dropped packet
must be retransmitted (and sometimes a significant amount of data after the
dropped packet that did make it to the destination, but could not be acked
because fo the missing packet).
This "bubble" in the data flow can be greatly compressed by configuring the
AQM algorithm to send an ECN packet to the sender when it drops a packet in
a flow. The sender can then adapt faster, slowing down it's new data, and
re-sending the dropped packet without having to wait for the timeout. This
has two major effects by allowing the sender to retransmit the packet
sooner the dealy on the dropped data is not as long, and because the
replacement data can arrive before the timeout of the following packets,
they may not need to be re-sent. by configuring the AQM algorithm to send
the ECN notification to the sender only when the packet is being dropped,
the effect of failure of the ECN packet to get through to the sender (the
notification packet runs into congestion and gets dropped, some network
device blocks it, etc) is that the ECN enabled case devolves to match the
non-ECN case in that the sender will still detect the dropped packet via
the timeout waiting for the ack as if ENCN was not enabled.
<insert link to possible problems that can happen here, including the
potential for an app to 'game' things if packets are marked at a different
level than when they are dropped.>
So a very strong recommendation to enable Active Queue Management, while
the different algorithms have different advantages and levels of testing,
even the 'worst' of the set results in a night-and-day improvement for
usability compared to unmanaged buffers.
Enabling ECN at the same point as dropping packets as part of enabling any
AQM algorithm results in a noticable improvement over the base algorithm
without ECN. When compared to the baseline, the improvement added by ECN is
tiny compared to the improvement from enabling AQM.
Is it fair to say that plain aqm vs aqm+ecn variation is on the same
order of difference as the differences between the different AQM
algorithms?
Future research items (which others here may already have done, and would
not be part of my 'elevator pitch')
I believe that currently ECn triggers the exact same slowdown that a missed
packet does, and it may be appropriate to have the sender do a less drastic
slowdown.
It would be very interesing to provide soem way for the application sending
the traffic to detect dropped packets and ECN responses. For example, a
streaming media source (especially an interactive one like video
conferencing) could adjust the bitrate that it's sending.
David Lang
_______________________________________________
aqm mailing list
aqm@ietf.org
https://www.ietf.org/mailman/listinfo/aqm
________________________________________________________________
Bob Briscoe, BT
_______________________________________________
aqm mailing list
aqm@ietf.org
https://www.ietf.org/mailman/listinfo/aqm
