Re: [aqm] WGLC on draft-ietf-aqm-eval-guidelines
Hello, We (Roland and Polina) need some time to review the new document and the rest of the email. According to the charter, the document is still in last call. What is the next deadline? Best Regards, Polina On 09/22/2015 08:38 AM, Nicolas Kuhn wrote: Hi all, Sorry for the delay of our answer. We have just posted a new version of the draft that, we hope, assesses the comments of Wolfram, Roland and Polina. Following Wolfram's comments, we: - changed the units of the equation ( FCT [s] = Fs [b] / ( G [bps] ) ); - mention only one reference for the latency/goodput trade-off graphs; - updated the "long-term UDP" terminology. Following the comments of Polina and Roland, we: -moved the " Methodology section " at the beginning of the document. - included discussions on ECN and scheduling in the methodology section. I think it is clearer that way. - fixed the table at the end with updated section numbers. For the details of the changes, please see below. ___ aqm mailing list aqm@ietf.org https://www.ietf.org/mailman/listinfo/aqm
Re: [aqm] WGLC on draft-ietf-aqm-eval-guidelines
Hi Wes, On 19.08.2015 at 04:52 Wesley Eddy wrote: > There were only a couple of the "major issues" that I thought I should > comment on as a co-chair of the WG: > > >> 3) the overall number of tests and parameter combinations is really >> high > > Are there particular permutations (or classes of permutations) that you > can suggest to remove? There's a balancing act between including enough > to satisfy people that want to find edge cases and thoroughly > characterize an algorithm, and the desire for a more easily tractable > suite of tests. Right now, I can't, it was merely an observation... >> 4) from the discussed end-to-end metrics only latency/goodput metrics >> are used in the scenarios and for some of the scenarios these metrics >> are not suitable to show the desired behavior > > It would be easier for the editors to improve this if you could suggest > specific metrics to add to specific scenarios, I think. We included some considerations in the comments to the individual sections already. As a general notice, The document could benefit by cross-referencing the scenarios against the documents referenced in Major Issue 6. Regarding particular metrics, suggesting one would require exact understanding of particular test goals, proper argumentation, and thus more time than LC deadline allowed. Regards, Roland ___ aqm mailing list aqm@ietf.org https://www.ietf.org/mailman/listinfo/aqm
Re: [aqm] WGLC on draft-ietf-aqm-eval-guidelines
On 8/18/2015 6:03 PM, Roland Bless wrote: > Hi, > > Am 10.08.2015 um 15:43 schrieb Wesley Eddy: >> As chairs, Richard and I would like to start a 2-week working >> group last call on the AQM characterization guidelines: >> >> https://datatracker.ietf.org/doc/draft-ietf-aqm-eval-guidelines/ >> >> Please make a review of this, and send comments to the list or >> chairs. Any comments that you might have will be useful to us, >> even if it's just to say that you've read it and have no other >> comments. > > "Unfortunately", we (Polina and I) did a thorough review, > which is attached. TL;DR: from our point-of-view > the I-D needs a major revision. > Many thanks for the detailed review. I think a majority of the comments could be addressed in an update, if the authors agree. There were only a couple of the "major issues" that I thought I should comment on as a co-chair of the WG: > 3) the overall number of tests and parameter combinations is really > high Are there particular permutations (or classes of permutations) that you can suggest to remove? There's a balancing act between including enough to satisfy people that want to find edge cases and thoroughly characterize an algorithm, and the desire for a more easily tractable suite of tests. > 4) from the discussed end-to-end metrics only latency/goodput metrics > are used in the scenarios and for some of the scenarios these metrics > are not suitable to show the desired behavior It would be easier for the editors to improve this if you could suggest specific metrics to add to specific scenarios, I think. > 5) some sections in this document (e.g., 7.3, 10, 13) specify requirements > for an AQM standard(/draft) and not requirements for a performance > evaluation, so these sections should be moved to [draft-ietf-aqm-recommendation] That one is now an RFC (7567), so hopefully they're already reflected if they were critical requirements. In any case, I agree with you that requirements themselves should not be conveyed in this document, but rather it should be just aimed at characterizing algorithm behavior with regard to the requirements (for ones that are applicable to verification by testing). -- Wes Eddy MTI Systems ___ aqm mailing list aqm@ietf.org https://www.ietf.org/mailman/listinfo/aqm
Re: [aqm] WGLC on draft-ietf-aqm-eval-guidelines
On 8/18/2015 6:07 PM, Dave Taht wrote: > On Tue, Aug 18, 2015 at 3:03 PM, Roland Bless wrote: >> Hi, >> >> Am 10.08.2015 um 15:43 schrieb Wesley Eddy: >>> As chairs, Richard and I would like to start a 2-week working >>> group last call on the AQM characterization guidelines: >>> >>> https://datatracker.ietf.org/doc/draft-ietf-aqm-eval-guidelines/ >>> >>> Please make a review of this, and send comments to the list or >>> chairs. Any comments that you might have will be useful to us, >>> even if it's just to say that you've read it and have no other >>> comments. >> >> "Unfortunately", we (Polina and I) did a thorough review, >> which is attached. TL;DR: from our point-of-view >> the I-D needs a major revision. > > I am so tired of this document that I can hardly bear to read it > again, but I agree with the majority of the comments. > > Sometimes I do wish we could do graphics and charts as the IEEE does. > We can add any type of graphics that are necessary, they will just only show up in the PDF version of the RFC, with only references to the PDF version in the TXT copy. See, for instance: https://tools.ietf.org/pdf/rfc6687.pdf Are there particular figures that need to be added to this AQM document to strengthen it? -- Wes Eddy MTI Systems ___ aqm mailing list aqm@ietf.org https://www.ietf.org/mailman/listinfo/aqm
Re: [aqm] WGLC on draft-ietf-aqm-eval-guidelines
On Tue, Aug 18, 2015 at 3:03 PM, Roland Bless wrote: > Hi, > > Am 10.08.2015 um 15:43 schrieb Wesley Eddy: >> As chairs, Richard and I would like to start a 2-week working >> group last call on the AQM characterization guidelines: >> >> https://datatracker.ietf.org/doc/draft-ietf-aqm-eval-guidelines/ >> >> Please make a review of this, and send comments to the list or >> chairs. Any comments that you might have will be useful to us, >> even if it's just to say that you've read it and have no other >> comments. > > "Unfortunately", we (Polina and I) did a thorough review, > which is attached. TL;DR: from our point-of-view > the I-D needs a major revision. I am so tired of this document that I can hardly bear to read it again, but I agree with the majority of the comments. Sometimes I do wish we could do graphics and charts as the IEEE does. > Regards, > Roland > > > ___ > aqm mailing list > aqm@ietf.org > https://www.ietf.org/mailman/listinfo/aqm > -- Dave Täht worldwide bufferbloat report: http://www.dslreports.com/speedtest/results/bufferbloat And: What will it take to vastly improve wifi for everyone? https://plus.google.com/u/0/explore/makewififast ___ aqm mailing list aqm@ietf.org https://www.ietf.org/mailman/listinfo/aqm
Re: [aqm] WGLC on draft-ietf-aqm-eval-guidelines
Hi,
Am 10.08.2015 um 15:43 schrieb Wesley Eddy:
> As chairs, Richard and I would like to start a 2-week working
> group last call on the AQM characterization guidelines:
>
> https://datatracker.ietf.org/doc/draft-ietf-aqm-eval-guidelines/
>
> Please make a review of this, and send comments to the list or
> chairs. Any comments that you might have will be useful to us,
> even if it's just to say that you've read it and have no other
> comments.
"Unfortunately", we (Polina and I) did a thorough review,
which is attached. TL;DR: from our point-of-view
the I-D needs a major revision.
Regards,
Roland
I completed my review for draft-ietf-aqm-eval-guidelines-07
and discussed it also with Polina, who did her own review
which we eventually aggregated here. We both think that this
document needs a major revision due to the amount of issues
we identified.
Major issues:
-
1) Structure, overview, rationale and requirements
The structure should/could be improved.
The goal and methodology should be put first. Some
motivation given in Section 14 should be moved to the
beginning, e.g., the goal of this document is stated in
Section 14.3.
2) It is unclear whether the tests from Sections 4-9 should be
carried out without or with ECN. Section 12 discusses this
much too late.
3) the overall number of tests and parameter combinations is really
high
4) from the discussed end-to-end metrics only latency/goodput metrics
are used in the scenarios and for some of the scenarios these metrics
are not suitable to show the desired behavior
5) some sections in this document (e.g., 7.3, 10, 13) specify requirements
for an AQM standard(/draft) and not requirements for a performance
evaluation, so these sections should be moved to
[draft-ietf-aqm-recommendation]
6) Related Work: There are several works that deal with
evaluation of TCP or congestion control performance:
- RFC 5166 https://tools.ietf.org/html/rfc5166
(Metrics for the Evaluation of Congestion Control Mechanisms)
is IMHO higly relevant but neither referenced nor discussed
- Yee-Ting Li, Douglas Leith, and Robert
N. Shorten. 2007. Experimental evaluation of TCP protocols for
high-speed networks. IEEE/ACM Trans. Netw. 15, 5 (October 2007),
1109-1122. DOI=10.1109/TNET.2007.896240
http://dx.doi.org/10.1109/TNET.2007.896240
- Andrew et al.: Towards a Common TCP Evaluation Suite,
Proceedings of the International Workshop on Protocols for Fast
Long-Distance Networks (PFLDnet), Manchester, United Kingdom,
March 2008
Detailed comments per section:
==
(the %% just separates different issues within the section comments)
{Section 1}
---
AQM schemes aim at reducing mean buffer occupancy, and
therefore both end-to-end delay and jitter.
==> is this true for every AQM ?
%%
In real implementations of switches, a global
memory is shared between the available devices:
This may be a common architecture nowadays, but not necessarily
always be the case...
=> In real implementations of switches, a global
memory is _often_ shared between the available devices:
%%
the size of the buffer for a given communication does not
make sense ...
and then...
The rest of this memo therefore refers to the
maximum queue depth as the size of the buffer for a given
communication.
=> I don't understand what you mean here. First you say
it doesn't make sense, then you define maximum queue depth
as exactly the size of the buffer for a given
communication.
- Do you mean buffer occupancy?
- Is "communication" here an end-to-end data flow or an aggregated flow?
- the term "maximum queue depth" is never used in the document again...
but "maximum queue size", "maximum buffer size"
I think it is essential to understand the difference between the
buffer size and the buffer occupancy that the AQM tries to control.
Due to shared memory architectures the buffer size may not be fixed
and thus vary for a given interface.
Is the buffer (size) here meant in both directions for bidirectional traffic?
%%
Bufferbloat [BB2011] is the consequence of deploying large unmanaged
buffers on the Internet, which has lead to an increase in end-to-end
delay: the buffering has often been measured to be ten times or
hundred times larger than needed.
Large buffers per se are not a real problem unless combined with TCP bandwidth
probing or unresponsive flows that fill buffers.
%%
The Active Queue Management and Packet Scheduling Working Group (AQM
WG) was recently formed within the TSV area to address the problems
with large unmanaged buffers in the Internet. Specifically, the AQM
IMHO this and the following paragraphs should be rephrased so that the
statement is also true in some years
after the WG has concluded...
%%
Missing: The use of ECN is also an incentiv
Re: [aqm] WGLC on draft-ietf-aqm-eval-guidelines
Hi all, I read the latest version of the draft, and I found it useful. The draft addresses a comprehensive range of topics for AQM characterization. What I am not so happy with, is the description of the corresponding experiments. Some critical points of my first review https://mailarchive.ietf.org/arch/msg/aqm/OwPTGmXLpEmCChpgE7ZFqFnnT64 still persist. I would like to regard these experiments as initial proposals (which is good to have) that might undergo substantial revision in practice later on. In general I have the feeling that the combinatorial number of mandatory experiments is close to infinity. Not only that I doubt this will ever be done; but who is subsequently going to judge the huge amount of results? Here are some minor comments: Section 2.7 defines goodput/delay scatter plots in two different ways: On with reference to [HAYE2013], the other definition with reference to [WINS2014]. I would prefer to have only one definition, namely [WINS2014]. - [HAYE2013] depends on a parameter variation across certain range (e.g. traffic load, or buffer size) that is not defined in most of our experiments. - [WINS2014] depends only on randomized replication of otherwise identical experiments. This should be applicable to any of the evaluation experiments. (In fact, it is unavoidable anyway.) Section 4.3: The term "long-lived non application-limited UDP" is somewhat infinite bandwidth. What the authors probably mean is "long-lived UDP flow from unresponsive application" to make it clear that no application layer congestion control is present like in NFS. Section 2.1: Formula on flow completion time: mismatch of dimensions (Byte vs. Mbps) Wolfram Lautenschlaeger -Ursprüngliche Nachricht- Von: aqm [mailto:aqm-boun...@ietf.org] Im Auftrag von Wesley Eddy Gesendet: Montag, 10. August 2015 15:44 An: aqm@ietf.org Betreff: [aqm] WGLC on draft-ietf-aqm-eval-guidelines As chairs, Richard and I would like to start a 2-week working group last call on the AQM characterization guidelines: https://datatracker.ietf.org/doc/draft-ietf-aqm-eval-guidelines/ Please make a review of this, and send comments to the list or chairs. Any comments that you might have will be useful to us, even if it's just to say that you've read it and have no other comments. Thanks! -- Wes Eddy MTI Systems ___ aqm mailing list aqm@ietf.org https://www.ietf.org/mailman/listinfo/aqm smime.p7s Description: S/MIME cryptographic signature ___ aqm mailing list aqm@ietf.org https://www.ietf.org/mailman/listinfo/aqm
[aqm] WGLC on draft-ietf-aqm-eval-guidelines
As chairs, Richard and I would like to start a 2-week working group last call on the AQM characterization guidelines: https://datatracker.ietf.org/doc/draft-ietf-aqm-eval-guidelines/ Please make a review of this, and send comments to the list or chairs. Any comments that you might have will be useful to us, even if it's just to say that you've read it and have no other comments. Thanks! -- Wes Eddy MTI Systems ___ aqm mailing list aqm@ietf.org https://www.ietf.org/mailman/listinfo/aqm
