Re: [aqm] I-D Action: draft-ietf-aqm-eval-guidelines-09.txt

[Polina Goltsman]

Dear Wesley,

I apologize for the long delay. Below are responses to your emails 
regarding the draft-ietf-aqm-eval-guidelines.




On 22.01.16 15:50, Wesley Eddy wrote:

On 12/7/2015 7:32 AM, Polina Goltsman wrote:


In the abstract, the document says that it describes characterization 
guidelines for an AQM proposal, to decide whether it should be 
adopted by the AQM WG. The WG currently has two AQMs 
(dropping/marking policy) in last call. Did someone evaluate these 
AQMs according to the specified guidelines?





In my opinion, for "standardization" it would be good to have crisp 
guidelines.  For simply publishing RFCs that are experimental (not 
standardized), it is much less important.


I have a question regarding "standardization". As I understand, no 
evaluation was performed based on these guidelines. Since the intended 
status of draft-ietf-aqm-eval-guidelines is "informational" too, this is 
also not an issue. Am I correct?

(see also an email from Dave Täht, quoted at the end of this email)


Moreover, it seems to me that the WG is about to conclude. What 
exactly is the purpose of standardizing this document then ?





It's definitely true that the activity level has been low, and so 
we're trying to wrap up the outstanding work items before it flattens 
completely.  This document is not proposed for standards track, only 
"Informational".  The current goal as I see it (with co-chair hat on) 
is to see if we have rough consensus that this is a useful 
contribution to the community going forward, and that any small issues 
can be addressed.


As I understood your review (please correct if I'm wrong), a main 
issue you see is that there isn't specific guidance about numeric 
values or ranges to use in evaluations?  Nicolas explained why this 
might be a bit difficult to do in general (and specific values 
published in 2016 may be moot by 2018), so as I understand, one of the 
limitations of this document is that it is only going to be able to 
provide general descriptive guidance in terms of what kinds of tests 
to run, and what types of things to look for, but not prescriptive 
values and conditions to be met.  Do you think that's okay, even 
though it's probably less directly useful than if there were specific 
values


Yes, my concern was that the document did not contain enough detailed 
description of a test setup (capacities, RTT, etc). However, as Nicolas 
explained, the intention is that the test setup is selected by a tester  
for some specific environment. The results of an evaluation will then 
provide guidance for selecting an AQM for this specific environment. 
From my point of view this is completely reasonable approach, however 
IMO  *the document should state this explicitly*.


From answers to our (mine and Roland's) reviews we found out that the 
goals of the document are:



We  believe that



"


>  AQM schemes need to be compared against both performance and

>  deployment categories.


"


(this one is present in the document in section 4.3)


The main idea was to present the  available and useful metrics early



in the document, so that the  tester can choose between them those that



are of interest for the considered  scenario.


and finally

The  scope of this document is to present how to assess the performance of 

AQMs – without focusing on a specific context ...

...
The unspecified  parameters do not require specific knowledge on AQMs – the unspecified 
parameters are (1) the RTT or the capacities, which are related to the 
network underneath; (2) and the characteristics of the traffic 
generation, which is related to the >traffic that is run between clients 
and servers.


(which I understand as stated above, correct me if I am wrong)
(this goal will be referred as [*] below)

Section 1.2 is intended to specify the goals of the document. I am not 
sure that everybody will understand the above stated goals from the text 
of Section 1.2. and the document would benefit if these goals were 
stated more explicitly (Well, I could not understand them without 
explanations from the authors).




On 22.01.16 19:32, Klatsky, Carl wrote:


Wes and all,

My comment is in regard to Polina’s comment “The WG currently has two 
AQMs (dropping/marking policy) in last call. Did someone evaluate 
these AQMs according to the specified guidelines?”.  As I read over 
draft-ietf-aqm-eval-guidelines, I did not think the objective of this 
memo was to arrive at consensus to select one specific AQM.  I thought 
the objective was to publish guidelines for implementers & service 
providers on how they can select an AQM that is best for their 
environment. And further that both AQMs in last call would complete 
the process as valid AQMs for implementers & service providers as 
available to use in their environment, with the guidelines helping 

Re: [aqm] I-D Action: draft-ietf-aqm-eval-guidelines-09.txt

[Wesley Eddy]

On 1/22/2016 2:17 PM, Fred Baker (fred) wrote:

On Jan 22, 2016, at 10:47 AM, Wesley Eddy  wrote:

I do also (personally) think that if there's a desire to go standards-track 
(rather than just experimental) with AQM algorithms, that having a fairly 
explicit evaluation of the algorithms with regard to the guidelines would be 
very helpful, exactly as Polina asked about.  But I don't think this has really 
happened, and don't think it's necessary at all for experimental RFCs.

As I recall the discussion, we decided up front that since there is no 
interoperability requirement among AQM algorithms (the requirement is that they 
interoperate well with TCP and UDP based applications; the AQM algorithms don' 
actually talk to each other, and the point is to drop or mark at the right rate 
and with the right pattern to encourage transport layer sessions to behave 
well), we didn't need to recommend a single AQM algorithm for all equipment or 
all uses. What we did need to do was identify some AQM algorithms that actually 
worked, and give guidance to the vendors and operators on their use.


I agree with all of this.  The characterization guidelines are aimed at 
helping to identify the AQM algorithms that actually work, or cases 
where they don't work as well (i.e. where some harmful or unintended 
consequence results).



___
aqm mailing list
aqm@ietf.org
https://www.ietf.org/mailman/listinfo/aqm

Re: [aqm] I-D Action: draft-ietf-aqm-eval-guidelines-09.txt

[Klatsky, Carl]

Wes and all,

My comment is in regard to Polina's comment "The WG currently has two AQMs 
(dropping/marking policy) in last call. Did someone evaluate these AQMs 
according to the specified guidelines?".  As I read over 
draft-ietf-aqm-eval-guidelines, I did not think the objective of this memo was 
to arrive at consensus to select one specific AQM.  I thought the objective was 
to publish guidelines for implementers & service providers on how they can 
select an AQM that is best for their environment.  And further that both AQMs 
in last call would complete the process as valid AQMs for implementers & 
service providers as available to use in their environment, with the guidelines 
helping them to decide which is best for them.  Some may chose PIE, some may 
chose FQ_CODEL.  And possibly other future AQMs would go through the IETF 
process for publication, and that would be an additional option for 
implementers & service providers to evaluate according to the guidelines as 
best for their environment.

Is my understand of draft-ietf-aqm-eval-guidelines correct?

Regards,

Carl Klatsky
Comcast

From: aqm [mailto:aqm-boun...@ietf.org] On Behalf Of Wesley Eddy
Sent: Friday, January 22, 2016 9:51 AM
To: aqm@ietf.org; Polina Goltsman
Subject: Re: [aqm] I-D Action: draft-ietf-aqm-eval-guidelines-09.txt

On 12/7/2015 7:32 AM, Polina Goltsman wrote:


In the abstract, the document says that it describes characterization 
guidelines for an AQM proposal, to decide whether it should be adopted by the 
AQM WG. The WG currently has two AQMs (dropping/marking policy) in last call. 
Did someone evaluate these AQMs according to the specified guidelines?


In my opinion, for "standardization" it would be good to have crisp guidelines. 
 For simply publishing RFCs that are experimental (not standardized), it is 
much less important.




Moreover, it seems to me that the WG is about to conclude. What exactly is the 
purpose of standardizing this document then ?


It's definitely true that the activity level has been low, and so we're trying 
to wrap up the outstanding work items before it flattens completely.  This 
document is not proposed for standards track, only "Informational".  The 
current goal as I see it (with co-chair hat on) is to see if we have rough 
consensus that this is a useful contribution to the community going forward, 
and that any small issues can be addressed.

As I understood your review (please correct if I'm wrong), a main issue you see 
is that there isn't specific guidance about numeric values or ranges to use in 
evaluations?  Nicolas explained why this might be a bit difficult to do in 
general (and specific values published in 2016 may be moot by 2018), so as I 
understand, one of the limitations of this document is that it is only going to 
be able to provide general descriptive guidance in terms of what kinds of tests 
to run, and what types of things to look for, but not prescriptive values and 
conditions to be met.  Do you think that's okay, even though it's probably less 
directly useful than if there were specific values?

___
aqm mailing list
aqm@ietf.org
https://www.ietf.org/mailman/listinfo/aqm

Re: [aqm] I-D Action: draft-ietf-aqm-eval-guidelines-09.txt

[Dave Täht]

The evaluation guide needs to be executable, or rather, turned into
public code and a standardized benchmark suite. Eventually.

Iteratively, flent has many tests that have proven valuable and quite a
few that have not.

The tests in the aqm guide, need to be created, iterated on, and examined.


___
aqm mailing list
aqm@ietf.org
https://www.ietf.org/mailman/listinfo/aqm

Re: [aqm] I-D Action: draft-ietf-aqm-eval-guidelines-09.txt

[Wesley Eddy]

On 1/22/2016 1:32 PM, Klatsky, Carl wrote:


Wes and all,

My comment is in regard to Polina’s comment “The WG currently has two 
AQMs (dropping/marking policy) in last call. Did someone evaluate 
these AQMs according to the specified guidelines?”.  As I read over 
draft-ietf-aqm-eval-guidelines, I did not think the objective of this 
memo was to arrive at consensus to select one specific AQM.  I thought 
the objective was to publish guidelines for implementers & service 
providers on how they can select an AQM that is best for their 
environment. And further that both AQMs in last call would complete 
the process as valid AQMs for implementers & service providers as 
available to use in their environment, with the guidelines helping 
them to decide which is best for them. Some may chose PIE, some may 
chose FQ_CODEL.  And possibly other future AQMs would go through the 
IETF process for publication, and that would be an additional option 
for implementers & service providers to evaluate according to the 
guidelines as best for their environment.


Is my understand of draft-ietf-aqm-eval-guidelines correct?




Yes, you're correct!  My assumption is that someone like a service 
provider would have an idea of some of the ranges of values (rates, 
delays, asymmetries, etc) appropriate for their environment, and would 
be able to use the evaluation guidelines effectively.


I do also (personally) think that if there's a desire to go 
standards-track (rather than just experimental) with AQM algorithms, 
that having a fairly explicit evaluation of the algorithms with regard 
to the guidelines would be very helpful, exactly as Polina asked about.  
But I don't think this has really happened, and don't think it's 
necessary at all for experimental RFCs.




___
aqm mailing list
aqm@ietf.org
https://www.ietf.org/mailman/listinfo/aqm

Re: [aqm] I-D Action: draft-ietf-aqm-eval-guidelines-09.txt

[Fred Baker (fred)]

On Jan 22, 2016, at 10:47 AM, Wesley Eddy  wrote:
> I do also (personally) think that if there's a desire to go standards-track 
> (rather than just experimental) with AQM algorithms, that having a fairly 
> explicit evaluation of the algorithms with regard to the guidelines would be 
> very helpful, exactly as Polina asked about.  But I don't think this has 
> really happened, and don't think it's necessary at all for experimental RFCs.

As I recall the discussion, we decided up front that since there is no 
interoperability requirement among AQM algorithms (the requirement is that they 
interoperate well with TCP and UDP based applications; the AQM algorithms don' 
actually talk to each other, and the point is to drop or mark at the right rate 
and with the right pattern to encourage transport layer sessions to behave 
well), we didn't need to recommend a single AQM algorithm for all equipment or 
all uses. What we did need to do was identify some AQM algorithms that actually 
worked, and give guidance to the vendors and operators on their use.




signature.asc
Description: Message signed with OpenPGP using GPGMail
___
aqm mailing list
aqm@ietf.org
https://www.ietf.org/mailman/listinfo/aqm

Re: [aqm] I-D Action: draft-ietf-aqm-eval-guidelines-09.txt

[Polina Goltsman]

Dear Nicolas,

Thank you very much for the explanation, I have however two questions 
regarding your comments:


1) why is the explanation about why are parameters unspecified not 
present in the document?


2) in this case, I assume that the remark about "asymmetric link 
scenario evaluates an AQM mechanism in a more realistic setup;" in 
section 3.1 and all cases where the parameters for capacity and 
round-trip time ARE present (see items 2 and 4 below), should be removed ?


I guess now I have to write a longer review...  :(

Regards,
Polina


On 12/11/2015 03:20 PM, Kuhn Nicolas wrote:


Dear Polina, all,

Thank you for your email and I understand your concerns. I will 
provide more detail answer below for each of your points, but I would 
like first to clarify a general issue.


The scope of this document is to present how to assess the performance 
of AQMs – without focusing on a specific context (3G networks, 
data-centers, satellite networks, home routers, ISP core network) 
where Bufferbloat may (or may not) occur (yet). This is the main 
reason why this draft does not provide specific details on the 
bottleneck capacities, numbers of flows, or traffic generation. Every 
strict number or specific flow characteristic limits the scope of this 
document that is not only focusing on web traffic latency reduction in 
home routers.


Please see inline for specific answer to your concerns.

Cheers,

Nicolas

*De :*aqm [mailto:aqm-boun...@ietf.org] *De la part de* Polina Goltsman
*Envoyé :* lundi 7 décembre 2015 13:32
*À :* Wesley Eddy; aqm@ietf.org
*Cc :* Bless, Roland (TM)
*Objet :* Re: [aqm] I-D Action: draft-ietf-aqm-eval-guidelines-09.txt

Dear all,

Apologies for the long delay.

I have review the latest version of the draft. In my opinion the 
document still needs a detailed review about the representation (the 
text): the structure of the document would make more sense if the 
information in section 2, 3, and 4 would appear in different order, 
the document still could use a more consistent terminology, and the 
structure of section 5-12 could be more consistent. I am happy to 
provide the detailed review, but since it is a lot of work, I have 
several concerns about the content of the document, which I would 
prefer to clear first.


NK : we changed the ToC of the draft several times and I think the 
first sections (2, 3 and 4) are quite independent. I do not see a 
specific need for changing their order.


Regarding sections 5-12, we could still add some “Motivation” and 
“Recommended tests” for the section 5 for increasing the consistency. 
The sections that do not need performance evaluations have another 
organization (Motivation then Recommended discussion), but that seems 
consistent to me. We had several checks within the list for the 
terminology (such as the one done by Gorry), but if you spot anything, 
feel free to send your comments.



The document proposes the evaluation methodology for AQMs which 
consists of the following topology and scenarios:


0. the experiments should be performed on the topology, described in 
Figure 1 in Section 3, where the bottleneck between routers L and R is 
of *unspecified* capacity, which SHOULD include*both symmetric and 
asymmetric*, and the buffer MAY be set to a BDP. If I understand 
correctly, it is further RECOMMENDED to use a range of input 
parameters for the evaluated AQM (this may only be required if several 
AQMs are being compared)


NK : The Figure 1 in section 3 MAY be used. The dumbbell topology is 
useful to evaluate the performance of congestion control (such as 
mentioned in the common TCP evaluation suite of the IRTF).


NK: Such as for the different parameters (buffer sizing, bottleneck 
capacity, etc.), we understand that having unspecified parameters may 
be annoying, but I do not see how we can define such parameters for 
satellite networks, data-centers, home gateways, core Internet 
providers’ network, etc. As one example, the capacity is (not only) 
related to the mechanisms from lower layers, such as link layer 
reliability schemes, channel access mechanisms, etc. which highly 
varies from one context to the other. Also, the buffer sizing depends 
on the BDP, the amount of traffic, the presence of PEP (such as in 
satellite gateways), thus I am afraid we can hardly define a buffer 
sizing that suits every use case. The amount of input parameters for 
the evaluated AQM that is to be looked at is the amount of parameters 
that are exhibited by the AQM developers (i.e. to what extent the AQM 
is a black box or not). I hardly see how we can have guidelines that 
specify number and parameterization that work for any use case 
scenario and deployment context.



1. (section 5) several experiments with different TCP congestion 
control schemes (+UDP) which should be performed on *unspecified* 
bottleneck capacity and *unspecified* RTT, which should or should not 
be the same as the one with which the AQM is config

Re: [aqm] I-D Action: draft-ietf-aqm-eval-guidelines-09.txt

[Polina Goltsman]

Dear all,

Apologies for the long delay.

I have review the latest version of the draft. In my opinion the 
document still needs a detailed review about the representation (the 
text): the structure of the document would make more sense if the 
information in section 2, 3, and 4 would appear in different order, the 
document still could use a more consistent terminology, and the 
structure of section 5-12 could be more consistent. I am happy to 
provide the detailed review, but since it is a lot of work, I have 
several concerns about the content of the document, which I would prefer 
to clear first.


The document proposes the evaluation methodology for AQMs which consists 
of the following topology and scenarios:


   0. the experiments should be performed on the topology, described in
   Figure 1 in Section 3, where the bottleneck between routers L and R
   is of *unspecified* capacity, which SHOULD include*both symmetric
   and asymmetric*, and the buffer MAY be set to a BDP. If I understand
   correctly, it is further RECOMMENDED to use a range of input
   parameters for the evaluated AQM (this may only be required if
   several AQMs are being compared)

   1. (section 5) several experiments with different TCP congestion
   control schemes (+UDP) which should be performed on *unspecified*
   bottleneck capacity and *unspecified* RTT, which should or should
   not be the same as the one with which the AQM is configured.

   2. (section 6) RTT fairness, where two groups of *unspecified*
   number of flows share a bottleneck of *unspecified* capacity, where
   the first group sees RTT of 100ms and the second is in range [5ms;560ms]

   3. (section 7) burst absorption, is a group of four scenarios, which
   include web-traffic, bursty video frames, which should be generated
   in an unspecified manner (*left to the tester to decide*), CBR
   traffic of *unspecified* rate, and TCP flow, all of which is
   evaluated using a set of metrics that MAY be generated. I might be
   missing something, but I don't understand how any of these metrics
   can be used to  characterize burst absorption.

   4. (section 8) stability, including (a) varying the number of flows
   on a bottleneck of *unspecified* capacity with *unspecified* RTT,
   according to the provided formulas, (b) varying network capacity
   between 10Mbps an 100Mbps for one TCP flow (if I understand it
   correctly)

   5. (section 9) which includes (a) traffic mix of a combination
   applications, including TCP, web-traffic, bi-direction VoIP using
   *unspecified congestion* control (I am not sure, but I think there
   are options), CBR of *unspecified* rate, and adaptive video
   streaming also with *unspecified* congestion control, with one
   combination required and other left to the tester to decide; and (b)
   bi-directional traffic, which should evaluate the effect on dropping
   DNS/TCP Syn packets* using a specified number of bulk TCP flows
   using the the throughput-delay tradeoff graph; (in both scenarios
   capacities and RTT are *unspecified* as well)

* I assume that DNS/SYN packets are mentioned in this section by mistake

Do you think that a person without a substantial background knowledge on 
the evaluation of AQM schemes can perform this evaluation (resolve all 
unspecified conditions) in a reasonable amount of time?


In the abstract, the document says that it describes characterization 
guidelines for an AQM proposal, to decide whether it should be adopted 
by the AQM WG. The WG currently has two AQMs (dropping/marking policy) 
in last call. Did someone evaluate these AQMs according to the specified 
guidelines?


Moreover, it seems to me that the WG is about to conclude. What exactly 
is the purpose of standardizing this document then ?


Regards,
Polina

On 12/01/2015 08:19 PM, Wesley Eddy wrote:

Thanks for making this update.

I don't think Roland and Polina who had made last call comments had 
yet had the time to check that the changes in the last revision met 
what they were expecting, so I'd like to give them (and anyone else 
who has comments) a couple of weeks to check this revision out, and 
assuming there are no major issues or objections by around 12/15, will 
plan to end the working group last call, and send the document to the 
area director for publication.




On 11/27/2015 5:50 AM, Kuhn Nicolas wrote:

Dear all,

This updated version integrates:
- modification on the buffer sizes, following some discussion points 
raised by Michael Scharf on the tcpm mailing list [1];

- some nits raised by Greg Skinner.

Kind regards,

Nicolas KUHN

[1] https://www.ietf.org/mail-archive/web/tcpm/current/msg09894.html

-Message d'origine-
De : aqm [mailto:aqm-boun...@ietf.org] De la part de 
internet-dra...@ietf.org

Envoyé : vendredi 27 novembre 2015 11:44
À : i-d-annou...@ietf.org
Cc : aqm@ietf.org
Objet : [aqm] I-D Action: draft-ietf-aqm-eval-guidelines-09.txt


A New Internet-Draft is available from the on-line

[aqm] I-D Action: draft-ietf-aqm-eval-guidelines-09.txt

[internet-drafts]

A New Internet-Draft is available from the on-line Internet-Drafts directories.
 This draft is a work item of the Active Queue Management and Packet Scheduling 
Working Group of the IETF.

Title   : AQM Characterization Guidelines
Authors : Nicolas Kuhn
  Preethi Natarajan
  Naeem Khademi
  David Ros
Filename: draft-ietf-aqm-eval-guidelines-09.txt
Pages   : 35
Date: 2015-11-27

Abstract:
   Unmanaged large buffers in today's networks have given rise to a slew
   of performance issues.  These performance issues can be addressed by
   some form of Active Queue Management (AQM) mechanism, optionally in
   combination with a packet scheduling scheme such as fair queuing.
   The IETF Active Queue Management and Packet Scheduling working group
   was formed to standardize AQM schemes that are robust, easily
   implementable, and successfully deployable in today's networks.  This
   document describes various criteria for performing precautionary
   characterizations of AQM proposals.  This document also helps in
   ascertaining whether any given AQM proposal should be taken up for
   standardization by the AQM WG.


The IETF datatracker status page for this draft is:
https://datatracker.ietf.org/doc/draft-ietf-aqm-eval-guidelines/

There's also a htmlized version available at:
https://tools.ietf.org/html/draft-ietf-aqm-eval-guidelines-09

A diff from the previous version is available at:
https://www.ietf.org/rfcdiff?url2=draft-ietf-aqm-eval-guidelines-09


Please note that it may take a couple of minutes from the time of submission
until the htmlized version and diff are available at tools.ietf.org.

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

___
aqm mailing list
aqm@ietf.org
https://www.ietf.org/mailman/listinfo/aqm

Re: [aqm] I-D Action: draft-ietf-aqm-eval-guidelines-09.txt

[Kuhn Nicolas]

Dear all, 

This updated version integrates:
- modification on the buffer sizes, following some discussion points raised by 
Michael Scharf on the tcpm mailing list [1];
- some nits raised by Greg Skinner.

Kind regards, 

Nicolas KUHN

[1] https://www.ietf.org/mail-archive/web/tcpm/current/msg09894.html

-Message d'origine-
De : aqm [mailto:aqm-boun...@ietf.org] De la part de internet-dra...@ietf.org
Envoyé : vendredi 27 novembre 2015 11:44
À : i-d-annou...@ietf.org
Cc : aqm@ietf.org
Objet : [aqm] I-D Action: draft-ietf-aqm-eval-guidelines-09.txt


A New Internet-Draft is available from the on-line Internet-Drafts directories.
 This draft is a work item of the Active Queue Management and Packet Scheduling 
Working Group of the IETF.

Title   : AQM Characterization Guidelines
Authors : Nicolas Kuhn
  Preethi Natarajan
  Naeem Khademi
  David Ros
Filename: draft-ietf-aqm-eval-guidelines-09.txt
Pages   : 35
Date: 2015-11-27

Abstract:
   Unmanaged large buffers in today's networks have given rise to a slew
   of performance issues.  These performance issues can be addressed by
   some form of Active Queue Management (AQM) mechanism, optionally in
   combination with a packet scheduling scheme such as fair queuing.
   The IETF Active Queue Management and Packet Scheduling working group
   was formed to standardize AQM schemes that are robust, easily
   implementable, and successfully deployable in today's networks.  This
   document describes various criteria for performing precautionary
   characterizations of AQM proposals.  This document also helps in
   ascertaining whether any given AQM proposal should be taken up for
   standardization by the AQM WG.


The IETF datatracker status page for this draft is:
https://datatracker.ietf.org/doc/draft-ietf-aqm-eval-guidelines/

There's also a htmlized version available at:
https://tools.ietf.org/html/draft-ietf-aqm-eval-guidelines-09

A diff from the previous version is available at:
https://www.ietf.org/rfcdiff?url2=draft-ietf-aqm-eval-guidelines-09


Please note that it may take a couple of minutes from the time of submission 
until the htmlized version and diff are available at tools.ietf.org.

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

___
aqm mailing list
aqm@ietf.org
https://www.ietf.org/mailman/listinfo/aqm

___
aqm mailing list
aqm@ietf.org
https://www.ietf.org/mailman/listinfo/aqm