Re: [aqm] Benoit Claise's Discuss on draft-ietf-aqm-eval-guidelines-11: (with DISCUSS and COMMENT)
All, We have pushed an updated version that integrates the suggested changes. Thanks, Nico -Message d'origine- De : aqm [mailto:aqm-boun...@ietf.org] De la part de Mirja Kuehlewind (IETF) Envoyé : mardi 14 juin 2016 09:13 À : Benoit Claise Cc : MORTON, ALFRED C (AL); w...@mti-systems.com; aqm-cha...@ietf.org; The IESG; draft-ietf-aqm-eval-guideli...@ietf.org; Schulthess Nicolas (F); aqm@ietf.org Objet : Re: [aqm] Benoit Claise's Discuss on draft-ietf-aqm-eval-guidelines-11: (with DISCUSS and COMMENT) Yes, already contacted the authors! Thanks all! > Am 14.06.2016 um 08:42 schrieb Benoit Claise <bcla...@cisco.com>: > > >>> -Original Message- >>> From: Mirja Kühlewind [mailto:mirja.kuehlew...@tik.ee.ethz.ch] >>> Sent: Monday, June 13, 2016 3:41 PM >> ... >>> Hi Al, >>> >>> I believe, we agree here. However, I’m not really sure what needs to >>> be changed/added in the draft now. The only concrete item I have is >>> replacing "application-level“ by "transport-layer payload“. Anything >>> else? >>> >>> Mirja >> [ACM] >> Thanks, that would resolve the biggest ambiguity for me. >> Like I said last week, I think we're done (with that change). > Thank you Al and Mirja. > I'll clear the DISCUSS on that basis, trusting the AD that the addition will > be introduced. > > Regards, Benoit >> >> Al >> >>> >>>> Am 10.06.2016 um 19:16 schrieb MORTON, ALFRED C (AL) >>> <acmor...@att.com>: >>>> more below, thanks for the clarifications, Mirja! >>>> Al >>>> >>>>> -Original Message- >>>>> From: Mirja Kühlewind [mailto:mirja.kuehlew...@tik.ee.ethz.ch] >>>>> Sent: Friday, June 10, 2016 12:55 PM >>>>> To: MORTON, ALFRED C (AL); Mirja Kühlewind; Benoit Claise >>>>> Cc: w...@mti-systems.com; aqm-cha...@ietf.org; The IESG; >>>>> draft-ietf- >>> aqm- >>>>> eval-guideli...@ietf.org; Schulthess Nicolas (F); aqm@ietf.org >>>>> Subject: Re: [aqm] Benoit Claise's Discuss on draft-ietf-aqm-eval- >>>>> guidelines-11: (with DISCUSS and COMMENT) >>>>> >>>>> Hi Al, >>>>> >>>>> see below. >>>>> >>>>> On 10.06.2016 18:41, MORTON, ALFRED C (AL) wrote: >>>>>> Hi, see below, >>>>>> >>>>>>> -Original Message- >>>>>>> From: Mirja Kühlewind [mailto:i...@kuehlewind.net] >>>>>>> Sent: Friday, June 10, 2016 9:15 AM >>>>>>> To: Benoit Claise; MORTON, ALFRED C (AL) >>>>>>> Cc: w...@mti-systems.com; aqm-cha...@ietf.org; The IESG; >>>>>>> draft-ietf- >>>>> aqm- >>>>>>> eval-guideli...@ietf.org; Schulthess Nicolas (F); aqm@ietf.org >>>>>>> Subject: Re: [aqm] Benoit Claise's Discuss on >>>>>>> draft-ietf-aqm-eval- >>>>>>> guidelines-11: (with DISCUSS and COMMENT) >>>>>>> >>>>>>> Benoit, >>>>>>> >>>>>>> waiting for Al. But in the mean time see below. >>>>>>> >>>>>>> On 10.06.2016 11:57, Benoit Claise wrote: >>>>>>>> Al, assuming that someone would like to register this metric in >>>>>>>> a >>>>>>> registry >>>>>>>> (RFC6390), are they any grey areas in the performance metric >>>>>>> definitions in >>>>>>>> the draft? >>>>>>>> From what I understand, a point such this one (from Al) is: >>>>>>>> >>>>>>>> Because we are using Goodput, G, I take as given that there >>>>>>>> must be a protocol with retransmission capability. >>>>>>>> Otherwise, further simplification is possible (with dummy >>>>>>> traffic). >>>>>>> >>>>>>> Not really if you have not retransmission, simply your >>>>>>> goodout=throughput. >>>>>>> Don't see a problem here. >>>>>> [ACM] >>>>>> Although Goodput == Throughput for UDP, you can make a simpler >>>>>> measurement, you don't have to check for uniqueness. >>>>> >>>>> That's the view from someone measuring in the network. But if you >>>>> do simula
Re: [aqm] Benoit Claise's Discuss on draft-ietf-aqm-eval-guidelines-11: (with DISCUSS and COMMENT)
Yes, already contacted the authors! Thanks all! > Am 14.06.2016 um 08:42 schrieb Benoit Claise <bcla...@cisco.com>: > > >>> -Original Message- >>> From: Mirja Kühlewind [mailto:mirja.kuehlew...@tik.ee.ethz.ch] >>> Sent: Monday, June 13, 2016 3:41 PM >> ... >>> Hi Al, >>> >>> I believe, we agree here. However, I’m not really sure what needs to be >>> changed/added in the draft now. The only concrete item I have is >>> replacing "application-level“ by "transport-layer payload“. Anything >>> else? >>> >>> Mirja >> [ACM] >> Thanks, that would resolve the biggest ambiguity for me. >> Like I said last week, I think we're done (with that change). > Thank you Al and Mirja. > I'll clear the DISCUSS on that basis, trusting the AD that the addition will > be introduced. > > Regards, Benoit >> >> Al >> >>> >>>> Am 10.06.2016 um 19:16 schrieb MORTON, ALFRED C (AL) >>> <acmor...@att.com>: >>>> more below, thanks for the clarifications, Mirja! >>>> Al >>>> >>>>> -Original Message- >>>>> From: Mirja Kühlewind [mailto:mirja.kuehlew...@tik.ee.ethz.ch] >>>>> Sent: Friday, June 10, 2016 12:55 PM >>>>> To: MORTON, ALFRED C (AL); Mirja Kühlewind; Benoit Claise >>>>> Cc: w...@mti-systems.com; aqm-cha...@ietf.org; The IESG; draft-ietf- >>> aqm- >>>>> eval-guideli...@ietf.org; Schulthess Nicolas (F); aqm@ietf.org >>>>> Subject: Re: [aqm] Benoit Claise's Discuss on draft-ietf-aqm-eval- >>>>> guidelines-11: (with DISCUSS and COMMENT) >>>>> >>>>> Hi Al, >>>>> >>>>> see below. >>>>> >>>>> On 10.06.2016 18:41, MORTON, ALFRED C (AL) wrote: >>>>>> Hi, see below, >>>>>> >>>>>>> -Original Message- >>>>>>> From: Mirja Kühlewind [mailto:i...@kuehlewind.net] >>>>>>> Sent: Friday, June 10, 2016 9:15 AM >>>>>>> To: Benoit Claise; MORTON, ALFRED C (AL) >>>>>>> Cc: w...@mti-systems.com; aqm-cha...@ietf.org; The IESG; draft-ietf- >>>>> aqm- >>>>>>> eval-guideli...@ietf.org; Schulthess Nicolas (F); aqm@ietf.org >>>>>>> Subject: Re: [aqm] Benoit Claise's Discuss on draft-ietf-aqm-eval- >>>>>>> guidelines-11: (with DISCUSS and COMMENT) >>>>>>> >>>>>>> Benoit, >>>>>>> >>>>>>> waiting for Al. But in the mean time see below. >>>>>>> >>>>>>> On 10.06.2016 11:57, Benoit Claise wrote: >>>>>>>> Al, assuming that someone would like to register this metric in a >>>>>>> registry >>>>>>>> (RFC6390), are they any grey areas in the performance metric >>>>>>> definitions in >>>>>>>> the draft? >>>>>>>> From what I understand, a point such this one (from Al) is: >>>>>>>> >>>>>>>> Because we are using Goodput, G, I take as given that there >>>>>>>> must be a protocol with retransmission capability. >>>>>>>> Otherwise, further simplification is possible (with dummy >>>>>>> traffic). >>>>>>> >>>>>>> Not really if you have not retransmission, simply your >>>>>>> goodout=throughput. >>>>>>> Don't see a problem here. >>>>>> [ACM] >>>>>> Although Goodput == Throughput for UDP, you can make a >>>>>> simpler measurement, you don't have to check for uniqueness. >>>>> >>>>> That's the view from someone measuring in the network. But if you do >>>>> simulations or have a controlled testbed, the easiest things is to >>>>> measure in >>>>> the application (and you automatically get the right thing). As we >>> don't >>>>> know >>>>> what exactly people do in the end, I think it is right to leave this >>>>> open >>>>> (and leave it as simple as possible in the description text). >>>> [ACM] >>>> Ok, but what layer of the application? The raw media stream(s)? >>>> Or everything in the TCP/UDP payload? >>>> >>>> In lab benchmarki
Re: [aqm] Benoit Claise's Discuss on draft-ietf-aqm-eval-guidelines-11: (with DISCUSS and COMMENT)
-Original Message- From: Mirja Kühlewind [mailto:mirja.kuehlew...@tik.ee.ethz.ch] Sent: Monday, June 13, 2016 3:41 PM ... Hi Al, I believe, we agree here. However, I’m not really sure what needs to be changed/added in the draft now. The only concrete item I have is replacing "application-level“ by "transport-layer payload“. Anything else? Mirja [ACM] Thanks, that would resolve the biggest ambiguity for me. Like I said last week, I think we're done (with that change). Thank you Al and Mirja. I'll clear the DISCUSS on that basis, trusting the AD that the addition will be introduced. Regards, Benoit Al Am 10.06.2016 um 19:16 schrieb MORTON, ALFRED C (AL) <acmor...@att.com>: more below, thanks for the clarifications, Mirja! Al -Original Message- From: Mirja Kühlewind [mailto:mirja.kuehlew...@tik.ee.ethz.ch] Sent: Friday, June 10, 2016 12:55 PM To: MORTON, ALFRED C (AL); Mirja Kühlewind; Benoit Claise Cc: w...@mti-systems.com; aqm-cha...@ietf.org; The IESG; draft-ietf- aqm- eval-guideli...@ietf.org; Schulthess Nicolas (F); aqm@ietf.org Subject: Re: [aqm] Benoit Claise's Discuss on draft-ietf-aqm-eval- guidelines-11: (with DISCUSS and COMMENT) Hi Al, see below. On 10.06.2016 18:41, MORTON, ALFRED C (AL) wrote: Hi, see below, -Original Message- From: Mirja Kühlewind [mailto:i...@kuehlewind.net] Sent: Friday, June 10, 2016 9:15 AM To: Benoit Claise; MORTON, ALFRED C (AL) Cc: w...@mti-systems.com; aqm-cha...@ietf.org; The IESG; draft-ietf- aqm- eval-guideli...@ietf.org; Schulthess Nicolas (F); aqm@ietf.org Subject: Re: [aqm] Benoit Claise's Discuss on draft-ietf-aqm-eval- guidelines-11: (with DISCUSS and COMMENT) Benoit, waiting for Al. But in the mean time see below. On 10.06.2016 11:57, Benoit Claise wrote: Al, assuming that someone would like to register this metric in a registry (RFC6390), are they any grey areas in the performance metric definitions in the draft? From what I understand, a point such this one (from Al) is: Because we are using Goodput, G, I take as given that there must be a protocol with retransmission capability. Otherwise, further simplification is possible (with dummy traffic). Not really if you have not retransmission, simply your goodout=throughput. Don't see a problem here. [ACM] Although Goodput == Throughput for UDP, you can make a simpler measurement, you don't have to check for uniqueness. That's the view from someone measuring in the network. But if you do simulations or have a controlled testbed, the easiest things is to measure in the application (and you automatically get the right thing). As we don't know what exactly people do in the end, I think it is right to leave this open (and leave it as simple as possible in the description text). [ACM] Ok, but what layer of the application? The raw media stream(s)? Or everything in the TCP/UDP payload? In lab benchmarking, it's sometimes about measuring at link speed x number of ports, so every operation makes a difference! But yes, Fs and G need to be reported on payload at the same layer, so the protocol layer chosen is an input parameter for this metric. Yes, it need to be the same layer for all your tests; but the goal is not be compatible with other tests. So it's your decision. It's guidance how you would test AQMs to decide if you want to deploy them in the future (or to show that your AQM has benefits compared to other AQMs such that another guy might deploy this in future). [ACM] The current text mentions the "application layer" but needs to add the note that the layer chosen needs to be specified/included in with the results, so that someone reading results later will know what was tested. There actually is now a sentence saying: "Where flow size is the size of the application-level flow in bits and goodput is the application-level transfer time (described in Section 2.5)." Is this sufficient? [ACM] I don't mean to prolong this, but I haven't been clear: The term "application-level" is ambiguous, it could be RTP, or some other container layer, or one of the MPEG layers, or the raw media/program stream (with our without meta data). If by saying "application-level", the transport-layer payload is meant, I suggest to say that. are we there yet? I know I am :-), it's 19:15 down the road in Geneva! Al Mirja Al ___ aqm mailing list aqm@ietf.org https://www.ietf.org/mailman/listinfo/aqm ___ aqm mailing list aqm@ietf.org https://www.ietf.org/mailman/listinfo/aqm ___ aqm mailing list aqm@ietf.org https://www.ietf.org/mailman/listinfo/aqm
Re: [aqm] Benoit Claise's Discuss on draft-ietf-aqm-eval-guidelines-11: (with DISCUSS and COMMENT)
> -Original Message- > From: Mirja Kühlewind [mailto:mirja.kuehlew...@tik.ee.ethz.ch] > Sent: Monday, June 13, 2016 3:41 PM ... > Hi Al, > > I believe, we agree here. However, I’m not really sure what needs to be > changed/added in the draft now. The only concrete item I have is > replacing "application-level“ by "transport-layer payload“. Anything > else? > > Mirja [ACM] Thanks, that would resolve the biggest ambiguity for me. Like I said last week, I think we're done (with that change). Al > > > > > > Am 10.06.2016 um 19:16 schrieb MORTON, ALFRED C (AL) > <acmor...@att.com>: > > > > more below, thanks for the clarifications, Mirja! > > Al > > > >> -Original Message- > >> From: Mirja Kühlewind [mailto:mirja.kuehlew...@tik.ee.ethz.ch] > >> Sent: Friday, June 10, 2016 12:55 PM > >> To: MORTON, ALFRED C (AL); Mirja Kühlewind; Benoit Claise > >> Cc: w...@mti-systems.com; aqm-cha...@ietf.org; The IESG; draft-ietf- > aqm- > >> eval-guideli...@ietf.org; Schulthess Nicolas (F); aqm@ietf.org > >> Subject: Re: [aqm] Benoit Claise's Discuss on draft-ietf-aqm-eval- > >> guidelines-11: (with DISCUSS and COMMENT) > >> > >> Hi Al, > >> > >> see below. > >> > >> On 10.06.2016 18:41, MORTON, ALFRED C (AL) wrote: > >>> Hi, see below, > >>> > >>>> -Original Message- > >>>> From: Mirja Kühlewind [mailto:i...@kuehlewind.net] > >>>> Sent: Friday, June 10, 2016 9:15 AM > >>>> To: Benoit Claise; MORTON, ALFRED C (AL) > >>>> Cc: w...@mti-systems.com; aqm-cha...@ietf.org; The IESG; draft-ietf- > >> aqm- > >>>> eval-guideli...@ietf.org; Schulthess Nicolas (F); aqm@ietf.org > >>>> Subject: Re: [aqm] Benoit Claise's Discuss on draft-ietf-aqm-eval- > >>>> guidelines-11: (with DISCUSS and COMMENT) > >>>> > >>>> Benoit, > >>>> > >>>> waiting for Al. But in the mean time see below. > >>>> > >>>> On 10.06.2016 11:57, Benoit Claise wrote: > >>>>> Al, assuming that someone would like to register this metric in a > >>>> registry > >>>>> (RFC6390), are they any grey areas in the performance metric > >>>> definitions in > >>>>> the draft? > >>>>> From what I understand, a point such this one (from Al) is: > >>>>> > >>>>> Because we are using Goodput, G, I take as given that there > >>>>> must be a protocol with retransmission capability. > >>>>> Otherwise, further simplification is possible (with dummy > >>>> traffic). > >>>> > >>>> Not really if you have not retransmission, simply your > >>>> goodout=throughput. > >>>> Don't see a problem here. > >>> [ACM] > >>> Although Goodput == Throughput for UDP, you can make a > >>> simpler measurement, you don't have to check for uniqueness. > >> > >> > >> That's the view from someone measuring in the network. But if you do > >> simulations or have a controlled testbed, the easiest things is to > >> measure in > >> the application (and you automatically get the right thing). As we > don't > >> know > >> what exactly people do in the end, I think it is right to leave this > >> open > >> (and leave it as simple as possible in the description text). > > [ACM] > > Ok, but what layer of the application? The raw media stream(s)? > > Or everything in the TCP/UDP payload? > > > > In lab benchmarking, it's sometimes about measuring at > > link speed x number of ports, so every operation makes a difference! > > > >> > >> > >>> > >>>> > >>>>> > >>>>> But yes, Fs and G need to be reported on payload > >>>>> at the same layer, so the protocol layer chosen is > >>>>> an input parameter for this metric. > >>>> > >>>> Yes, it need to be the same layer for all your tests; but the goal > is > >>>> not be > >>>> compatible with other tests. So it's your decision. It's guidance > how > >>>> you > >>>> would test AQMs to decide if you want to deploy them in the future > >> (or > >>>> to > >>>> show
Re: [aqm] Benoit Claise's Discuss on draft-ietf-aqm-eval-guidelines-11: (with DISCUSS and COMMENT)
Hi Al, I believe, we agree here. However, I’m not really sure what needs to be changed/added in the draft now. The only concrete item I have is replacing "application-level“ by "transport-layer payload“. Anything else? Mirja > > Am 10.06.2016 um 19:16 schrieb MORTON, ALFRED C (AL) <acmor...@att.com>: > > more below, thanks for the clarifications, Mirja! > Al > >> -Original Message- >> From: Mirja Kühlewind [mailto:mirja.kuehlew...@tik.ee.ethz.ch] >> Sent: Friday, June 10, 2016 12:55 PM >> To: MORTON, ALFRED C (AL); Mirja Kühlewind; Benoit Claise >> Cc: w...@mti-systems.com; aqm-cha...@ietf.org; The IESG; draft-ietf-aqm- >> eval-guideli...@ietf.org; Schulthess Nicolas (F); aqm@ietf.org >> Subject: Re: [aqm] Benoit Claise's Discuss on draft-ietf-aqm-eval- >> guidelines-11: (with DISCUSS and COMMENT) >> >> Hi Al, >> >> see below. >> >> On 10.06.2016 18:41, MORTON, ALFRED C (AL) wrote: >>> Hi, see below, >>> >>>> -Original Message- >>>> From: Mirja Kühlewind [mailto:i...@kuehlewind.net] >>>> Sent: Friday, June 10, 2016 9:15 AM >>>> To: Benoit Claise; MORTON, ALFRED C (AL) >>>> Cc: w...@mti-systems.com; aqm-cha...@ietf.org; The IESG; draft-ietf- >> aqm- >>>> eval-guideli...@ietf.org; Schulthess Nicolas (F); aqm@ietf.org >>>> Subject: Re: [aqm] Benoit Claise's Discuss on draft-ietf-aqm-eval- >>>> guidelines-11: (with DISCUSS and COMMENT) >>>> >>>> Benoit, >>>> >>>> waiting for Al. But in the mean time see below. >>>> >>>> On 10.06.2016 11:57, Benoit Claise wrote: >>>>> Al, assuming that someone would like to register this metric in a >>>> registry >>>>> (RFC6390), are they any grey areas in the performance metric >>>> definitions in >>>>> the draft? >>>>> From what I understand, a point such this one (from Al) is: >>>>> >>>>> Because we are using Goodput, G, I take as given that there >>>>> must be a protocol with retransmission capability. >>>>> Otherwise, further simplification is possible (with dummy >>>> traffic). >>>> >>>> Not really if you have not retransmission, simply your >>>> goodout=throughput. >>>> Don't see a problem here. >>> [ACM] >>> Although Goodput == Throughput for UDP, you can make a >>> simpler measurement, you don't have to check for uniqueness. >> >> >> That's the view from someone measuring in the network. But if you do >> simulations or have a controlled testbed, the easiest things is to >> measure in >> the application (and you automatically get the right thing). As we don't >> know >> what exactly people do in the end, I think it is right to leave this >> open >> (and leave it as simple as possible in the description text). > [ACM] > Ok, but what layer of the application? The raw media stream(s)? > Or everything in the TCP/UDP payload? > > In lab benchmarking, it's sometimes about measuring at > link speed x number of ports, so every operation makes a difference! > >> >> >>> >>>> >>>>> >>>>> But yes, Fs and G need to be reported on payload >>>>> at the same layer, so the protocol layer chosen is >>>>> an input parameter for this metric. >>>> >>>> Yes, it need to be the same layer for all your tests; but the goal is >>>> not be >>>> compatible with other tests. So it's your decision. It's guidance how >>>> you >>>> would test AQMs to decide if you want to deploy them in the future >> (or >>>> to >>>> show that your AQM has benefits compared to other AQMs such that >> another >>>> guy >>>> might deploy this in future). >>> [ACM] >>> >>> The current text mentions the "application layer" but needs to add the >> note >>> that the layer chosen needs to be specified/included in with the >> results, so that >>> someone reading results later will know what was tested. >> >> There actually is now a sentence saying: >> >> "Where flow size is the size of the application-level flow in bits and >> goodput is the application-level transfer time (described in >> Section 2.5)." >> >> Is this sufficient? > [ACM] > > I don't mean to prolong this, but I haven't been clear: > The term "application-level" is ambiguous, it could be > RTP, or some other container layer, or one of the MPEG layers, > or the raw media/program stream (with our without meta data). > > If by saying "application-level", the transport-layer payload > is meant, I suggest to say that. > > are we there yet? I know I am :-), it's 19:15 down the road in Geneva! > Al > >> >> Mirja >> >> >>> >>> Al >>> >>> >>> ___ >>> aqm mailing list >>> aqm@ietf.org >>> https://www.ietf.org/mailman/listinfo/aqm >>> > ___ > aqm mailing list > aqm@ietf.org > https://www.ietf.org/mailman/listinfo/aqm ___ aqm mailing list aqm@ietf.org https://www.ietf.org/mailman/listinfo/aqm
Re: [aqm] Benoit Claise's Discuss on draft-ietf-aqm-eval-guidelines-11: (with DISCUSS and COMMENT)
more below, thanks for the clarifications, Mirja! Al > -Original Message- > From: Mirja Kühlewind [mailto:mirja.kuehlew...@tik.ee.ethz.ch] > Sent: Friday, June 10, 2016 12:55 PM > To: MORTON, ALFRED C (AL); Mirja Kühlewind; Benoit Claise > Cc: w...@mti-systems.com; aqm-cha...@ietf.org; The IESG; draft-ietf-aqm- > eval-guideli...@ietf.org; Schulthess Nicolas (F); aqm@ietf.org > Subject: Re: [aqm] Benoit Claise's Discuss on draft-ietf-aqm-eval- > guidelines-11: (with DISCUSS and COMMENT) > > Hi Al, > > see below. > > On 10.06.2016 18:41, MORTON, ALFRED C (AL) wrote: > > Hi, see below, > > > >> -Original Message- > >> From: Mirja Kühlewind [mailto:i...@kuehlewind.net] > >> Sent: Friday, June 10, 2016 9:15 AM > >> To: Benoit Claise; MORTON, ALFRED C (AL) > >> Cc: w...@mti-systems.com; aqm-cha...@ietf.org; The IESG; draft-ietf- > aqm- > >> eval-guideli...@ietf.org; Schulthess Nicolas (F); aqm@ietf.org > >> Subject: Re: [aqm] Benoit Claise's Discuss on draft-ietf-aqm-eval- > >> guidelines-11: (with DISCUSS and COMMENT) > >> > >> Benoit, > >> > >> waiting for Al. But in the mean time see below. > >> > >> On 10.06.2016 11:57, Benoit Claise wrote: > >>> Al, assuming that someone would like to register this metric in a > >> registry > >>> (RFC6390), are they any grey areas in the performance metric > >> definitions in > >>> the draft? > >>> From what I understand, a point such this one (from Al) is: > >>> > >>> Because we are using Goodput, G, I take as given that there > >>> must be a protocol with retransmission capability. > >>> Otherwise, further simplification is possible (with dummy > >> traffic). > >> > >> Not really if you have not retransmission, simply your > >> goodout=throughput. > >> Don't see a problem here. > > [ACM] > > Although Goodput == Throughput for UDP, you can make a > > simpler measurement, you don't have to check for uniqueness. > > > That's the view from someone measuring in the network. But if you do > simulations or have a controlled testbed, the easiest things is to > measure in > the application (and you automatically get the right thing). As we don't > know > what exactly people do in the end, I think it is right to leave this > open > (and leave it as simple as possible in the description text). [ACM] Ok, but what layer of the application? The raw media stream(s)? Or everything in the TCP/UDP payload? In lab benchmarking, it's sometimes about measuring at link speed x number of ports, so every operation makes a difference! > > > > > >> > >>> > >>> But yes, Fs and G need to be reported on payload > >>> at the same layer, so the protocol layer chosen is > >>> an input parameter for this metric. > >> > >> Yes, it need to be the same layer for all your tests; but the goal is > >> not be > >> compatible with other tests. So it's your decision. It's guidance how > >> you > >> would test AQMs to decide if you want to deploy them in the future > (or > >> to > >> show that your AQM has benefits compared to other AQMs such that > another > >> guy > >> might deploy this in future). > > [ACM] > > > > The current text mentions the "application layer" but needs to add the > note > > that the layer chosen needs to be specified/included in with the > results, so that > > someone reading results later will know what was tested. > > There actually is now a sentence saying: > > "Where flow size is the size of the application-level flow in bits and > goodput is the application-level transfer time (described in > Section 2.5)." > > Is this sufficient? [ACM] I don't mean to prolong this, but I haven't been clear: The term "application-level" is ambiguous, it could be RTP, or some other container layer, or one of the MPEG layers, or the raw media/program stream (with our without meta data). If by saying "application-level", the transport-layer payload is meant, I suggest to say that. are we there yet? I know I am :-), it's 19:15 down the road in Geneva! Al > > Mirja > > > > > > Al > > > > > > ___ > > aqm mailing list > > aqm@ietf.org > > https://www.ietf.org/mailman/listinfo/aqm > > ___ aqm mailing list aqm@ietf.org https://www.ietf.org/mailman/listinfo/aqm
Re: [aqm] Benoit Claise's Discuss on draft-ietf-aqm-eval-guidelines-11: (with DISCUSS and COMMENT)
Hi, see below, > -Original Message- > From: Mirja Kühlewind [mailto:i...@kuehlewind.net] > Sent: Friday, June 10, 2016 9:15 AM > To: Benoit Claise; MORTON, ALFRED C (AL) > Cc: w...@mti-systems.com; aqm-cha...@ietf.org; The IESG; draft-ietf-aqm- > eval-guideli...@ietf.org; Schulthess Nicolas (F); aqm@ietf.org > Subject: Re: [aqm] Benoit Claise's Discuss on draft-ietf-aqm-eval- > guidelines-11: (with DISCUSS and COMMENT) > > Benoit, > > waiting for Al. But in the mean time see below. > > On 10.06.2016 11:57, Benoit Claise wrote: > > Al, assuming that someone would like to register this metric in a > registry > > (RFC6390), are they any grey areas in the performance metric > definitions in > > the draft? > > From what I understand, a point such this one (from Al) is: > > > > Because we are using Goodput, G, I take as given that there > > must be a protocol with retransmission capability. > > Otherwise, further simplification is possible (with dummy > traffic). > > Not really if you have not retransmission, simply your > goodout=throughput. > Don't see a problem here. [ACM] Although Goodput == Throughput for UDP, you can make a simpler measurement, you don't have to check for uniqueness. > > > > > But yes, Fs and G need to be reported on payload > > at the same layer, so the protocol layer chosen is > > an input parameter for this metric. > > Yes, it need to be the same layer for all your tests; but the goal is > not be > compatible with other tests. So it's your decision. It's guidance how > you > would test AQMs to decide if you want to deploy them in the future (or > to > show that your AQM has benefits compared to other AQMs such that another > guy > might deploy this in future). [ACM] The current text mentions the "application layer" but needs to add the note that the layer chosen needs to be specified/included in with the results, so that someone reading results later will know what was tested. Al ___ aqm mailing list aqm@ietf.org https://www.ietf.org/mailman/listinfo/aqm
Re: [aqm] Benoit Claise's Discuss on draft-ietf-aqm-eval-guidelines-11: (with DISCUSS and COMMENT)
Hi Mirja, Hi Benoit, I would not expect to see a bis doc here. As I said I would rather expect a new, separate document to specify and register the metrics (and providing more details). Given the very different scope and use case, I would not even see it as a problem if those two documents don't align fully. I can go back to the authors and ask if they would be interested in a directorate review. But I guess that would also mean that they first have to change the text in section 2.X to the format/structure guidelines in RFC6390 (sec 5.4.2 only in this case, I'd say), correct? Not necessarily. The only question is that important in my mind is: Al, assuming that someone would like to register this metric in a registry (RFC6390), are they any grey areas in the performance metric definitions in the draft? From what I understand, a point such this one (from Al) is: Because we are using Goodput, G, I take as given that there must be a protocol with retransmission capability. Otherwise, further simplification is possible (with dummy traffic). But yes, Fs and G need to be reported on payload at the same layer, so the protocol layer chosen is an input parameter for this metric. I still have to say that looking at RFC6390 again I find it not fully applicable for this case; e.g. Measurement Timing is not a great fit when you talk about a one time measurement (for one test run) of the flow completion time. I'd fear that using the template provided in RFC6390 would make it rather more confusing than clear. Fine with that. Regards, Benoit Mirja On 10.06.2016 09:36, Benoit Claise wrote: Thanks Al, This is exactly the type of feedback requested from the Performance Metric Directorate. Mirja, I buy into your argument: I guess as soon as we have a registry, maybe there is someone interest in IPPM to catch up these metrics again and provide a RFC6390 definition but I would rather not like this document doing it. As such, we probably don't need the RFC6390 template in this document. Fair enough. However, what would be a pity is that if/when someone would like to register this metric, we end with questions such Al's one below, because the metric definitions in the published RFC are not precise. That could result in a RFCbis. Therefore, going through the Performance Metric Directorate feedback now is important IMO. Regards, Benoit Because we are using Goodput, G, I take as given that there must be a protocol with retransmission capability. Otherwise, further simplification is possible (with dummy traffic). But yes, Fs and G need to be reported on payload at the same layer, so the protocol layer chosen is an input parameter for this metric. Al -Original Message- From: Mirja Kuehlewind (IETF) [mailto:i...@kuehlewind.net] Sent: Wednesday, June 08, 2016 7:21 AM To: MORTON, ALFRED C (AL) Cc:w...@mti-systems.com;aqm-cha...@ietf.org; The IESG; draft-ietf-aqm- eval-guideli...@ietf.org; Benoit Claise;aqm@ietf.org Subject: Re: [aqm] Benoit Claise's Discuss on draft-ietf-aqm-eval- guidelines-11: (with DISCUSS and COMMENT) Actually, it really doesn't matter that much in this case, I’d say. As we are talking about a lab environment, you might use dummy traffic that has some headers or not, that you might take into account of not, mostly depending on which information can be more easily accessed. What is important is that you do the same thing for all schemes that you compare. I guess one could add a note that there are different ways to measure this and that it is important to measure G at the same layer. Does that make sense? Mirja Am 08.06.2016 um 13:03 schrieb MORTON, ALFRED C (AL) <acmor...@att.com>: Here's one area which could use more detail: ...The Flow Completion Time (FCT) is related to the flow size (Fs) and the goodput for the flow (G) as follows: FCT [s] = Fs [Byte] / ( G [Bit/s] / 8 [Bit/Byte] ) What protocol layers are included and excluded from Fs? Also, G needs to be measured at the same layer, and the definition in RFC 2647 is a bit vague about layers, too. It would be good to clarify which bytes to count here. Al -Original Message- From: Mirja Kuehlewind (IETF) [mailto:i...@kuehlewind.net] Sent: Wednesday, June 08, 2016 5:40 AM To: MORTON, ALFRED C (AL) Cc: Benoit Claise;w...@mti-systems.com;aqm-cha...@ietf.org; aqm@ietf.org;draft-ietf-aqm-eval-guideli...@ietf.org; The IESG Subject: Re: [aqm] Benoit Claise's Discuss on draft-ietf-aqm-eval- guidelines-11: (with DISCUSS and COMMENT) Hi Al, what kind of detail are you looking for? Because I thought with the given equation this one was pretty clear. Do you have a reference to the benchmarking work? Mirja Am 08.06.2016 um 11:18 schrieb MORTON, ALFRED C (AL) <acmor...@att.com>: Hi Mirja, That sounds fairly reasonable to me. Would it be possible to ask the authors provide a bit more detail on Flow Completion Time? Flow Com
Re: [aqm] Benoit Claise's Discuss on draft-ietf-aqm-eval-guidelines-11: (with DISCUSS and COMMENT)
Hi Benoit, I would not expect to see a bis doc here. As I said I would rather expect a new, separate document to specify and register the metrics (and providing more details). Given the very different scope and use case, I would not even see it as a problem if those two documents don't align fully. I can go back to the authors and ask if they would be interested in a directorate review. But I guess that would also mean that they first have to change the text in section 2.X to the format/structure guidelines in RFC6390 (sec 5.4.2 only in this case, I'd say), correct? I still have to say that looking at RFC6390 again I find it not fully applicable for this case; e.g. Measurement Timing is not a great fit when you talk about a one time measurement (for one test run) of the flow completion time. I'd fear that using the template provided in RFC6390 would make it rather more confusing than clear. Mirja On 10.06.2016 09:36, Benoit Claise wrote: Thanks Al, This is exactly the type of feedback requested from the Performance Metric Directorate. Mirja, I buy into your argument: I guess as soon as we have a registry, maybe there is someone interest in IPPM to catch up these metrics again and provide a RFC6390 definition but I would rather not like this document doing it. As such, we probably don't need the RFC6390 template in this document. Fair enough. However, what would be a pity is that if/when someone would like to register this metric, we end with questions such Al's one below, because the metric definitions in the published RFC are not precise. That could result in a RFCbis. Therefore, going through the Performance Metric Directorate feedback now is important IMO. Regards, Benoit Because we are using Goodput, G, I take as given that there must be a protocol with retransmission capability. Otherwise, further simplification is possible (with dummy traffic). But yes, Fs and G need to be reported on payload at the same layer, so the protocol layer chosen is an input parameter for this metric. Al -Original Message- From: Mirja Kuehlewind (IETF) [mailto:i...@kuehlewind.net] Sent: Wednesday, June 08, 2016 7:21 AM To: MORTON, ALFRED C (AL) Cc:w...@mti-systems.com;aqm-cha...@ietf.org; The IESG; draft-ietf-aqm- eval-guideli...@ietf.org; Benoit Claise;aqm@ietf.org Subject: Re: [aqm] Benoit Claise's Discuss on draft-ietf-aqm-eval- guidelines-11: (with DISCUSS and COMMENT) Actually, it really doesn't matter that much in this case, I’d say. As we are talking about a lab environment, you might use dummy traffic that has some headers or not, that you might take into account of not, mostly depending on which information can be more easily accessed. What is important is that you do the same thing for all schemes that you compare. I guess one could add a note that there are different ways to measure this and that it is important to measure G at the same layer. Does that make sense? Mirja Am 08.06.2016 um 13:03 schrieb MORTON, ALFRED C (AL) <acmor...@att.com>: Here's one area which could use more detail: ...The Flow Completion Time (FCT) is related to the flow size (Fs) and the goodput for the flow (G) as follows: FCT [s] = Fs [Byte] / ( G [Bit/s] / 8 [Bit/Byte] ) What protocol layers are included and excluded from Fs? Also, G needs to be measured at the same layer, and the definition in RFC 2647 is a bit vague about layers, too. It would be good to clarify which bytes to count here. Al -Original Message- From: Mirja Kuehlewind (IETF) [mailto:i...@kuehlewind.net] Sent: Wednesday, June 08, 2016 5:40 AM To: MORTON, ALFRED C (AL) Cc: Benoit Claise;w...@mti-systems.com;aqm-cha...@ietf.org; aqm@ietf.org;draft-ietf-aqm-eval-guideli...@ietf.org; The IESG Subject: Re: [aqm] Benoit Claise's Discuss on draft-ietf-aqm-eval- guidelines-11: (with DISCUSS and COMMENT) Hi Al, what kind of detail are you looking for? Because I thought with the given equation this one was pretty clear. Do you have a reference to the benchmarking work? Mirja Am 08.06.2016 um 11:18 schrieb MORTON, ALFRED C (AL) <acmor...@att.com>: Hi Mirja, That sounds fairly reasonable to me. Would it be possible to ask the authors provide a bit more detail on Flow Completion Time? Flow Completion Time is close to a definition for a new metric, and could benefit from more attention, perhaps a few more details. RFC6390 will provide some areas for improvement. I imagine that related benchmarking efforts may wish to measure this metric, and there would be independent implementations based on the description provided here. regards from Geneve' Al -Original Message- From: Mirja Kuehlewind (IETF) [mailto:i...@kuehlewind.net] Sent: Wednesday, June 08, 2016 4:46 AM To: MORTON, ALFRED C (AL); Benoit Claise Cc:w...@mti-systems.com;aqm@ietf.org; The IESG; draft-ietf-aqm- eval- guideli...@ietf.org;aqm-cha...@ietf.org Subject: Re: [aqm] Benoit Claise's Discuss on draft-i
Re: [aqm] Benoit Claise's Discuss on draft-ietf-aqm-eval-guidelines-11: (with DISCUSS and COMMENT)
Thanks Al, This is exactly the type of feedback requested from the Performance Metric Directorate. Mirja, I buy into your argument: I guess as soon as we have a registry, maybe there is someone interest in IPPM to catch up these metrics again and provide a RFC6390 definition but I would rather not like this document doing it. As such, we probably don't need the RFC6390 template in this document. Fair enough. However, what would be a pity is that if/when someone would like to register this metric, we end with questions such Al's one below, because the metric definitions in the published RFC are not precise. That could result in a RFCbis. Therefore, going through the Performance Metric Directorate feedback now is important IMO. Regards, Benoit Because we are using Goodput, G, I take as given that there must be a protocol with retransmission capability. Otherwise, further simplification is possible (with dummy traffic). But yes, Fs and G need to be reported on payload at the same layer, so the protocol layer chosen is an input parameter for this metric. Al -Original Message- From: Mirja Kuehlewind (IETF) [mailto:i...@kuehlewind.net] Sent: Wednesday, June 08, 2016 7:21 AM To: MORTON, ALFRED C (AL) Cc: w...@mti-systems.com; aqm-cha...@ietf.org; The IESG; draft-ietf-aqm- eval-guideli...@ietf.org; Benoit Claise; aqm@ietf.org Subject: Re: [aqm] Benoit Claise's Discuss on draft-ietf-aqm-eval- guidelines-11: (with DISCUSS and COMMENT) Actually, it really doesn't matter that much in this case, I’d say. As we are talking about a lab environment, you might use dummy traffic that has some headers or not, that you might take into account of not, mostly depending on which information can be more easily accessed. What is important is that you do the same thing for all schemes that you compare. I guess one could add a note that there are different ways to measure this and that it is important to measure G at the same layer. Does that make sense? Mirja Am 08.06.2016 um 13:03 schrieb MORTON, ALFRED C (AL) <acmor...@att.com>: Here's one area which could use more detail: ...The Flow Completion Time (FCT) is related to the flow size (Fs) and the goodput for the flow (G) as follows: FCT [s] = Fs [Byte] / ( G [Bit/s] / 8 [Bit/Byte] ) What protocol layers are included and excluded from Fs? Also, G needs to be measured at the same layer, and the definition in RFC 2647 is a bit vague about layers, too. It would be good to clarify which bytes to count here. Al -Original Message- From: Mirja Kuehlewind (IETF) [mailto:i...@kuehlewind.net] Sent: Wednesday, June 08, 2016 5:40 AM To: MORTON, ALFRED C (AL) Cc: Benoit Claise; w...@mti-systems.com; aqm-cha...@ietf.org; aqm@ietf.org; draft-ietf-aqm-eval-guideli...@ietf.org; The IESG Subject: Re: [aqm] Benoit Claise's Discuss on draft-ietf-aqm-eval- guidelines-11: (with DISCUSS and COMMENT) Hi Al, what kind of detail are you looking for? Because I thought with the given equation this one was pretty clear. Do you have a reference to the benchmarking work? Mirja Am 08.06.2016 um 11:18 schrieb MORTON, ALFRED C (AL) <acmor...@att.com>: Hi Mirja, That sounds fairly reasonable to me. Would it be possible to ask the authors provide a bit more detail on Flow Completion Time? Flow Completion Time is close to a definition for a new metric, and could benefit from more attention, perhaps a few more details. RFC6390 will provide some areas for improvement. I imagine that related benchmarking efforts may wish to measure this metric, and there would be independent implementations based on the description provided here. regards from Geneve' Al -Original Message- From: Mirja Kuehlewind (IETF) [mailto:i...@kuehlewind.net] Sent: Wednesday, June 08, 2016 4:46 AM To: MORTON, ALFRED C (AL); Benoit Claise Cc: w...@mti-systems.com; aqm@ietf.org; The IESG; draft-ietf-aqm- eval- guideli...@ietf.org; aqm-cha...@ietf.org Subject: Re: [aqm] Benoit Claise's Discuss on draft-ietf-aqm-eval- guidelines-11: (with DISCUSS and COMMENT) Hi Benoit, I finally had another look at the document as well as at RFC6390. I guess the metrics in question are Flow completion time (sec 2.1.), Flow start up time (2.2.) and Packet loss synchronization (2.4.). And you are right that these metric could be see as Application-Specific Performance Metric as defined in RFC6390. However, I agree with Al that given the scope of this document is providing "a generic list of scenarios against which an AQM proposal should be evaluated, considering both potential performance gain and safety of deployment.“, I don’t think these metric need to be defined and registered this way. I guess as soon as we have a registry, maybe there is someone interest in IPPM to catch up these metrics again and provide a RFC6390 definition but I would rather not like this document doing it. Is that acceptable for you? We could add
Re: [aqm] Benoit Claise's Discuss on draft-ietf-aqm-eval-guidelines-11: (with DISCUSS and COMMENT)
Actually, it really doesn't matter that much in this case, I’d say. As we are talking about a lab environment, you might use dummy traffic that has some headers or not, that you might take into account of not, mostly depending on which information can be more easily accessed. What is important is that you do the same thing for all schemes that you compare. I guess one could add a note that there are different ways to measure this and that it is important to measure G at the same layer. Does that make sense? Mirja > Am 08.06.2016 um 13:03 schrieb MORTON, ALFRED C (AL) <acmor...@att.com>: > > Here's one area which could use more detail: > > ...The Flow Completion Time (FCT) is > related to the flow size (Fs) and the goodput for the flow (G) as > follows: > > FCT [s] = Fs [Byte] / ( G [Bit/s] / 8 [Bit/Byte] ) > > What protocol layers are included and excluded from Fs? > > Also, G needs to be measured at the same layer, and the > definition in RFC 2647 is a bit vague about layers, too. > It would be good to clarify which bytes to count here. > > Al > >> -Original Message- >> From: Mirja Kuehlewind (IETF) [mailto:i...@kuehlewind.net] >> Sent: Wednesday, June 08, 2016 5:40 AM >> To: MORTON, ALFRED C (AL) >> Cc: Benoit Claise; w...@mti-systems.com; aqm-cha...@ietf.org; >> aqm@ietf.org; draft-ietf-aqm-eval-guideli...@ietf.org; The IESG >> Subject: Re: [aqm] Benoit Claise's Discuss on draft-ietf-aqm-eval- >> guidelines-11: (with DISCUSS and COMMENT) >> >> Hi Al, >> >> what kind of detail are you looking for? Because I thought with the >> given equation this one was pretty clear. >> >> Do you have a reference to the benchmarking work? >> >> Mirja >> >> >>> Am 08.06.2016 um 11:18 schrieb MORTON, ALFRED C (AL) >> <acmor...@att.com>: >>> >>> Hi Mirja, >>> >>> That sounds fairly reasonable to me. >>> Would it be possible to ask the authors provide a bit more >>> detail on Flow Completion Time? >>> >>>>>> Flow Completion Time is close to a definition for a new metric, >>>>>> and could benefit from more attention, perhaps a few more details. >>>>>> RFC6390 will provide some areas for improvement. >>> >>> I imagine that related benchmarking efforts may wish to measure this >>> metric, and there would be independent implementations based on >>> the description provided here. >>> >>> regards from Geneve' >>> Al >>> >>>> -Original Message----- >>>> From: Mirja Kuehlewind (IETF) [mailto:i...@kuehlewind.net] >>>> Sent: Wednesday, June 08, 2016 4:46 AM >>>> To: MORTON, ALFRED C (AL); Benoit Claise >>>> Cc: w...@mti-systems.com; aqm@ietf.org; The IESG; draft-ietf-aqm-eval- >>>> guideli...@ietf.org; aqm-cha...@ietf.org >>>> Subject: Re: [aqm] Benoit Claise's Discuss on draft-ietf-aqm-eval- >>>> guidelines-11: (with DISCUSS and COMMENT) >>>> >>>> Hi Benoit, >>>> >>>> I finally had another look at the document as well as at RFC6390. I >>>> guess the metrics in question are Flow completion time (sec 2.1.), >> Flow >>>> start up time (2.2.) and Packet loss synchronization (2.4.). And you >> are >>>> right that these metric could be see as Application-Specific >> Performance >>>> Metric as defined in RFC6390. However, I agree with Al that given the >>>> scope of this document is providing >>>> "a generic list of scenarios against which an >>>> AQM proposal should be evaluated, considering both potential >>>> performance gain and safety of deployment.“, >>>> I don’t think these metric need to be defined and registered this >> way. >>>> >>>> I guess as soon as we have a registry, maybe there is someone >> interest >>>> in IPPM to catch up these metrics again and provide a RFC6390 >> definition >>>> but I would rather not like this document doing it. >>>> >>>> Is that acceptable for you? >>>> >>>> We could add one more sentence in the abstract to make the scope on >> lab >>>> testing during development and before deployment clear. Would that >> help? >>>> >>>> Mirja >>>> >>>> >>>> >>>>> Am 25.05.2016 um 14:22 schrieb Mirja Kuehlewind (IETF) >>>> <i...@kuehlewind.net>: >>>>> >>>>
Re: [aqm] Benoit Claise's Discuss on draft-ietf-aqm-eval-guidelines-11: (with DISCUSS and COMMENT)
Here's one area which could use more detail: ...The Flow Completion Time (FCT) is related to the flow size (Fs) and the goodput for the flow (G) as follows: FCT [s] = Fs [Byte] / ( G [Bit/s] / 8 [Bit/Byte] ) What protocol layers are included and excluded from Fs? Also, G needs to be measured at the same layer, and the definition in RFC 2647 is a bit vague about layers, too. It would be good to clarify which bytes to count here. Al > -Original Message- > From: Mirja Kuehlewind (IETF) [mailto:i...@kuehlewind.net] > Sent: Wednesday, June 08, 2016 5:40 AM > To: MORTON, ALFRED C (AL) > Cc: Benoit Claise; w...@mti-systems.com; aqm-cha...@ietf.org; > aqm@ietf.org; draft-ietf-aqm-eval-guideli...@ietf.org; The IESG > Subject: Re: [aqm] Benoit Claise's Discuss on draft-ietf-aqm-eval- > guidelines-11: (with DISCUSS and COMMENT) > > Hi Al, > > what kind of detail are you looking for? Because I thought with the > given equation this one was pretty clear. > > Do you have a reference to the benchmarking work? > > Mirja > > > > Am 08.06.2016 um 11:18 schrieb MORTON, ALFRED C (AL) > <acmor...@att.com>: > > > > Hi Mirja, > > > > That sounds fairly reasonable to me. > > Would it be possible to ask the authors provide a bit more > > detail on Flow Completion Time? > > > >>>> Flow Completion Time is close to a definition for a new metric, > >>>> and could benefit from more attention, perhaps a few more details. > >>>> RFC6390 will provide some areas for improvement. > > > > I imagine that related benchmarking efforts may wish to measure this > > metric, and there would be independent implementations based on > > the description provided here. > > > > regards from Geneve' > > Al > > > >> -Original Message- > >> From: Mirja Kuehlewind (IETF) [mailto:i...@kuehlewind.net] > >> Sent: Wednesday, June 08, 2016 4:46 AM > >> To: MORTON, ALFRED C (AL); Benoit Claise > >> Cc: w...@mti-systems.com; aqm@ietf.org; The IESG; draft-ietf-aqm-eval- > >> guideli...@ietf.org; aqm-cha...@ietf.org > >> Subject: Re: [aqm] Benoit Claise's Discuss on draft-ietf-aqm-eval- > >> guidelines-11: (with DISCUSS and COMMENT) > >> > >> Hi Benoit, > >> > >> I finally had another look at the document as well as at RFC6390. I > >> guess the metrics in question are Flow completion time (sec 2.1.), > Flow > >> start up time (2.2.) and Packet loss synchronization (2.4.). And you > are > >> right that these metric could be see as Application-Specific > Performance > >> Metric as defined in RFC6390. However, I agree with Al that given the > >> scope of this document is providing > >> "a generic list of scenarios against which an > >> AQM proposal should be evaluated, considering both potential > >> performance gain and safety of deployment.“, > >> I don’t think these metric need to be defined and registered this > way. > >> > >> I guess as soon as we have a registry, maybe there is someone > interest > >> in IPPM to catch up these metrics again and provide a RFC6390 > definition > >> but I would rather not like this document doing it. > >> > >> Is that acceptable for you? > >> > >> We could add one more sentence in the abstract to make the scope on > lab > >> testing during development and before deployment clear. Would that > help? > >> > >> Mirja > >> > >> > >> > >>> Am 25.05.2016 um 14:22 schrieb Mirja Kuehlewind (IETF) > >> <i...@kuehlewind.net>: > >>> > >>> Hi Al, Benoit, hi all, > >>> > >>> thanks for the feedback. Sorry for me delaying this maybe a little > but > >> I need t have another look at the document which will be next week at > >> this point. In general I agree that this does not need to only rely > on > >> registered metrics because is mostly for lab tests; further this > might > >> probably not the right doc to register new metrics. However, I would > >> still like to have another look at the doc and see if we can improve > >> anything or figure out if any of the ’new’/non-registed metrics > >> should/could be taken up by e.g. ippm. > >>> > >>> Mirja > >>> > >>> > >>> > >>>> Am 20.05.2016 um 14:53 schrieb MORTON, ALFRED C (AL) > >> <acmor...@att.com>: &g
Re: [aqm] Benoit Claise's Discuss on draft-ietf-aqm-eval-guidelines-11: (with DISCUSS and COMMENT)
Hi Al, what kind of detail are you looking for? Because I thought with the given equation this one was pretty clear. Do you have a reference to the benchmarking work? Mirja > Am 08.06.2016 um 11:18 schrieb MORTON, ALFRED C (AL) <acmor...@att.com>: > > Hi Mirja, > > That sounds fairly reasonable to me. > Would it be possible to ask the authors provide a bit more > detail on Flow Completion Time? > >>>> Flow Completion Time is close to a definition for a new metric, >>>> and could benefit from more attention, perhaps a few more details. >>>> RFC6390 will provide some areas for improvement. > > I imagine that related benchmarking efforts may wish to measure this > metric, and there would be independent implementations based on > the description provided here. > > regards from Geneve' > Al > >> -Original Message- >> From: Mirja Kuehlewind (IETF) [mailto:i...@kuehlewind.net] >> Sent: Wednesday, June 08, 2016 4:46 AM >> To: MORTON, ALFRED C (AL); Benoit Claise >> Cc: w...@mti-systems.com; aqm@ietf.org; The IESG; draft-ietf-aqm-eval- >> guideli...@ietf.org; aqm-cha...@ietf.org >> Subject: Re: [aqm] Benoit Claise's Discuss on draft-ietf-aqm-eval- >> guidelines-11: (with DISCUSS and COMMENT) >> >> Hi Benoit, >> >> I finally had another look at the document as well as at RFC6390. I >> guess the metrics in question are Flow completion time (sec 2.1.), Flow >> start up time (2.2.) and Packet loss synchronization (2.4.). And you are >> right that these metric could be see as Application-Specific Performance >> Metric as defined in RFC6390. However, I agree with Al that given the >> scope of this document is providing >> "a generic list of scenarios against which an >> AQM proposal should be evaluated, considering both potential >> performance gain and safety of deployment.“, >> I don’t think these metric need to be defined and registered this way. >> >> I guess as soon as we have a registry, maybe there is someone interest >> in IPPM to catch up these metrics again and provide a RFC6390 definition >> but I would rather not like this document doing it. >> >> Is that acceptable for you? >> >> We could add one more sentence in the abstract to make the scope on lab >> testing during development and before deployment clear. Would that help? >> >> Mirja >> >> >> >>> Am 25.05.2016 um 14:22 schrieb Mirja Kuehlewind (IETF) >> <i...@kuehlewind.net>: >>> >>> Hi Al, Benoit, hi all, >>> >>> thanks for the feedback. Sorry for me delaying this maybe a little but >> I need t have another look at the document which will be next week at >> this point. In general I agree that this does not need to only rely on >> registered metrics because is mostly for lab tests; further this might >> probably not the right doc to register new metrics. However, I would >> still like to have another look at the doc and see if we can improve >> anything or figure out if any of the ’new’/non-registed metrics >> should/could be taken up by e.g. ippm. >>> >>> Mirja >>> >>> >>> >>>> Am 20.05.2016 um 14:53 schrieb MORTON, ALFRED C (AL) >> <acmor...@att.com>: >>>> >>>> All, >>>> a few replies in-line below, >>>> Al >>>> >>>>> -Original Message- >>>>> From: Benoit Claise [mailto:bcla...@cisco.com] >>>>> Sent: Thursday, May 19, 2016 5:38 AM >>>>> To: The IESG >>>>> Cc: draft-ietf-aqm-eval-guideli...@ietf.org; w...@mti-systems.com; >> aqm- >>>>> cha...@ietf.org; w...@mti-systems.com; aqm@ietf.org; linda Dunbar; >>>>> MORTON, ALFRED C (AL) >>>>> Subject: Benoit Claise's Discuss on draft-ietf-aqm-eval-guidelines- >> 11: >>>>> (with DISCUSS and COMMENT) >>>>> >>>>> Benoit Claise has entered the following ballot position for >>>>> draft-ietf-aqm-eval-guidelines-11: Discuss >>>>> >>>> ... >>>>> >> -- >>>>> DISCUSS: >>>>> >> -- >>>>> >>>>> Has a RFC6390 performance directorate review done for the 2.X >> metrics? >>>>> It >>>>> should. >>>> [ACM] >>>> I reviewed this draft about 18 mon
Re: [aqm] Benoit Claise's Discuss on draft-ietf-aqm-eval-guidelines-11: (with DISCUSS and COMMENT)
Hi Mirja, That sounds fairly reasonable to me. Would it be possible to ask the authors provide a bit more detail on Flow Completion Time? > >> Flow Completion Time is close to a definition for a new metric, > >> and could benefit from more attention, perhaps a few more details. > >> RFC6390 will provide some areas for improvement. I imagine that related benchmarking efforts may wish to measure this metric, and there would be independent implementations based on the description provided here. regards from Geneve' Al > -Original Message- > From: Mirja Kuehlewind (IETF) [mailto:i...@kuehlewind.net] > Sent: Wednesday, June 08, 2016 4:46 AM > To: MORTON, ALFRED C (AL); Benoit Claise > Cc: w...@mti-systems.com; aqm@ietf.org; The IESG; draft-ietf-aqm-eval- > guideli...@ietf.org; aqm-cha...@ietf.org > Subject: Re: [aqm] Benoit Claise's Discuss on draft-ietf-aqm-eval- > guidelines-11: (with DISCUSS and COMMENT) > > Hi Benoit, > > I finally had another look at the document as well as at RFC6390. I > guess the metrics in question are Flow completion time (sec 2.1.), Flow > start up time (2.2.) and Packet loss synchronization (2.4.). And you are > right that these metric could be see as Application-Specific Performance > Metric as defined in RFC6390. However, I agree with Al that given the > scope of this document is providing > "a generic list of scenarios against which an >AQM proposal should be evaluated, considering both potential >performance gain and safety of deployment.“, > I don’t think these metric need to be defined and registered this way. > > I guess as soon as we have a registry, maybe there is someone interest > in IPPM to catch up these metrics again and provide a RFC6390 definition > but I would rather not like this document doing it. > > Is that acceptable for you? > > We could add one more sentence in the abstract to make the scope on lab > testing during development and before deployment clear. Would that help? > > Mirja > > > > > Am 25.05.2016 um 14:22 schrieb Mirja Kuehlewind (IETF) > <i...@kuehlewind.net>: > > > > Hi Al, Benoit, hi all, > > > > thanks for the feedback. Sorry for me delaying this maybe a little but > I need t have another look at the document which will be next week at > this point. In general I agree that this does not need to only rely on > registered metrics because is mostly for lab tests; further this might > probably not the right doc to register new metrics. However, I would > still like to have another look at the doc and see if we can improve > anything or figure out if any of the ’new’/non-registed metrics > should/could be taken up by e.g. ippm. > > > > Mirja > > > > > > > >> Am 20.05.2016 um 14:53 schrieb MORTON, ALFRED C (AL) > <acmor...@att.com>: > >> > >> All, > >> a few replies in-line below, > >> Al > >> > >>> -Original Message- > >>> From: Benoit Claise [mailto:bcla...@cisco.com] > >>> Sent: Thursday, May 19, 2016 5:38 AM > >>> To: The IESG > >>> Cc: draft-ietf-aqm-eval-guideli...@ietf.org; w...@mti-systems.com; > aqm- > >>> cha...@ietf.org; w...@mti-systems.com; aqm@ietf.org; linda Dunbar; > >>> MORTON, ALFRED C (AL) > >>> Subject: Benoit Claise's Discuss on draft-ietf-aqm-eval-guidelines- > 11: > >>> (with DISCUSS and COMMENT) > >>> > >>> Benoit Claise has entered the following ballot position for > >>> draft-ietf-aqm-eval-guidelines-11: Discuss > >>> > >> ... > >>> > -- > >>> DISCUSS: > >>> > -- > >>> > >>> Has a RFC6390 performance directorate review done for the 2.X > metrics? > >>> It > >>> should. > >> [ACM] > >> I reviewed this draft about 18 months ago. > >> Mostly, it points to existing RFCs for fundamental metrics, > >> and discusses others. I read this: > >> ...This document provides characterization guidelines that > >> can be used to assess the deployability of an AQM, whether it is > >> candidate for standardization at IETF or not. > >> as restricted to lab testing. > >> > >>> See http://www.ietf.org/iesg/directorate/performance-metrics.html > >>> I guess that the metrics will be recorded in the future (See > >>> draft-ietf-ippm-metric-registry-06 > >>> ), right? > >> [ACM] >
Re: [aqm] Benoit Claise's Discuss on draft-ietf-aqm-eval-guidelines-11: (with DISCUSS and COMMENT)
Hi Al, Benoit, hi all, thanks for the feedback. Sorry for me delaying this maybe a little but I need t have another look at the document which will be next week at this point. In general I agree that this does not need to only rely on registered metrics because is mostly for lab tests; further this might probably not the right doc to register new metrics. However, I would still like to have another look at the doc and see if we can improve anything or figure out if any of the ’new’/non-registed metrics should/could be taken up by e.g. ippm. Mirja > Am 20.05.2016 um 14:53 schrieb MORTON, ALFRED C (AL) <acmor...@att.com>: > > All, > a few replies in-line below, > Al > >> -Original Message- >> From: Benoit Claise [mailto:bcla...@cisco.com] >> Sent: Thursday, May 19, 2016 5:38 AM >> To: The IESG >> Cc: draft-ietf-aqm-eval-guideli...@ietf.org; w...@mti-systems.com; aqm- >> cha...@ietf.org; w...@mti-systems.com; aqm@ietf.org; linda Dunbar; >> MORTON, ALFRED C (AL) >> Subject: Benoit Claise's Discuss on draft-ietf-aqm-eval-guidelines-11: >> (with DISCUSS and COMMENT) >> >> Benoit Claise has entered the following ballot position for >> draft-ietf-aqm-eval-guidelines-11: Discuss >> > ... >> -- >> DISCUSS: >> -- >> >> Has a RFC6390 performance directorate review done for the 2.X metrics? >> It >> should. > [ACM] > I reviewed this draft about 18 months ago. > Mostly, it points to existing RFCs for fundamental metrics, > and discusses others. I read this: > ...This document provides characterization guidelines that > can be used to assess the deployability of an AQM, whether it is > candidate for standardization at IETF or not. > as restricted to lab testing. > >> See http://www.ietf.org/iesg/directorate/performance-metrics.html >> I guess that the metrics will be recorded in the future (See >> draft-ietf-ippm-metric-registry-06 >> ), right? > [ACM] > That's up to the authors, they might simply point to > metrics in the registry contributed by others > (when following these guidelines at a future time). > >> For example, Flow Completion Time and Packet Loss Synchronization are >> new, I believe. > [ACM] > Flow Completion Time is close to a definition for a new metric, > and could benefit from more attention, perhaps a few more details. > RFC6390 will provide some areas for improvement. > > Packet loss sync full methodology is described in [JAY006], > according to the text. > >> And some other metrics are already documented in RFC6390 compliant >> documents. Pointers should be provided. > [ACM] > Most others are discussion sections and provide references. > >> See >> https://tools.ietf.org/html/draft-ietf-xrblock-independent-burst-gap- >> discard-01#appendix-A >> for an example >> >> >> -- >> COMMENT: >> -- >> >> - Random Early Detection (RED), BLUE, and Proportional Integral >> controller (PI) >> Would you have references? >> >> - BDP is mentioned a few times. Please expand. >> >> - Glossary section = terminology section, right? If we want to be >> consistent across documents >> >> - section 12.2. Why not a MUST below? >> In order to understand an AQM's deployment considerations and >> performance under a specific environment, AQM proposals SHOULD >> describe the parameters that control the macroscopic AQM behavior, >> and identify any parameters that require tuning to operational >> conditions. >> > ___ aqm mailing list aqm@ietf.org https://www.ietf.org/mailman/listinfo/aqm
Re: [aqm] Benoit Claise's Discuss on draft-ietf-aqm-eval-guidelines-11: (with DISCUSS and COMMENT)
Many thanks Al. Regards, Benoit All, a few replies in-line below, Al -Original Message- From: Benoit Claise [mailto:bcla...@cisco.com] Sent: Thursday, May 19, 2016 5:38 AM To: The IESG Cc: draft-ietf-aqm-eval-guideli...@ietf.org; w...@mti-systems.com; aqm- cha...@ietf.org; w...@mti-systems.com; aqm@ietf.org; linda Dunbar; MORTON, ALFRED C (AL) Subject: Benoit Claise's Discuss on draft-ietf-aqm-eval-guidelines-11: (with DISCUSS and COMMENT) Benoit Claise has entered the following ballot position for draft-ietf-aqm-eval-guidelines-11: Discuss ... -- DISCUSS: -- Has a RFC6390 performance directorate review done for the 2.X metrics? It should. [ACM] I reviewed this draft about 18 months ago. Mostly, it points to existing RFCs for fundamental metrics, and discusses others. I read this: ...This document provides characterization guidelines that can be used to assess the deployability of an AQM, whether it is candidate for standardization at IETF or not. as restricted to lab testing. See http://www.ietf.org/iesg/directorate/performance-metrics.html I guess that the metrics will be recorded in the future (See draft-ietf-ippm-metric-registry-06 ), right? [ACM] That's up to the authors, they might simply point to metrics in the registry contributed by others (when following these guidelines at a future time). For example, Flow Completion Time and Packet Loss Synchronization are new, I believe. [ACM] Flow Completion Time is close to a definition for a new metric, and could benefit from more attention, perhaps a few more details. RFC6390 will provide some areas for improvement. Packet loss sync full methodology is described in [JAY006], according to the text. And some other metrics are already documented in RFC6390 compliant documents. Pointers should be provided. [ACM] Most others are discussion sections and provide references. See https://tools.ietf.org/html/draft-ietf-xrblock-independent-burst-gap- discard-01#appendix-A for an example -- COMMENT: -- - Random Early Detection (RED), BLUE, and Proportional Integral controller (PI) Would you have references? - BDP is mentioned a few times. Please expand. - Glossary section = terminology section, right? If we want to be consistent across documents - section 12.2. Why not a MUST below? In order to understand an AQM's deployment considerations and performance under a specific environment, AQM proposals SHOULD describe the parameters that control the macroscopic AQM behavior, and identify any parameters that require tuning to operational conditions. ___ aqm mailing list aqm@ietf.org https://www.ietf.org/mailman/listinfo/aqm
Re: [aqm] Benoit Claise's Discuss on draft-ietf-aqm-eval-guidelines-11: (with DISCUSS and COMMENT)
All, a few replies in-line below, Al > -Original Message- > From: Benoit Claise [mailto:bcla...@cisco.com] > Sent: Thursday, May 19, 2016 5:38 AM > To: The IESG > Cc: draft-ietf-aqm-eval-guideli...@ietf.org; w...@mti-systems.com; aqm- > cha...@ietf.org; w...@mti-systems.com; aqm@ietf.org; linda Dunbar; > MORTON, ALFRED C (AL) > Subject: Benoit Claise's Discuss on draft-ietf-aqm-eval-guidelines-11: > (with DISCUSS and COMMENT) > > Benoit Claise has entered the following ballot position for > draft-ietf-aqm-eval-guidelines-11: Discuss > ... > -- > DISCUSS: > -- > > Has a RFC6390 performance directorate review done for the 2.X metrics? > It > should. [ACM] I reviewed this draft about 18 months ago. Mostly, it points to existing RFCs for fundamental metrics, and discusses others. I read this: ...This document provides characterization guidelines that can be used to assess the deployability of an AQM, whether it is candidate for standardization at IETF or not. as restricted to lab testing. > See http://www.ietf.org/iesg/directorate/performance-metrics.html > I guess that the metrics will be recorded in the future (See > draft-ietf-ippm-metric-registry-06 > ), right? [ACM] That's up to the authors, they might simply point to metrics in the registry contributed by others (when following these guidelines at a future time). > For example, Flow Completion Time and Packet Loss Synchronization are > new, I believe. [ACM] Flow Completion Time is close to a definition for a new metric, and could benefit from more attention, perhaps a few more details. RFC6390 will provide some areas for improvement. Packet loss sync full methodology is described in [JAY006], according to the text. > And some other metrics are already documented in RFC6390 compliant > documents. Pointers should be provided. [ACM] Most others are discussion sections and provide references. > See > https://tools.ietf.org/html/draft-ietf-xrblock-independent-burst-gap- > discard-01#appendix-A > for an example > > > -- > COMMENT: > -- > > - Random Early Detection (RED), BLUE, and Proportional Integral > controller (PI) > Would you have references? > > - BDP is mentioned a few times. Please expand. > > - Glossary section = terminology section, right? If we want to be > consistent across documents > > - section 12.2. Why not a MUST below? >In order to understand an AQM's deployment considerations and >performance under a specific environment, AQM proposals SHOULD >describe the parameters that control the macroscopic AQM behavior, >and identify any parameters that require tuning to operational >conditions. > ___ aqm mailing list aqm@ietf.org https://www.ietf.org/mailman/listinfo/aqm
