Dear Toke, On Feb 13, 2014, at 5:09 PM, Toke Høiland-Jørgensen <t...@toke.dk> wrote:
> Nicolas KUHN <nicolas.k...@telecom-bretagne.eu> writes: > >> We tried to be the clearer the possible. >> If you have any comments or proposals, we would be glad to hear them. > > Well, one thing I believe is worth addressing is the issue of simulation > vs experiments on real systems. I would not consider an algorithm that > has never been subjected to real-world testing to be sufficiently > evaluated. > I believe the draft must be the most generic the possible. These guidelines provide the tools and aspects that must be looked at, however the AQM is tested. Even-driven simulations (such as in NS-2,OMNET) enable to achieve an economical and fast protocol experimentation [http://arxiv.org/pdf/1002.2827.pdf]. These guidelines cannot reject a proposal because it has not been tested in the 'real' world. I will add in the "methodology" section something about that issue, saying that proposals SHOULD be evaluated in testbeds, and MAY be evaluated with event-driven simulations to better understand their internal behaviour. > Another issue is how to obtain the data: if the AQM itself is inspected > while running that might affect its performance, whereas if packet > traces or application level measurements at another point are inspected, > they might be showing behaviour that has very little to do with the > behaviour of the AQM algorithm. As an example of the latter, on Linux it > is very easy to end up measuring buffers in the network driver, unless > care is taken to avoid it. > This is the kind of discussion that should be avoided in the draft. Indeed, I am afraid we may open a "pandora's box" as we do not want to list all the issues encountered by any evaluation tool set. We may end discussing why choosing emulation, vs simulation, vs mathematical model, vs real-workd experiment. > I would like to see the draft address these issues and provide some > guidance on how the most obvious pitfalls can be avoided. I'll be happy > to draft some text on it, if it's not part of the planned methodology > section. > >> I agree that methodology should be integrated in the draft. I will >> include a dedicated section in the next version of the draft. > > Great (also, comments above)! > >> I think this is dealt with in section 3.3.1.5. > > (I'm assuming you mean 3.2.1.5 here). > Yes, I integrated the "Methodology" section and 3.2.1.5 became 3.3.1.5. Sorry for that > I suppose that scenario could cover this; however, I was also alluding > to the methodological issue of distinguishing between the case where > latency is measured by a separate flow (such as a straight 'ping' > alongside the test) vs directly measuring the queueing delay of the > packets making up the traffic scenario; I believe both kinds of > measurements provide valuable data, but about different aspects of the > algorithm behaviour… > I think that this is more a metric linked to the evaluation tool set. If we want the guidelines to be "generic" and "evaluation tool set" free, this should not be discussed. >> The next version will be more specific with the expected output >> (measuring the metrics separately for each class of traffic). The next >> version will also detail that the TCp transfer can be operated with an >> LBE congestion control. > > ACK. What about IW4 vs IW10? I am not sure that it makes sense here, that would be another pandora's box: the guidelines should not consider every possible TCP improvements. Otherwise, others would say "do you consider RENO? CUBIC? Delay based CC?" or "what about MultiPath?". We must generic and speak generally about TCP and UDP - without much more details. > >> The idea behind the draft was to be quite generic. Wi-Fi is a use case >> for "Medium bandwidth, medium delay" scenario. We could go deeper into >> details, but I am afraid that the draft will lose some of its >> generically. This should be further discussed with the list. > > I'm not saying "get rid of the scenario", I'm saying "Don't call it > wifi, because that mischaracterises the behaviour of wifi". And perhaps > also "add a scenario where bandwidth varies on very small time scales". > Yes I updated the Section "Medium bandwidth, medium delay: Wi-Fi" so that the tester may include "small time scales varying bandwidth" >> I am not sure the current draft mention that AQM and scheduling are >> interchangeable. I think the draft mentions that they may both reduce >> the latency and the distinction between the two is clear. If some >> sections lack of clarity, please point them out. > > I think it could be made clearer whether or not his document is meant as > only a way to evaluate AQM algorithms, or if hybrid AQM/scheduling > algorithms should also be in scope (I would argue for the latter). > I agree as well - the draft would then need some rewording I will add an entry in the glossary explaining that by AQM, the draft consider hybrid AQM+scheduling > In the first many sections, whenever scheduling is mentioned (sections 2, > 2.2.3, 3.2.3), the wording suggests that indeed AQM and scheduling can > be complimentary and an algorithm might feasibly be a tightly coupled > hybrid. However, the section specifically dealing with scheduling (4.4), > discussing adding scheduling "on top of" the algorithm is mentioned as a > requirement, which might imply that an algorithm that already includes > (say) a tightly coupled scheduler would not be liable for consideration? > I will update the Section (4.4) considering that scheduling is a feature of an AQM, and is not introduced "on top" of it. > > Also, this paragraph: > > Both schedulers and AQMs can be introduced when packet are either > enqued or dequed. If both schedulers and AQM are implemented when > packet are enqued, their interaction should not be a major issue. > However, if one is introduced when packets are enqued and the others > when they are dequed, there may be destructive interactions. > > seems a bit speculative to me. Has the fact that the algorithms' > activation time is a predictor of adverse side effects actually been > demonstrated? > The guidelines just mention that this MAY be destructive and this issue should be carefully discussed. >> The draft claims that the impact of every externally tuned parameter >> MUST be discussed. The tester SHOULD provide background material >> showing control theoretic analysis but COULD use other ways to discuss >> stability. > > That formulation would be fine with me. However, the last part of the > latter sentence (the COULD part) is missing from the draft... :) > ACK Cheers, Nicolas > -Toke > _______________________________________________ > aqm mailing list > aqm@ietf.org > https://www.ietf.org/mailman/listinfo/aqm _______________________________________________ aqm mailing list aqm@ietf.org https://www.ietf.org/mailman/listinfo/aqm
