Re: Discuss criteria for documents that advance on the standards track
[BA] My comment is that having guidelines for this could help make the advancement process more predictable. Thank you for working on this. Jari Arkko said: During the discussion of the two maturity levels change, a question was brought up about DISCUSSes appropriate for documents that advance on the standards track. We discussed this in the IESG and I drafted some suggested guidelines. Feedback on these suggestions would be welcome. The intent is to publish an IESG statement to complement the already existing general-purpose DISCUSS criteria IESG statement (http://www.ietf.org/iesg/statement/discuss-criteria.html). Here are the suggested guidelines for documents that advance to IS: http://www.arkko.com/ietf/iesg/discuss-criteria-advancing.txt Comments appreciated. Please send comments either on this list or to the IESG (i...@ietf.org) in time before our next telechat (Thursday, September 8, 2011). ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Discuss criteria for documents that advance on the standards track
--On Tuesday, August 30, 2011 14:51 -0700 Fred Baker f...@cisco.com wrote: What's also not fair game is to raise the bar - to expect the document at DS to meet more stringent criteria than it was required to meet at the time of PS approval. Hmmm, the demonstrated interoperability requirement of DS/IS is in fact a raising of the bar,and one that has served us well. We don't (although IMHO we should) require even an implementation to go to PS. I know it is controversial, but there is at least one other area in which we should be raising the bar for DS/IS by dropping the bar for Proposed. If we really want to get PS specs out quickly while the percentage of people who easily write very high quality technical English in the IETF continues to go down, we need to stop the behavior of various IESG members simulating technical editors or translators to fix PS text (or insisting that the author or WG do so, which, IMO, is less bad but still often a problem). Doing that will get documents out faster, perhaps even a lot faster in some cases, but will inevitably result in PS documents that need significant editorial work before being approved at DS. I think that would actually be a good thing. I think that stuffing explicit placeholders to the effect of this needs to be rewritten to be completely clear to folks who haven't participated in the WG into PS documents would be a fine idea -- it would get those documents finished and out while making their preliminary nature very clear. But it implies a higher editorial quality standard --a higher bar-- for DS/IS than for PS. john ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Discuss criteria for documents that advance on the standards track
Keith, thank you for the feedback. Some responses inline: 1. Fix the broken IESG voting system before you try to establish more decision criteria. I do agree with your general thinking here. The way that you describe the different positions is what I personally try to achieve in my IESG reviews. But I think you are overemphasizing the role of the vote designations. Again, I try to do this already, though maybe not always succeeding. In general, if the Area Directors are not doing their job of stopping bad stuff and moving good stuff along, they don't need a new voting system, they need to grow a spine. Elaborating a bit more about this: We have plenty of cases where a DISCUSS has been left standing, and today that acts as your NO vote. (Obviously, in many cases this was an error, or lack of effort. I'm quilty of this for sure, even right now I have a queue of DISCUSSes and their proposed resolutions that I need to go check.) But I think a DISCUSS should stand, if there is a serious issue and it is not rectified. There are corner cases where a single AD has an opinion that is not shared by the rest of the community. For those cases we have an override procedure. This has been never been invoked, but that's probably because it gets pretty ugly way before that -- there are often heated discussions between ADs about discusses. We have the ABSTAIN vote, which some ADs use to vote NO, often together with other ADs who feel the same. There's never been a case where this would have blocked a document from proceeding, as we've never collected the necessary 1/3 number of ADs to vote ABSTAIN for any single document. My conclusion is that ABSTAIN stands for NO-OBJECTION in practical terms. I don't recommend its use... 2. Don't overconstrain the use of DISCUSS. In particular, don't ever create a situation where a reviewer can't cite a problem with a document, regardless of whether that problem has previously been enumerated. I agree, and that's why the guidelines I posted are just that -- guidelines. They are not binding rules, they leave room for a judgment call. 3. I take serious issue with the statement in the draft that IESG reviews are reviews of last resort and the implication that WG reviews are sufficient. In numerous situations this has not been the case. Of course. But I don't see a conflict between a review of last resort and having the last resort find issues. I wish we'd find less issues, but at least I still view the IESG as the final check, that should catch issues if others have not. Its not a last resort in the sense that it would not be invoked; we do review all documents very carefully. Its a last resort only in the sense that there is an expectation that previous stages should have produced a quality result without issues. Jari ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Discuss criteria for documents that advance on the standards track
Would having professional editors make a difference here? On Aug 31, 2011, at 2:31 AM, John C Klensin wrote: --On Tuesday, August 30, 2011 14:51 -0700 Fred Baker f...@cisco.com wrote: What's also not fair game is to raise the bar - to expect the document at DS to meet more stringent criteria than it was required to meet at the time of PS approval. Hmmm, the demonstrated interoperability requirement of DS/IS is in fact a raising of the bar,and one that has served us well. We don't (although IMHO we should) require even an implementation to go to PS. I know it is controversial, but there is at least one other area in which we should be raising the bar for DS/IS by dropping the bar for Proposed. If we really want to get PS specs out quickly while the percentage of people who easily write very high quality technical English in the IETF continues to go down, we need to stop the behavior of various IESG members simulating technical editors or translators to fix PS text (or insisting that the author or WG do so, which, IMO, is less bad but still often a problem). Doing that will get documents out faster, perhaps even a lot faster in some cases, but will inevitably result in PS documents that need significant editorial work before being approved at DS. I think that would actually be a good thing. I think that stuffing explicit placeholders to the effect of this needs to be rewritten to be completely clear to folks who haven't participated in the WG into PS documents would be a fine idea -- it would get those documents finished and out while making their preliminary nature very clear. But it implies a higher editorial quality standard --a higher bar-- for DS/IS than for PS. john ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf smime.p7s Description: S/MIME cryptographic signature ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Discuss criteria for documents that advance on the standards track
Eric, John, Would having professional editors make a difference here? I know it is controversial, but there is at least one other area in which we should be raising the bar for DS/IS by dropping the bar for Proposed. If we really want to get PS specs out quickly while the percentage of people who easily write very high quality technical English in the IETF continues to go down, we need to stop the behavior of various IESG members simulating technical editors or translators to fix PS text (or insisting that the author or WG do so, which, IMO, is less bad but still often a problem). I think the existing Discuss criteria already says very clearly that editorial comments cannot be blocking DISCUSSes. I see a lot of language feedback from IESG and directorate reviews, but its rare to have them appear in the DISCUSSes. If they do, its inappropriate, you should push back. And I'm sure you will :-) Besides, we pay the RFC Editor a large amount of money every year to do the editing. Documents need to be clear enough to be understood, but the RFC editor can handle most of the editorial problems. (That being said, I've seen documents that were sent back because they really were not understandable. Obviously there is some bar under which you should not go, or the document cannot advance at all. This happens more in WG stages than in the IESG. But if you can't communicate your idea clearly then I think it should be up to you to hire co-workers/editors to help clarify your idea... not the IETF's problem, IMHO.) Jari ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Discuss criteria for documents that advance on the standards track
On Aug 31, 2011, at 2:31 AM, John C Klensin wrote: --On Tuesday, August 30, 2011 14:51 -0700 Fred Baker f...@cisco.com wrote: What's also not fair game is to raise the bar - to expect the document at DS to meet more stringent criteria than it was required to meet at the time of PS approval. Hmmm, the demonstrated interoperability requirement of DS/IS is in fact a raising of the bar,and one that has served us well. We don't (although IMHO we should) require even an implementation to go to PS. I know it is controversial, but there is at least one other area in which we should be raising the bar for DS/IS by dropping the bar for Proposed. If we really want to get PS specs out quickly while the percentage of people who easily write very high quality technical English in the IETF continues to go down, we need to stop the behavior of various IESG members simulating technical editors or translators to fix PS text (or insisting that the author or WG do so, which, IMO, is less bad but still often a problem). Doing that will get documents out faster, perhaps even a lot faster in some cases, but will inevitably result in PS documents that need significant editorial work before being approved at DS. Given that people commonly implement and deploy Proposed Standards in products, sometimes even prior to their approval as Proposed Standards, I think this is a very bad idea. I think it's likely to create interoperability problems between PS and DS versions of products, and sometimes between implementations of the same PS version. For better or worse, the widespread desire to implement at PS (recommendations in 2026 notwithstanding) pretty much forces IETF to try to produce high quality prose at PS. I agree that IESG should not generally be acting as technical editors, so maybe IETF needs to find some way to provide technical editors who are proficient in technical English before documents are reviewed by IESG. Documents written by native English speakers would also benefit. Maybe the GEN-ART review could be expanded, or maybe the RFC Editor could assume some role here (awkward though that might be at first). Keith ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Discuss criteria for documents that advance on the standards track
On Aug 31, 2011, at 2:36 AM, Jari Arkko wrote: Keith, thank you for the feedback. Some responses inline: 1. Fix the broken IESG voting system before you try to establish more decision criteria. I do agree with your general thinking here. The way that you describe the different positions is what I personally try to achieve in my IESG reviews. But I think you are overemphasizing the role of the vote designations. Again, I try to do this already, though maybe not always succeeding. In general, if the Area Directors are not doing their job of stopping bad stuff and moving good stuff along, they don't need a new voting system, they need to grow a spine. Elaborating a bit more about this: We have plenty of cases where a DISCUSS has been left standing, and today that acts as your NO vote. (Obviously, in many cases this was an error, or lack of effort. I'm quilty of this for sure, even right now I have a queue of DISCUSSes and their proposed resolutions that I need to go check.) But I think a DISCUSS should stand, if there is a serious issue and it is not rectified. There are corner cases where a single AD has an opinion that is not shared by the rest of the community. For those cases we have an override procedure. This has been never been invoked, but that's probably because it gets pretty ugly way before that -- there are often heated discussions between ADs about discusses. We have the ABSTAIN vote, which some ADs use to vote NO, often together with other ADs who feel the same. There's never been a case where this would have blocked a document from proceeding, as we've never collected the necessary 1/3 number of ADs to vote ABSTAIN for any single document. My conclusion is that ABSTAIN stands for NO-OBJECTION in practical terms. I don't recommend its use... The biggest problem with the current voting system (other than misleading labels, which do cause real problems of their own) is the presumption that the document should go forward no matter how few IESG members read the document. So No Objection votes from ADs who didn't read the document count as Yes votes, but there's also a presumption in the rules (as well as pressure from other ADs who want to get documents off the agenda) to clear Discuss votes in favor of moving a document forward whether or not the identified issues have been adequately addressed. (One thing that I didn't mention that also needs to be fixed if it's still the case is the presumption that the responsible AD votes Yes for the document. I don't know what the tools do now, but this Yes vote used to be automatically filled-in.) 2. Don't overconstrain the use of DISCUSS. In particular, don't ever create a situation where a reviewer can't cite a problem with a document, regardless of whether that problem has previously been enumerated. I agree, and that's why the guidelines I posted are just that -- guidelines. They are not binding rules, they leave room for a judgment call. 3. I take serious issue with the statement in the draft that IESG reviews are reviews of last resort and the implication that WG reviews are sufficient. In numerous situations this has not been the case. Of course. But I don't see a conflict between a review of last resort and having the last resort find issues. I wish we'd find less issues, but at least I still view the IESG as the final check, that should catch issues if others have not. Its not a last resort in the sense that it would not be invoked; we do review all documents very carefully. Its a last resort only in the sense that there is an expectation that previous stages should have produced a quality result without issues. That is not a valid expectation, in either theory or practice. That's my point. Arguably when a poor quality document gets to IESG, it's a failure on the part of the WG to do due diligence. But the problem is actually deeper than that - it's partially structural (in that IETF partitions almost all work into narrowly focused WGs who don't represent a sufficiently broad spectrum of interests), and partially due to a failure to consistently apply good engineering principles across all of IETF. IESG's assuming that the WG has produced a quality result basically works to mask the other problems with IETF's way of doing work. But even if WGs generally did produce high quality results without issues (which I don't think is the case now), IESG review should still not presume that they do. There will always be some failures at the WG level, and the IESG's job is to try to catch those. Keith ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Discuss criteria for documents that advance on the standards track
On Aug 31, 2011, at 4:34 AM, Jari Arkko wrote: Eric, John, Would having professional editors make a difference here? I know it is controversial, but there is at least one other area in which we should be raising the bar for DS/IS by dropping the bar for Proposed. If we really want to get PS specs out quickly while the percentage of people who easily write very high quality technical English in the IETF continues to go down, we need to stop the behavior of various IESG members simulating technical editors or translators to fix PS text (or insisting that the author or WG do so, which, IMO, is less bad but still often a problem). I think the existing Discuss criteria already says very clearly that editorial comments cannot be blocking DISCUSSes. So nobody has the job of making sure that the documents are well-written in clear English? Besides, we pay the RFC Editor a large amount of money every year to do the editing. Documents need to be clear enough to be understood, but the RFC editor can handle most of the editorial problems. If the document is edited for clarity after final review, that's really a botch in the process. It means that the review doesn't apply to the version of the document being published. Of course the RFC Editor's resources shouldn't be used for documents that aren't otherwise deemed worthy of publication, and you'd like to avoid the RFC Editor having to do multiple reviews, so I admit that this is tricky to solve. (That being said, I've seen documents that were sent back because they really were not understandable. Obviously there is some bar under which you should not go, or the document cannot advance at all. This happens more in WG stages than in the IESG. But if you can't communicate your idea clearly then I think it should be up to you to hire co-workers/editors to help clarify your idea... not the IETF's problem, IMHO.) If writing clear specifications isn't the IETF's problem, I'm not sure what is. Keith ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Discuss criteria for documents that advance on the standards track
--On Wednesday, August 31, 2011 11:34 +0300 Jari Arkko jari.ar...@piuha.net wrote: Eric, John, Would having professional editors make a difference here? I know it is controversial, but there is at least one other ... I think the existing Discuss criteria already says very clearly that editorial comments cannot be blocking DISCUSSes. This issue drifts quickly from DISCUSS criteria into how we review, approve, and edit documents, but I think they are completely intertwined... so please forgive the apparent digression. I see a lot of language feedback from IESG and directorate reviews, but its rare to have them appear in the DISCUSSes. If they do, its inappropriate, you should push back. And I'm sure you will :-) Yes, but... First of all, a long list of editorial comments from one or more ADs can do fully as good a job of slowing a document down as a DISCUSS or two. And anyone smart enough to be on the IESG is more than capable of disguising what is really an editorial comment as a substantive issue that gets a DISCUSS. Part of the problem is that there are a whole series of judgment calls tangled up with this topic. For example: * If an AD says this particular paragraph is really important to the specification and, after several readings, I can't tell what is actually means or whether a particular implementation would be conforming, that is a problem, certainly worthy of a DISCUSS (or, given Keith's typology, NO until it is fixed. On the other hand, this particular paragraph is important and it is much harder to understand than it should be, even though the intent is clear is an editorial comment that should be non-blocking. I think we all know that the boundary between those two can be very thin and even a matter for debate. * I can't speak for others, but I'm often tired with a document for which I've got responsibility by the time I get it past the small revisions and nit-picking or WG LC and pre-IETF-LC comments from ADs. Faced with what I consider an inappropriate editorial comment or even an inappropriate DISCUSS, I have a choice between pushing back (thereby guaranteeing that I will have to deal with the document for additional weeks or months) or just making the requested changes and getting the document off my plate. I'm probably more stubborn about editorial matters than most people, but even I usually just give in. Besides, we pay the RFC Editor a large amount of money every year to do the editing. Documents need to be clear enough to be understood, but the RFC editor can handle most of the editorial problems. Yes, and I've been saying that, along with variations on let them do their job for years. But it introduces another set of issues. I don't think I'm giving away a secret by saying that they believe that their contracts and the way their performance is measured focus on pages of published output per month. They also believe that the IESG has told them things that amount to minimal editing or copy editing only on many occasions. Unless we can change things to shift from measuring what is easy to include a focus on quality improvements, the RFC Editor ends up in an impossible situation. From a standardization perspective, it also isn't desirable to make lots of editorial changes post-approval. AUTH48 is a rather dangerous process because it is easy to make mistakes there that no one catches or, again, for an exhausted author to make it is easier to let this go than to fight about it decisions. One could, in principle, send all such documents back to a producing WG for final, pre-publication review but, in my experience, most of the people in most WGs get more exhausted by the document end-game than most authors (and most other participants want to reopen long-dead issues by questioning the language used to describe them). This is precisely why many SDOs use a two-stage vote in which the first vote approves the general concepts and technical content, the document then gets edited into final form by profession staff, and then the first and only actual approval vote occurs (possibly by a different body with a different mission). That wouldn't work well in the IETF... and not only because we have a more-than-one-step standards process. I think we could adopt a process in which a WG, or an AD, could look at a document before IETF LC and say this is going to need a lot of editorial work before approval but the technical content looks ok, so let's hand it off to the RFC Editor now and get it cleaned up before a final WG review and IETF LC. But that has a lot of procedural and budgetary implications and I don't know if we are ready to deal with them. (That being said, I've seen documents that were sent back because they really were not understandable. Obviously there is some bar under which you should not go, or the document cannot advance at all. This happens more in WG stages than in the IESG. But if you can't communicate your idea clearly then I
Re: Discuss criteria for documents that advance on the standards track
--On Wednesday, August 31, 2011 08:02 -0400 Keith Moore mo...@network-heretics.com wrote: I think the existing Discuss criteria already says very clearly that editorial comments cannot be blocking DISCUSSes. So nobody has the job of making sure that the documents are well-written in clear English? Keith, I don't think either Jari or I were saying anything remotely close to that. Think about it this way, moving away from the language in 2026 and back to the much-earlier assumptions on which that language, and the multiple-stage standards process, were based. With the understanding that these are oversimplifications, we can have three types of documents: (1) Complete spec, so complete and so clear that someone with reasonable familiarity with the technology but no contact with the authors, anyone who participated the relevant WG, or the WG mailing list can produce an implementation that will interoperate with other implementations --both those produced with the same level of knowledge and those produced by, or in coordination with, the authors. (2) Specs that are good and clear enough to act as aids to memory and documentation about agreement so that someone who participated in the WG and who can ask questions of a WG mailing list and/or the authors can produce an interoperable implementation. (3) A spec that just outlines an idea for others to try out. Because I want to avoid losing the distinction, I would point out that only the first is really suitable for use in a procurement spec (although people often accept the second and sometimes the third). From a standardization standpoint, we should consider the third as potentially appropriate for Experimental, especially if the purpose of the experiment is to help understand what the details should be. We've been trying to hold everything (even, in many cases, experimental documents) up until we get to that first criterion, but, if one looks at long-term history and reads between the lines, only DS and above actually require documentation at that level. We ought to, IMO, be permitting publication of PS documents at the second level as long as there are no _obvious_ ambiguities that cannot be figured out (the same way) by people of good will acting in good faith and with help from WG lists and as long as there are no other known [technical] defects. I suggest that we have called that second category Proposed Standard and then interpreted that term in ways that have caused us harm. It is closer to what ISO has called a Type II Technical Report (known in internal jargon as a pre-standard or sub-standard) and what IEEE calls (or used to call) a Draft Standard for Trial Use -- a document that represents consensus about the idea and about publication, including publication as a standard when it is mature enough, and good enough to form a basis for experimentation and serious evaluation, but not expected to meet criterion (1) above. From that particular narrow perspective, if one wants PS documents published quickly rather than killing them and the standards process through delays and nit-picking that are almost irrelevant to the technical content of the specification, then we either need to make some really major changes in how we do things or we need to be willing to publish things that, explicitly or implicitly contain notes like the following paragraph is horribly written and barely intelligible and will need to be fixed up for DS but, right now, everyone involved knows what it means. Insisting that all PS documents be well-written in clear English may actually be our enemy. That doesn't mean they shouldn't be. And it would be perfectly reasonable for some WGs and authors to decide the situation is do the work now (and accept the added delay) or do it later, so better now. But we should not be trying to force every PS document to meet such a standard... at least if we want to get them out quickly and have them treated as interim steps rather than final products. Besides, we pay the RFC Editor a large amount of money every year to do the editing. Documents need to be clear enough to be understood, but the RFC editor can handle most of the editorial problems. If the document is edited for clarity after final review, that's really a botch in the process. It means that the review doesn't apply to the version of the document being published. Of course the RFC Editor's resources shouldn't be used for documents that aren't otherwise deemed worthy of publication, and you'd like to avoid the RFC Editor having to do multiple reviews, so I admit that this is tricky to solve. Yes. See my previous note. (That being said, I've seen documents that were sent back because they really were not understandable. Obviously there is some bar under which you should not go, or the document cannot
Re: Discuss criteria for documents that advance on the standards track
Keith Moore mo...@network-heretics.com wrote: The biggest problem with the current voting system (other than misleading labels, which do cause real problems of their own) is the presumption that the document should go forward no matter how few IESG members read the document. Keith makes a good point here; but I wouldn't agree to any rule that a particular number must read a document. Some ADs quite properly defer actual reading to review teams. So No Objection votes from ADs who didn't read the document count as Yes votes, No, they don't. (But I can't ask folks posting to this thread to actually _understand_ the difference.) Roughly, the rule says enough ADs must enter a position before a document can be approved. Basically, No-Objection says the AD is somewhat familiar with the document and actively consents to approving it. These positions are _NOT_ votes. but there's also a presumption in the rules (as well as pressure from other ADs who want to get documents off the agenda) to clear Discuss votes in favor of moving a document forward whether or not the identified issues have been adequately addressed. This is somewhat true, but the pressure is highly variable. The agendas _are_ too crowded (IMHO); but in most cases sufficient progress will have been made before the telechat that only a few seconds are needed to agree to AD-followup status. When there is no progress between telechats (most often due to unresponsive authors/editors), there _is_ some pressure to reduce what a DISCUSS asks for. It might help for non-IESG folk to chime in on whether this is good or bad... (There _are_ cases where the responsible AD puts quite a bit of pressure on the DISCUSS-holder: that's really an internal issue which I don't believe this list should be discussing.) (One thing that I didn't mention that also needs to be fixed if it's still the case is the presumption that the responsible AD votes Yes for the document. I don't know what the tools do now, but this Yes vote used to be automatically filled-in.) We're seeing a number of cases where the responsible AD holds a DISCUSS at the time of the telechat -- generally because LastCall hadn't ended yet when the document was placed on the agenda. IMHO, there's nothing there that needs fixing. Arguably when a poor quality document gets to IESG, it's a failure on the part of the WG to do due diligence. IMHO, there _are_ poor-quality documents that get on the IESG agenda. I'm not sure it helps to allocate blame... But there is a huge variance between WGs on diligence. I am alarmed by the sheer number of COMMENTS saying in essence, This document is not specific enough to guide an implementor to an interoperable implementation. To me, that's really-close to DISCUSS territory... However, we really don't have a process for improving situations like that -- other than for it to be a DISCUSS and for authors to actually be responsive (which would probably require repeating at least one LastCall). :^( In the absence of such a process, I really can't blame ADs for reducing such issues from DISCUSS to COMMENT, and entering ABSTAIN if they think the issue is serious. But the problem is actually deeper than that - it's partially structural (in that IETF partitions almost all work into narrowly focused WGs who don't represent a sufficiently broad spectrum of interests), and partially due to a failure to consistently apply good engineering principles across all of IETF. +1 This problem, of course, is endemic in organizations that depend on volunteers. And I really don't have suggestions on how to ensure sufficient wide-area review, though review teams certainly help. I wonder if there's some room for process-improvement by formalizing some role for review teams... IESG's assuming that the WG has produced a quality result basically works to mask the other problems with IETF's way of doing work. I don't agree that's what IESG members assume -- they IMHO instead presume that documenting ideas (even not-fully-baked ideas) is a mostly-unmitigated good-thing. But even if WGs generally did produce high quality results without issues (which I don't think is the case now), IESG review should still not presume that they do. There will always be some failures at the WG level, and the IESG's job is to try to catch those. I don't think we have universal agreement on that as a goal. Of course, neither do we have universal agreement that anyone else has the job to catch these. Some of us quite plainly believe it shouldn't be anyone's job to catch these: that ideas should be published and we should see whether rough consensus emerges later. (All of which brings us to the actual question: when advancing a maturity-level, what constitutes sufficient consensus? Arguably, folks will expect a higher maturity level to indicate that the standard is ready to be handed to an implementor, and
Re: Discuss criteria for documents that advance on the standards track
On Aug 31, 2011, at 10:42 AM, John C Klensin wrote: We ought to, IMO, be permitting publication of PS documents at the second level as long as there are no _obvious_ ambiguities that cannot be figured out (the same way) by people of good will acting in good faith and with help from WG lists and as long as there are no other known [technical] defects. That would be fine with me if we could somehow effectively discourage deployment of proposed standards in products. But that's a lost cause, IMO. Keith ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Discuss criteria for documents that advance on the standards track
Dear Jari, During the discussion of the two maturity levels change, a question was brought up about DISCUSSes appropriate for documents that advance on the standards track. We discussed this in the IESG and I drafted some suggested guidelines. Feedback on these suggestions would be welcome. The intent is to publish an IESG statement to complement the already existing general-purpose DISCUSS criteria IESG statement (http://www.ietf.org/iesg/statement/discuss-criteria.html). Here are the suggested guidelines for documents that advance to IS: http://www.arkko.com/ietf/iesg/discuss-criteria-advancing.txt Comments appreciated. Please send comments either on this list or to the IESG (i...@ietf.org) in time before our next telechat (Thursday, September 8, 2011). Jari As with 2119bis, thank you guys for putting this together. The text in http://www.arkko.com/ietf/iesg/discuss-criteria-advancing.txt says IESG reviews should be considered as a review of last resort. Most documents reviewed by the IESG are produced and reviewed in the context of IETF working groups. In those cases, the IESG cannot overrule working group consensus without good reason; informed community consensus should prevail. I might have thought that the point for documents advancing on the standards track is that they already have *IETF* consensus (especially if they are being advanced without text changes), so the IESG cannot overrule *IETF* consensus without good reason ... which seems like a higher bar (one can more easily assume the existence of a rogue working group producing a rogue Internet-Draft, than a rogue IETF that is still sane enough to NomCom-select non-rogue ADs who will push back on rogue standards-track documents advancing :-) So perhaps s/informed working group consensus/informed IETF consensus/? I note that this isn't quite tying the IESG's hands (I'm reading this descriptive text as the moral equivalent of MUST NOT overrule IETF consensus without good reason, which leaves one hand free). Spencer ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Discuss criteria for documents that advance on the standards track
thanks Spencer for pointing this part out. On Aug 31, 2011, at 11:23 AM, Spencer Dawkins wrote: IESG reviews should be considered as a review of last resort. Most documents reviewed by the IESG are produced and reviewed in the context of IETF working groups. In those cases, the IESG cannot overrule working group consensus without good reason; informed community consensus should prevail. The idea that WG consensus should prevail is simply incorrect. It biases IESG in an inappropriate way. There are a number of very good reasons for overriding WG consensus, e.g. - there is no evidence of broad community consensus or a clear lack of broad community consensus - the document does not meet the criteria specified in 2026 (or other document when applicable) - the document is ambiguous in such a way that it is likely to degrade interoperability The WG DOES NOT represent the entire community.Far too often, WGs are deliberately chartered in such a way as to marginalize parts of the community Keith ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Discuss criteria for documents that advance on the standards track
--On Wednesday, August 31, 2011 11:08 -0400 Keith Moore mo...@network-heretics.com wrote: On Aug 31, 2011, at 10:42 AM, John C Klensin wrote: We ought to, IMO, be permitting publication of PS documents at the second level as long as there are no _obvious_ ambiguities that cannot be figured out (the same way) by people of good will acting in good faith and with help from WG lists and as long as there are no other known [technical] defects. That would be fine with me if we could somehow effectively discourage deployment of proposed standards in products. But that's a lost cause, IMO. Keith, IMO, there are two possibilities here. At this point, sadly, both involve a chicken-and-egg problem. Such is life. (1) We proceed as if Proposed Standards are what 2026 (and the earlier culture) claims they are and work on ways to reinforce that notion in the community. If that is our goal, than getting documents out sooner and having them look a bit rough as well as being a bit rough is A Good Thing and reinforces the deploying at PS, much less at I-D, is taking a big risk. The idea isn't foreign to the industry: to take a handy example, we saw products identified as 801.11-draft-N floating around for years. Every vendor who shipped one (and their more sophisticated customers) knew that there could be serious incompatibilities between their approximations to the drafts and the real 802.11n. (2) We accept, and effectively encourage, deployment of proposed standards in products, either because it is a lost cause or because we think it is a good idea. As part of that, we move further down the path of ensuring that PS documents are complete and polished (my group (1)) and of guaranteeing that there will be no significant changes in the future (either in going to DS or in grade).If that is really our position, then we better stop grumbling about how long it takes to get PS documents out (and the partially-consequential deployment of technologies from I-Ds), accept the fact that the SDOs who used to be very concerned about how much faster the IETF was than they were have now won and become faster than us (in large measure because we have slowed down so much). And we should stop wasting time trying to figure out how many maturity levels we need or what those criteria should be because the answer is one level is what we get and pretending anything else, e.g., by trying to fine-tune procedures, is an embarrassing public act of self-delusion (stronger language occurs to me but would probably get the message killed by spam filters). Remember that, while ignoring procedures and category definitions that we don't follow is not desirable, fixing them to reflect a model that doesn't (and won't) exist either is a public demonstration that we are disconnected from reality. I'd much rather leave that distinction to some other SDOs than join them. YMMD. john ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Discuss criteria for documents that advance on the standards track
On Aug 31, 2011, at 11:36 AM, John C Klensin wrote: We ought to, IMO, be permitting publication of PS documents at the second level as long as there are no _obvious_ ambiguities that cannot be figured out (the same way) by people of good will acting in good faith and with help from WG lists and as long as there are no other known [technical] defects. That would be fine with me if we could somehow effectively discourage deployment of proposed standards in products. But that's a lost cause, IMO. Keith, IMO, there are two possibilities here. At this point, sadly, both involve a chicken-and-egg problem. Such is life. (1) We proceed as if Proposed Standards are what 2026 (and the earlier culture) claims they are and work on ways to reinforce that notion in the community. [...] (2) We accept, and effectively encourage, deployment of proposed standards in products, either because it is a lost cause or because we think it is a good idea. [...] Agree with your description of the two possibilities, but I think the decision of possibility 2 has long since been forced on us by market expectation and habit. We could fix it, perhaps, by drastically changing how we label our products (dropping the whole notion of Proposed Standard, refusing to publish your category 2 documents as RFCs, etc.). But we can't significantly change market perception of what RFC and Proposed Standard mean. (which, BTW, is why I think that moving to a two-step process, while probably Mostly Harmless, won't do a bit of good in the overall scheme of things) Remember that, while ignoring procedures and category definitions that we don't follow is not desirable, fixing them to reflect a model that doesn't (and won't) exist either is a public demonstration that we are disconnected from reality. I'd much rather leave that distinction to some other SDOs than join them. YMMD. And my assertion is that the model inherent in your possibility #1 above doesn't exist and won't exist absent from drastic change. Insisting on high quality for proposed standards is recognizing current market reality. And I'm not one of those people who believes that market perceptions are inherently unchangeable. But expecting the market to change how it interprets our documents without us bothering to significantly change how we present them to the public - THAT would indeed be a serious disconnect from reality. Keith ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Discuss criteria for documents that advance on the standards track
Keith, Yes, to what you are saying, but I was pointing out that the text we're discussing isn't intended to apply to moving what a working group has consensus for onto the standards track, it's intended to apply to what the *IETF* already has consensus for, that's already on the standards track, further along the standards track. I think there's a higher bar, especially when a document is advancing unchanged, because I don't think there's any meaningful distinction between a widely deployed PS that could be improved, and advancing it to be a widely deployed IS that could be improved. I don't see the value-add from DISCUSSing the advancement, because the same document is sitting there as a proposed standard, and widely deployed. If ADs look at a document proposed for advancement and see real and meaningful opportunities to improve the specification, I think it makes just as much sense to advance the document, as is, and start looking for people who will produce an improved version, as to slow down the document for DISCUSSion in the hope you end up with an improved document, that can then advance. Finding those people, and chartering that work, falls well within the S in IESG, AFAICT. Thanks, Spencer - Original Message - From: Keith Moore To: Spencer Dawkins Cc: Jari Arkko ; IETF Discussion Sent: Wednesday, August 31, 2011 10:35 AM Subject: Re: Discuss criteria for documents that advance on the standards track thanks Spencer for pointing this part out. On Aug 31, 2011, at 11:23 AM, Spencer Dawkins wrote: IESG reviews should be considered as a review of last resort. Most documents reviewed by the IESG are produced and reviewed in the context of IETF working groups. In those cases, the IESG cannot overrule working group consensus without good reason; informed community consensus should prevail. The idea that WG consensus should prevail is simply incorrect. It biases IESG in an inappropriate way. There are a number of very good reasons for overriding WG consensus, e.g. - there is no evidence of broad community consensus or a clear lack of broad community consensus - the document does not meet the criteria specified in 2026 (or other document when applicable) - the document is ambiguous in such a way that it is likely to degrade interoperability The WG DOES NOT represent the entire community.Far too often, WGs are deliberately chartered in such a way as to marginalize parts of the community Keith ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Discuss criteria for documents that advance on the standards track
--On Wednesday, August 31, 2011 11:47 -0400 Keith Moore mo...@network-heretics.com wrote: ... IMO, there are two possibilities here. At this point, sadly, both involve a chicken-and-egg problem. Such is life. (1) We proceed as if Proposed Standards are what 2026 (and the earlier culture) claims they are and work on ways to reinforce that notion in the community. [...] (2) We accept, and effectively encourage, deployment of proposed standards in products, either because it is a lost cause or because we think it is a good idea. [...] Agree with your description of the two possibilities, but I think the decision of possibility 2 has long since been forced on us by market expectation and habit. We could fix it, perhaps, by drastically changing how we label our products (dropping the whole notion of Proposed Standard, refusing to publish your category 2 documents as RFCs, etc.). But we can't significantly change market perception of what RFC and Proposed Standard mean. Whether we agree on how best to manage a perception change and what it takes, I agree that there is no single magic bullet, much less a painless one. I would turn your comment around slightly and say that a change in names is likely to have little or no effect on perceptions unless we actually change what we do. As I said in and earlier note, there is something of a chicken-and-egg problem here. while probably Mostly Harmless, won't do a bit of good in the overall scheme of things) For reasons I've explained in previous notes, I don't think procedural changes, especially from not followed and doesn't work but ancient or contemporary and [still] doesn't work or reflect reality, are harmless (mostly or otherwise). Remember that, while ignoring procedures and category definitions that we don't follow is not desirable, fixing them to reflect a model that doesn't (and won't) exist either is a public demonstration that we are disconnected from reality. I'd much rather leave that distinction to some other SDOs than join them. YMMD. And my assertion is that the model inherent in your possibility #1 above doesn't exist and won't exist absent from drastic change. Insisting on high quality for proposed standards is recognizing current market reality. If you are right, then fussing around with how many maturity levels we have, what criteria we claim we are using, and even how we name things, are dangerous as well as a waste of time and resources. That is where we are maybe in violent agreement -- we either ought to be identifying real problems and fixing them or just staying with what we have until we have the knowledge and will needed to make real changes. In that regard, I applaud one aspect of Jari's note, which is that it affects and IETF administrative procedure and not a fundamental (and unnecessary) procedural change. By contrast, suppose draft-housley-two-maturity-levels were just shelved for a while and replaced by an announcement by an AD or two (ideally from areas that have really low advancement rates even as compared to the already-low IETF average) that they intended to (i) accept a public assertion of interoperability as an implementation report, issue an IETF LC for DS to see if anyone with real experience with the protocol objects loudly, and then move to advance the document, and (ii) that they would construe the passage of the relevant number of month and public claims of deployment as sufficient incentive to issue an IETF LC for advancement to IS. If the rest of the IESG were willing to go along with that, we'd have a real experiment without permanent changes to the official procedures. People who saw a problem advancing a particular protocol or document could object and it would presumably work only for PS documents that were of high quality. If it worked out, we would have a basis for coming back and evaluating whether there really was enough of a difference between DS and IS to be worth the trouble and whether the practical differences between that sort of implementation report and today's interpretation of what is required was. And then we could take ...two-maturity-levels back up with some confidence about both utility and potential harm. And, if you are right and this is all hopeless without really major changes, then that experiment might give us the information needed to finally come to grips with that and either make the major change or accept the status quo and stop spending energy trying to turn the knobs a degree or two. And I'm not one of those people who believes that market perceptions are inherently unchangeable. But expecting the market to change how it interprets our documents without us bothering to significantly change how we present them to the public - THAT would indeed be a serious disconnect from reality. I actually agree. john
Re: Discuss criteria for documents that advance on the standards track
On Aug 31, 2011, at 12:19 PM, John C Klensin wrote: we either ought to be identifying real problems and fixing them or just staying with what we have until we have the knowledge and will needed to make real changes. That would certainly be my preference. Keith ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Discuss criteria for documents that advance on the standards track
There's something inherently wrong with trying to establish criteria for voting DISCUSS. My understanding was always that DISCUSS was supposed to be an indication that, at a minimum, the AD needs to understand the situation better before casting a yea or nay vote. The resolution of a DISCUSS might end up being a yes vote, a no objection vote, a request for clarification of the document. It should also be possible for an AD to say now that I've understood better, I can't support this going forward (for which ABSTAIN is not an appropriate label). It should never be wrong for an AD to vote DISCUSS, though it's reasonable to limit the amount of time during which a DISCUSS can stand for a particular document. It should also never be wrong for an AD to vote NO if the AD has initially voted DISCUSS, made a sincere effort to understand the issues, believes that he does so, and can state 2026 or other documented criteria for why the document is not suitable for approval. So my recommendations are: 1. Fix the broken IESG voting system before you try to establish more decision criteria. I'm serious. The system you have now is just too broken, and has been broken for a long time. It places too much pressure on ADs to cave in when a WG produces flawed output that isn't easily fixed. It is weighed too heavily in favor of approving a document - such that one AD can vote YES, another vote DISCUSS, no other ADs read the document and all vote NO OBJECTION, and the AD that votes DISCUSS will be expected to change that vote to ABSTAIN because nobody else felt like reading it. And the idea that an AD should ever, ever, be compelled to change his vote to an ABSTAIN is completely unacceptable. And the votes of ADs who haven't read the document should not count AT ALL. Here is a stab at better voting criteria: - The possible votes are: Yes, No, No Objection, Discuss, Abstain, Recuse - Yes means I've read it, I believe it meets applicable criteria (2026 or whatever else is applicable), that there's community-wide consensus to support the document, and that all issues raised by reviewers (including directorate, last call comments, etc.) have been adequately addressed. - No means I've read it, but one or more of the above criteria have failed to have been met. The criteria must be documented. - No Objection means I've read it, and I think there are issues with it, but I don't think those issues are sufficient to block the document. The criteria should be documented, but the AD isn't compelled to do so. - Discuss means I have read the document and see at least one potential issue which needs to be discussed. Discuss should always be raised before voting No. However, Discuss votes should time out and be replaced by Abstain (if not explicitly changed by the AD) after say 45 days. (however, nothing should ever stop an AD from changing his vote to No.) Note that the ADs, WG chairs, authors, etc. should attempt to resolve the issue offline (not during the telechat or other IESG meeting) but the discussion should be archived. - Abstain means I did not read this, and/or I don't consider myself competent to review this - Recuse means I have a conflict of interest with stating a position on this document All Discuss votes must be changed (whether explicitly by the AD or by automatic timeout) before a ballot is finished. If there are two or more No votes, all ADs without a conflict of interest are expected to read the document, and the vote is taken again. In order to approve a document or standards action, there must be twice as many Yes + No Objection votes as No votes. 2. Don't overconstrain the use of DISCUSS. In particular, don't ever create a situation where a reviewer can't cite a problem with a document, regardless of whether that problem has previously been enumerated.It's fine to have guidelines for the benefit of new ADs but they should be nonbinding. You need to have other ways of dealing with the occasional stubborn AD who votes DISCUSS or whatever without a defensible reason. This applies to all IESG voting actions, not just moving from PS-DS/IS. 3. I take serious issue with the statement in the draft that IESG reviews are reviews of last resort and the implication that WG reviews are sufficient. In numerous situations this has not been the case. I'm all for having IESG place greater reliance on directorates (especially if this is formalized to some degree), so as to lessen IESG's workload to to get individual ADs out of the hot seat. But for IESG to presume by default that the WG has done due diligence is for IESG to shirk its responsibility. 4. When evaluating PS-DS/IS actions, IESG should avoid repeating the PS review. Instead it should look at what has changed between PS and DS/IS, and what has been learned as a result of implementation and deployment experience. The changes have to be minimal and safe.No new features can
Re: Discuss criteria for documents that advance on the standards track
On Aug 30, 2011, at 2:17 PM, Keith Moore wrote: My understanding was always that DISCUSS was supposed to be an indication that, at a minimum, the AD needs to understand the situation better before casting a yea or nay vote. The resolution of a DISCUSS might end up being a yes vote, a no objection vote, a request for clarification of the document. It should also be possible for an AD to say now that I've understood better, I can't support this going forward (for which ABSTAIN is not an appropriate label). It should never be wrong for an AD to vote DISCUSS, though it's reasonable to limit the amount of time during which a DISCUSS can stand for a particular document. It should also never be wrong for an AD to vote NO if the AD has initially voted DISCUSS, made a sincere effort to understand the issues, believes that he does so, and can state 2026 or other documented criteria for why the document is not suitable for approval. I agree in part. My problem is that even some ADs tell me that ADs sometimes behave as if they haven't done their job if they haven't produced a discuss. A few months ago, I got a discuss on a document that in essence said what should I be writing a 'discuss' about?. I have no problem with valid concerns, and many of the concerns raised are valid. It should be possible for a working group to produce a quality document and have the IESG simply approve it, however. I'm not sure it is. Discuss votes should time out and be replaced by Abstain (if not explicitly changed by the AD) after say 45 days. There I disagree. If the AD raised a valid issue, the ball is in the author/wg's court to address it. They can game this rule by not responding until after 45 days. Personally, I would rather see the document returned to the author/wg if review results in a request for major changes. In my working group, we recently had one draft completely rewritten in the response to discusses, and I pulled it back for the WG to decide whether it still had consensus around the resulting draft. I would say the same about a discuss that is not adequately responded to in 45 days; the IESG should clean it off their plate. What's also not fair game is to raise the bar - to expect the document at DS to meet more stringent criteria than it was required to meet at the time of PS approval. Hmmm, the demonstrated interoperability requirement of DS/IS is in fact a raising of the bar,and one that has served us well. We don't (although IMHO we should) require even an implementation to go to PS. ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Discuss criteria for documents that advance on the standards track
On Aug 30, 2011, at 5:51 PM, Fred Baker wrote: On Aug 30, 2011, at 2:17 PM, Keith Moore wrote: My understanding was always that DISCUSS was supposed to be an indication that, at a minimum, the AD needs to understand the situation better before casting a yea or nay vote. The resolution of a DISCUSS might end up being a yes vote, a no objection vote, a request for clarification of the document. It should also be possible for an AD to say now that I've understood better, I can't support this going forward (for which ABSTAIN is not an appropriate label). It should never be wrong for an AD to vote DISCUSS, though it's reasonable to limit the amount of time during which a DISCUSS can stand for a particular document. It should also never be wrong for an AD to vote NO if the AD has initially voted DISCUSS, made a sincere effort to understand the issues, believes that he does so, and can state 2026 or other documented criteria for why the document is not suitable for approval. I agree in part. My problem is that even some ADs tell me that ADs sometimes behave as if they haven't done their job if they haven't produced a discuss. A few months ago, I got a discuss on a document that in essence said what should I be writing a 'discuss' about?. I have no problem with valid concerns, and many of the concerns raised are valid. It should be possible for a working group to produce a quality document and have the IESG simply approve it, however. I'm not sure it is. The first time I reviewed a document as an AD, I realized I was working very hard to find something wrong. And I did - I found a bona fide (and uncontroversial) technical flaw - actually an incorrect value for a constant. But after that I got over needing to find things that were wrong. I found plenty of issues without trying to do anything more than read and understand the documents - and had quite enough work to do just reading and understanding them and writing up those issues without making more work for myself. (Especially when other ADs would often demand that I not only identify the issues but also specify the fixes - something I always regarded as out-of-scope for an AD and even potentially tampering with WG output.) So I don't disagree that ADs sometime believe that they need to vote Discuss or they're not doing their jobs. Though I have a hard time believing that the drafts coming out of WGs these days are so good that ADs have to work hard to find things that are wrong with many of them. If they're trying to find a reason to Discuss every document, well I guess that is a problem. But overall, it seems like there are far more incentives to avoid finding reasons to object to a document, than there are to find them. Discuss votes should time out and be replaced by Abstain (if not explicitly changed by the AD) after say 45 days. There I disagree. If the AD raised a valid issue, the ball is in the author/wg's court to address it. They can game this rule by not responding until after 45 days. I don't think I stated that rule very well. If the author/wg fail to address the issue within 45 days, the AD should of course be able to change the vote to No. The point is that the Discuss should inherently be a temporary state, a time during which the AD doesn't vote No (yet) but lets the author/WG know that he has issues with a document. And the idea is to encourage those parties enough time to understand each other and work out their differences before bringing the issue to a firm vote. But I don't think that the WG should be able to game the rule to force the AD's objection to go away, nor should the AD be able to singlehandedly block a document. Personally, I would rather see the document returned to the author/wg if review results in a request for major changes. In my working group, we recently had one draft completely rewritten in the response to discusses, and I pulled it back for the WG to decide whether it still had consensus around the resulting draft. I would say the same about a discuss that is not adequately responded to in 45 days; the IESG should clean it off their plate. IMO, any time a document is significantly revised, IESG should consider it a separate item, requiring a new Last Call, writeup, ballot, etc. What's also not fair game is to raise the bar - to expect the document at DS to meet more stringent criteria than it was required to meet at the time of PS approval. Hmmm, the demonstrated interoperability requirement of DS/IS is in fact a raising of the bar,and one that has served us well. We don't (although IMHO we should) require even an implementation to go to PS. I should have been more specific. By raising the bar I was thinking about things like design decisions, and document quality. If feature FOO was judged to be of adequate design at PS, it's not fair to say it's not adequate on DS review unless
Re: Discuss criteria for documents that advance on the standards track
On 2011-08-31 09:51, Fred Baker wrote: ... If the AD raised a valid issue, the ball is in the author/wg's court to address it. They can game this rule by not responding until after 45 days. Not if the draft has been updated and the AD doesn't either cancel the DISCUSS within a reasonable time*, or explain why the update is insufficient. This happens, probably as often as authors failing to address an issue within a reasonable time. Whichever party is responsible for the delay should be subject to a timeout, surely. That said, I disagree with Keith. The current DISCUSS criteria were written for a very specific reason - to get rid of the type of DISCUSS listed in the non-criteria. There was a lot of analysis done of documents that were stuck in the IESG for months in the 2004/2005 timeframe, and the non-criteria describe the problem cases - DISCUSSes that were not actionable or simply expressed the fact that the AD didn't like the WG's informed consensus. Of course nothing's perfect and I'm sure the criteria could be improved. But not having them was much worse. * for example, is IESG Evaluation::AD Followup (for 32 days) a reasonable delay for a Discussing AD to review an updated draft [hint, hint]? Brian ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Discuss criteria for documents that advance on the standards track
On 2011-08-31 08:18, Jari Arkko wrote: ... Here are the suggested guidelines for documents that advance to IS: http://www.arkko.com/ietf/iesg/discuss-criteria-advancing.txt Comments appreciated. To answer Jari's original request: +1 to these new guidelines. Not worth nit-picking until we have some experience. Brian ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf
