Re: Qualitative Analysis of IETF and IESG trends (Re: Measuring IETF and IESG trends)
Ted, The big problem others have been pointing to is that DISCUSSes are not being used to say here is a technical issue, for which any solution acceptable to the community is fine, but are instead being used to say here is a technical issue, and here's what it would take to satisfy me that it is resolved. The second formulation shortens easily in the minds of listeners to satisfy me, and when there is text presented, it becomes add/change this as below to remove my hold on your document. Ack. I agree that this is a concern, and something that I forgot to put on my list. Of course, as Joel and Brian pointed out, identifying this problem is not always as simple as looking at whether text came from the AD. Also, *if* you assume the Discuss was appropriate, presumably the resolution, whatever it is, has to satisfy some criteria so that the original problem goes away. If an AD is not happy about a particular text proposal, is it because the criteria was not met, or was it because he or she insisted on particular text? Obviously the former is appropriate and the latter is not. And how well were the criteria described? Many debates about resolutions involve either unclear criteria or disagreements about whether all criteria need to be fulfilled, more than the actual words in the resolution. The statement above is offensive, Jari. Blaming working groups for exhaustion after a late surprise is insensitive I'm sorry you found it offensive. I did not mean to be insensitive. If it helps, this item was on a long list of reasons why WG involvement isn't being handled as well as it should be. Not the biggest reason, or very commonly occurring one. (But I think I've seen a few cases where the author/WG was not interested in the particular way to resolve an issue, as long as it was resolved. Can't speak about why they were in that state.) Many, if not most reasons on my list rest on the ADs and some on the shepherds. I blame myself for not doing a better job in involving the WGs and I plan to improve this for the documents that I sponsor. Jari ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf
RE: Qualitative Analysis of IETF and IESG trends (Re: Measuring IETF and IESG trends)
Speaking as an individual who has also participated in the work of other standards organizations - In other SDOs, the IEEE 802 for example, suggesting a fix for a problem detected in the text at ballot time is not only welcome, but sometimes the recommended if not mandatory practice. Dan -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Brian E Carpenter Sent: Wednesday, July 02, 2008 12:58 AM To: Joel M. Halpern Cc: ietf@ietf.org Subject: Re: Qualitative Analysis of IETF and IESG trends (Re: Measuring IETF and IESG trends) On 2008-07-02 09:07, Joel M. Halpern wrote: Of course, we also get complaints whenever anyone raises an issue without providing text. So, by a strict reading of the argument, the AD is hanged if he provides text (directing the working group) and hanged if he does not provide text (you didn't make clear what your problem is, and how to fix it.) There is, I think a big difference between an AD writing (a) Here is the fix for my problem and (b) Here is my proposal for one way to fix this issue; there may of course be other ways to do so, so please let me know what the WG prefers to do. But that takes time to type in, and an overloaded AD (or reviewer) will be very tempted just to write Suggested fix:. Maybe we should assign specific (b) semantics to SUGGESTION and use that as shorthand? Brian ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Qualitative Analysis of IETF and IESG trends (Re: Measuring IETF and IESG trends)
The problems with the Discussing AD proposing text are more in the area of scalability. I prefer seeing the authors (or shepherds) be active and propose ways to resolve an issue. Or at least the initial proposal, review and suggestions from both sides may be needed to converge. This is not the big problem that other folks have been pointing to. The big problem others have been pointing to is that DISCUSSes are not being used to say here is a technical issue, for which any solution acceptable to the community is fine, but are instead being used to say here is a technical issue, and here's what it would take to satisfy me that it is resolved. The second formulation shortens easily in the minds of listeners to satisfy me, and when there is text presented, it becomes add/change this as below to remove my hold on your document. The other clause (or I won't remove my hold) is clearly heard even in the cases where the AD doesn't say it out loud. Whether you realize it or not, there are ADs who either say it about their own positions or ascribe it to other ADs pretty freely (That will never get past the X ADs, unless you change to Y being a formulation heard in the halls all to often). This not just about scaling problems. snip - WGs that for some reason have stopped caring about anything else than getting the document published. Not care about the particular hoop that they have to jump through to resolve a Discuss. (And by the same token, not care about Comment level review issues at all). The statement above is offensive, Jari. Blaming working groups for exhaustion after a late surprise is insensitive to say the least, and that is the case where the late surprise is warranted by a technical issue that does rise to DISCUSS levels. Blaming them for exhaustion after intransigence by specific ADs who really do mean satisfy me is worse than insensitive. It's blaming the victim. Some of these issues could be improved with a clearer definition of roles, and some additional guidelines on how to involve the WG. You know, members of the IESG acting as a check on each other and resisting efforts to force specific text changes would be useful too. If you would like to help personally rather than simply spread the blame Ted ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Qualitative Analysis of IETF and IESG trends (Re: Measuring IETF and IESG trends)
Of course, we also get complaints whenever anyone raises an issue without providing text. So, by a strict reading of the argument, the AD is hanged if he provides text (directing the working group) and hanged if he does not provide text (you didn't make clear what your problem is, and how to fix it.) In practice, we need to allow both sides of that to occur. But we need to be careful in adjusting the process not to say things like ADs providing text is a bad thing, because it is all to easily read as a demand. (And it is true that such text often is seen as a demand.) Or, I suppose, we could say that ADs should never provide text :-) Joel Ted Hardie wrote: The problems with the Discussing AD proposing text are more in the area of scalability. I prefer seeing the authors (or shepherds) be active and propose ways to resolve an issue. Or at least the initial proposal, review and suggestions from both sides may be needed to converge. This is not the big problem that other folks have been pointing to. The big problem others have been pointing to is that DISCUSSes are not being used to say here is a technical issue, for which any solution acceptable to the community is fine, but are instead being used to say here is a technical issue, and here's what it would take to satisfy me that it is resolved. The second formulation shortens easily in the minds of listeners to satisfy me, and when there is text presented, it becomes add/change this as below to remove my hold on your document. The other clause (or I won't remove my hold) is clearly heard even in the cases where the AD doesn't say it out loud. Whether you realize it or not, there are ADs who either say it about their own positions or ascribe it to other ADs pretty freely (That will never get past the X ADs, unless you change to Y being a formulation heard in the halls all to often). This not just about scaling problems. snip - WGs that for some reason have stopped caring about anything else than getting the document published. Not care about the particular hoop that they have to jump through to resolve a Discuss. (And by the same token, not care about Comment level review issues at all). The statement above is offensive, Jari. Blaming working groups for exhaustion after a late surprise is insensitive to say the least, and that is the case where the late surprise is warranted by a technical issue that does rise to DISCUSS levels. Blaming them for exhaustion after intransigence by specific ADs who really do mean satisfy me is worse than insensitive. It's blaming the victim. Some of these issues could be improved with a clearer definition of roles, and some additional guidelines on how to involve the WG. You know, members of the IESG acting as a check on each other and resisting efforts to force specific text changes would be useful too. If you would like to help personally rather than simply spread the blame Ted ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Qualitative Analysis of IETF and IESG trends (Re: Measuring IETF and IESG trends)
On 2008-07-02 09:07, Joel M. Halpern wrote: Of course, we also get complaints whenever anyone raises an issue without providing text. So, by a strict reading of the argument, the AD is hanged if he provides text (directing the working group) and hanged if he does not provide text (you didn't make clear what your problem is, and how to fix it.) There is, I think a big difference between an AD writing (a) Here is the fix for my problem and (b) Here is my proposal for one way to fix this issue; there may of course be other ways to do so, so please let me know what the WG prefers to do. But that takes time to type in, and an overloaded AD (or reviewer) will be very tempted just to write Suggested fix:. Maybe we should assign specific (b) semantics to SUGGESTION and use that as shorthand? Brian ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Qualitative Analysis of IETF and IESG trends (Re: Measuring IETF and IESG trends)
Laksminath, My point was this: if a WG actually missed anything substantial and that comes out during an IETF last call, and the shepherding AD agrees, the document gets sent back to the WG. If the shepherding AD also misses or misjudges, any member of the IESG can send it back to the WG for resolution. What I think is not acceptable is for the author and one or more DISCUSS ADs to hack up the document and publish it. I think it would be useful to separate this discussion into different parts: - what issues can be raised as a Discuss - balancing the IETF opinion vs. WG opinion when there is a conflict - how text proposals and draft revisions get created - the role of shepherds, authors, and sponsors during this process - how WGs become involved in changes from IETF Last Call and IESG reviews The Discuss criteria document says quite a bit of the first topic. Earlier in this thread we talked about the second topic. But I wanted to say a few words about changing text to resolve an issue. I said earlier that I would rather not be sending text proposals. I didn't want to imply that text coming from the IESG would be inappropriate. And certainly text coming from the author would not be inappropriate either. In fact, it is quite typical that the Discussing AD, possible directorate experts (if involved), and the author are the most likely suspects to come up with a way to resolve an issue. And most motived to get to a resolution. For instance, some of my documents have recently had issues with PMTU, and I asked the Discussing transport ADs to work with the authors to design a solution; this worked well. In this sense hacking the document is not just OK but it may actually be the right thing. The problems with the Discussing AD proposing text are more in the area of scalability. I prefer seeing the authors (or shepherds) be active and propose ways to resolve an issue. Or at least the initial proposal, review and suggestions from both sides may be needed to converge. The big problem with any of the key players (author, Discussing AD, sponsoring AD) making changes relates mostly to the fifth item on my list. We are not very good at keeping the WGs in the loop. This is often done right, but far from always. I know I have problems in this area. One of the consequences is less control on the WG's side on controversial topics. Another one is reduced review of new text; errors might creep in. While the hacking part may have been OK, the publication is the problem. I don't want to make excuses, but it may be helpful to understand some of the reasons behind this: - Some issues are too small or obvious to warrant engaging the WG; getting the document approved on the given telechat day is seen as a higher priority. A fairly large number of Discusses get resolved on the day of the telechat, having been filed just days or hours before. - The author - AD team works at a much faster pace in many cases than, say, the shepherd or the entire WG. - Discussing AD is not on the WG list, does not know status of the document in other respects than his or hers own Discuss. - Things falling in the cracks, e.g., Discussing AD thinks that the shepherd or the sponsoring AD is responsible for talking to the WG. - No guidelines or processes relating to how the WG is actually involved in the discussion. In some cases we inform the WG what the Discusses are and what is being done about them. In other cases we actually run something resembling a WGLC or a consensus call. In yet other cases the new draft version announcement goes to the list as the sole announcement, and people are expected to look at the tracker for the Discusses. - Desire to avoid lengthy discussions. - Sponsoring AD trusts that the Discussing AD and author have made the right choices. - WGs that for some reason have stopped caring about anything else than getting the document published. Not care about the particular hoop that they have to jump through to resolve a Discuss. (And by the same token, not care about Comment level review issues at all). Some of these issues could be improved with a clearer definition of roles, and some additional guidelines on how to involve the WG. Jari ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Qualitative Analysis of IETF and IESG trends (Re: Measuring IETF and IESG trends)
Hi Lakshminath, At 07:11 27-06-2008, Lakshminath Dondeti wrote: Is it really necessary for all the battles to repeat on the IETF list? Why can't the shepherding AD judge the overall consensus? No, it is not necessary. One could read the WG discussion of the topic instead of rehashing the same arguments on the IETF list. The shepherd is there to gather information, get the document through the various stages and provide coordination. The PACT I-D may be a useful read: An IETF effort is designed to resolve engineering choices for one issue and then move to a new issue. It is not reasonable to permit arbitrary criticisms to be raised late in the process, derailing the incremental effort of a WG. It is always reasonable to raise fundamental engineering problems, but it is essential to distinguish these from matters of engineering aesthetics. In particular, the IETF Last Call and IESG review periods are not intended for second-guessing a WG's design choices -- the purpose of an IETF Last Call and IESG review is to focus on the overall viability of the document. I'll also highlight a few points from RFC 3774: Participants are frequently allowed to re-open previously closed issues just to replay parts of the previous discussion without introducing new material. This may be either because the decision has not been clearly documented, or it may be a maneuver to try to get a decision changed because the participant did not concur with the consensus originally. On the other hand, the decision making process must allow discussions to be re-opened if significant new information comes to light or additional experience is gained which appears to justify alternative conclusions for a closed issue. One cause that can lead to legitimate attempts to re-open an apparently closed issue is the occurrence of 'consensus by exhaustion'. The IETF culture of openness also tends to tolerate participants who, whilst understanding the principles of the IETF, disagree with them and actively ignore them. This can be confusing for newer participants, but they need to be made aware that the IETF does not exclude such people. Regards, -sm ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Qualitative Analysis of IETF and IESG trends (Re: Measuring IETF and IESG trends)
Brian, Thanks for your response. Please see inline: On 6/26/2008 4:23 PM, Brian E Carpenter wrote: Lakshminath, On 2008-06-26 23:43, Lakshminath Dondeti wrote: On 6/25/2008 2:41 PM, Brian E Carpenter wrote: ... Our fundamental collective job is defined in RFC 3935: The mission of the IETF is to produce high quality, relevant technical and engineering documents that influence the way people design, use, and manage the Internet in such a way as to make the Internet work better. That means that it is *not* our collective job to ensure that a WG consensus survives critical review by the IETF as a whole and by the IESG, if there's reason to believe that the IETF as a whole doesn't agree with the WG consensus. And it's clearly the IESG's job to ensure that the critical review and final consensus (or lack of consensus) occur. But, surely the WG consensus counts as part of the overall IETF consensus process, doesn't it? Please see the example in my response to Jari. The shepherding AD (or at least the document shepherd) has an idea of the WG consensus as well as the IETF consensus. We cannot simply weigh the latest opinions more than all the discussions that have happened as part of the WG consensus. At one level I agree. But suppose that the set of people who are active in the SXFG7M WG are so focused on the sxfg7m protocol that they have all missed the fact that it's extremely damaging to normal operations of the m7gfxs protocol? And this includes the responsible AD, who has no deep knowledge of m7gfxs? This is the sort of problem that IETF Last Call and IESG review is intended to find, and it may well mean that the WG consensus ends up being irrelevant to the IETF non-consensus. (I'm not in the least suggesting that this applies to the draft that led to the appeal that led to this thread.) For what it's worth, I am not talking about a specific draft or a specific WG at this point. I am of the opinion that we are not discussing a one-off issue. If protocol X disrupts protocol Y, we get into very interesting situations. It is also going to get us into a rathole that I want to avoid. My point was this: if a WG actually missed anything substantial and that comes out during an IETF last call, and the shepherding AD agrees, the document gets sent back to the WG. If the shepherding AD also misses or misjudges, any member of the IESG can send it back to the WG for resolution. What I think is not acceptable is for the author and one or more DISCUSS ADs to hack up the document and publish it. If it so happens that the issue raised was considered and ruled out as a non-issue by the WG, then the shepherding AD knows the situation already. Strong consensus in the working group damaging a protocol that matters to very few people (ok, that's a rathole) -- but here is where judgment is necessary. And as you note, any of the judgment calls are appealable. regards, Lakshminath My conclusion, again, is that in the end this is the sort of judgment call that we *expect* the IESG to make. And when we feel they've misjudged, we appeal, and that tunes their judgment for the future. Brian ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Qualitative Analysis of IETF and IESG trends (Re: Measuring IETF and IESG trends)
On 6/26/2008 6:35 PM, SM wrote: At 04:43 26-06-2008, Lakshminath Dondeti wrote: But, surely the WG consensus counts as part of the overall IETF consensus process, doesn't it? Please see the example in my response to Jari. The shepherding AD (or at least the document shepherd) has an idea of the WG consensus as well as the IETF consensus. We cannot simply weigh the latest opinions more than all the discussions that have happened as part of the WG consensus. The document may be a product of WG consensus. It still has to pass through the community and the IESG to be published as an IETF document. The WG knows about the internals of the document and generally have a focused view. The last call allows a wider range of input and to gauge the impact the proposal may have in other areas. It is not about weighing the latest opinions more. The author/shepard can always post an explanation. The participants in the WG should be aware that there will be an IETF-wide last call. You cannot blame the process if they choose to remain silent instead of taking part in the last call. Note that letter-writing campaigns in a last call have been proven to be ineffective. Is it really necessary for all the battles to repeat on the IETF list? Why can't the shepherding AD judge the overall consensus? thanks, Lakshminath Regards, -sm ___ IETF mailing list IETF@ietf.org https://www.ietf.org/mailman/listinfo/ietf ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Qualitative Analysis of IETF and IESG trends (Re: Measuring IETF and IESG trends)
Lakshminath: Consider a hypothetical case: a large WG has strong consensus on one of their documents, they believe it is within the charter's scope and think that the document is in the best interest of the Internet. The WG chairs assess the consensus, and forward the document to the shepherding AD. The shepherding AD takes one last look, determines everything is in order and sends it to last call. A few people on the IETF Discussion list think that the proposed specification is about to doom the Internet. A few others who have not even read the document agree based on emails. Most of the WG members are either not on the IETF list or choose to stay silent. The shepherding AD considers those comments, thinks that those issues have been addressed already and puts the document on the IESG agenda. All other ADs (except one) think that everything is fine and vote No Objection. One AD agrees with the few people on the IETF Discussion list and decides to put a DISCUSS and proceeds to hack the document. In the current model, other than the very few exceptions cited recently, the AD gets what he or she wants for the most part. It is plausible that AD may do this even if no one else identified a problem. Actually, this sounds very similar to the case where an override vote was almost used. Scheduling the override vote was sufficient for the DISCUSS-holding AD to ask for a strawpoll, and based on those results, the DISCUSS-holding AD cleared the DISCUSS position. Russ ___ Ietf mailing list Ietf@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Qualitative Analysis of IETF and IESG trends (Re: Measuring IETF and IESG trends)
On 6/25/2008 9:19 AM, Melinda Shore wrote: On 6/25/08 11:44 AM, Lakshminath Dondeti [EMAIL PROTECTED] wrote: I would like to hear others' opinions (I was going to put together a draft with some ideas on how we might define these roles, but I want to hear others' thoughts before I do that) on this topic. I think your points are valid, but I'm not sure what the effect would be if you controlled for quality coming out of the working groups. That is to say, I think that occasionally working groups are coming to consensus on bad documents or bad ideas, and that the incidence of that is increasing. Well that is a disturbing trend as well. A long while ago now, one of the then ADs mentioned that he needed to put a DISCUSS on a few successive MSEC documents that I was shepherding and mentioned that he wants to have a chat with the chairs on the quality of documents that we are forwarding. I asked him to come to one of our meetings and explain the expectations directly to the WG. That never happened. But that is the kind of direction or steering an area director might do if working groups are indeed producing bad documents and advancing bad ideas. Presumably they are several cases here, viz., going against charter or just being plain terrible at writing interoperable specifications. Pushing a document back to the WG is actually a better thing, I am beginning to think. Sure that may introduce delays initially as we learn how to operate in that mode, but the process of writing specifications is more transparent and more consensus based than in the current model of operation. One of the problems with the current model is that comments toward the end of the process, either AD's or reviewers', are weighted more than comments during the working group discussions, at least in some cases. If that's true it once again raises the very familiar question of picking up quality problems earlier in the process. Actually, that latter question applies regardless. Yes, I have heard it mentioned several times before, but I haven't seen any concrete steps to achieve that. I am now making the case that if a document makes it beyond the shepherding AD and put on the IESG telechat, changes to the text should be relatively a big deal. There can be changes, but they should be reviewed by the WG. In some cases, there have been bitter disputes about taking change control from authors and putting it in the hands of a WG, but the current model toward the end of the process seems to put change control in too few hands. That is what I am trying to highlight. regards, Lakshminath Melinda ___ IETF mailing list IETF@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Qualitative Analysis of IETF and IESG trends (Re: Measuring IETF and IESG trends)
Melinda Shore wrote: I think your points are valid, but I'm not sure what the effect would be if you controlled for quality coming out of the working groups. The IETF works without any effort to measure quality or even uptake. As a consequence, we have no way of determining whether our protocols succeed in the long-run, or whether AD Discusses are true diligence or mere whimsy. What we are left with is isolated cases and reactions. d/ -- Dave Crocker Brandenburg InternetWorking bbiw.net ___ IETF mailing list IETF@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Qualitative Analysis of IETF and IESG trends (Re: Measuring IETF and IESG trends)
Jari, Thanks. Some thoughts inline: On 6/25/2008 11:30 AM, Jari Arkko wrote: Lakshminath, Better understanding of the type of behaviors in this space would certainly be useful. And I don't want to disagree with your assessment of the behaviors; many of them sound like something that appears in practice. In particular, the shepherds are far less involved in the Discuss resolutions than the authors. And we do not involve the WGs as much as we should. I think writing guidelines on what the role of the various persons in the process is would be very useful. And obviously we should start by better following of the existing documents, like the Proto Shepherd RFC or the Discuss criteria document. Of course. I will try and see if I can put something of an initial draft coherently. However, with regards to blocking consensus of a WG, please remember that the WG is not necessarily the only place where consensus is tested. I recently had a case which had significant IETF Last Call discussion. I held a Discuss to ensure that the (fairly clear, IMO) conclusion from the discussion would be taken into the document. It did eventually, but only after significant back-and-forth with the authors. Overriding the original WG consensus? Yes. Right thing to do? I think so, not only was it right technically but it was something backed up by the Last Call. Did we get the details right, did the text go too far or did it fall short? I don't know, its a judgment call. The end result was somewhere between the LC guidance and authors' opinions. Painful for the WG? Sure. So, here is where I am probably confused as to how consensus is determined. I understand that the documents are eventually IETF Consensus with the IESG determining what the consensus is. The sponsoring AD has the overall view of any given document better than anyone else, at least in the authoritative role (for this discussion, I am just considering WG documents). Consider a hypothetical case: a large WG has strong consensus on one of their documents, they believe it is within the charter's scope and think that the document is in the best interest of the Internet. The WG chairs assess the consensus, and forward the document to the shepherding AD. The shepherding AD takes one last look, determines everything is in order and sends it to last call. A few people on the IETF Discussion list think that the proposed specification is about to doom the Internet. A few others who have not even read the document agree based on emails. Most of the WG members are either not on the IETF list or choose to stay silent. The shepherding AD considers those comments, thinks that those issues have been addressed already and puts the document on the IESG agenda. All other ADs (except one) think that everything is fine and vote No Objection. One AD agrees with the few people on the IETF Discussion list and decides to put a DISCUSS and proceeds to hack the document. In the current model, other than the very few exceptions cited recently, the AD gets what he or she wants for the most part. It is plausible that AD may do this even if no one else identified a problem. Responding to Brian here: Note that I am not pointing fingers at IESG alone, but yes, I am pointing fingers at our process in that the hypothetical situation described above is allowed to happen at the moment. Brian suggests external factors, but I am sorry I am not convinced. If there are WGs that have forgotten the IETF's mission, it is one of the primary roles of the ADs and the IESG to steer those WGs in the right direction. If our technology is becoming harder and more complex, it calls for involving more people and increasing transparency and not the other way around. On text that comes from the IESG: this is more common in recent years than it was before. I am one of the ADs who tends to do that, both for the documents that I sponsor and for resolving my Discusses. But I would rather not do it. But I often end up doing it if there is no progress otherwise; I want to get my sponsored documents approved and I want to reduce the list of my outstanding Discusses. If I can help my authors by proposing text, I will do it. Jari, as I have noted before, if the status quo is considered the best way forward, I would rather you continue to propose text. I as an author and document shepherd have appreciated that compared to the alternative. That alternative, the iterative guess work, takes forever. But I would really like to see the document shepherds in active role here. Or at least the authors. The general guidance for authors whose document gets a Discuss is to first confirm whether the raised issue is a real one or not. If it is not, ask the Discuss to be cleared. Fight if needed! If it is real, work with your shepherd and WG to develop a proposal to fix the problem. Mail the proposal to the Discussing ADs in a timely manner.
Re: Qualitative Analysis of IETF and IESG trends (Re: Measuring IETF and IESG trends)
On 6/25/2008 4:28 PM, John C Klensin wrote: --On Thursday, 26 June, 2008 09:41 +1200 Brian E Carpenter [EMAIL PROTECTED] wrote: ... And of course, individual ADs have to think carefully whether a given issue is or is not worthy of a DISCUSS, and sometimes they get it wrong. But that will always be true, however we tune the process and procedures. Brian, While I agree with this, I also believe that there have to be effective safeguards against bad judgments prevailing for too long. And I believe that those have largely slipped away from us... unless we believe that making changes, unvalidated by WG or mailing list consensus, simply to get a DISCUSS removed is the right way to get to high quality, relevant technical and engineering documents Indeed that is the issue! regards, Lakshminath john ___ IETF mailing list IETF@ietf.org https://www.ietf.org/mailman/listinfo/ietf ___ IETF mailing list IETF@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Qualitative Analysis of IETF and IESG trends (Re: Measuring IETF and IESG trends)
On 6/25/2008 2:41 PM, Brian E Carpenter wrote: On 2008-06-26 06:30, Jari Arkko wrote: Lakshminath, Better understanding of the type of behaviors in this space would certainly be useful. And I don't want to disagree with your assessment of the behaviors; many of them sound like something that appears in practice. In particular, the shepherds are far less involved in the Discuss resolutions than the authors. And we do not involve the WGs as much as we should. I think writing guidelines on what the role of the various persons in the process is would be very useful. And obviously we should start by better following of the existing documents, like the Proto Shepherd RFC or the Discuss criteria document. However, with regards to blocking consensus of a WG, please remember that the WG is not necessarily the only place where consensus is tested. I recently had a case which had significant IETF Last Call discussion. I held a Discuss to ensure that the (fairly clear, IMO) conclusion from the discussion would be taken into the document. It did eventually, but only after significant back-and-forth with the authors. Overriding the original WG consensus? Yes. Right thing to do? I think so, not only was it right technically but it was something backed up by the Last Call. Did we get the details right, did the text go too far or did it fall short? I don't know, its a judgment call. The end result was somewhere between the LC guidance and authors' opinions. Painful for the WG? Sure. On text that comes from the IESG: this is more common in recent years than it was before. I am one of the ADs who tends to do that, both for the documents that I sponsor and for resolving my Discusses. But I would rather not do it. But I often end up doing it if there is no progress otherwise; I want to get my sponsored documents approved and I want to reduce the list of my outstanding Discusses. If I can help my authors by proposing text, I will do it. But I would really like to see the document shepherds in active role here. Or at least the authors. The general guidance for authors whose document gets a Discuss is to first confirm whether the raised issue is a real one or not. If it is not, ask the Discuss to be cleared. Fight if needed! If it is real, work with your shepherd and WG to develop a proposal to fix the problem. Mail the proposal to the Discussing ADs in a timely manner. Address explicitly all components raised in a Discuss, either by explaining how they are not issues or providing a solution to resolve the issue. Our fundamental collective job is defined in RFC 3935: The mission of the IETF is to produce high quality, relevant technical and engineering documents that influence the way people design, use, and manage the Internet in such a way as to make the Internet work better. That means that it is *not* our collective job to ensure that a WG consensus survives critical review by the IETF as a whole and by the IESG, if there's reason to believe that the IETF as a whole doesn't agree with the WG consensus. And it's clearly the IESG's job to ensure that the critical review and final consensus (or lack of consensus) occur. But, surely the WG consensus counts as part of the overall IETF consensus process, doesn't it? Please see the example in my response to Jari. The shepherding AD (or at least the document shepherd) has an idea of the WG consensus as well as the IETF consensus. We cannot simply weigh the latest opinions more than all the discussions that have happened as part of the WG consensus. That, IMHO, is the proper role of a DISCUSS and the proper reason for delays in document approval. And if we see fluctuation in these delays, and fluctuation in the amount of active intervention by the ADs, it does not follow that the IESG is to blame. Maybe there are external factors, maybe there are WGs that are forgetting the IETF's mission, maybe our technology is getting harder and more complex. So I'm very dubious about using either quantitative *or* qualitative observations to point the finger at the IESG, or at process issues in general, without digging much deeper. Of course, the IESG needs to pay attention to delays, so Jari's data (like the earlier data that Bill Fenner used to produce) are very valuable. And of course, individual ADs have to think carefully whether a given issue is or is not worthy of a DISCUSS, and sometimes they get it wrong. But that will always be true, however we tune the process and procedures. I am suggesting that we make further progress along the lines of the definition of the role of the document shepherd and reassert (or clearly define and state the expectations) on the roles of the document shepherd and shepherding AD. regards, Lakshminath Brian ___ IETF mailing list IETF@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Qualitative Analysis of IETF and IESG trends (Re: Measuring IETF and IESG trends)
Lakshminath, On 2008-06-26 23:43, Lakshminath Dondeti wrote: On 6/25/2008 2:41 PM, Brian E Carpenter wrote: ... Our fundamental collective job is defined in RFC 3935: The mission of the IETF is to produce high quality, relevant technical and engineering documents that influence the way people design, use, and manage the Internet in such a way as to make the Internet work better. That means that it is *not* our collective job to ensure that a WG consensus survives critical review by the IETF as a whole and by the IESG, if there's reason to believe that the IETF as a whole doesn't agree with the WG consensus. And it's clearly the IESG's job to ensure that the critical review and final consensus (or lack of consensus) occur. But, surely the WG consensus counts as part of the overall IETF consensus process, doesn't it? Please see the example in my response to Jari. The shepherding AD (or at least the document shepherd) has an idea of the WG consensus as well as the IETF consensus. We cannot simply weigh the latest opinions more than all the discussions that have happened as part of the WG consensus. At one level I agree. But suppose that the set of people who are active in the SXFG7M WG are so focused on the sxfg7m protocol that they have all missed the fact that it's extremely damaging to normal operations of the m7gfxs protocol? And this includes the responsible AD, who has no deep knowledge of m7gfxs? This is the sort of problem that IETF Last Call and IESG review is intended to find, and it may well mean that the WG consensus ends up being irrelevant to the IETF non-consensus. (I'm not in the least suggesting that this applies to the draft that led to the appeal that led to this thread.) My conclusion, again, is that in the end this is the sort of judgment call that we *expect* the IESG to make. And when we feel they've misjudged, we appeal, and that tunes their judgment for the future. Brian ___ IETF mailing list IETF@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Qualitative Analysis of IETF and IESG trends (Re: Measuring IETF and IESG trends)
--On Friday, 27 June, 2008 11:23 +1200 Brian E Carpenter [EMAIL PROTECTED] wrote: ... At one level I agree. But suppose that the set of people who are active in the SXFG7M WG are so focused on the sxfg7m protocol that they have all missed the fact that it's extremely damaging to normal operations of the m7gfxs protocol? And this includes the responsible AD, who has no deep knowledge of m7gfxs? This is the sort of problem that IETF Last Call and IESG review is intended to find, and it may well mean that the WG consensus ends up being irrelevant to the IETF non-consensus. (I'm not in the least suggesting that this applies to the draft that led to the appeal that led to this thread.) My conclusion, again, is that in the end this is the sort of judgment call that we *expect* the IESG to make. And when we feel they've misjudged, we appeal, and that tunes their judgment for the future. Brian, Again, I agree. And this sort of example is precisely one of the major reasons why proposals to let a WG override an IESG conclusion about consensus are problematic (the other involves a WG that has been captured by a particular interest in order to get a particular protocol output, but I'd expect an AD to notice that and deal with it long before documents go into Last Call). However, it means that we need to treat appeals about IESG judgment calls as a completely normal and orderly part of the process, rather than something sufficiently unusual that either (i) the IESG treats an appeal as an attack on them collectively and starts circling the wagons rather than considering the appeal as a normal and formal request from the community for reconsideration, or that (ii) IETF participants believe that they need to fear retaliation from one or more ADs if they generate appeals independent of what might or might not be going on with recent cases, I believe we have seen both phenomena in the last several years. This is not an easy problem. john ___ IETF mailing list IETF@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Qualitative Analysis of IETF and IESG trends (Re: Measuring IETF and IESG trends)
At 04:43 26-06-2008, Lakshminath Dondeti wrote: But, surely the WG consensus counts as part of the overall IETF consensus process, doesn't it? Please see the example in my response to Jari. The shepherding AD (or at least the document shepherd) has an idea of the WG consensus as well as the IETF consensus. We cannot simply weigh the latest opinions more than all the discussions that have happened as part of the WG consensus. The document may be a product of WG consensus. It still has to pass through the community and the IESG to be published as an IETF document. The WG knows about the internals of the document and generally have a focused view. The last call allows a wider range of input and to gauge the impact the proposal may have in other areas. It is not about weighing the latest opinions more. The author/shepard can always post an explanation. The participants in the WG should be aware that there will be an IETF-wide last call. You cannot blame the process if they choose to remain silent instead of taking part in the last call. Note that letter-writing campaigns in a last call have been proven to be ineffective. Regards, -sm ___ IETF mailing list IETF@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Qualitative Analysis of IETF and IESG trends (Re: Measuring IETF and IESG trends)
Hi all, I am concerned by the following trends: * Number of outstanding Discusses is growing. (Thanks to Jari's data) * The extent of text changes as part of Discuss Resolution is increasing (I have only anecdotal evidence on this; perhaps others have statistics). * In some cases, members of the IESG are pretty much writing core parts of documents (or worse yet make authors iterate until the text is satisfactory), overruling WG consensus, based on personal opinions or citing solicited reviews. There are a number of derivative trends as well: * ADs who cannot convince working groups seem to use the threat of DISCUSS to get their way. * I would have thought document shepherds represent and defend working group consensus and shepherding ADs defend the completeness, accuracy, relevance of the drafts and defend their assessment of the IETF consensus. But, increasingly, it seems to fall on an AD holding a DISCUSS position, and authors to discuss and agree on text which becomes finalized without any other review. (Note: I am a guilty party in some of these cases, as document author and document shepherd. In all cases, I seem to have looked for an expedient way out rather than find a solution to what seems to be problematic). * One of the new trends seems to be to use DISCUSS to include applicability statements in current drafts to avoid potential DISCUSS positions on future drafts. That or some variation of that is status quo. It should not be that way. I would like to see a better definition of the roles of the WG, authors of a document, document shepherds, shepherding ADs and other ADs in our standards process. I would like to hear others' opinions (I was going to put together a draft with some ideas on how we might define these roles, but I want to hear others' thoughts before I do that) on this topic. thanks, Lakshminath On 6/25/2008 4:37 AM, Jari Arkko wrote: deleted text That being said, it is beneficial to understand what is happening and what changes are occurring in the process. Or understand where improvements might have a negligible vs. visible impact. One of the things that the IESG has been concerned about recently is that the number of outstanding Discusses is growing. We talked about this in our May retreat and identified some actions to help reduce this problem. For instance, better tool support so that the ADs would better see the different things that are waiting for their action, getting the document shepherds better involved in the resolution process, informing authors how they should respond to Discusses, using RFC Editor notes to make small changes when the authors are slow in producing a new version, better checks of documents before they come to telechats (e.g., to ensure that formal syntax in a document is free of errors), etc. These would be in addition to the usual things we'd do, like debate whether the Discuss was within the Discuss criteria, whether the issue is real, try to ensure that the AD and the authors are being responsive over e-mail, etc. Another interesting area to think about is the time that our processes take. For instance, documents that go through me take on the average five months from publication request to approval. But there is a lot of variation. This time includes everything from AD review, IETF Last Call to IESG review and waiting for document revisions, etc. One interesting piece of information is that documents that require no revision during this process are very rare, but they go through the system about five times as fast. If we look at the (unreliable) substates, they appear to indicate that the IESG processing time is divided roughly 1:2:1 for waiting on AD, authors, and mandatory process waits like last call periods. I am working to extend the analysis a little bit further by including individual draft and WG document stages. You see some of the results of this in the third URL above, but I'd like to understand what fraction of the overall draft-smith-00 to RFC time is taken by the different stages for all IETF work, and how the stages have developed over time. Comments and suggestions on what would be useful to measure are welcome. Jari ___ IETF mailing list IETF@ietf.org https://www.ietf.org/mailman/listinfo/ietf ___ IETF mailing list IETF@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Qualitative Analysis of IETF and IESG trends (Re: Measuring IETF and IESG trends)
On 6/25/08 11:44 AM, Lakshminath Dondeti [EMAIL PROTECTED] wrote: I would like to hear others' opinions (I was going to put together a draft with some ideas on how we might define these roles, but I want to hear others' thoughts before I do that) on this topic. I think your points are valid, but I'm not sure what the effect would be if you controlled for quality coming out of the working groups. That is to say, I think that occasionally working groups are coming to consensus on bad documents or bad ideas, and that the incidence of that is increasing. If that's true it once again raises the very familiar question of picking up quality problems earlier in the process. Actually, that latter question applies regardless. Melinda ___ IETF mailing list IETF@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Qualitative Analysis of IETF and IESG trends (Re: Measuring IETF and IESG trends)
Lakshminath, Better understanding of the type of behaviors in this space would certainly be useful. And I don't want to disagree with your assessment of the behaviors; many of them sound like something that appears in practice. In particular, the shepherds are far less involved in the Discuss resolutions than the authors. And we do not involve the WGs as much as we should. I think writing guidelines on what the role of the various persons in the process is would be very useful. And obviously we should start by better following of the existing documents, like the Proto Shepherd RFC or the Discuss criteria document. However, with regards to blocking consensus of a WG, please remember that the WG is not necessarily the only place where consensus is tested. I recently had a case which had significant IETF Last Call discussion. I held a Discuss to ensure that the (fairly clear, IMO) conclusion from the discussion would be taken into the document. It did eventually, but only after significant back-and-forth with the authors. Overriding the original WG consensus? Yes. Right thing to do? I think so, not only was it right technically but it was something backed up by the Last Call. Did we get the details right, did the text go too far or did it fall short? I don't know, its a judgment call. The end result was somewhere between the LC guidance and authors' opinions. Painful for the WG? Sure. On text that comes from the IESG: this is more common in recent years than it was before. I am one of the ADs who tends to do that, both for the documents that I sponsor and for resolving my Discusses. But I would rather not do it. But I often end up doing it if there is no progress otherwise; I want to get my sponsored documents approved and I want to reduce the list of my outstanding Discusses. If I can help my authors by proposing text, I will do it. But I would really like to see the document shepherds in active role here. Or at least the authors. The general guidance for authors whose document gets a Discuss is to first confirm whether the raised issue is a real one or not. If it is not, ask the Discuss to be cleared. Fight if needed! If it is real, work with your shepherd and WG to develop a proposal to fix the problem. Mail the proposal to the Discussing ADs in a timely manner. Address explicitly all components raised in a Discuss, either by explaining how they are not issues or providing a solution to resolve the issue. Jari ___ IETF mailing list IETF@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Qualitative Analysis of IETF and IESG trends (Re: Measuring IETF and IESG trends)
On 2008-06-26 06:30, Jari Arkko wrote: Lakshminath, Better understanding of the type of behaviors in this space would certainly be useful. And I don't want to disagree with your assessment of the behaviors; many of them sound like something that appears in practice. In particular, the shepherds are far less involved in the Discuss resolutions than the authors. And we do not involve the WGs as much as we should. I think writing guidelines on what the role of the various persons in the process is would be very useful. And obviously we should start by better following of the existing documents, like the Proto Shepherd RFC or the Discuss criteria document. However, with regards to blocking consensus of a WG, please remember that the WG is not necessarily the only place where consensus is tested. I recently had a case which had significant IETF Last Call discussion. I held a Discuss to ensure that the (fairly clear, IMO) conclusion from the discussion would be taken into the document. It did eventually, but only after significant back-and-forth with the authors. Overriding the original WG consensus? Yes. Right thing to do? I think so, not only was it right technically but it was something backed up by the Last Call. Did we get the details right, did the text go too far or did it fall short? I don't know, its a judgment call. The end result was somewhere between the LC guidance and authors' opinions. Painful for the WG? Sure. On text that comes from the IESG: this is more common in recent years than it was before. I am one of the ADs who tends to do that, both for the documents that I sponsor and for resolving my Discusses. But I would rather not do it. But I often end up doing it if there is no progress otherwise; I want to get my sponsored documents approved and I want to reduce the list of my outstanding Discusses. If I can help my authors by proposing text, I will do it. But I would really like to see the document shepherds in active role here. Or at least the authors. The general guidance for authors whose document gets a Discuss is to first confirm whether the raised issue is a real one or not. If it is not, ask the Discuss to be cleared. Fight if needed! If it is real, work with your shepherd and WG to develop a proposal to fix the problem. Mail the proposal to the Discussing ADs in a timely manner. Address explicitly all components raised in a Discuss, either by explaining how they are not issues or providing a solution to resolve the issue. Our fundamental collective job is defined in RFC 3935: The mission of the IETF is to produce high quality, relevant technical and engineering documents that influence the way people design, use, and manage the Internet in such a way as to make the Internet work better. That means that it is *not* our collective job to ensure that a WG consensus survives critical review by the IETF as a whole and by the IESG, if there's reason to believe that the IETF as a whole doesn't agree with the WG consensus. And it's clearly the IESG's job to ensure that the critical review and final consensus (or lack of consensus) occur. That, IMHO, is the proper role of a DISCUSS and the proper reason for delays in document approval. And if we see fluctuation in these delays, and fluctuation in the amount of active intervention by the ADs, it does not follow that the IESG is to blame. Maybe there are external factors, maybe there are WGs that are forgetting the IETF's mission, maybe our technology is getting harder and more complex. So I'm very dubious about using either quantitative *or* qualitative observations to point the finger at the IESG, or at process issues in general, without digging much deeper. Of course, the IESG needs to pay attention to delays, so Jari's data (like the earlier data that Bill Fenner used to produce) are very valuable. And of course, individual ADs have to think carefully whether a given issue is or is not worthy of a DISCUSS, and sometimes they get it wrong. But that will always be true, however we tune the process and procedures. Brian ___ IETF mailing list IETF@ietf.org https://www.ietf.org/mailman/listinfo/ietf
Re: Qualitative Analysis of IETF and IESG trends (Re: Measuring IETF and IESG trends)
--On Thursday, 26 June, 2008 09:41 +1200 Brian E Carpenter [EMAIL PROTECTED] wrote: ... And of course, individual ADs have to think carefully whether a given issue is or is not worthy of a DISCUSS, and sometimes they get it wrong. But that will always be true, however we tune the process and procedures. Brian, While I agree with this, I also believe that there have to be effective safeguards against bad judgments prevailing for too long. And I believe that those have largely slipped away from us... unless we believe that making changes, unvalidated by WG or mailing list consensus, simply to get a DISCUSS removed is the right way to get to high quality, relevant technical and engineering documents john ___ IETF mailing list IETF@ietf.org https://www.ietf.org/mailman/listinfo/ietf
