Re: New root cause problems?
My historical data on document progress in the I-D tracker, collected since the beginning of 2003, is available at https://rtg.ietf.org/phpmyadmin username ietf password ietf; pick database trackerdata and table dochistory. (Ignore the permission denied error; go straight for the table list on the left or the SQL tab on the top of the right pane) A starter query to somewhat understand the table relationships is SELECT docname, changedate, statename, substatename FROM dochistory, docs LEFT JOIN states ON stateid = newstate LEFT JOIN substates ON substateid = newsubstate WHERE docs.docid = dochistory.docid ORDER BY dochistory.docid, histid; (This will return some NULL transitions where something other than the substate or substate transitioned, like the version number, etc.) I don't have any great ideas on how to do the analysis, but at least I have some data to supply. Bill ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: New root cause problems?
In addition to choosing the right way to look at the distribution of total action times, I strongly recommend breaking down the transactions into component parts and looking at the details. The exchange I had with Sam Hartman was a good example. On 5/23. Sam wrote: > I'd think two months would be doing good for IESG processing. That > includes AD review, IETf last call, telechat delays and a bit of slop > for interaction with the authors. However that's time to travel > through the queue, not actually time spent on that document. > > Because of the delays involved with telechats and last call, getting a > WG document done in less than a month of wall time or an individual > document done in less than 1.5 months is very unlikely. What catches my attention -- and brings shudders as I remember my time as an AD -- is that he paints a picture that looks like T[IESG] = T[AD] + T[last call] + T[telechat delay] + T[authors] I see two problems with this formulation: (1) T[AD] is in fact composed of several separate things, and the differences between them are almost certainly critical in understanding process delays. In particular, AD time is spent in several different ways: (a) Initial document review (done by all ADs) (b) AD followups (done by the sheparding AD) (c) AD rereview and discussion to clear discussses (done by all ADs) There are timeouts associated with (a) - once a document is on the IESG's agenda, ADs are required to have their review done by the next telechat. They can get an extension, but only once. This sohuld therefore end up being similar to telechat timinig. There are, however, no timeouts associated with (b) and (c). And more to the point, the pressures on sheparding AD are quite different than those on the other ADs to act in a timely fashion. My experience has been that the distribution for (b) is bimodal (or trimodal if you count the case where the document just passes without any discusses). In some cases the sheparding AD only has to pass on the discuss comments to the authors, done. In others, however, the AD has to spend a bunch of time in effect figuring out what amounts to a compromise. This can take a lot of time. In my experience (c) is the truly problematic case. In my experience it is both highly variable and the average time taken seems to be much longer than it should be. Additionally, as sheparding AD I often found it was only possible to clear discusses by explicitly bringing the document up on a subsequent telechat, so there's a periodicity here as well. I have in the past advocated putting a timeout on (c), and I continue to think this is a really good idea. (2) When I was an AD I went to a good deal of trouble to overlap the various phases of the process whenever possible in order to save time. Nothing prevents ADs from entering comments early. Nothing prevents documents from being revised during last call. I don't think I was terribly successful in this, but much of my time as AD was spent without the tools we have now. I therefore think the additive nature of your formula has to be characterized as a worst case, and it needs to be noted that substantial gains can be had by overlapping terms. Now each of these terms has quite different characteristics. The last call and telechat delays add up to a few weeks, to the overall time is at least a couple of months. But those terms probably have moderately low variance. On the other hand, the T[AD] and T[authors] terms have very low minimums but extremely high variance, and it's within those terms that the real issues emerge. Agreed. I suspect if you break those terms down even further, interesting and useful dynamics will emerge. Exactly so. Breaking them down further is IMO essential to getting anything useful out of the analysis. Ned ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: New root cause problems?
> From: Brian E Carpenter <[EMAIL PROTECTED]> > Dave Crocker wrote: > > use the median, rather than the mean. > > I'm actually more interested in the mode, I think, but first, > some actual data would be good. Agreed. And I would add that other than mode it matters what is actually being measured -- in the case of 0 documents for IESG review in 8 years it makes a wee bit of difference to some measures if the statistic is presented as years/document or documents/year... On the other hand, the mode isn't very informative if the sample size is small. ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: New root cause problems?
Dave Crocker wrote: o be slightly provocative, if the average times are forced upwards by a long tail of WGs/drafts/RFCs that take extremely long times to get done due to one-of-a-kind reasons, it would seem fair to remove thoses cases from consideration. use the median, rather than the mean. I'm actually more interested in the mode, I think, but first, some actual data would be good. Brian ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: New root cause problems?
In addition to choosing the right way to look at the distribution of total action times, I strongly recommend breaking down the transactions into component parts and looking at the details. The exchange I had with Sam Hartman was a good example. On 5/23. Sam wrote: I'd think two months would be doing good for IESG processing. That includes AD review, IETf last call, telechat delays and a bit of slop for interaction with the authors. However that's time to travel through the queue, not actually time spent on that document. Because of the delays involved with telechats and last call, getting a WG document done in less than a month of wall time or an individual document done in less than 1.5 months is very unlikely. What catches my attention -- and brings shudders as I remember my time as an AD -- is that he paints a picture that looks like T[IESG] = T[AD] + T[last call] + T[telechat delay] + T[authors] Now each of these terms has quite different characteristics. The last call and telechat delays add up to a few weeks, to the overall time is at least a couple of months. But those terms probably have moderately low variance. On the other hand, the T[AD] and T[authors] terms have very low minimums but extremely high variance, and it's within those terms that the real issues emerge. I suspect if you break those terms down even further, interesting and useful dynamics will emerge. I'm certainly not picking on Sam but merely using his responses as a good clue for untangling the separate effects. In a subsequent note, in regard to T[AD], Sam said It depends a lot on a document. I can often do a document in in two hours if it is reasonably short and I understand the technology and the document quality is good. I have one document languishing somewhat in my queue because I need to block out an entire day for it and finding a full day to work on one document is hard. Keep in mind that AD review can easily have round trips with the authors. Also, as you are well aware, finding the time among all the other things is difficult. To me, this suggests it would be useful to try to identify the issues that make it easy and quick to process some documents, but harder and longer to process other documents. There may be some issues related to just the quantity of work, but I suspect the bigger issues are qualitative and we may learn something if we probe more deeply and not just study the distributions of the overall processing times. Steve Dave Crocker wrote: o be slightly provocative, if the average times are forced upwards by a long tail of WGs/drafts/RFCs that take extremely long times to get done due to one-of-a-kind reasons, it would seem fair to remove thoses cases from consideration. use the median, rather than the mean. d/ --- Dave Crocker Brandenburg InternetWorking +1.408.246.8253 dcrocker a t ... WE'VE MOVED to: www.bbiw.net ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: New root cause problems?
> o be slightly provocative, if the average > times are forced upwards by a long tail of WGs/drafts/RFCs that > take extremely long times to get done due to one-of-a-kind reasons, > it would seem fair to remove thoses cases from consideration. use the median, rather than the mean. d/ --- Dave Crocker Brandenburg InternetWorking +1.408.246.8253 dcrocker a t ... WE'VE MOVED to: www.bbiw.net ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: New root cause problems?
Bruce Lilly wrote: On Tue May 24 2005 09:18, Hollenbeck, Scott wrote: On Tue May 24 2005 08:35, Margaret Wasserman wrote: At 5:57 PM -0400 5/10/05, Bruce Lilly wrote: OK, I'll bite -- where are the statistics (I know of one WG that has been active for more than 8 years and has set to produce a single document for IESG review; that's got to skew the statistics a bit)? I can't talk to whatever went on in USEFOR prior to 2004, but I have appointed two new co-chairs to help the group evaluate and meet it's goals. Something (either document production or a working group action) should happen within the next few months. I should clarify a few points: 1. sorry for the typo in the original "set" should be "yet" 2. the intent was not to cast aspersions on the cognizant ADs (past or present) or WG Chairs (past or present), or to discuss possible causes for the situation -- indeed, obviously I didn't even name the WG until asked -- but merely to observe that the history of at least one WG seemed to be out of whack with what was asserted was shown by some statistics. I'd like statistics too. To be slightly provocative, if the average times are forced upwards by a long tail of WGs/drafts/RFCs that take extremely long times to get done due to one-of-a-kind reasons, it would seem fair to remove thoses cases from consideration. We should probably concentrate on causes of delay that affect the mode of the distribution, not the tail. Brian ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
RE: New root cause problems?
On Tue May 24 2005 09:18, Hollenbeck, Scott wrote: > > On Tue May 24 2005 08:35, Margaret Wasserman wrote: > > > At 5:57 PM -0400 5/10/05, Bruce Lilly wrote: > > > >OK, I'll bite -- where are the statistics (I know of one > > WG that has been > > > >active for more than 8 years and has set to produce a > > single document for > > > >IESG review; that's got to skew the statistics a bit)? > I can't talk to whatever went on in USEFOR prior to 2004, but I have > appointed two new co-chairs to help the group evaluate and meet it's > goals. Something (either document production or a working group action) > should happen within the next few months. I should clarify a few points: 1. sorry for the typo in the original "set" should be "yet" 2. the intent was not to cast aspersions on the cognizant ADs (past or present) or WG Chairs (past or present), or to discuss possible causes for the situation -- indeed, obviously I didn't even name the WG until asked -- but merely to observe that the history of at least one WG seemed to be out of whack with what was asserted was shown by some statistics. ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
RE: New root cause problems?
> -Original Message- > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On > Behalf Of Bruce Lilly > Sent: Tuesday, May 24, 2005 8:59 AM > To: ietf@ietf.org > Cc: Margaret Wasserman > Subject: Re: New root cause problems? > > On Tue May 24 2005 08:35, Margaret Wasserman wrote: > > > > > > > > Hi Bruce, > > > > At 5:57 PM -0400 5/10/05, Bruce Lilly wrote: > > >OK, I'll bite -- where are the statistics (I know of one > WG that has been > > >active for more than 8 years and has set to produce a > single document for > > >IESG review; that's got to skew the statistics a bit)? > > > > What WG is that? > > http://www.ietf.org/html.charters/usefor-charter.html > > N.B. the "charter rewrite/update is underway" has been there > for years -- it is NOT related to the missed Apr 05 milestone. > > I'm still interested in seeing those statistics... I can't talk to whatever went on in USEFOR prior to 2004, but I have appointed two new co-chairs to help the group evaluate and meet it's goals. Something (either document production or a working group action) should happen within the next few months. -Scott- ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: New root cause problems?
On Tue May 24 2005 08:35, Margaret Wasserman wrote: > > > > Hi Bruce, > > At 5:57 PM -0400 5/10/05, Bruce Lilly wrote: > >OK, I'll bite -- where are the statistics (I know of one WG that has been > >active for more than 8 years and has set to produce a single document for > >IESG review; that's got to skew the statistics a bit)? > > What WG is that? http://www.ietf.org/html.charters/usefor-charter.html N.B. the "charter rewrite/update is underway" has been there for years -- it is NOT related to the missed Apr 05 milestone. I'm still interested in seeing those statistics... ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: New root cause problems?
Hi Bruce, At 5:57 PM -0400 5/10/05, Bruce Lilly wrote: OK, I'll bite -- where are the statistics (I know of one WG that has been active for more than 8 years and has set to produce a single document for IESG review; that's got to skew the statistics a bit)? What WG is that? Margaret ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: New root cause problems?
Last year, Tim Simcoe, now on the faculty at the Rotman School of Management at the University of Toronto, completed a study entitled: "Delays and de Jure Standards: What Caused the Slowdown in Internet Standard Setting?" For those interested, here are pointers to the work (which is continuing): http://groups.haas.berkeley.edu/bpp/phd/simcoe/profile.htm http://www.rotman.utoronto.ca/strategy/research/working%20papers/Simcoe%20-%20Delays.pdf ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: New root cause problems?
Sorry for taking so long to follow up here: From: "Margaret Wasserman" <[EMAIL PROTECTED]> I have one new "root cause" issue that I don't believe was included in the original Problem Statement: It takes too long to publish an RFC after final approval. It currently takes several months for an RFC to be published after it is approved by the IESG. At the SIPPING-OMA ad hoc meeting at Minneapolis, we noted that the P-Alerting-Mode "private header" in draft-allen-sipping-poc-p-headers-01 was really generally applicable (not limited to OMA environments), and that it would be really nice to take the text out of the existing private headers draft and publish it as a standards-track RFC. Then we said, "OMA needs a stable reference for this Really Soon, but maybe we could fast-track a short specification with one header based on existing text to approval as a standards-track RFC by Paris." And then someone said, "but then it will take another six months to come out as an RFC, anyway." And no one argued that it wouldn't. And, IIRC, that wasn't the only time taking a long time to publish an approved document came up during the same ad hoc meeting. Spencer ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: New root cause problems?
On 5/11/05, Margaret Wasserman <[EMAIL PROTECTED]> wrote: > Also, if there is the equivalent of the I-D tracker for the RFC > Editor Queue (where correspondence and major state transitions for a > document are captured and can be seen later), I am not sure where to > find it. The RFC editor queue is a text document that only seems to > capture the current state of the document (EDIT, AUTH48, etc.). I've been collecting a daily snapshot of the RFC Editor's queue since mid-2002; http://rtg.ietf.org/~fenner/iesg/analyze-rfcq.txt (updated daily) has ananalysis of every single document and state it's gone through since then. (That's where Thomas's example came from.) It doesn't have correspondence, and you have to download the whole file and search it for the document you're looking for, but at least the state change data is there. Bill ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: Re: New root cause problems?
Margaret, > At 10:50 AM +0200 5/11/05, Brian E Carpenter wrote: > >But that gives very limited insight into what is holding it up, except > >for a few cases. If it's in EDIT state you get no useful information about > >issues and progress, for example. > > Also, if there is the equivalent of the I-D tracker for the RFC > Editor Queue (where correspondence and major state transitions for a > document are captured and can be seen later), I am not sure where to > find it. The RFC editor queue is a text document that only seems to > capture the current state of the document (EDIT, AUTH48, etc.). I'm personally not to sure about the accuracy of some of those states. I found a couple that I am personally involved in that are still listed as in the AUTH state, but where all of the authors have signed off on the document & the RFC Editor has confirmed that we have signed off on it. ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: Re: New root cause problems?
Margaret, This is a huge problem, IMO. If you couple your point with James sub-point about drafts expiring in the RFC editors' queue, you really do have a mess. I often have to discuss with other SDOs on IETF drafts; often they need an RFC number for their own processes. It might be enough to say to some IETFers to look at the RFC Editor's Queue, but it is not a reasonable way to manage things inter-SDO. I am a co-author of a draft that was initially requested in August 2003, by 3GPP for Release 5, but it had not been accepted yet as a WG item, so it was not included in 3GPP Release 5. The authors worked hard to get the document done, and the current version is dated August 12, 2004. It is currently listed in the RFC Editors Queue with the note "Fast Track requested." It is in the AUTH = Awaiting Author Action state, but all of the authors signed off on it on March 23rd. I'll admit that the authors made a bit of a mess during AUTH48 because we had quite many changes requestes, but that should only account for about 1 month of delay. I'm actually catching a lot of grief about this via private mail because it looks like incompetence to folks outside of the IETF. I really think these type of things really strains any credibility that the IETF has. John Loughney > From: Margaret Wasserman <[EMAIL PROTECTED]> > Date: 2005/05/10 Tue PM 04:36:51 EEST > To: Brian E Carpenter <[EMAIL PROTECTED]>, ietf@ietf.org > Subject: Re: New root cause problems? > > > I have one new "root cause" issue that I don't believe was included > in the original Problem Statement: > > It takes too long to publish an RFC after final approval. > > It currently takes several months for an RFC to be published after it > is approved by the IESG. > > This results in several problems: > > - RFCs come out much later than they should, contributing greatly to > the perception that the iETF is slow. The time to move from approved > I-D to RFC is often a significant percentage (perhaps averaging 20% > of more?) of the time required to develop and publish a specification. > - Stable references are not available until months after a > specification is fully approved, resulting in ambiguity about the > status of a document and encouraging the use of I-D names as > references, thus blurring the distinction between WG I-Ds and > approved specifications. > - Too many changes are made to documents after WG and IESG approval > (copy editing, changes to reflect updated thinking/terminology, > etc.). These changes are often not reviewed or approved by the WG or > the community. > - Some RFCs are already out-of-date by the time they are published. > The document then contains the RFC publication date, which may result > in a mistaken impression about when the IETF held a specific view or > encouraged a particular practice. > > I believe that the IETF should consider modifications to our document > handling process to reduce or eliminate the delay in publishing > approved specifications. > > Margaret > > > At 2:36 PM +0200 5/10/05, Brian E Carpenter wrote: > >Having finally read the list traffic up to date, I have a question. > > > >Can anybody identify a *new* "root cause problem" at the same level > >of abstraction as those identified in RFC 3774? Or is it the case > >that (at that level of abstraction) we have only been re-discussing > >the RFC 3774 problem set? > > > >(Please try to be succinct... or change the subject header if you want > >to change the subject.) > > > >Thanks > >Brian > > > > > > > > > >___ > >Ietf mailing list > >Ietf@ietf.org > >https://www1.ietf.org/mailman/listinfo/ietf > > > ___ > Ietf mailing list > Ietf@ietf.org > https://www1.ietf.org/mailman/listinfo/ietf > ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: New root cause problems?
On Wed, 2005-05-11 at 10:34, Bill Fenner wrote: > You may be interested in > http://rtg.ietf.org/~fenner/iesg/rfc-deps.pdf for a visualization that > I've been fine-tuning for a couple of years; it's auto-generated > daily. I bow to your superior graphviz skills. We need a link to this from the RFC-Editor pages. It might make sense to show the larger hairballs in the plenary as something of a cautionary note. - Bill ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: New root cause problems?
On 5/11/05, Dave Crocker <[EMAIL PROTECTED]> wrote: > > You may be interested in > > > > http://rtg.ietf.org/~fenner/iesg/rfc-deps.pdf for a visualization that > > I've been fine-tuning for a couple of years; it's auto-generated > > daily. > > wow. neat! > > what is the difference between the green and red circles? > > what is the difference between the black and gold lines? Sorry, the key is on the previous page, http://rtg.ietf.org/~fenner/iesg/ - oval colors map to RFC-Editor state as below. Arrows indicate dependency. Box with orange arrows inside indicates a cycle; large cycles are hard to detect without looking for them. color state green EDIT red REF blueIANA black other dotted blackNot in queue (The orange arrows were the original reason for this; I thought the big glorp of ccamp documents was stuck because it was a 7-hop interdependency so each document had what appeared to be unresolved dependencies) Bill ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: New root cause problems?
> You may be interested in > > http://rtg.ietf.org/~fenner/iesg/rfc-deps.pdf for a visualization that > I've been fine-tuning for a couple of years; it's auto-generated > daily. wow. neat! what is the difference between the green and red circles? what is the difference between the black and gold lines? d/ --- Dave Crocker Brandenburg InternetWorking +1.408.246.8253 dcrocker a t ... WE'VE MOVED to: www.bbiw.net ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: New root cause problems?
Bill et al, You may be interested in http://rtg.ietf.org/~fenner/iesg/rfc-deps.pdf for a visualization that I've been fine-tuning for a couple of years; it's auto-generated daily. Bill ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: New root cause problems?
At 10:50 AM +0200 5/11/05, Brian E Carpenter wrote: But that gives very limited insight into what is holding it up, except for a few cases. If it's in EDIT state you get no useful information about issues and progress, for example. Also, if there is the equivalent of the I-D tracker for the RFC Editor Queue (where correspondence and major state transitions for a document are captured and can be seen later), I am not sure where to find it. The RFC editor queue is a text document that only seems to capture the current state of the document (EDIT, AUTH48, etc.). Margaret ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: New root cause problems?
On Wed, 2005-05-11 at 06:35, Bill Sommerfeld wrote: > When I went looking at the queue a month or so ago it required work to > distinguish between a REF to a document already in the queue vs. a REF > to a document not yet in the editor queue. .. and, having just done this a second time, I wrote some quick and dirty code: http://blogs.sun.com/roller/resources/sommerfeld/queue-to-dot using awk, sed, and the "graphviz" toolkit's "dot" program. feed it a slightly cleaned up copy of the text inside: http://www.rfc-editor.org/queue.html and it produces: http://blogs.sun.com/roller/resources/sommerfeld/rfc-editor-queue-20050511.png (This is a 18559-pixel wide .png; mozilla by default scales it to the window width which produces something resembling random dot noise but if you click on the noise to zoom in you get a horizontally scrollable image I am a neophyte user of graphviz so I'm sure I could give it some hints to produce a better layout and include more useful info but I figured this would be of interest sooner rather than later..) - Bill ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: New root cause problems?
On Wed, 2005-05-11 at 04:50, Brian E Carpenter wrote: > > A delay of that length is generally due to dependencies -- normative > > references to other documents that are held up. > > > > When a document is in the RFC Editor queue, you can query its state via > > their web site. > > But that gives very limited insight into what is holding it up, except > for a few cases. If it's in EDIT state you get no useful information about > issues and progress, for example. The dependency graph isn't all that approachable, either. When I went looking at the queue a month or so ago it required work to distinguish between a REF to a document already in the queue vs. a REF to a document not yet in the editor queue. - Bill ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: New root cause problems?
below... Steven M. Bellovin wrote: In message <[EMAIL PROTECTED]>, "Eric A. Hall" writes: On 5/10/2005 12:45 PM, Thomas Narten wrote: One example (and I'm just using it because it was it comes to mind, and one that I think is symptomatic of the broader problem): October 15, 2004: IESG approves 4-document set. Within one week: authors send xml source to RFC editor March 10, 2005: IESG requests expedited processing (target date: March 31) March 29, 2005: RFCs published Total time between IESG approval and publication, 5 1/2 months. That was expedited. Better example is iSCSI. Draft-20 was approved Feb 2003 [http://www.ietf.org/IESG/Announcements/draft-ietf-ips-iscsi.ann] but published as RFC3720 in April 2004, for a lag time of 14 months. I have no knowledge of this process and maybe there were a lot of changes needed or something, but for a whole year there were vendors releasing products marketed as conformant with "draft 20" A delay of that length is generally due to dependencies -- normative references to other documents that are held up. When a document is in the RFC Editor queue, you can query its state via their web site. But that gives very limited insight into what is holding it up, except for a few cases. If it's in EDIT state you get no useful information about issues and progress, for example. Brian ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: New root cause problems?
Argh, I should have known better than to hope that gmail wouldn't munch the tables. The category/status table is: mysql> select all_id.status, substring(rfcq.category, 1, 40) as "Category", count(all_id.status) as "Num" from rfcq, all_id where all_id.docname = rfcq.docname group by all_id.status, rfcq.category; +---+--+-+ | status| Category | Num | +---+--+-+ | active| INDEPENDENT SUBMISSIONS UNDER RFC EDITOR | 7 | | expired | INDEPENDENT SUBMISSIONS UNDER RFC EDITOR | 18 | | rfc | IAB DOCUMENTS (by date received) | 1 | | rfc | NON-WORKING GROUP INFORMATIONAL/EXPERIME | 25 | | rfc | NON-WORKING GROUP STANDARDS TRACK (by da | 10 | | rfc | WORKING GROUP INFORMATIONAL/EXPERIMENTAL | 24 | | rfc | WORKING GROUP STANDARDS TRACK (by date r | 56 | | withdrawn | NON-WORKING GROUP INFORMATIONAL/EXPERIME | 1 | | iesg | IAB DOCUMENTS (by date received) | 1 | | iesg | INDEPENDENT SUBMISSIONS UNDER RFC EDITOR | 12 | | iesg | NON-WORKING GROUP INFORMATIONAL/EXPERIME | 21 | | iesg | NON-WORKING GROUP STANDARDS TRACK (by da | 2 | | iesg | WORKING GROUP INFORMATIONAL/EXPERIMENTAL | 47 | | iesg | WORKING GROUP STANDARDS TRACK (by date r | 61 | +---+--+-+ 14 rows in set (0.05 sec) and the date table is: mysql> select min(all_id.lastup) as "Updated", substring(rfcq.category, 1, 50) as "Category" from rfcq, all_id where all_id.docname = rfcq.docname and all_id.status != 'rfc' group by rfcq.category; +++ | Updated| Category | +++ | 2004-10-22 | IAB DOCUMENTS (by date received) | | 2003-07-01 | INDEPENDENT SUBMISSIONS UNDER RFC EDITOR REVIEW ( | | 2001-07-24 | NON-WORKING GROUP INFORMATIONAL/EXPERIMENTAL/BCP ( | | 2004-10-01 | NON-WORKING GROUP STANDARDS TRACK (by date receive | | 2002-10-11 | WORKING GROUP INFORMATIONAL/EXPERIMENTAL/BCP (by d | | 2002-09-11 | WORKING GROUP STANDARDS TRACK (by date received) | +++ 6 rows in set (0.04 sec) Bill ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: New root cause problems?
On 5/10/05, James M. Polk <[EMAIL PROTECTED]> wrote: > Adding to this - and I'm not sure this is the kind of thing you were > looking for, but it adds to the overall problem - is that IDs timeout after > 6 months (which is fine), but that includes IDs that are in the RFC-Editor > queue process too. I believe it only includes IDs that are independent submissions to the RFC Editor and have not yet been sent to the IESG for its conflict check. mysql> select all_id.status, rfcq.category, count(all_id.status) from rfcq, all_id where all_id.docname = rfcq.docname group by all_id.status, rfcq.category; +---+-+--+ | status| category | count(all_id.status) | +---+-+--+ | active| INDEPENDENT SUBMISSIONS UNDER RFC EDITOR REVIEW (by date received) |7 | | expired | INDEPENDENT SUBMISSIONS UNDER RFC EDITOR REVIEW (by date received) | 18 | | rfc | IAB DOCUMENTS (by date received) |1 | | rfc | NON-WORKING GROUP INFORMATIONAL/EXPERIMENTAL/BCP (by date received) | 25 | | rfc | NON-WORKING GROUP STANDARDS TRACK (by date received) | 10 | | rfc | WORKING GROUP INFORMATIONAL/EXPERIMENTAL/BCP (by date received) | 24 | | rfc | WORKING GROUP STANDARDS TRACK (by date received) | 56 | | withdrawn | NON-WORKING GROUP INFORMATIONAL/EXPERIMENTAL/BCP (by date received) |1 | | iesg | IAB DOCUMENTS (by date received) |1 | | iesg | INDEPENDENT SUBMISSIONS UNDER RFC EDITOR REVIEW (by date received) | 12 | | iesg | NON-WORKING GROUP INFORMATIONAL/EXPERIMENTAL/BCP (by date received) | 21 | | iesg | NON-WORKING GROUP STANDARDS TRACK (by date received) |2 | | iesg | WORKING GROUP INFORMATIONAL/EXPERIMENTAL/BCP (by date received) | 47 | | iesg | WORKING GROUP STANDARDS TRACK (by date received) | 61 | +---+-+--+ 14 rows in set (0.04 sec) Forgive the too-wide presentation; but as you can see, all 18 expired documents are in the Independent Submissions category. (The status "rfc" documents are because the rfcq database doesn't age out old entries.) The oldest document in each category: mysql> select min(all_id.lastup) as "Updated", rfcq.category from rfcq, all_id where all_id.docname = rfcq.docname and all_id.status != 'rfc' group by rfcq.category; ++-+ | Updated| category | ++-+ | 2004-10-22 | IAB DOCUMENTS (by date received) | | 2003-07-01 | INDEPENDENT SUBMISSIONS UNDER RFC EDITOR REVIEW (by date received) | | 2001-07-24 | NON-WORKING GROUP INFORMATIONAL/EXPERIMENTAL/BCP (by date received) | | 2004-10-01 | NON-WORKING GROUP STANDARDS TRACK (by date received) | | 2002-10-11 | WORKING GROUP INFORMATIONAL/EXPERIMENTAL/BCP (by date received) | | 2002-09-11 | WORKING GROUP STANDARDS TRACK (by date received) | ++-+ 6 rows in set (0.03 sec) so clearly, given that there are I-Ds in the queue that were last updated in 2001 and 2002, the 6 months expiration is not a universal concern for documents in the RFC Editor's queue. [These databases are available from https://rtg.ietf.org/phpmyadmin/ ; login ietf password ietf] Bill ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: New root cause problems?
> Date: 2005-05-10 13:01 > From: Dave Crocker <[EMAIL PROTECTED]> > Let's say that it averages 4 months to go from IESG approval to RFC > publication. (I'm choosing that number because it is big enough to indicate > a > problem, but small enough not to insult the RFC Editor.) > > Using your estimate of 20% for this step, that means that working groups > average 16 months to do their part of the work, from start to finish. > > However that does not match either general impression or that statistics that > were produced awhile back. > > If we averaged 16 months for working groups to go from start to finish, > we would not have serious and consistent complaints that the IETF is too > slow. OK, I'll bite -- where are the statistics (I know of one WG that has been active for more than 8 years and has set to produce a single document for IESG review; that's got to skew the statistics a bit)? ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: New root cause problems?
Hi Dave, My impressions are, of course, based upon my own experiences... I checked the time spent in the "In RFC Editor Queue" state for the last 10 RFCs that have been published from my I-D Tracker queue, and the details are included below. According to my numbers, these ten documents spent a combined total of 201 months in the IETF process after -00 publication, which typically approximates when the WG accepted the work item (about 20 months per document). These documents spent a combined total of 50 months in the "In RFC Editor Queue" state (an average of 5 months per document) or just about 25% of the total time from WG acceptance to document publication. Therefore, given my own experience, I will stand by my impression that about 25% of the IETF's processing time for a given document is being spent in the RFC Editor Queue. This number may not capture the full extent of the issue, as my longest-standing documents in the "In RFC editor Queue" state haven't had a chance to emerge yet. I've only been an AD for about 22 months, and I am responsible for 11 documents that have been in the RFC Editor Queue for over 6 months, including 2 that have been in the RFC Editor Queue for over a year. Please note that I do understand that not all of the time spent in the "In RFC Editor Queue" state can be attributed to the RFC Editor, any more than all of the time spent "in the IESG" can be attributed to the IESG. Some of this time is spent waiting to resolve references, waiting for author feedback on IANA issues, waiting for author response to AUTH48, etc. However, I do believe that far too much time is being spent during the "In RFC Editor Queue" phase of the process. This phase of the process is too opaque for me to identify the specific reasons why this is taking so long, but I strongly believe that we need more visibilityy into this part of the process and that we need to apply our efforts to making this period shorter. Margaret draft-carpenter-obsolete-1888: (TOTAL TIME: 8 months) In RFC Editor Q: 2004-11-02 to 2005-04-26 (5-1/2 months) -00 Published: August 2004 draft-ietf-dhc-agentopt-radius: (TOTAL TIME: 36-1/2 months) In RFC Editor Q: 2004-09-27 to 2005-02-24 (5 months) -00 Published: 2002-02-14 draft-ietf-dhc-auth-suboption: (TOTAL TIME: 33-1/2 months) In RFC Editor Q: 2004-09-21 to 2005-04-06 (6-1/2 months) -00 Published: 2002-06-23 draft-ietf-dhc-dhcpv6-opt-nisconfig: (TOTAL TIME: 32 months) In RFC Editor Q: 2004-06-02 to 2004-10-12 (4-1/2 months) -00 Published: 2002-02-17 draft-ietf-dhc-rapid-commit-opt: (TOTAL TIME: 16 months) In RFC Editor Q: 2004-11-02 to 2005-04-06 (5 months) -00 Published: 2003-11-28 draft-ietf-dhc-reclassify-options: (TOTAL TIME: 11 months) In RFC Editor Q: 2004-07-12 to 2004-12-03 (4-1/2 months) -00 Published: 2004-01-05 draft-ietf-dhc-subscriber-id: (TOTAL TIME: 20 months) In RFC Editor Q: 2004-09-16 to 2005-03-22 (6 months) -00 Published: July 2003 draft-ietf-dhc-vendor: (TOTAL TIME: 14 months) In RFC Editor Q: 2004-07-12 to 2004-11-04 (4 months) -00 Published: September 2003 draft-ietf-eap-rfc2284bis: (TOTAL TIME: 17 months) In RFC Editor Q: 2004-02-29 to 2004-06-22 (4 months) -00 Published: January 2003 draft-ietf-ipv6-deprecate-site-local: (TOTAL TIME: 13 months) In RFC Editor Q: 2004-04-16 to 2004-09-21 (5 months) -00 Published: 2003-08-16 At 10:01 AM -0700 5/10/05, Dave Crocker wrote: The time to move from approved I-D to RFC is often a significant percentage (perhaps averaging 20% of more?) of the time required to develop and publish a specification. Let's say that it averages 4 months to go from IESG approval to RFC publication. (I'm choosing that number because it is big enough to indicate a problem, but small enough not to insult the RFC Editor.) Using your estimate of 20% for this step, that means that working groups average 16 months to do their part of the work, from start to finish. However that does not match either general impression or that statistics that were produced awhile back. If we averaged 16 months for working groups to go from start to finish, we would not have serious and consistent complaints that the IETF is too slow. So the actual, incremental delay for the RFC publication process is probably no worse than 10% and I wouldn't be surprised if the real statistic were more like 5%. As much as it is worth trying to make everything better, it is difficult to see a 5 or 10% overhead to the process as being one of our strategic problems. d/ --- Dave Crocker Brandenburg InternetWorking +1.408.246.8253 dcrocker a t ... WE'VE MOVED to: www.bbiw.net ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: New root cause problems?
"Eric A. Hall" <[EMAIL PROTECTED]> writes: > On 5/10/2005 12:45 PM, Thomas Narten wrote: > > One example (and I'm just using it because it was it comes to mind, > > and one that I think is symptomatic of the broader problem): > > October 15, 2004: IESG approves 4-document set. > > Within one week: authors send xml source to RFC editor > > March 10, 2005: IESG requests expedited processing (target date: March 31) > > March 29, 2005: RFCs published > > > > Total time between IESG approval and publication, 5 1/2 months. For the record, I cited this one because as AD shepherd for this one, I am certain there were no normative dependencies, the IANA work was relatively minor (no new allocations where made), the authors didn't ask for massive changes during 48 hours, and I know there was frustration/puzzlement from the editors directed at me while we waited for the document to pop out. More generally, there are a number of reasons why documents get hung up after IESG approval. They include: 1) normative references to IDs that are not also in the queue. Unfortunately, there are too many cases of this, and IMO, the IESG (and WGs) should really not be approving such documents. Too often, once they are in the RFC queue, people forget about the normative depedencies which may themselves be stalled. As AD, I saw this _many_ times. :-( 2) IANA considerations. IANA needs to do its part before an RFC can be published. IANA delays have also stretched into multiple months at times. 3) Additional "changes" requested by the WG and or authors. I know of several cases where WGs/authors submitted extremely large sets of "editorial changes" during 48 hours. Because all those changes need to be vetted appropriately, this can set back documents a long time. E.g, the l2tpv3 base spec ended up going back to the WG for review of all 250+ changes... 4) And of course the RFC editor processing itself. > That was expedited. Better example is iSCSI. Draft-20 was approved Feb > 2003 [http://www.ietf.org/IESG/Announcements/draft-ietf-ips-iscsi.ann] but > published as RFC3720 in April 2004, for a lag time of 14 months. In the case of this document, one or more normative references accounts for part of the problem. Here is the history, as indicated by the RFC Editor's queue: draft-ietf-ips-iscsi 20030212 category WORKING GROUP STANDARDS TRACK (by date received) draft-ietf-ips-iscsi 20030212 current status EDIT draft-ietf-ips-iscsi 20030212 entered queue at 2003/02/11 (1 days in unknown state) draft-ietf-ips-iscsi 20030212 is version 20 draft-ietf-ips-iscsi 20030503 entered status IANA draft-ietf-ips-iscsi 20030503 was in EDIT since 20030212 (80 days) draft-ietf-ips-iscsi 20030506 entered status RFC-EDITOR draft-ietf-ips-iscsi 20030506 was in IANA since 20030503 (3 days) draft-ietf-ips-iscsi 20030619 entered status REF draft-ietf-ips-iscsi 20030619 was in RFC-EDITOR since 20030506 (44 days) draft-ietf-ips-iscsi 20040120 entered status RFC-EDITOR draft-ietf-ips-iscsi 20040120 was in REF since 20030619 (215 days) draft-ietf-ips-iscsi 20040323 entered status AUTH48 draft-ietf-ips-iscsi 20040323 was in RFC-EDITOR since 20040120 (63 days) draft-ietf-ips-iscsi 20040421 entered status AUTH draft-ietf-ips-iscsi 20040421 was in AUTH48 since 20040323 (29 days) draft-ietf-ips-iscsi 20040429 no longer in queue draft-ietf-ips-iscsi 20040429 was in AUTH since 20040421 (8 days) draft-ietf-ips-iscsi 20040429 was in queue since 20030212 (442 days) >From the above, one can see that the REF issue added 215 days. But that accounts for only about 1/2 of the total time. Plenty of other delays there. Thomas ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: New root cause problems?
In message <[EMAIL PROTECTED]>, "Eric A. Hall" writes: > >On 5/10/2005 12:45 PM, Thomas Narten wrote: > >> One example (and I'm just using it because it was it comes to mind, >> and one that I think is symptomatic of the broader problem): > >> October 15, 2004: IESG approves 4-document set. >> Within one week: authors send xml source to RFC editor >> March 10, 2005: IESG requests expedited processing (target date: March 31) >> March 29, 2005: RFCs published >> >> Total time between IESG approval and publication, 5 1/2 months. > >That was expedited. Better example is iSCSI. Draft-20 was approved Feb >2003 [http://www.ietf.org/IESG/Announcements/draft-ietf-ips-iscsi.ann] but >published as RFC3720 in April 2004, for a lag time of 14 months. > >I have no knowledge of this process and maybe there were a lot of changes >needed or something, but for a whole year there were vendors releasing >products marketed as conformant with "draft 20" > A delay of that length is generally due to dependencies -- normative references to other documents that are held up. When a document is in the RFC Editor queue, you can query its state via their web site. --Prof. Steven M. Bellovin, http://www.cs.columbia.edu/~smb ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: New root cause problems?
At 12:45 PM 5/10/2005 -0400, Thomas Narten wrote: Total time between IESG approval and publication, 5 1/2 months. And to get to Margaret's other points, I agree that these delays damage the IETF. It contributes to the perception that we are too slow, causes additional confusion about a document's true status, etc. Implementors and other SDOs need access to the final RFCs in a timely fashion. so do customers who feel an ID isn't sufficiently stable to specify as a customer requirement (for an RFP, for example) I have already had a customer get so frustrated with the IETF process they wrote an extension themselves into H.323 and got it ratified in 9 months (H.460.14) and our effort (that was first presented at the Adelaide meeting) just went to the IESG for review 5 years and 2 months later. Thomas ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf cheers, James *** Truth is not to be argued... it is to be presented. Alas, few *truths* exist without the math. ...all else is a matter of perspective ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: New root cause problems?
On 5/10/2005 12:45 PM, Thomas Narten wrote: > One example (and I'm just using it because it was it comes to mind, > and one that I think is symptomatic of the broader problem): > October 15, 2004: IESG approves 4-document set. > Within one week: authors send xml source to RFC editor > March 10, 2005: IESG requests expedited processing (target date: March 31) > March 29, 2005: RFCs published > > Total time between IESG approval and publication, 5 1/2 months. That was expedited. Better example is iSCSI. Draft-20 was approved Feb 2003 [http://www.ietf.org/IESG/Announcements/draft-ietf-ips-iscsi.ann] but published as RFC3720 in April 2004, for a lag time of 14 months. I have no knowledge of this process and maybe there were a lot of changes needed or something, but for a whole year there were vendors releasing products marketed as conformant with "draft 20" -- Eric A. Hallhttp://www.ehsco.com/ Internet Core Protocols http://www.oreilly.com/catalog/coreprot/ ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: New root cause problems?
> The time to move from approved > > I-D to RFC is often a significant percentage (perhaps averaging 20% > of more?) of the time required to develop and publish a specification. Let's say that it averages 4 months to go from IESG approval to RFC publication. (I'm choosing that number because it is big enough to indicate a problem, but small enough not to insult the RFC Editor.) Using your estimate of 20% for this step, that means that working groups average 16 months to do their part of the work, from start to finish. However that does not match either general impression or that statistics that were produced awhile back. If we averaged 16 months for working groups to go from start to finish, we would not have serious and consistent complaints that the IETF is too slow. So the actual, incremental delay for the RFC publication process is probably no worse than 10% and I wouldn't be surprised if the real statistic were more like 5%. As much as it is worth trying to make everything better, it is difficult to see a 5 or 10% overhead to the process as being one of our strategic problems. d/ --- Dave Crocker Brandenburg InternetWorking +1.408.246.8253 dcrocker a t ... WE'VE MOVED to: www.bbiw.net ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: New root cause problems?
> I have one new "root cause" issue that I don't believe was included > in the original Problem Statement: > It takes too long to publish an RFC after final approval. I agree with this. Over the last year especially, I'm seeing a significant number of cases where it is taking much more time to get an RFC published than seems justified. One example (and I'm just using it because it was it comes to mind, and one that I think is symptomatic of the broader problem): The DNSSEC bis document set (RFCs 4033-4035). October 15, 2004: IESG approves 4-document set. Within one week: authors send xml source to RFC editor March 10, 2005: IESG requests expedited processing (target date: March 31) March 29, 2005: RFCs published Total time between IESG approval and publication, 5 1/2 months. And to get to Margaret's other points, I agree that these delays damage the IETF. It contributes to the perception that we are too slow, causes additional confusion about a document's true status, etc. Implementors and other SDOs need access to the final RFCs in a timely fashion. Thomas ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: New root cause problems?
Adding to this - and I'm not sure this is the kind of thing you were looking for, but it adds to the overall problem - is that IDs timeout after 6 months (which is fine), but that includes IDs that are in the RFC-Editor queue process too. For example, look at the RFC-Editor queue site: http://www.rfc-editor.org/queue.html and try and look at a document that has a original submission date more than 6 months ago (from today)? The search link will fail, and this is a problem. Some of these are quite valuable (especially to their WGs), yet the IETF process eliminates their viewing. Solution: perhaps an exception to the 6 month timeout should be made for every document upon entry into the RFC-Editor's queue such that every document remains until the document has been published as an RFC (or won't be). At 09:36 AM 5/10/2005 -0400, Margaret Wasserman wrote: I have one new "root cause" issue that I don't believe was included in the original Problem Statement: It takes too long to publish an RFC after final approval. It currently takes several months for an RFC to be published after it is approved by the IESG. This results in several problems: - RFCs come out much later than they should, contributing greatly to the perception that the iETF is slow. The time to move from approved I-D to RFC is often a significant percentage (perhaps averaging 20% of more?) of the time required to develop and publish a specification. - Stable references are not available until months after a specification is fully approved, resulting in ambiguity about the status of a document and encouraging the use of I-D names as references, thus blurring the distinction between WG I-Ds and approved specifications. - Too many changes are made to documents after WG and IESG approval (copy editing, changes to reflect updated thinking/terminology, etc.). These changes are often not reviewed or approved by the WG or the community. - Some RFCs are already out-of-date by the time they are published. The document then contains the RFC publication date, which may result in a mistaken impression about when the IETF held a specific view or encouraged a particular practice. I believe that the IETF should consider modifications to our document handling process to reduce or eliminate the delay in publishing approved specifications. Margaret At 2:36 PM +0200 5/10/05, Brian E Carpenter wrote: Having finally read the list traffic up to date, I have a question. Can anybody identify a *new* "root cause problem" at the same level of abstraction as those identified in RFC 3774? Or is it the case that (at that level of abstraction) we have only been re-discussing the RFC 3774 problem set? (Please try to be succinct... or change the subject header if you want to change the subject.) Thanks Brian ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf cheers, James *** Truth is not to be argued... it is to be presented. Alas, few *truths* exist without the math. ...all else is a matter of perspective ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
Re: New root cause problems?
I have one new "root cause" issue that I don't believe was included in the original Problem Statement: It takes too long to publish an RFC after final approval. It currently takes several months for an RFC to be published after it is approved by the IESG. This results in several problems: - RFCs come out much later than they should, contributing greatly to the perception that the iETF is slow. The time to move from approved I-D to RFC is often a significant percentage (perhaps averaging 20% of more?) of the time required to develop and publish a specification. - Stable references are not available until months after a specification is fully approved, resulting in ambiguity about the status of a document and encouraging the use of I-D names as references, thus blurring the distinction between WG I-Ds and approved specifications. - Too many changes are made to documents after WG and IESG approval (copy editing, changes to reflect updated thinking/terminology, etc.). These changes are often not reviewed or approved by the WG or the community. - Some RFCs are already out-of-date by the time they are published. The document then contains the RFC publication date, which may result in a mistaken impression about when the IETF held a specific view or encouraged a particular practice. I believe that the IETF should consider modifications to our document handling process to reduce or eliminate the delay in publishing approved specifications. Margaret At 2:36 PM +0200 5/10/05, Brian E Carpenter wrote: Having finally read the list traffic up to date, I have a question. Can anybody identify a *new* "root cause problem" at the same level of abstraction as those identified in RFC 3774? Or is it the case that (at that level of abstraction) we have only been re-discussing the RFC 3774 problem set? (Please try to be succinct... or change the subject header if you want to change the subject.) Thanks Brian ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
New root cause problems?
Having finally read the list traffic up to date, I have a question. Can anybody identify a *new* "root cause problem" at the same level of abstraction as those identified in RFC 3774? Or is it the case that (at that level of abstraction) we have only been re-discussing the RFC 3774 problem set? (Please try to be succinct... or change the subject header if you want to change the subject.) Thanks Brian ___ Ietf mailing list Ietf@ietf.org https://www1.ietf.org/mailman/listinfo/ietf
