[Gen-art] timing of reviews and discussions with authors, and AD reactions to various outcomes

[Jari Arkko]

This is a reminder of procedures that I use when i look at Gen-ART reviews and 
an upcoming telechat.

Most of the time, I get to the list of documents for the telechat documents 
couple of days before the actual meeting. 

When a review has arrived by then, it is very, very much appreciated. 90% of 
the time there has been a review and a reaction from the authors and any issues 
have already been resolved. Even happier about that :-) No need for anyone to 
do anything additional, except for me to thank both sides.

If there is no response from the authors I will ask them. I may not ask in case 
of trivial editorial issues. In a very serious issue, I may raise a blocking 
Discuss. But most of the time I will say “please look at this, and I’d like 
resolve this discussion before the telechat”. And then, only if that fails to 
work I will raise a discuss just before telechat.

If there is no review, I will most likely send the Gen-ART reviewer a question. 
There has been cases when I’ve lost an e-mail, or my Mac has :-) Most of the 
time I’ll get a response and 75% of the time I will see the review a bit later. 
Thank you.

If I have no review by the telechat day, I will most likely review the draft 
myself (and I review a small fraction anyway even if you have reviewed). I of 
course very much appreciate your reviews. But it is OK also to miss a review, 
it is not the end of the world. If you do miss a review, after the telechat you 
are better off concentrating on the next review task rather than delivering a 
review for a potentially approved document.

From your perspective the main points are: 1) expectation that authors will 
engage in discussion 2) the AD will step in if needed, but I expect that most 
of the time the authors do the right things 3) I use a Discuss as a last resort 
mechanism if sanity does not prevail :-) and 4) let Jean or me know if you are 
unable to do a review.

Thanks again for your service. The system seems to be working very well from my 
perspective, with much added value to the IETF.

Jari



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
Gen-art mailing list
Gen-art@ietf.org
https://www.ietf.org/mailman/listinfo/gen-art

Re: [Gen-art] Timing of reviews: conclusion

[Martin Thomson]

On 11 July 2013 13:03, Brian E Carpenter  wrote:
> but IMHO it would be better
> if we submitted our reviews via the review tool, which could then
> send them on as email automatically.

I have all the same problems as Brian.  And this might also make
access control easier, such that reviews could be sent to WG lists
without requiring sign-up (though on reflection, that's a little
optimistic).
___
Gen-art mailing list
Gen-art@ietf.org
https://www.ietf.org/mailman/listinfo/gen-art

Re: [Gen-art] Timing of reviews: conclusion

[Romascanu, Dan (Dan)]

+1



> -Original Message-
> From: gen-art-boun...@ietf.org [mailto:gen-art-boun...@ietf.org] On
> Behalf Of Brian E Carpenter
> Sent: Thursday, July 11, 2013 11:04 PM
> To: A. Jean Mahoney
> Cc: gen-art@ietf.org
> Subject: Re: [Gen-art] Timing of reviews: conclusion
> 
> On 12/07/2013 06:49, A. Jean Mahoney wrote:
> > I'm also willing to help with the experiment, but I suggest that the
> > review tool be updated later, after we have a better idea of the new
> > requirements. Updating the review tool would probably require updating
> > datatracker also.
> 
> Yes. The following is not a complaint, since the review tool is a much-
> appreciated volunteer effort, but IMHO it would be better if we
> submitted our reviews via the review tool, which could then send them on
> as email automatically. At the moment I usually forget to enter the
> review URL into the tool until some days or weeks later!
> 
> (People who've used EDAS know what I mean.)
> 
> It would also make sense for reviewers to be remembered by the tracker,
> so that they can get status updates automatically.
> 
> Since the tracker already knows who is interested in each draft,
> integrating the two would make a lot of sense.
> 
>Brian
> 
> >
> > I can easily capture early review assignments in the spreadsheets for
> now.
> >
> > Jean
> >
> >
> > On 7/11/13 10:23 AM, Jari Arkko wrote:
> >>> I am also in favor of the experiment and willing to participate.
> >>> Besides the changes pointed to by Brian we may also need to make
> >>> some adjustments to the Gen-ART tools (new column in the table at
> >>> http://art.tools.ietf.org/tools/art/genart/index.cgi/t=3634/fullqueu
> >>> e to mark the WG review due date) and the process used by the
> >>> secretary (separate mail for WG early review?).
> >> That's a good point. Thanks.
> >>
> >> Jari
> >>
> >>
> >> ___
> >> Gen-art mailing list
> >> Gen-art@ietf.org
> >> https://www.ietf.org/mailman/listinfo/gen-art
> >
> > ___
> > Gen-art mailing list
> > Gen-art@ietf.org
> > https://www.ietf.org/mailman/listinfo/gen-art
> >
> ___
> Gen-art mailing list
> Gen-art@ietf.org
> https://www.ietf.org/mailman/listinfo/gen-art
___
Gen-art mailing list
Gen-art@ietf.org
https://www.ietf.org/mailman/listinfo/gen-art

Re: [Gen-art] Timing of reviews: conclusion

[Brian E Carpenter]

On 12/07/2013 06:49, A. Jean Mahoney wrote:
> I'm also willing to help with the experiment, but I suggest that the
> review tool be updated later, after we have a better idea of the new
> requirements. Updating the review tool would probably require updating
> datatracker also.

Yes. The following is not a complaint, since the review tool is a
much-appreciated volunteer effort, but IMHO it would be better
if we submitted our reviews via the review tool, which could then
send them on as email automatically. At the moment I usually forget
to enter the review URL into the tool until some days or weeks later!

(People who've used EDAS know what I mean.)

It would also make sense for reviewers to be remembered by the tracker,
so that they can get status updates automatically.

Since the tracker already knows who is interested in each draft,
integrating the two would make a lot of sense.

   Brian

> 
> I can easily capture early review assignments in the spreadsheets for now.
> 
> Jean
> 
> 
> On 7/11/13 10:23 AM, Jari Arkko wrote:
>>> I am also in favor of the experiment and willing to participate.
>>> Besides the changes pointed to by Brian we may also need to make some
>>> adjustments to the Gen-ART tools (new column in the table at
>>> http://art.tools.ietf.org/tools/art/genart/index.cgi/t=3634/fullqueue
>>> to mark the WG review due date) and the process used by the secretary
>>> (separate mail for WG early review?).
>> That's a good point. Thanks.
>>
>> Jari
>>
>>
>> ___
>> Gen-art mailing list
>> Gen-art@ietf.org
>> https://www.ietf.org/mailman/listinfo/gen-art
> 
> ___
> Gen-art mailing list
> Gen-art@ietf.org
> https://www.ietf.org/mailman/listinfo/gen-art
> 
___
Gen-art mailing list
Gen-art@ietf.org
https://www.ietf.org/mailman/listinfo/gen-art

Re: [Gen-art] Timing of reviews: conclusion

[A. Jean Mahoney]

I'm also willing to help with the experiment, but I suggest that the 
review tool be updated later, after we have a better idea of the new 
requirements. Updating the review tool would probably require updating 
datatracker also.


I can easily capture early review assignments in the spreadsheets for now.

Jean


On 7/11/13 10:23 AM, Jari Arkko wrote:

I am also in favor of the experiment and willing to participate. Besides the 
changes pointed to by Brian we may also need to make some adjustments to the 
Gen-ART tools (new column in the table at 
http://art.tools.ietf.org/tools/art/genart/index.cgi/t=3634/fullqueue to mark 
the WG review due date) and the process used by the secretary (separate mail 
for WG early review?).

That's a good point. Thanks.

Jari


___
Gen-art mailing list
Gen-art@ietf.org
https://www.ietf.org/mailman/listinfo/gen-art


___
Gen-art mailing list
Gen-art@ietf.org
https://www.ietf.org/mailman/listinfo/gen-art

Re: [Gen-art] Timing of reviews: conclusion

[Robert Sparks]

I would like start discussing a more general way to track reviews when 
they are needed. Dan's new-column tweak to the existing tool makes me a 
little nervous in the long run - the column won't apply to all 
documents, and other types will come in making new columns leading to a 
painfully sparse matrix.


Rather, I would like to work towards a system where we can request 
reviews on a document at any time, for any reason.


Instead of scraping last calls to identify documents that need review, 
we can make the parts of the system that issue last calls automatically 
make a review request, and that request can tailor the data we need to 
track for that particular review (like any constraining dates such as WG 
LC or IETF LC completion).


RjS

On 7/11/13 10:23 AM, Jari Arkko wrote:

I am also in favor of the experiment and willing to participate. Besides the 
changes pointed to by Brian we may also need to make some adjustments to the 
Gen-ART tools (new column in the table at 
http://art.tools.ietf.org/tools/art/genart/index.cgi/t=3634/fullqueue to mark 
the WG review due date) and the process used by the secretary (separate mail 
for WG early review?).

That's a good point. Thanks.

Jari


___
Gen-art mailing list
Gen-art@ietf.org
https://www.ietf.org/mailman/listinfo/gen-art


___
Gen-art mailing list
Gen-art@ietf.org
https://www.ietf.org/mailman/listinfo/gen-art

Re: [Gen-art] Timing of reviews: conclusion

[Jari Arkko]

> I am also in favor of the experiment and willing to participate. Besides the 
> changes pointed to by Brian we may also need to make some adjustments to the 
> Gen-ART tools (new column in the table at 
> http://art.tools.ietf.org/tools/art/genart/index.cgi/t=3634/fullqueue to mark 
> the WG review due date) and the process used by the secretary (separate mail 
> for WG early review?). 

That's a good point. Thanks.

Jari


___
Gen-art mailing list
Gen-art@ietf.org
https://www.ietf.org/mailman/listinfo/gen-art

Re: [Gen-art] Timing of reviews: conclusion

[Romascanu, Dan (Dan)]

I am also in favor of the experiment and willing to participate. Besides the 
changes pointed to by Brian we may also need to make some adjustments to the 
Gen-ART tools (new column in the table at 
http://art.tools.ietf.org/tools/art/genart/index.cgi/t=3634/fullqueue to mark 
the WG review due date) and the process used by the secretary (separate mail 
for WG early review?). 

Regards,

Dan






> -Original Message-
> From: gen-art-boun...@ietf.org [mailto:gen-art-boun...@ietf.org] On
> Behalf Of Jari Arkko
> Sent: Wednesday, July 10, 2013 7:53 PM
> To:  Team
> Subject: [Gen-art] Timing of reviews: conclusion
> 
> I wanted to conclude our earlier discussion on the timing of directorate
> reviews. Overall I read the thread as cautiously positive for
> experimenting if this helps or not. But we need to take care of a
> multiple issues:
> 
> - Figure out the best way for the reviewers to be able to send mail to
> WG lists
> - Limit the time/scope of the experiment in order to find out what the
> real effect is on, e.g., reviewer workload
> - Determine whether to review at WG time, IETF last call time, and IESG
> review time. Right now we are doing the last two.
> - Determine how to start the review, and what the timing is.
> - Carefully select review timing and documents so that they are ready
> for an external review
> - Shield reviewers from being shouted down by a large group of WG
> participants
> 
> The suggestions for resolving these items is:
> 
> - We can start the effort by having the WG chairs able to ask for this
> service. The team can then throttle the requests to a level they see
> they can handle.
> - Start the experiment now and review in six months, determine success
> based on interviewing participating reviewers and WG chairs
> - Perform reviews after the WGLC (in parallel with AD reviews)
> - Review documents while they are in the working group, and attempt to
> have the same person do a review of the new version that is entering the
> IESG telechat (similar to what we are doing now, but replacing IETF last
> call review with WGLC review)
> - Require working group chairs to mediate and monitor discussions
> between the reviewer and the working group
> - Whitelisting might be a feasible solution for making the reviewers
> able to post to the WG list. Or is there something else that we could
> do? Rely on the WG chairs approving posts?
> 
> Comments?
> 
> Jari
> 
> ___
> Gen-art mailing list
> Gen-art@ietf.org
> https://www.ietf.org/mailman/listinfo/gen-art
___
Gen-art mailing list
Gen-art@ietf.org
https://www.ietf.org/mailman/listinfo/gen-art

Re: [Gen-art] Timing of reviews: conclusion

[Jari Arkko]

Brian:

Good points. And I like the template. Thanks.

Jari


___
Gen-art mailing list
Gen-art@ietf.org
https://www.ietf.org/mailman/listinfo/gen-art

Re: [Gen-art] Timing of reviews: conclusion

[Brian E Carpenter]

Hi Jari,

Personally, I'm willing to try this. I was just wondering what our
"standard" template would be for this. Something like this perhaps:

The WG Chairs requested a Gen-ART review for this draft. For background on
Gen-ART, please see the FAQ at
.

Please resolve these comments along with any Area Director comments
you may receive.

Document: draft-ietf-foobar...
Reviewer: Brian Carpenter
Review Date: 2013-07-10
IETF LC End Date: TBD
IESG Telechat date: TBD

Summary:  Needs work...

Also: we would need to update the GenART FAQ slightly. The answer
to "Q: Why is my document being reviewed?" needs changing.

Regards
   Brian

On 11/07/2013 04:53, Jari Arkko wrote:
> I wanted to conclude our earlier discussion on the timing of directorate 
> reviews. Overall I read the thread as cautiously positive for experimenting 
> if this helps or not. But we need to take care of a multiple issues:
> 
> - Figure out the best way for the reviewers to be able to send mail to WG 
> lists
> - Limit the time/scope of the experiment in order to find out what the real 
> effect is on, e.g., reviewer workload
> - Determine whether to review at WG time, IETF last call time, and IESG 
> review time. Right now we are doing the last two.
> - Determine how to start the review, and what the timing is.
> - Carefully select review timing and documents so that they are ready for an 
> external review
> - Shield reviewers from being shouted down by a large group of WG participants
> 
> The suggestions for resolving these items is:
> 
> - We can start the effort by having the WG chairs able to ask for this 
> service. The team can then throttle the requests to a level they see they can 
> handle.
> - Start the experiment now and review in six months, determine success based 
> on interviewing participating reviewers and WG chairs
> - Perform reviews after the WGLC (in parallel with AD reviews)
> - Review documents while they are in the working group, and attempt to have 
> the same person do a review of the new version that is entering the IESG 
> telechat (similar to what we are doing now, but replacing IETF last call 
> review with WGLC review)
> - Require working group chairs to mediate and monitor discussions between the 
> reviewer and the working group
> - Whitelisting might be a feasible solution for making the reviewers able to 
> post to the WG list. Or is there something else that we could do? Rely on the 
> WG chairs approving posts?
> 
> Comments?
> 
> Jari
> 
> ___
> Gen-art mailing list
> Gen-art@ietf.org
> https://www.ietf.org/mailman/listinfo/gen-art
> 
___
Gen-art mailing list
Gen-art@ietf.org
https://www.ietf.org/mailman/listinfo/gen-art

[Gen-art] Timing of reviews: conclusion

[Jari Arkko]

I wanted to conclude our earlier discussion on the timing of directorate 
reviews. Overall I read the thread as cautiously positive for experimenting if 
this helps or not. But we need to take care of a multiple issues:

- Figure out the best way for the reviewers to be able to send mail to WG lists
- Limit the time/scope of the experiment in order to find out what the real 
effect is on, e.g., reviewer workload
- Determine whether to review at WG time, IETF last call time, and IESG review 
time. Right now we are doing the last two.
- Determine how to start the review, and what the timing is.
- Carefully select review timing and documents so that they are ready for an 
external review
- Shield reviewers from being shouted down by a large group of WG participants

The suggestions for resolving these items is:

- We can start the effort by having the WG chairs able to ask for this service. 
The team can then throttle the requests to a level they see they can handle.
- Start the experiment now and review in six months, determine success based on 
interviewing participating reviewers and WG chairs
- Perform reviews after the WGLC (in parallel with AD reviews)
- Review documents while they are in the working group, and attempt to have the 
same person do a review of the new version that is entering the IESG telechat 
(similar to what we are doing now, but replacing IETF last call review with 
WGLC review)
- Require working group chairs to mediate and monitor discussions between the 
reviewer and the working group
- Whitelisting might be a feasible solution for making the reviewers able to 
post to the WG list. Or is there something else that we could do? Rely on the 
WG chairs approving posts?

Comments?

Jari

___
Gen-art mailing list
Gen-art@ietf.org
https://www.ietf.org/mailman/listinfo/gen-art

Re: [Gen-art] timing of reviews

[Romascanu, Dan (Dan)]

> -Original Message-
> From: Jari Arkko [mailto:jari.ar...@piuha.net]
> Sent: Monday, May 27, 2013 11:18 AM
> To: Brian E Carpenter
> Cc: Romascanu, Dan (Dan); General Area Review Team; Mary Barnes; The
> IESG
> Subject: Re: [Gen-art] timing of reviews
> 
> Dan, Brian,
> 
> > On 26/05/2013 20:57, Romascanu, Dan (Dan) wrote:
> > ...
> >> I do believe that there is however a solution to this problem, but it
> >> does not come for free either. Let the WGLC take place as it happens
> >> right now, but move the Expert Reviews (a.k.a. Directorate Reviews)
> >> also within the responsibility of the WG, after the WGLC completes.
> >
> > I think this would be a very good compromise - ask WG chairs to obtain
> > and respond to the cross-area reviews (including
> > Gen-ART) just before requesting AD review. This would mean we
> > reviewers would be looking at reasonably finished work as we do now,
> > and it should incent the WG to take notice of the reviews, and relieve
> > the AD to some extent too.
> >
> > Very good suggestion!
> 
> I agree. I like this model because it puts the chairs and the WG in
> charge. Thanks, Dan!
> 
> The other potentially helpful technique is to let the WG chairs have
> some room to decide when to ask for reviews. The chairs typically have
> some understanding of where their documents are in terms of readiness :-
> ) even when opening a last call. A complex, controversial document is
> probably better off first complete its round of WGLC(s) before giving it
> to directorates...
> 
> Jari

[[DR]] My experience is that WG chairs already feel that they have the room to 
ask for early directorate reviews for more complex documents - this is at least 
my experience with the MIB doctors reviews. These are considered however 'early 
reviews' according to the current process and they are an exception. If we go 
ahead with this proposal, we make of reviews performed after WGLC but before 
submission to the IESG the rule. 

Regards,

Dan

___
Gen-art mailing list
Gen-art@ietf.org
https://www.ietf.org/mailman/listinfo/gen-art

Re: [Gen-art] timing of reviews

[Jari Arkko]

Dan, Brian,

> On 26/05/2013 20:57, Romascanu, Dan (Dan) wrote:
> ...
>> I do believe that there is however a solution to this
>> problem, but it does not come for free either. Let the WGLC
>> take place as it happens right now, but move the Expert
>> Reviews (a.k.a. Directorate Reviews) also within the
>> responsibility of the WG, after the WGLC completes. 
> 
> I think this would be a very good compromise - ask WG chairs
> to obtain and respond to the cross-area reviews (including
> Gen-ART) just before requesting AD review. This would mean
> we reviewers would be looking at reasonably finished work
> as we do now, and it should incent the WG to take notice of
> the reviews, and relieve the AD to some extent too.
> 
> Very good suggestion!

I agree. I like this model because it puts the chairs and the WG in charge. 
Thanks, Dan!

The other potentially helpful technique is to let the WG chairs have some room 
to decide when to ask for reviews. The chairs typically have some understanding 
of where their documents are in terms of readiness :-) even when opening a last 
call. A complex, controversial document is probably better off first complete 
its round of WGLC(s) before giving it to directorates...

Jari

___
Gen-art mailing list
Gen-art@ietf.org
https://www.ietf.org/mailman/listinfo/gen-art

Re: [Gen-art] timing of reviews

[Romascanu, Dan (Dan)]

Indeed. In the XRBLOCK WG we are actually soliciting both PMDIR and SDP 
Directorate reviews, and update the documents according to the directorate 
comments before submitting them to the IESG. So we can say that there is some 
runing code for this proposal.

Regards,

Dan



> -Original Message-
> From: Gonzalo Camarillo [mailto:gonzalo.camari...@ericsson.com]
> Sent: Monday, May 27, 2013 10:42 AM
> To: Romascanu, Dan (Dan)
> Cc: Mary Barnes; Jari Arkko; General Area Review Team; The IESG
> Subject: Re: [Gen-art] timing of reviews
> 
> Hi,
> 
> > I do believe that there is however a solution to this problem, but it
> > does not come for free either. Let the WGLC take place as it happens
> > right now, but move the Expert Reviews (a.k.a. Directorate Reviews)
> > also within the responsibility of the WG, after the WGLC completes.
> > This will make the WG cycle longer by a few weeks, but will allow for
> > the Directorate comments to be processed by the WG, but on a stable
> > version, which the WG considers as having passed the WGLC.
> 
> what Dan is proposing here is already happening to a certain extent in
> RAI, in particular in the XRBLOCK WG, which Dan cochairs. The chairs
> request Benoit's directorate to perform a review before they send the
> documents to me (i.e., before pub req).
> 
> Cheers,
> 
> Gonzalo
___
Gen-art mailing list
Gen-art@ietf.org
https://www.ietf.org/mailman/listinfo/gen-art

Re: [Gen-art] timing of reviews

[Gonzalo Camarillo]

Hi,

> I do believe that there is however a solution to this problem, but it
> does not come for free either. Let the WGLC take place as it happens
> right now, but move the Expert Reviews (a.k.a. Directorate Reviews)
> also within the responsibility of the WG, after the WGLC completes.
> This will make the WG cycle longer by a few weeks, but will allow for
> the Directorate comments to be processed by the WG, but on a stable
> version, which the WG considers as having passed the WGLC.

what Dan is proposing here is already happening to a certain extent in
RAI, in particular in the XRBLOCK WG, which Dan cochairs. The chairs
request Benoit's directorate to perform a review before they send the
documents to me (i.e., before pub req).

Cheers,

Gonzalo
___
Gen-art mailing list
Gen-art@ietf.org
https://www.ietf.org/mailman/listinfo/gen-art

Re: [Gen-art] timing of reviews

[Brian E Carpenter]

On 26/05/2013 20:57, Romascanu, Dan (Dan) wrote:
...
> I do believe that there is however a solution to this
> problem, but it does not come for free either. Let the WGLC
> take place as it happens right now, but move the Expert
> Reviews (a.k.a. Directorate Reviews) also within the
> responsibility of the WG, after the WGLC completes. 

I think this would be a very good compromise - ask WG chairs
to obtain and respond to the cross-area reviews (including
Gen-ART) just before requesting AD review. This would mean
we reviewers would be looking at reasonably finished work
as we do now, and it should incent the WG to take notice of
the reviews, and relieve the AD to some extent too.

Very good suggestion!

Brian
___
Gen-art mailing list
Gen-art@ietf.org
https://www.ietf.org/mailman/listinfo/gen-art

Re: [Gen-art] timing of reviews

[Romascanu, Dan (Dan)]

I am sharing Mary's concerns - also some of the ones expressed by other.

The principal problem I see with moving the Gen-ART review in parallel with the 
WGLC rather than IETF LC is related to the level of maturity of documents when 
they reach WGLC. This is of course variable according to the area and WG, but 
it is clear that most of the documents are in a less stable status when they 
reach WGLC than when they reach IETFLC. As a result the Gen-ART reviewers will 
see their load increase not only as a result of having to review a number of 
documents that will never exit the WG (I do not know how many these are, can we 
see statistics?), but also because they will need to follow the full WGLC cycle 
which may include several revisions, extend over multiple Last Calls, and lead 
to considerable changes in the documents resulting from other WGLC comments, 
changes which also need to be reviewed. 

I do believe that there is however a solution to this problem, but it does not 
come for free either. Let the WGLC take place as it happens right now, but move 
the Expert Reviews (a.k.a. Directorate Reviews) also within the responsibility 
of the WG, after the WGLC completes. This will make the WG cycle longer by a 
few weeks, but will allow for the Directorate comments to be processed by the 
WG, but on a stable version, which the WG considers as having passed the WGLC. 

Would this work? We may want to hear the wg-chairs opinion, I am concerned 
about to intensive cross-posting. 

Regards,

Dan




> -Original Message-
> From: gen-art-boun...@ietf.org [mailto:gen-art-boun...@ietf.org] On
> Behalf Of Mary Barnes
> Sent: Friday, May 24, 2013 4:15 PM
> To: Jari Arkko
> Cc: General Area Review Team; The IESG
> Subject: Re: [Gen-art] timing of reviews
> 
> I will caveat my comments with the fact that I am no longer taking on
> new documents for review. However, having been involved in the process
> for 10 years now on the gen-art side and having done the reviews for 3
> IETF chairs, and having much experience as a chair and document editor,
> I do have some concerns as laid out below.
> 
> Mary.
> 
> On Fri, May 24, 2013 at 1:11 AM, Jari Arkko 
> wrote:
> >
> >> I think time will tell whether it's small. Getting into the thick of
> >> WG discussion in an area one doesn't know well can be quite a shock
> >> to the system. That's not to say we shouldn't try it, but I would ask
> >> for it to be a time-limited experiment initially.
> >
> > Yes - or scope limited (only some WGs or areas at the beginning).
> [MB] Absolutely, it needs to have a limited scope initially.  As Brian
> said, the concept is not new, it's just that few took advantage of it.
>I would prefer that this stage remain optimal and up to the WG chairs
> as to whether it's necessary. Without  the tool support that is
> available for the regular LC and Telechat reviews, I don't think we want
> to burden Jean too much until we have a very good idea of the level of
> effort and until there is tool support (e.g., chairs enter the docs into
> a tool that gets the doc into Jean's incoming queue for assignment). I
> also believe this increases the burden on the reviewers significantly
> more than the burden when we added the LC reviews.  For folks that
> haven't been around a while, the original Gen-ART process only did
> reviews prior to the telechats.  Adding reviews at LC did increase the
> workload for reviewers, however, we did see a somewhat
> increase in documents ready at Telechat time.   [/MB]
> >
> >> It seems important to inform WGs explicitly that outside review is
> >> taking place *and* has to be taken seriously (because if not, the
> >> same reviewer will be back at a later stage with the same
> >> questions).'
> >
> > Good point.
> [MB] Exactly.  [/MB]
> 
> [MB] My biggest concern is that many documents that go through WGLC in
> the WGs in which I am active are NOT ready at that stage.  I also have
> seen in cases of Gen-ART reviews, that documents clearly weren't
> reviewed by the shepherd or ADs due to the level of editorial issues and
> technical (the p2psip reload document is a good example of this).
> There were concerns raised in the WG with regards to its readiness but
> it was still progressed. Certainly, I hope this is the exception.
> However, my guess is that many documents will change quite a bit after
> LC.  When I chaired the SIPPING WG, we had multiple stages of review in
> the WG where we had assigned reviewers.  We had an initial review when
> the doc was first accepted as a WG document, then we had a pre-WGLC
> review and then the real WGLC. Many of the documents went through a 2nd
> last call. If we could get

Re: [Gen-art] timing of reviews

[Spencer Dawkins]

On 5/24/2013 8:14 AM, Mary Barnes wrote:


Yes - or scope limited (only some WGs or areas at the beginning).

[MB] Absolutely, it needs to have a limited scope initially.  As Brian
said, the concept is not new, it's just that few took advantage of it.


Mary is the best person to comment on the time period when I was a 
Gen-ART reviewer, but I do remember a few working group chairs asking 
for early reviews (which were granted in every case I heard about). I'm 
not sure how many chairs knew they could ask for early reviews, and of 
course, I don't know why they wanted early reviews. But it's fair to say 
that nobody discouraged early reviews, at least not where I could hear that.


To all the reviewers:

Thank all of you for your review work. I dropped off Gen-ART when I 
joined the IAB, but still read your reviews (and am now reading them 
more closely :-)


Spencer
___
Gen-art mailing list
Gen-art@ietf.org
https://www.ietf.org/mailman/listinfo/gen-art

Re: [Gen-art] timing of reviews

[Elwyn Davies]

The comments from David, Brian, Mary and Joel are broadly in line with
my thinking on this.  One additional point: during the discussion that
Jari's original question provoked I suggested that a written summary of
the design decisions that had led to the chosen protocol design could be
helpful for the 'tail' discussion process.  Although it would make some
extra work for authors and WG chairs, having this information would make
the gen-art (and other) reviewing easier and make interpreting what is
going on during WGLC somewhat easier.

One way of doing this would be to make the summary an appendix to the
proto write up.  Doing this might also serve to bring out any remaining
issues that need to be sorted out during WGLC - and thereby provide
evidence of the draft maturity as highlighted by David. And, perhaps,
lessen the chance (or at least the impact) of attempts to reopen decided
issues.

All this being said, I would certainly be prepared to give it a go on a
trial basis if our secretary can cope with he extra work until the tools
could be extended.

Regards,
Elwyn

On Fri, 2013-05-24 at 12:00 -0400, Black, David wrote:
> I concur with Brian's concerns - I would strongly suggest that the WG chair(s)
> take on two responsibilities for this experiment with Gen-ART reviews at WGLC:
> 
> (1) Commit to monitoring/mediating the WG mailing list interaction between the
> WG members and the Gen-ART reviewer.  Letting the WG membership loose "no 
> holds
> barred" on a Gen-ART reviewer who is not an expert in the protocol being 
> reviewed
> may not work out well, e.g., it could generate list reactions along the lines 
> of
> "if the Gen-ART reviewer doesn't understand this basic concept, why should we 
> pay
> serious attention to anything else in the review?"  I've recently had a behind
> the scenes interaction on a technical issue from a Gen-ART review where the
> involvement of the responsible WG chair made a significant difference.
> 
> (2) Determine whether the draft is mature enough for a Gen-ART review at WGLC.
> At a high level, the question to be asked is roughly "if WG LC resulted in no
> changes to this draft, would you be comfortable submitting it to your AD with
> a publication request?"  If the answer is "no", a Gen-ART review is probably
> premature.  A potentially useful mechanism could be to ask the WG chair to
> prepare a publication request writeup (Barry's short form should suffice) at
> the start of WGLC in order to request a Gen-ART review.  If the WG chair 
> believes
> that a writeup is premature, then the Gen-ART review is probably also 
> premature.
> For example, if a major purpose of a WGLC is to drive open technical issues
> to resolution, Gen-ART review is probably premature, and a writeup would note
> the open technical issues that need resolution.
> 
> Thanks,
> --David
> 
> > -Original Message-
> > From: gen-art-boun...@ietf.org [mailto:gen-art-boun...@ietf.org] On Behalf 
> > Of
> > Brian E Carpenter
> > Sent: Thursday, May 23, 2013 10:27 PM
> > To: Joel M. Halpern
> > Cc: General Area Review Team; The IESG
> > Subject: Re: [Gen-art] timing of reviews
> > 
> > On 24/05/2013 11:43, Joel M. Halpern wrote:
> > > In principle I am happy perform reviews in conjunction with WG last
> > > call, even though that would probably produce a small increase in loading.
> > 
> > I think time will tell whether it's small. Getting into the thick of
> > WG discussion in an area one doesn't know well can be quite a shock
> > to the system. That's not to say we shouldn't try it, but I would ask
> > for it to be a time-limited experiment initially.
> > 
> > It seems important to inform WGs explicitly that outside review
> > is taking place *and* has to be taken seriously (because if not,
> > the same reviewer will be back at a later stage with the same
> > questions).
> > 
> > Then there is a question: will Gen-ART re-review at the time of
> > IETF Last Call as well as when the draft reaches the IESG agenda?
> > 
> > BTW, the concept of early review by Gen-ART is not new, but I think
> > it was only tried for one or two drafts in the past.
> > 
> > Brian
> > 
> > >
> > > There is however a complication.  Many WG email lists are "ember only",
> > > for good reason.  I do not wish to end up on all those lists.  Which
> > > means that the moderator has to let the message through, and then
> > > probably remember to whitelist the reviewer.  Ad folk have to keep
> > > copying the reviewer explicitly with responses.  I think this can be
> > >

Re: [Gen-art] timing of reviews

[Black, David]

I concur with Brian's concerns - I would strongly suggest that the WG chair(s)
take on two responsibilities for this experiment with Gen-ART reviews at WGLC:

(1) Commit to monitoring/mediating the WG mailing list interaction between the
WG members and the Gen-ART reviewer.  Letting the WG membership loose "no holds
barred" on a Gen-ART reviewer who is not an expert in the protocol being 
reviewed
may not work out well, e.g., it could generate list reactions along the lines of
"if the Gen-ART reviewer doesn't understand this basic concept, why should we 
pay
serious attention to anything else in the review?"  I've recently had a behind
the scenes interaction on a technical issue from a Gen-ART review where the
involvement of the responsible WG chair made a significant difference.

(2) Determine whether the draft is mature enough for a Gen-ART review at WGLC.
At a high level, the question to be asked is roughly "if WG LC resulted in no
changes to this draft, would you be comfortable submitting it to your AD with
a publication request?"  If the answer is "no", a Gen-ART review is probably
premature.  A potentially useful mechanism could be to ask the WG chair to
prepare a publication request writeup (Barry's short form should suffice) at
the start of WGLC in order to request a Gen-ART review.  If the WG chair 
believes
that a writeup is premature, then the Gen-ART review is probably also premature.
For example, if a major purpose of a WGLC is to drive open technical issues
to resolution, Gen-ART review is probably premature, and a writeup would note
the open technical issues that need resolution.

Thanks,
--David

> -Original Message-
> From: gen-art-boun...@ietf.org [mailto:gen-art-boun...@ietf.org] On Behalf Of
> Brian E Carpenter
> Sent: Thursday, May 23, 2013 10:27 PM
> To: Joel M. Halpern
> Cc: General Area Review Team; The IESG
> Subject: Re: [Gen-art] timing of reviews
> 
> On 24/05/2013 11:43, Joel M. Halpern wrote:
> > In principle I am happy perform reviews in conjunction with WG last
> > call, even though that would probably produce a small increase in loading.
> 
> I think time will tell whether it's small. Getting into the thick of
> WG discussion in an area one doesn't know well can be quite a shock
> to the system. That's not to say we shouldn't try it, but I would ask
> for it to be a time-limited experiment initially.
> 
> It seems important to inform WGs explicitly that outside review
> is taking place *and* has to be taken seriously (because if not,
> the same reviewer will be back at a later stage with the same
> questions).
> 
> Then there is a question: will Gen-ART re-review at the time of
> IETF Last Call as well as when the draft reaches the IESG agenda?
> 
> BTW, the concept of early review by Gen-ART is not new, but I think
> it was only tried for one or two drafts in the past.
> 
> Brian
> 
> >
> > There is however a complication.  Many WG email lists are "ember only",
> > for good reason.  I do not wish to end up on all those lists.  Which
> > means that the moderator has to let the message through, and then
> > probably remember to whitelist the reviewer.  Ad folk have to keep
> > copying the reviewer explicitly with responses.  I think this can be
> > done, but we should remember such procedural glitches.
> >
> > Yours,
> > Joel
> >
> > On 5/23/2013 6:28 PM, Jari Arkko wrote:
> >> Thanks again for all your hard work in doing the reviews. They make
> >> it possible for me to concentrate on reviewing just those documents
> >> where there are problems or I have deeper expertise. And the quality
> >> of the protocol specifications coming out of the IETF is obviously
> >> very important, particularly for protocols that are gaining
> >> significant use.
> >>
> >> As you may have seen, the IESG and the community has been wondering
> >> if there'd be something that we could do about the IETF process, in
> >> the sense that there's quite many things happening at the very end of
> >> the document's life cycle. This results in some surprises, and often
> >> also moves some important decisions out of the working group and to
> >> author/shepherd/AD hands. A while ago we met for the IESG retreat and
> >> wanted to experiment with three specific changes:
> >>
> >> - sending work back to the WGs when significant changes are needed,
> >> and making the WG the central place of the edits - moving some
> >> directorate reviews earlier, without impact reviewer effort too much
> >> - inviting some of the shepherds onto tele chats
> >>
&g

Re: [Gen-art] timing of reviews

[Stephen Farrell]

FYI, I started the same discussion on the secdir list. [1]

I'll summarise to the IESG when appropriate.

S.

[1] http://www.ietf.org/mail-archive/web/secdir/current/msg03968.html

On 05/23/2013 11:28 PM, Jari Arkko wrote:
> Thanks again for all your hard work in doing the reviews. They make it 
> possible for me to concentrate on reviewing just those documents where there 
> are problems or I have deeper expertise. And the quality of the protocol 
> specifications coming out of the IETF is obviously very important, 
> particularly for protocols that are gaining significant use.
> 
> As you may have seen, the IESG and the community has been wondering if 
> there'd be something that we could do about the IETF process, in the sense 
> that there's quite many things happening at the very end of the document's 
> life cycle. This results in some surprises, and often also moves some 
> important decisions out of the working group and to author/shepherd/AD hands. 
> A while ago we met for the IESG retreat and wanted to experiment with three 
> specific changes:
> 
> - sending work back to the WGs when significant changes are needed, and 
> making the WG the central place of the edits
> - moving some directorate reviews earlier, without impact reviewer effort too 
> much
> - inviting some of the shepherds onto tele chats
> 
> I am writing to you in order to discuss the second item. How big of a change 
> would it be to have Gen-ART reviews be invoked during WGLC, while the 
> documents are still in the working groups? The goal of the reviews would 
> still be the same, e.g., I would be checking that the reviews were positive 
> and/or that the issues brought up have been properly addressed.
> 
> There are important details to consider, however, and I would like to get 
> your feedback on how you would seem them having an effect, and what would be 
> the best way to organise this, if we decide to go ahead with the change for 
> the Gen-ART.
> 
> Triggering the review would have to be done by something else than IETF last 
> call announcement. Is the best approach is to have the WG chairs manually 
> request for this? Note that sometimes there are multiple WGLCs. I presume 
> that it would be preferable to have a Gen-ART review be done only once at 
> this stage, as otherwise the work load would increase. The chairs may have 
> some idea of whether they are likely to need another WGLC before they start 
> one. 
> 
> There may be possibly bigger changes and time lag between the first Gen-ART 
> review and the one that checks that the changes are ok.
> 
> Some specs may not make it through WGLC, and a review at that stage may 
> increase the effort you guys are putting in, by reviewing those specs that 
> will fail.
> 
> Jari
> 
___
Gen-art mailing list
Gen-art@ietf.org
https://www.ietf.org/mailman/listinfo/gen-art

Re: [Gen-art] timing of reviews

[Mary Barnes]

I will caveat my comments with the fact that I am no longer taking on
new documents for review. However, having been involved in the process
for 10 years now on the gen-art side and having done the reviews for 3
IETF chairs, and having much experience as a chair and document
editor,  I do have some concerns as laid out below.

Mary.

On Fri, May 24, 2013 at 1:11 AM, Jari Arkko  wrote:
>
>> I think time will tell whether it's small. Getting into the thick of
>> WG discussion in an area one doesn't know well can be quite a shock
>> to the system. That's not to say we shouldn't try it, but I would ask
>> for it to be a time-limited experiment initially.
>
> Yes - or scope limited (only some WGs or areas at the beginning).
[MB] Absolutely, it needs to have a limited scope initially.  As Brian
said, the concept is not new, it's just that few took advantage of it.
   I would prefer that this stage remain optimal and up to the WG
chairs as to whether it's necessary. Without  the tool support that is
available for the regular LC and Telechat reviews, I don't think we
want to burden Jean too much until we have a very good idea of the
level of effort and until there is tool support (e.g., chairs enter
the docs into a tool that gets the doc into Jean's incoming queue for
assignment). I also believe this increases the burden on the reviewers
significantly more than the burden when we added the LC reviews.  For
folks that haven't been around a while, the original Gen-ART process
only did reviews prior to the telechats.  Adding reviews at LC did
increase the workload for reviewers, however, we did see a somewhat
increase in documents ready at Telechat time.   [/MB]
>
>> It seems important to inform WGs explicitly that outside review
>> is taking place *and* has to be taken seriously (because if not,
>> the same reviewer will be back at a later stage with the same
>> questions).'
>
> Good point.
[MB] Exactly.  [/MB]

[MB] My biggest concern is that many documents that go through WGLC in
the WGs in which I am active are NOT ready at that stage.  I also have
seen in cases of Gen-ART reviews, that documents clearly weren't
reviewed by the shepherd or ADs due to the level of editorial issues
and technical (the p2psip reload document is a good example of this).
There were concerns raised in the WG with regards to its readiness but
it was still progressed. Certainly, I hope this is the exception.
However, my guess is that many documents will change quite a bit after
LC.  When I chaired the SIPPING WG, we had multiple stages of review
in the WG where we had assigned reviewers.  We had an initial review
when the doc was first accepted as a WG document, then we had a
pre-WGLC review and then the real WGLC. Many of the documents went
through a 2nd last call. If we could get more WGs to be more
disciplined about ensuring that documents receive adequate review
prior to WGLC, then this idea might work.  At this point, I see very
few WGs that have an effective process in place. One also has to
consider that in some WGs (many that I participate in and write docs
for), there are very often folks that don't review any version of a
document in detail until it goes through WGLC. So, I am personally
quite skeptical that this proposal adds significant value to the
overall process, in particular based on the level of effort required
by both the gen-art reviewers.  I think this is just moving an
existing problem around and asking the wrong set of folks to help
solve the problem. I think a more effective change would be for ADs to
expect WGs to ensure the docs are thoroughly reviewed earlier in the
stage by the WG.

 [/MB]
>
>> Then there is a question: will Gen-ART re-review at the time of
>> IETF Last Call as well as when the draft reaches the IESG agenda?
>
> My initial though was that we'd do the same as we do today for drafts that 
> change, i.e., re-review by the same person. The likelihood for draft changes 
> is of course bigger when starting from the WGLC. Another possible work load 
> increase to watch for. I'd be interested in your thoughts on other approaches 
> to deal with re-reviews though.
>
> Jari
>
> ___
> Gen-art mailing list
> Gen-art@ietf.org
> https://www.ietf.org/mailman/listinfo/gen-art
___
Gen-art mailing list
Gen-art@ietf.org
https://www.ietf.org/mailman/listinfo/gen-art

Re: [Gen-art] timing of reviews

[Jari Arkko]

> I think time will tell whether it's small. Getting into the thick of
> WG discussion in an area one doesn't know well can be quite a shock
> to the system. That's not to say we shouldn't try it, but I would ask
> for it to be a time-limited experiment initially.

Yes - or scope limited (only some WGs or areas at the beginning).

> It seems important to inform WGs explicitly that outside review
> is taking place *and* has to be taken seriously (because if not,
> the same reviewer will be back at a later stage with the same
> questions).'

Good point.

> Then there is a question: will Gen-ART re-review at the time of
> IETF Last Call as well as when the draft reaches the IESG agenda?

My initial though was that we'd do the same as we do today for drafts that 
change, i.e., re-review by the same person. The likelihood for draft changes is 
of course bigger when starting from the WGLC. Another possible work load 
increase to watch for. I'd be interested in your thoughts on other approaches 
to deal with re-reviews though.

Jari

___
Gen-art mailing list
Gen-art@ietf.org
https://www.ietf.org/mailman/listinfo/gen-art

Re: [Gen-art] timing of reviews

[Jari Arkko]

Joel:

> There is however a complication.  Many WG email lists are "ember only", for 
> good reason.  I do not wish to end up on all those lists.  Which means that 
> the moderator has to let the message through, and then probably remember to 
> whitelist the reviewer.  Ad folk have to keep copying the reviewer explicitly 
> with responses.  I think this can be done, but we should remember such 
> procedural glitches.

Good point. Thanks for bringing that up.

Jari

___
Gen-art mailing list
Gen-art@ietf.org
https://www.ietf.org/mailman/listinfo/gen-art

Re: [Gen-art] timing of reviews

[Brian E Carpenter]

On 24/05/2013 11:43, Joel M. Halpern wrote:
> In principle I am happy perform reviews in conjunction with WG last
> call, even though that would probably produce a small increase in loading.

I think time will tell whether it's small. Getting into the thick of
WG discussion in an area one doesn't know well can be quite a shock
to the system. That's not to say we shouldn't try it, but I would ask
for it to be a time-limited experiment initially.

It seems important to inform WGs explicitly that outside review
is taking place *and* has to be taken seriously (because if not,
the same reviewer will be back at a later stage with the same
questions).

Then there is a question: will Gen-ART re-review at the time of
IETF Last Call as well as when the draft reaches the IESG agenda?

BTW, the concept of early review by Gen-ART is not new, but I think
it was only tried for one or two drafts in the past.

Brian

> 
> There is however a complication.  Many WG email lists are "ember only",
> for good reason.  I do not wish to end up on all those lists.  Which
> means that the moderator has to let the message through, and then
> probably remember to whitelist the reviewer.  Ad folk have to keep
> copying the reviewer explicitly with responses.  I think this can be
> done, but we should remember such procedural glitches.
> 
> Yours,
> Joel
> 
> On 5/23/2013 6:28 PM, Jari Arkko wrote:
>> Thanks again for all your hard work in doing the reviews. They make
>> it possible for me to concentrate on reviewing just those documents
>> where there are problems or I have deeper expertise. And the quality
>> of the protocol specifications coming out of the IETF is obviously
>> very important, particularly for protocols that are gaining
>> significant use.
>>
>> As you may have seen, the IESG and the community has been wondering
>> if there'd be something that we could do about the IETF process, in
>> the sense that there's quite many things happening at the very end of
>> the document's life cycle. This results in some surprises, and often
>> also moves some important decisions out of the working group and to
>> author/shepherd/AD hands. A while ago we met for the IESG retreat and
>> wanted to experiment with three specific changes:
>>
>> - sending work back to the WGs when significant changes are needed,
>> and making the WG the central place of the edits - moving some
>> directorate reviews earlier, without impact reviewer effort too much
>> - inviting some of the shepherds onto tele chats
>>
>> I am writing to you in order to discuss the second item. How big of a
>> change would it be to have Gen-ART reviews be invoked during WGLC,
>> while the documents are still in the working groups? The goal of the
>> reviews would still be the same, e.g., I would be checking that the
>> reviews were positive and/or that the issues brought up have been
>> properly addressed.
>>
>> There are important details to consider, however, and I would like to
>> get your feedback on how you would seem them having an effect, and
>> what would be the best way to organise this, if we decide to go ahead
>> with the change for the Gen-ART.
>>
>> Triggering the review would have to be done by something else than
>> IETF last call announcement. Is the best approach is to have the WG
>> chairs manually request for this? Note that sometimes there are
>> multiple WGLCs. I presume that it would be preferable to have a
>> Gen-ART review be done only once at this stage, as otherwise the work
>> load would increase. The chairs may have some idea of whether they
>> are likely to need another WGLC before they start one.
>>
>> There may be possibly bigger changes and time lag between the first
>> Gen-ART review and the one that checks that the changes are ok.
>>
>> Some specs may not make it through WGLC, and a review at that stage
>> may increase the effort you guys are putting in, by reviewing those
>> specs that will fail.
>>
>> Jari
>>
>> ___ Gen-art mailing list
>> Gen-art@ietf.org https://www.ietf.org/mailman/listinfo/gen-art
>>
> ___
> Gen-art mailing list
> Gen-art@ietf.org
> https://www.ietf.org/mailman/listinfo/gen-art
> 
___
Gen-art mailing list
Gen-art@ietf.org
https://www.ietf.org/mailman/listinfo/gen-art

Re: [Gen-art] timing of reviews

[Joel M. Halpern]

In principle I am happy perform reviews in conjunction with WG last 
call, even though that would probably produce a small increase in loading.


There is however a complication.  Many WG email lists are "ember only", 
for good reason.  I do not wish to end up on all those lists.  Which 
means that the moderator has to let the message through, and then 
probably remember to whitelist the reviewer.  Ad folk have to keep 
copying the reviewer explicitly with responses.  I think this can be 
done, but we should remember such procedural glitches.


Yours,
Joel

On 5/23/2013 6:28 PM, Jari Arkko wrote:

Thanks again for all your hard work in doing the reviews. They make
it possible for me to concentrate on reviewing just those documents
where there are problems or I have deeper expertise. And the quality
of the protocol specifications coming out of the IETF is obviously
very important, particularly for protocols that are gaining
significant use.

As you may have seen, the IESG and the community has been wondering
if there'd be something that we could do about the IETF process, in
the sense that there's quite many things happening at the very end of
the document's life cycle. This results in some surprises, and often
also moves some important decisions out of the working group and to
author/shepherd/AD hands. A while ago we met for the IESG retreat and
wanted to experiment with three specific changes:

- sending work back to the WGs when significant changes are needed,
and making the WG the central place of the edits - moving some
directorate reviews earlier, without impact reviewer effort too much
- inviting some of the shepherds onto tele chats

I am writing to you in order to discuss the second item. How big of a
change would it be to have Gen-ART reviews be invoked during WGLC,
while the documents are still in the working groups? The goal of the
reviews would still be the same, e.g., I would be checking that the
reviews were positive and/or that the issues brought up have been
properly addressed.

There are important details to consider, however, and I would like to
get your feedback on how you would seem them having an effect, and
what would be the best way to organise this, if we decide to go ahead
with the change for the Gen-ART.

Triggering the review would have to be done by something else than
IETF last call announcement. Is the best approach is to have the WG
chairs manually request for this? Note that sometimes there are
multiple WGLCs. I presume that it would be preferable to have a
Gen-ART review be done only once at this stage, as otherwise the work
load would increase. The chairs may have some idea of whether they
are likely to need another WGLC before they start one.

There may be possibly bigger changes and time lag between the first
Gen-ART review and the one that checks that the changes are ok.

Some specs may not make it through WGLC, and a review at that stage
may increase the effort you guys are putting in, by reviewing those
specs that will fail.

Jari

___ Gen-art mailing list
Gen-art@ietf.org https://www.ietf.org/mailman/listinfo/gen-art


___
Gen-art mailing list
Gen-art@ietf.org
https://www.ietf.org/mailman/listinfo/gen-art

[Gen-art] timing of reviews

[Jari Arkko]

Thanks again for all your hard work in doing the reviews. They make it possible 
for me to concentrate on reviewing just those documents where there are 
problems or I have deeper expertise. And the quality of the protocol 
specifications coming out of the IETF is obviously very important, particularly 
for protocols that are gaining significant use.

As you may have seen, the IESG and the community has been wondering if there'd 
be something that we could do about the IETF process, in the sense that there's 
quite many things happening at the very end of the document's life cycle. This 
results in some surprises, and often also moves some important decisions out of 
the working group and to author/shepherd/AD hands. A while ago we met for the 
IESG retreat and wanted to experiment with three specific changes:

- sending work back to the WGs when significant changes are needed, and making 
the WG the central place of the edits
- moving some directorate reviews earlier, without impact reviewer effort too 
much
- inviting some of the shepherds onto tele chats

I am writing to you in order to discuss the second item. How big of a change 
would it be to have Gen-ART reviews be invoked during WGLC, while the documents 
are still in the working groups? The goal of the reviews would still be the 
same, e.g., I would be checking that the reviews were positive and/or that the 
issues brought up have been properly addressed.

There are important details to consider, however, and I would like to get your 
feedback on how you would seem them having an effect, and what would be the 
best way to organise this, if we decide to go ahead with the change for the 
Gen-ART.

Triggering the review would have to be done by something else than IETF last 
call announcement. Is the best approach is to have the WG chairs manually 
request for this? Note that sometimes there are multiple WGLCs. I presume that 
it would be preferable to have a Gen-ART review be done only once at this 
stage, as otherwise the work load would increase. The chairs may have some idea 
of whether they are likely to need another WGLC before they start one. 

There may be possibly bigger changes and time lag between the first Gen-ART 
review and the one that checks that the changes are ok.

Some specs may not make it through WGLC, and a review at that stage may 
increase the effort you guys are putting in, by reviewing those specs that will 
fail.

Jari

___
Gen-art mailing list
Gen-art@ietf.org
https://www.ietf.org/mailman/listinfo/gen-art