--On Saturday, 08 March, 2008 00:12 -0800 Lakshminath Dondeti
<[EMAIL PROTECTED]> wrote:
>> The ideal way to deal with them is to always respond, and to
>> get an "I am satisfied with your response" to close the
>> thread.
>
> "Ideal" being the keyword though. Not everyone, for any
> number of reasons, including cultural reasons, will come out
> and state "all clear." It is also asking too much to ask the
> reviewer to get into a debate with the authors. It also
> fosters an environment where the reviewer starts becoming an
> authority.
>...
>> Repeat above several times and intersperse with long periods
>> of time where nothing happens on a document. You now have an
>> idea of why it seems to take a long time to get documents
>> through the system.
>
> Indeed. What started out as a great idea -- I volunteered to
> be a GenART reviewer 3-4 years ago now -- is beginning to
> become yet another burden in the process.
I have to agree with this. One of the biggest risks we have to
quality in standards is the dark side of the review process, a
situation in which the effort to get a document as nearly
correct as is reasonable given its maturity level and possible
turns into "deal with that objection". A half-dozen years ago
we had extensive discussions of a concern in which ADs would
object to particular text and authors, exhausted from the
process of getting documents produced, would agree to any
suggested changes --as long as they were not really offensive--
in order to make the objection go away. Put differently, there
is a tendency for "satisfy him (or her) and make the DISCUSS go
away" to become a more important objecting in practice than "get
things right".
In at least some ways, the DISCUSS criteria were an attempt to
constrain that problem, at as as far as the ADs were concerned.
If we replace the "opportunity" to have to individually satisfy
a dozen or so ADs with the opportunity to have to individually
satisfy them plus a dozen more area-related reviewers, we are in
big trouble.
I am greatly in favor of these invited reviews if they ensure
that every document is carefully reviewed by someone who was not
part of its development process. I think that is where we
started out.
But the IESG has been selected by the community to take
responsibility for these evaluations. If the IESG isn't going
to do it, or can't, we need to be looking at our basic processes
and structure, not at who is reviewing what for whom. If a
review is done for a particular area, I expect that review to be
treated primarily as advice to the relevant ADs. I expect those
ADs to evaluate that advice carefully, not just to critically
accept it.
I definitely do not want to see a discussion between authors and
reviewers --especially Area-selected reviewers-- during Last
Call. It too easily deteriorates into a "satisfy him"
situation, and those reviewers are not anything special (or,
unlike the ADs themselves, selected by some community mechanism).
I think this whole process needs a little refocusing.
Especially for WG documents, no matter how many reviewers,
shepherds, and lions, tigers, and bears we introduce into the
system, the IESG should have one primary focus, which is making
a "go / no-go" decision on whether a document is ready for
approval and publication given maturity levels and any other
relevant issues. If the answer is "no-go" for anything but an
obvious matter about which there is no dissent or ambiguity, the
document ought to go back to the WG for resolution of issues
(which should clearly be identified as clearly as possible), not
turned into a negotiation process between whomever happened to
generate a comment and the authors about whether that person's
view of the comment can be satisfied. There is just too much
risk in the "satisfy comments" model of getting something
important wrong or of responding to the comment but missing the
main point of which the comment is a symptom.
I certainly don't object to Thomas's idea of using issue
trackers more and I think that making reviews public is an
important safeguard. But those issue trackers should be used to
* inform the IESG's decision about whether the document
is ready and to
* help inform the WG, presumably along with a careful
IESG summary, about the issues (not the specific
comments) it needs to address if the document is bounced
back to it.
Except perhaps for editorial matters and for clarification, a
dialogue between an author and someone who makes a comment
should have no place in the consensus process.
john
_______________________________________________
IETF mailing list
IETF@ietf.org
https://www.ietf.org/mailman/listinfo/ietf
