Curtis,
Good thoughtful comments. It did look like you were making a statement
without a ? but, regardless I am just interjecting my observation from
a perspective that I think is important. As I told the SC and PC,
ultimately it is up to them to come up with a relevant, compelling
program. Merit will support the PC and SC as much as we can in these
efforts.
Cheers,
Don
Curtis Villamizar wrote:
In message <[EMAIL PROTECTED]>
"Don Welch, Merit Network" writes:
I have served on many program and general committees for academic
conferences. In every case we took great pains to preserve the privacy
of the feedback. We did strive to provide constructive feedback to the
author(s) and in all cases the reviewer would also be kept confidential.
Don
Cat Okita wrote:
On Mon, 16 Oct 2006, Curtis Villamizar wrote:
Would it help if you could see anonymous ratings without the comments
that go with the ratings? Providing the comments would just mean
people wouldn't record some (maybe most) of their comments and would
have to make comments on the call.
While I'd definitely agree that having ratings for talk proposals does
make it much easier for the PC to process them, sharing those ratings
publically doesn't seem like a good way to encourage people to present.
<snip>
--
Donald J. Welch, Ph.D.
Donald,
First off I asked a question rather than advocated a change. Maybe
that was not clear in my email due to the second sentence which did
not have a question mark. More on that below.
Second its not a great idea to compare the NANOG PC to the acedemic
review process used in most journals and conferences.
And finally - please take this as just followup on the discussion of
the PC (not the original topic, hence change in subject line) and
nothing more.
The NANOG PC is not like a journal PC or most conference PCs were a
refereed paper accompanies a presentation. In those cases PC members
on a give track divide up the preliminary selection of papers and work
with the authors to improve them, then either the group of reviewers
for the track or the entire PC accepts or rejects the papers after
that process. If you take SigComm as an example, the entire process
from initial submission to presentation is about 8 months long. Most
acedemic journals and conferences have long review cycles.
The NANOG PC doesn't have a referreed paper and doesn't go through
multiple iterations of referee/author feedback before final
consideration. Its more of a cursory review of the slides. A very
short process might be necessary for presentation of timely
information on operational issues. (Or maybe its a bad idea).
Another difference in NANOG vs the typical acedemic conference is the
emphasis in NANOG as a whole on transparency rather than anonymity
(that's a word me thinks - I might have even spelled it right). An
objection back when I was on the PC was the lack of transparency and
it still gets raised occasionally which is why I posed **a question**.
It seems the objection of others on the list to what they saw as an
implied suggestion is "anonymous ratings without the comments" isn't
anonymous enough if it still reveals the title and author. The only
way to satisfy that would be to publish the ratings norms only (mean,
std.dev, etc), with no author, no title, no comments. Note that the
ratings are not the final decision. This is an increase in
transparency in that if the rejected talks all had very low mean
ratings it indicates that the PC was in agreement on those that needed
a lot more work or were not very worthwhile. Occasionally some talks
with higher mean ratings may be seen as rejected and there should be a
reason for this. Often there is a good reason. For example if there
is a preferred topic for presentations in the CFP and some very good
talks would be better for another meeting they may be rejected on that
grounds, or if a specific talk was a real good topic but the
presentation seemed to need a lot of work it may also be rejected. I
don't know if the PC still calls these "conditionally rejected" which
means the PC would like to see them back again either later as is or
with changes.
Again, I'm asking a question and not endorsing any change. Is more
transparency in the PC needed? I was replying to a comment that
seemed to indicate so. Would some view of the PC ratings, retaining
some level of anonymity help?
A second issue which you raise and which has been raised before many
times, is should the NANOG PC be more like an acedemic review and by
extension should NANOG presentations be more like acedemic conference
presentations. While on the PC I always looked for a set of slides
which stood on their own and was criticized by some for that. The
objection was that the presentation was important and the slides
should only be a visual to aid in the presentation not a record of the
presentation worthy of archive. If so we would have no useful archive
at all. I had also at one point suggested that maybe we get more
rigorous and try to get a brief paper so the slides and paper would
serve as a better archive and any movement in that direction was
soundly rejected.
There may still be aspects of an acedemic review that could be brought
into the NANOG PC. One is the review feedback and iteration cycle
which for many journals is repeated multiple times as needed. The
timeliness factor disallows this. NANOG presentations are due on a
given date and shortly after the PC decision is due.
So again not advocating any change myself but asking a question - what
aspect of the academic review process would you incorporate in the
present NANOG PC to improve it? Could any such change be accommodated
in the very short review time window currently given to the PC or
should that be changed?
btw - In practice one of the benefits of the preliminary presentation
ratings made by PC members was that it made it more likely that the PC
members at least looked over the presentations *before* the decision
making conference call (which wasn't always the case for some PC
members prior to that).
Curtis
ps - the reviewers in NANOG PC are not very confidential since all PC
members review all papers and there is not a lot of them and you just
have to look on the NANOG web page to see who they are. Its not like
most journals and conferences where the set of reviewers is not
acknowledged to maintain confidentiality. Only the editorial board is
listed in the publication.
!DSPAM:453470fd2619871331!
--
Donald J.
Welch, Ph.D.
President & CEO
Merit
Network, Inc.
www.merit.edu
734-615-0547
1000
Oakbrook Drive
Ann Arbor, MI
48104
|
