In message <[EMAIL PROTECTED]> "Don Welch, Merit Network" writes: > > > I have served on many program and general committees for academic > conferences. In every case we took great pains to preserve the privacy > of the feedback. We did strive to provide constructive feedback to the > author(s) and in all cases the reviewer would also be kept confidential. > Don > > Cat Okita wrote: > > On Mon, 16 Oct 2006, Curtis Villamizar wrote: > >> Would it help if you could see anonymous ratings without the comments > >> that go with the ratings? Providing the comments would just mean > >> people wouldn't record some (maybe most) of their comments and would > >> have to make comments on the call. > > > > While I'd definitely agree that having ratings for talk proposals does > > make it much easier for the PC to process them, sharing those ratings > > publically doesn't seem like a good way to encourage people to present. > > > <snip> > -- > > Donald J. Welch, Ph.D.
Donald, First off I asked a question rather than advocated a change. Maybe that was not clear in my email due to the second sentence which did not have a question mark. More on that below. Second its not a great idea to compare the NANOG PC to the acedemic review process used in most journals and conferences. And finally - please take this as just followup on the discussion of the PC (not the original topic, hence change in subject line) and nothing more. The NANOG PC is not like a journal PC or most conference PCs were a refereed paper accompanies a presentation. In those cases PC members on a give track divide up the preliminary selection of papers and work with the authors to improve them, then either the group of reviewers for the track or the entire PC accepts or rejects the papers after that process. If you take SigComm as an example, the entire process from initial submission to presentation is about 8 months long. Most acedemic journals and conferences have long review cycles. The NANOG PC doesn't have a referreed paper and doesn't go through multiple iterations of referee/author feedback before final consideration. Its more of a cursory review of the slides. A very short process might be necessary for presentation of timely information on operational issues. (Or maybe its a bad idea). Another difference in NANOG vs the typical acedemic conference is the emphasis in NANOG as a whole on transparency rather than anonymity (that's a word me thinks - I might have even spelled it right). An objection back when I was on the PC was the lack of transparency and it still gets raised occasionally which is why I posed **a question**. It seems the objection of others on the list to what they saw as an implied suggestion is "anonymous ratings without the comments" isn't anonymous enough if it still reveals the title and author. The only way to satisfy that would be to publish the ratings norms only (mean, std.dev, etc), with no author, no title, no comments. Note that the ratings are not the final decision. This is an increase in transparency in that if the rejected talks all had very low mean ratings it indicates that the PC was in agreement on those that needed a lot more work or were not very worthwhile. Occasionally some talks with higher mean ratings may be seen as rejected and there should be a reason for this. Often there is a good reason. For example if there is a preferred topic for presentations in the CFP and some very good talks would be better for another meeting they may be rejected on that grounds, or if a specific talk was a real good topic but the presentation seemed to need a lot of work it may also be rejected. I don't know if the PC still calls these "conditionally rejected" which means the PC would like to see them back again either later as is or with changes. Again, I'm asking a question and not endorsing any change. Is more transparency in the PC needed? I was replying to a comment that seemed to indicate so. Would some view of the PC ratings, retaining some level of anonymity help? A second issue which you raise and which has been raised before many times, is should the NANOG PC be more like an acedemic review and by extension should NANOG presentations be more like acedemic conference presentations. While on the PC I always looked for a set of slides which stood on their own and was criticized by some for that. The objection was that the presentation was important and the slides should only be a visual to aid in the presentation not a record of the presentation worthy of archive. If so we would have no useful archive at all. I had also at one point suggested that maybe we get more rigorous and try to get a brief paper so the slides and paper would serve as a better archive and any movement in that direction was soundly rejected. There may still be aspects of an acedemic review that could be brought into the NANOG PC. One is the review feedback and iteration cycle which for many journals is repeated multiple times as needed. The timeliness factor disallows this. NANOG presentations are due on a given date and shortly after the PC decision is due. So again not advocating any change myself but asking a question - what aspect of the academic review process would you incorporate in the present NANOG PC to improve it? Could any such change be accommodated in the very short review time window currently given to the PC or should that be changed? btw - In practice one of the benefits of the preliminary presentation ratings made by PC members was that it made it more likely that the PC members at least looked over the presentations *before* the decision making conference call (which wasn't always the case for some PC members prior to that). Curtis ps - the reviewers in NANOG PC are not very confidential since all PC members review all papers and there is not a lot of them and you just have to look on the NANOG web page to see who they are. Its not like most journals and conferences where the set of reviewers is not acknowledged to maintain confidentiality. Only the editorial board is listed in the publication.