I have always felt that the "outstanding program committee member" awards or somewhat quixotic, in as much as we often tend to pick a handful of people for the recognition. As the program committees continue to grow in size (IJCAI 2016 had over 2000 people in the program committee), and we are faced with continual deadlines and hyper-paper-productivity, it seems counterproductive to recognize just a vanishing percentage of people for reviewing, as it might inadvertently send the message that good reviewing is a rather unattainable "get a Nobel Prize" like goal.
So, for this IJCAI, we requested the program committee members to nominate their peers who, in their eyes, have done an outstanding job of reviewing/handling a paper. We explicitly told them that our goal was to recognize a significant percentage of the program committee. We made a simple google forms interface through which PC members could submit nominations and justifications.
As soon as we made the link public, we had an onslaught of nominations! Apparently there was a pent-up demand for such a mechanism..
Given the size and diversity of the program committee, we went through all the nominations and justifications manually to make sure that the system is not being gamed.
Sure enough, we did indeed catch a few instances of unsubstantive/nonsequitor nominations, such as
Prof. X has always submitted to IJCAI and I nominate X for outstanding reviewing
or
Dr. Y asked me to review some papers, and I nominate Y for outstanding reviewing
or in one particularly egregious case, a program committee member who was ejected from the IJCAI program committee for complete malfeasance, and had the audacity to nominate a bunch of his/her colleagues..
But the overwhelming majority of nominations were substantive and heartfelt. It was quite a gratifying experience to go through them!
Yesterday, we updated the IJCAI program committee listing on the website by putting a resplendent blue ribbon (
) in front of each program committee member for each nomination he/she received.You can now look at these beribboned program committee members--nearly 200 of them (or about 10% of the program committee)--at the webpage here:
http://ijcai-16.org/index.php/welcome/view/program_committee
If any of these are your colleagues, please take time to congratulate them. They did what they did despite knowing that there are few tangible rewards and incentives for good reviewing.
If you are an author who submitted a paper to IJCAI and got thoughtful reviews, you know that your experience was not by any means isolated!
For myself, as a graying member of the AI community, I am freshly tickled pink at how "good reviewing" is still a rule rather than exception for us! Thank you, IJCAI-16 program committee!
--Rao
