Spinning the Wheel of Fortune: Some quasi-humorous consequences of endless conference deadlines..

Not too long ago, there used to be a couple of real conference deadlines per year in AI.  You work on your papers through the year, submit them,  wait for their reviews, and  if those don't work out, revise and resubmit for the next cycle that is a year or at least six months away.

These days, clearly, things have changed--especially in AI--where there are a whole variety of conferences with endless and sometimes overlapping deadlines.

There are many things that can be said about this brave new world, but I want to use this post to share a couple of quasi-humorous consequences..

[Withdrawal after author response]: The  author response period for IJCAI-16 ended this Saturday, and we have been getting a  steady trickle of mails from authors to help them withdraw their paper. Interestingly most of them seem to have suddenly realized a "lethal error" [sic] in their experiments and thus want to withdraw the paper, as urgently as possible! The fact that a new set of conference deadlines are around the corner (e.g. ACL on 3/18 ) is just a mere coincidence.

Interestingly, submitting to another conference, in the middle of the review period of the current conference (perhaps because of the poor reviews) is considered a violation; but if the authors just send a mail requesting withdrawal of their paper, it magically becomes "legal" (but, does it become ethical?..).

[Reviews ready before papers!]: On the night we sent the assignments to the 2000 program committee members, I knew I still had to fine tune a couple of features of the IJCAI review form. However as it was an exhausting weekend, I thought I will do that in a couple of days and went to sleep. After all, reviewers will need time to read the papers right? And all I need to do is to finalize the review form before they start submitting reviews. Little did I know!

By the time I woke up from a brief 4-hour nap, my mail box already had some 30 messages from Easychair about newly submitted reviews! While scrambling to finalize the review form pronto,  I was curious as to how this superhuman feat of near instantaneous reviewing was possible. Turned out that several of those papers were being re-submitted from a couple of conferences whose cycles just got over, and the papers turned out to have overlapping reviewers! So reviewers can play the game as well as the authors: send the same papers and get the same reviews!  ;-)

To some extent the conferences are all complicit in this, given the excessive interest in talking about the number of submissions a conference receives. After all, the bigger the denominator, the lower the acceptance rate, and thus higher the perceived selectivity. Apparently ivy leagues are not the only ones running this racket..

We collectively bemoan the quality of submissions and reviews. May be we need to put the money where our mouths are, and work on designing mechanisms that don't incentivize the wrong things.. For starters, I hope we start caring more about the impact of a conference (measured by citations, for example), than its "selectivity".

Enough chit chat--I hear another conference deadline approaching... time to give the wheel another whirl..

Rao

Papers "written by" the program chair and other unexpected consequences of increased diversity in conference participation..

One of the beautiful things about the conferences in general, and AI conferences in particular, is how cosmopolitan they have become.

Gone are the days when all the participants and submissions were from just a couple of "usual suspects"  countries. It should be no surprise to anyone that neither US nor Europe are the top regions in terms of submissions to IJCAI. It is also worth noting that the allure of AI has broadened significantly and there are submissions from many authors who are first time IJCAI submitters. Similar increased internationalization has also occurred in the program committee.

All of this internationalization of IJCAI is truly a cause to celebrate. Knuth says that he wants the names of all the authors cited in his Art of Programming books to be in their native script . I don't know if I can ever get mentioned in the AoP books, but certainly look forward to seeing కంభంపాటి సుబ్బారావు in technical forums ;-). Heck, being an AI aficionado, I am sure of the day when people can write in the language they feel most fluent  in, and the AI Babel Fish will just render it flawlessly into the reader's preferred language.

In the mean time, however, the increased diversity and internationalization does bring up some challenges that we don't always anticipate. Here are a couple of interesting ones:

• There are several papers currently going through IJCAI review that have "Subbarao Kambhampati" as the sole author of the paper. Apparently some authors thought that the way to make their submissions compliant with double-blind reviewing requirement is to just put me as the author. (We decided to leave them in the reviewing pool as they do follow the spirit--if not the letter--of the double blind review process.  Of course it did lead to a couple of irate mails from some PC members asking (a) why am I submitting papers to the conference I am the program chair for and (b) why am I not even following the rules ;-)

• In some cases, the program committee members have developed surprising interpretations of "conflict of interest" for double blind reviewing. Basically, to ensure that they may not be reviewing a paper from an author that they have CoI with, they decided to (a) guess the identity of the authors (b) send mails to them to see if they are the authors and (c) declare CoI when they find that in fact the author is someone they guessed ;-). This would have been great entertainment,  if I didn't have to go find new reviewers for those papers :-(

The lesson to be  learned, I guess, is that we can't both hope for increased diversity and internationalization in participation, and at the same time assume that everyone has the same common understanding of the process that the old boys do. Things need to be put in writing, however obvious they might be for some (large) subset of the participants. 

Rao

So what papers are IJCAI-16 PC members most interested in reading (based on the bid popularity)?

One of the unsaid things about a large conference like IJCAI is that the quality of reviews a paper gets is critically correlated with the number of bids that the paper gets.

When a paper doesn't get enough bids, it needs to be matched manually to PC members--something that is very much error-prone when we are talking about 2000 papers and 1700+ reviewers (and the ever condensed reviewing time).

Having just completed making some 10,000+ reviewer assignments using 53,000 reviewer bids, I am struck not just by the well known long tail phenomena in the number of bids papers get, but also how many papers get almost "shutout" (get zero bids).

So we thought it would be fund to do an analysis on what papers are getting a lot of bids (based on a word cloud analysis of the paper titles).

52892 bids were made by 1723 program committee members over 2000 papers in the main track(or an average of 25 bids/paper and 30 bids/PC member). Here is how they were distributed




So, we naturally wanted to find out which papers are getting more vs. fewer bids. Numbering the bins 0 (for <=3 combined bids), 1 (for 4-10 combined bids) etc, we made word clouds for papers in each of the bins (based on the titles). Thus the cloud (helpfully shaped) 8 has the words occurring in the papers getting  >70 bids ;-)

Here then are the bins from the "least bids (0)" to "most bids (8)"

So there you go... I have my own interpretation of this data, but I would rather hear your interpretations ;-)

Rao
(with all the help from Lydia Manikonda-- IJCAI-16 Data Scientist)

What's hot in AI? (from IJCAI-16 Main Track Submission Ttitles)

Here is the word cloud of IJCAI-16 paper titles.




The above is with tf/idf so it throws out non-discriminative words (like "is", "of" and "deep" :-)


Here is another tf/idf normalized view...


                                                                                            (Clouds Credit: Lydia Manikonda)

So apparently the IJCAI does remain the conference for the Whole AI Enterprise ;-)


Finally, here is the normal frequency word cloud...




Rao
Feb 9th

My earliest IJCAI submissions... ;-)

As we are caught-up  in the  IJCAI-16 paper assignment phase, I recalled, with some fondness, my earliest IJCAI papers.

My very first submission was to IJCAI 1985. That paper was unfortunately rejected; I managed to find those reviews ;-)  It did go on to form the basis for my MS thesis and is currently my most cited paper...

My next submission was to IJCAI 1989 and it fared much better--getting accepted to the conference. I remember going to the huge conference at Detroit. I also remember that Tom Dean, who was tasked with giving a talk on the last day of the conference about what is new in planning and reasoning at IJCAI-89 asked me for a couple of my plastic transparencies, and covered them in his talk! Heady feeling.. ;-)  That paper went onto form the basis for my PhD thesis. I managed to find its reviews too .

I don't obviously have any submissions this time--but wish all the rest of you with submissions great reviews!

Rao
2/7/16