This week, I’m at the CHI conference in Montreal. This is my third CHI conference, in addition to having attended numerous other peer-reviewed/academic conferences. And I realized while sitting in some of the sessions where were lets just say not-so-interesting to put it mildly, that I would prefer a slightly (well, radically) different format for academic/peer-reviewed conferences.

Most major conferences now have multiple parallel tracks. CHI has 9 parallel tracks running concurrently with 5 different courses. So all in all, you can chose to be in one of 14 different places. The opportunity cost of missing some talks is high but unfortunately, since one doesn’t know which ones are the ones worth listening to, it’s ultimately an educated guess (some would call it more like a crap shoot, but it’s not really all that bad).

By the time a paper is presented at a conference there is nothing a person in the audience can really say or do in order to change that in any way. Regardless of how critical the comments may be, the paper/poster has already been published and is hence been condemned to be around forever. Since the paper has already gone through a peer-review process, I am willing to trust the judgment (though some would question that, but bear with me for a minute) of the reviewers who accepted the paper to have verified the methodology and the details presented in the paper. Therefore, there is no reason for me to have to sit through a half hour (20 minutes presentation and 10 minute Q&A) on a paper. Instead all I really care about is what is the motivation/topic for the paper, what was built/tested/evaluated and what the results were. 5 minutes at the most. The 5 minute presentation is sufficient to present a summary of the work. Anyone who is interested in the work, can read the paper and/or then meet with the presenter during his/her designated “discussion time” (no paper presentations should be on at the same time).

Back to the question of whether or not to trust the reviewers’ judgments — once the paper has already been accepted there is very little accountability in a public forum for the quality of the paper or the presentation. Therefore, rather than spend the time to listen to all the details, I prefer getting the executive summary and then having the opportunity to interact with the presenter in a special public Q&A session.

To that effect, I would also like to see some kind of accountability on the quality of the paper or presentation after the conference. Under the current system, while members of the audience can ask insightful questions, that interaction is not captured other than in the memories of others who were present and happened to be listening. It would be interesting to have members of the community post their critiques or questions in a public forum — similar to comments on a blog entry. Along the same lines I would also be in favor of having the paper reviewers comments and the authors rebuttal available since the real value in research is not just what actually gets published, but in the back-and-forth that happens between the community.

So here is what I would propose:
1) No paper presentation to exceed 5 minutes, maybe even 3 minutes.
2) Single (or at least fewer) track paper presentations so that one can at least get a comprehensive overview of all the papers in the conference.
3) The paper, abstract, reviews and rebuttals posted on a website, which allows others to view them and comment on them.
4) A designated “discussion session” for the paper, which is more interactive. Those who wish to question the methods or results can read the paper and then come and participate in the discussion. Those who wish to just listen and absorb are welcome as well. A scribe should be present to capture the interaction of the discussion so that it too can be archived along with the paper.

These are of course thoughts in progress, but I’d be happy to discuss them with anyone who is interested.