SIGCOMM Blog

I had a few questions about this:

1) Is there a reason that this is being applied to just the SIGCOMM Conference and not any of the other sponsored venues?

2) Have any of the other ACM SIGs placed such specific policies for their conferences?

3) I understand that the reviewing process is unpredictable and paper rejections can make it feel especially so. However, the Shadow PC experiments suggest that results of the reviewing process are reasonably fair. Is there a particular motivation for the change? If not, why not experiment with a less high-profile venue?

Hi Srini,

To answer your questions:

First, remember that the SIG directly oversees the SIGCOMM conference, but other conferences have steering committees. The policies for other SIG-sponsored conferences are set by their respective steering committees.

And second, yes, many steering committees have similar policies in which (1) the steering committee play an advisory role in PC selection, and/or (2) there are suggestions (even, in other cases, limits) on how PC-chair-authored papers are handled.

Finally, the motivation for the change is as stated -- it's important for SIGCOMM to have deep technical expertise that is balanced as possible over various criteria.

Your point about shadow PC experiments is interesting to me, and I'd like to know more. I'm not aware of any analysis of shadow PC experiments showing what you claim. If you have any data, please pass it on. (We have talked among the SIGCOMM EC about the need to study reviewing fairness more carefully and the idea of comparing shadow PC results to actual paper selections was discussed as one way to do that, so I'm curious what's been done elsewhere).

For the steering committee answer, could you provide some pointers to policy. I haven't seen any that are like the one Sigcomm is proposing.

The shadow PC analysis was published in CCR by Anja Feldman for Sigcomm 2005. See http://portal.acm.org/citation.cfm?id=1070889

I think the results are open to interpretation in terms of "bias". However, given what I have seen of the randomness within PC meetings, I found the results were surprisingly similar to the original results.

I'm somewhat confused. If the purpose of these changes is "fairness" then doesn't it behoove us to provide at least an argument a) that unfairness is occurring, b) that is is occurring due to a specific set of problems and c) that the new measures directly address these problems.

For my part, having chaired the SIGCOMM conference, having sat on SIGCOMM PCs for a number of years, and before that having decidedly been an outsider to the networking community, the SIGCOMM conference doesn't seem any more or less fair than any of the other communities I participate in. To wit, there are a number of aspects of the process and of the community that introduce randomness and, at times, capriciousness (e.g., Jon Crowcroft's wonderful tongue in cheek CCR paper "Cold Topics in Networking" is all to close to reality at times) but I have never witnessed systematic bias of the sort that any change in the PC's makeup would resolve. Perhaps part of the problem is that its not clear what "fair" means. If the idea is that research paper "goodness" exists as some Platonic quality and if only we could perfect the PC makeup and reviewing process then we'd get the "right" papers accepted, then I find this highly dubious. If the idea is that a different PC could have resulted in a different set of papers selected then I have little doubt that the answer is "yes". However, I argue that the same PC offset by six months would also make a different selection and that these three different selections would all be fine... none more fair than the other... each flawed, but generally accepting papers that are of high quality and valuable to the community. If there is deeper unfairness than this I would like to see the evidence.

If the purpose is not fairness, but to provide advice and insight to those running the conference, then an alternate approach, which I think might be superior for a number of reasons, would be to actually create a steering committee for the SIGCOMM conference. In particular the kinds of questions and challenges faced by the PC chairs and the general chairs are problems invariably faced by the leaders of past year's conferences. These are frequently problems that arise directly through experience, but may be otherwise non-obvious (e.g., the challenges in review assignment balancing during the later rounds of a multi-round process with variable light/heavy workloads with niche area specialties) or may have some sensitivity (e.g., who doesn't get their reviews done on time). If we're going to be formalizing an advisory body (distinct from the informal advice chairs invariably receives from many parties) I think one comprised of the two or three most recent generations of conference leaders would probably be best suited for providing this kind of advice.

Hi Mark,

Anja Feldmann arranged a Shadow TPC at SIGCOMM'05:

http://doi.acm.org/10.1145/1070873.1070889

More recently, there is a shadow TPC at CoNEXT:

http://doi.acm.org/10.1145/1355734.1355750

We organised two shadow TPCs for CoNEXT 2007 and CoNEXT 2008. There two shadow TPCs were educational TPCs, i.e. TPCs were young researchers learn the operation of a TPC by doing the entire selection process. Such an educational TPC cannot be used to evaluate the fairness of the review process in the official conference as the TPC members in the shadow TPC are not selected as in the real TPC, furthermore in the shadow TPC we did not request additional reviews outside the TPC as a real TPC would do.

I think Stefan has a good point about building a steering committee for SIGCOMM. I'm the PC chair for PAM this year, and I find the PAM SC extremely useful. In PAM, the SC is composed of the past PC chairs and general chairs. So, they have often encountered the same issues I'm having. As soon as I accepted the job, the SC sent me an email detailing all the tasks of the PC chair, including the guidelines usually used to pick PC members. These guidelines were not too far from what the SIGCOMM EC is proposing, but they were presented as advice. I did follow them and added other criteria that seemed important to me. Before inviting PC members, I sent the PC composition to the SC to verify.

I like the SIG initiative to publish the guidelines for PC composition to the SIGCOMM community. This will allow all of us to participate in building them. I just think that they should be presented as an advice and not a rule. Once the PC chairs know the expectations of the community for the PC composition, we should trust them to put together a strong and balanced PC. If there was an SC of past chairs, they could provide feedback throughout the process, including on the composition of the PC.

First, claiming randomness is unfair given the consistent success of a specific community at sigcomm, and of a certain style of papers. and this success is not due to misconduct or bad intentions of the PC or PC chairs, but to the fact that a specific communty is involved in the sigcomm PC and that this community has similar tastes for papers. Only diversity could fix this.

I agree that there is no platonic quality of papers. Sigcomm likes a certain type of papers. maybe we are simply wrong to try to change that :-) maybe this is what sigcomm is about. Now, what is wrong is that the carrier of a PhD student or a young faculty relies so strongly on SIGCOMM!

I see the role of a SC to help maintain the conference memory, which might not be always easy. By experience, it is difficult to maintain data about PC member performance :-). But it is damn useful! Also best practices are useful to know. Now, it should be natural behavior of the PC chairs to ask for feedback. I find it amusing when people claim "i didnt receive any advice". well, maybe they had to ask for some first :)

a small comment on PC chairs not submitting. It is healthy not to submit. But I find it unfair for co-authors. When I was at CCR, I never published a single article. I asked my co-authors to remove my name. Sometimes co-authors did not accept and we had to find another venue. I remember having the same issue couple of years ago with sigcomm (one of the few years where PC chairs decided not to submit). we ended not submitting because we considered unfair not to have our co-author on the paper while he was co-chair. given the number of venues, not submitting at a specific event should be easy. what makes it difficult though is the importance of a sigcomm paper on a resume!

summary: diversity and communication should be able to fix all problems :-)

I like Stefan's idea of a separate steering committee for the SIGCOMM conference, just as the other SIGCOMM-sponsored events have their own steering committee. The SIG executive committee acting as a de facto steering committee is arguably an artifact of the days when the SIGCOMM conference was the primary (only?) event the SIG sponsored.

As the semi-independent events have proliferated, having separate steering committees has seemed an appropriate way to spread the workload and give each event its own emphasis and personality. (In most cases, the events came under the SIG's umbrella with such a steering committee -- whether official or de facto -- already in place, but that needn't impede our creating a separate one for the "main event.")

-- Jen

Yes! I want to thank Christophe for advancing a position here on both unfairness and the source of it (that is the topic of this blog post after all).

One of the things that has long bothered me about these new rules is that they were created suddenly, without explanation and seemingly without public input (if Ion and I missed some venue where this came up I stand corrected, but I know at least that the two of us had no idea this was going on and thus provided no input). We all dithered a quite a bit at the business meeting, but no one seemed willing to put forth a rationale for why this was going on. Now we have one. That doesn't mean I agree with the argument, but now at least we can have an honest discussion about it :-)

> First, claiming randomness is unfair given the consistent success of a specific
> community at sigcomm, and of a certain style of papers. and this success is not
> due to misconduct or bad intentions of the PC or PC chairs, but to the fact that
> a specific communty is involved in the sigcomm PC and that this community has
> similar tastes for papers.
To be clear, I wasn't offering the canard that program committees are entirely random (I would never serve if I believed this, and I've served my share). Indeed, I think I claimed that different PCs could select a different set of high quality papers out the same submissions (ditto the same PC perturbed in reviewing assignment, time of year, caffination level, etc). My expectation in fact is that most PCs would select most of the same top ranked papers but that there would be a fair amount of variation around the second half (some in the top half too... I've been in PC's that ranked a paper in the top 10 that was submitted unchanged from an equivalent conference where it was in down in the 50s). However, my main claim was that I haven't seen any evidence of systemic bias -- and I've gone looking. The opposing hypothesis offered by Christophe is that papers authored by particular people, from particular institutions or in particular styles are preferred -- for reasons other than their differential quality/contribution -- over papers by other individuals, from other institutions or in other styles due to homogeneity in the makeup of the program committee. While I cannot categorically deny the possibility of this, I also haven't seen strong evidence for it. Can we suggest a test? I've tried the obvious ones and didn't find anything. Seriously, I think this is crux of the issue -- if we think there is bias, implicit or otherwise, then lets go look for evidence. Otherwise we're all blowing smoke.

> Only diversity could fix this.
So I think the real issue here is "diversity in what dimensions?". In my data I don't see any evidence that gender, institution or country of origin play a significant role in distinguishing reviewing scores for particular papers. That's not to say that we shouldn't try to achieve diversity across those dimensions, but rather that they don't seem to, by themselves, hedge against any particular reviewing bias. I suspect that a bigger issue is likely to be the particular specialties of the reviewers (and indeed, I can imagine an argument for trying to track/balance this kind of information -- awkward though it might be).

In my experience, a major source of non-determinism in the SIGCOMM PC process is the tremendous breadth of topics that we consider to belong under our umbrella. There are quite a few topics that require deep sub-area knowledge to evaluate (e.g., metarouting, wireless PHY/MAC issues, botnets, sensor nets, high-speed packet processing, LRD/wavelet/multifractal, etc.) While reviews from external experts are always helpful, these people can't norm against the rest of the papers (indeed, my experience is that most external experts are MUCH more negative than the PC in general for this reason) and generally can't be available for the PC meeting itself. Thus, there is a big difference in how "mainstream" topics get treated in a PC meeting -- many people can understand the topic/contribution even if they didn't review the paper and much of committee is involved in the discussion -- and how "niche" areas are treated, where much more deference is given to the domain "experts" since the rest of committee really doesn't understand the contribution let alone the material. Thus, I think there is much more variance in outcomes of niche area papers (positive and negative). FWIW, as someone who frequently submits in a niche area -- security -- I've experienced this variability first-hand (almost entirely on the negative side in my case; I haven't been able to get a security paper published in SIGCOMM since 2000). This is exacerbated in the current light/heavy structure because it becomes that much harder to balance competing demographic diversity interests and maintain a quorum of experts in every known subfield (note that the alternative of having a huge PC meeting tends to be a disaster for different reasons and a multi-track PC is a place that SIGCOMM has never gone). I contrast this to two other communities I serve in: systems and security. In these fields most submissions fall within a common "cannon" that everyone is responsible for and thus many more PC members are able to provide input on each paper in the PC meeting. The SIGCOMM PC generally doesn't have this luxury and I think it is unlikely to change simply because this community is so broad.

> I agree that there is no platonic quality of papers. Sigcomm likes a certain type
> of papers. maybe we are simply wrong to try to change that :-)
I also think its important to recognize that the "type" of papers that SIGCOMM likes changes over time. The first SIGCOMM paper I helped write was "The end-to-end effects of Internet path selection" which was really a performance justification for overlay routing. However, we received advice before submitting, reinforced by the content of the paper reviews themselves, that the SIGCOMM community was not friendly to the concept (BTW, this is why that particular paper is careful not to reach any explicit conclusions although the data itself is crystal clear). Fast-forward a few years and this becomes completely uncontroversial and overlay/DHT papers cover SIGCOMM like a plague. IMC was founded because of a perception that SIGCOMM wasn't taking measurement papers, but indeed its not clear that this has been true in any consistent sense (we took 5-7 papers this year that could have appeared in IMC). For a long time SIGCOMM was not viewed as friendly to wireless papers, but over the last few years there have been tons of wireless papers (even some not authored by Dina Katabi). At the same time there are clearly parts of networking (e.g. optical, switch scheduling) that currently are more at home at other venues like INFOCOM. Its hard to be all things to all people and still maintain a cohesive community. I have no advice here, but I personally would be disappointed if SIGCOMM tried to me more like INFOCOM.

Minor other stuff:

> Now, what is wrong is that the carrier of a PhD student or a young faculty relies
> so strongly on SIGCOMM!
FWIW, I agree completely and I think it creates perverse incentives all around. But what to do about it? How about we all refuse to hire graduating students with more than 2 SIGCOMM publications? :-)

> Now, it should be natural behavior of the PC chairs to ask for feedback. I find it
> amusing when people claim "i didnt receive any advice". well, maybe they had to
> ask for some first :)
I've yet to meet one of these PC chairs who don't ask for advice :-) Certainly, Ion and I talked to quite a few people including previous PC chairs and Dina approached me for input as soon as she and Luigi were chosen.

> a small comment on PC chairs not submitting. It is healthy not to submit. But I find
> it unfair for co-authors. When I was at CCR, I never published a single article. I
> asked my co-authors to remove my name. Sometimes co-authors did not accept
> and we had > to find another venue. I remember having the same issue couple
> of years ago with sigcomm (one of the few years where PC chairs decided not to
> submit). we ended not submitting because we considered unfair not to have our
> co-author on the paper while he was co-chair. given the number of venues, not
> submitting at a specific event should be easy. what makes it difficult though is
> the importance of a sigcomm paper on a resume!
I have strong feelings on this one. Consider that many members of our community are faculty who have students. It is inherently unfair that a student should be penalized in any way just because their advisor happens to be chair (or on the PC committee, or an EC member, etc). I would certainly never serve on a committee that had such rules. This leads to the other option that you describe above: removing the chair's name from the paper. I find this even less satisfactory although I know of chairs who have done this in the past. To me it seems like sophistry.. either you helped write the paper or you didn't. If I remove my name this doesn't change my interest in the paper, but it makes my interests invisible to any scrutiny. Thus, the only reason do this is perception (which I think is a crappy reason to do anything, but I recognize that well-meaning people can disagree on this), but I think it even fails this test. Everyone knows the score when your Ph.D. student gets in a single-authored paper on a topic you specialize in.

Stefan,

The idea of a steering committee for the SIGCOMM conference came up in Seattle and it seemed like a good idea to me at the time. Some people on the EC also discussed it after the conference and the reaction was generally positive.

This is a lot of devil in the details and I worry a bit that this whole issue has gotten so heated that it will be hard to retrofit an SC in place smoothly. But generally I like the idea of separating SC functions from the SIGCOMM EC.

> I worry a bit that this whole issue has gotten so heated that it will be hard
> to retrofit an SC in place smoothly.
I think there are two separate issues at play here. The first, is what kind of body is best suited to provide advice/feedback to those running the SIGCOMM Conference. Here I respectfully feel that a committee made of recent conference chairs is likely to offer to most benefit since they can speak from modern experience. This seems easy to do and relatively uncontroversial. The second issue is what kind of body is best suited to making changes in how the SIGCOMM Conference is run and what you think they should do (and on what basis). I think this one is a lot stickier and reflects much of the controversy here. How one decides the second issue is likely to impact how one feels about the first issue. For example, if one arrives with the presumption that the PC ecosystem itself is the problem, then giving conference governance to a steering committee composed primarily of past PC chairs will be like letting the wolf guard he hen house. If you feel otherwise, then having a unique kind of governance for the SIGCOMM conference feeds like imposing a solution absent a problem.

FWIW, I don't actually care that much who governs the SIGCOMM conference (although I suspect it would make the EC's life easier if they didn't) but I do have a strong bias towards us making decisions that are based on concrete arguments that can be defended and are falsifiable.