SIGCOMM Blog

I believe this is true and the cause is not intentional wish to reject some work but rather lack of openness toward new ideas. I've submitted papers to Sigcomm several times and got rejections. I didn't mind so much the outcome but I did mind reviewer's comments that showed that (1) paper was not carefully read and (2) bias was toward "more solutions" rather than accepting papers that help investigate the problem further.

I think it would be helpful if some kind of author feedback to reviewers could be established - of course, anonymously. I know this is hard but it may result in better reviews in the long run.

It would also be helpful if some clear, visible list of paper requirements could be compiled as guidelines both to authors and reviewers. Such as (1) what kind of topic is welcome (novel solutions vs problem investigation vs solution comparison...), (2) what kind of experimental section should Sigcomm papers have (I was surprised to see a few published lately that had NO experiments whatsoever), (3) some guidelines about sources of data and experimental techniques (I've had a paper rejected from multiple venues because of using RouteViews as source of Internet topology data. But is there any better source? If not, then it would not be fair to reject papers based on this. If yes, let's make a public announcement about a preferred source)

Hope this is more helpful than inflammatory.

> I've had a paper rejected from multiple venues because of using RouteViews
> as source of Internet topology data. But is there any better source? If not,
> then it would not be fair to reject papers based on this
I don't have a particular answer to this question, but it brings up a closely-related issue that has bothered me for some time. Much empirical networking research studies an artifact (e.g. the Internet, cell-phone networks, etc) for which there are both public and private data sources. Any researcher has access to the public data resources, but the private resources are frequently of much higher quality, coverage, etc. and are only available to certain organizations. For example, when you read a paper that describes how they looked at the routing policies used by several hundred of their customers you know there are a small number of outfits that could pull this off and they probably can't share. These results are great and even though most of the community can't repeat the measurements or explore them further, these are still very useful in a qualitative sense. Now, a problem emerges when those with access to the private data sources come to realize that the public resources are insufficiently accurate, broad, precise, etc to discern what's really going on. I've seen this kind of thing happen in areas that range from routing to malware. What is the right thing for the community to do here? Tell those who don't have access to the private data that they should abandon the topic if they can't get into the club? or let "public" researchers compete in analyzing data that is too noisy to support their conclusions? Neither seems very satisfactory.

In my view, the SIGCOMM conference is a closed group of Professors publishing papers in a so called 'highly selective' conference so that they can tout their publishing prowess to their dean in order to secure tenure. I have never submitted to SIGCOMM nor would I want to because it is a waste of time. The papers are usually well written (so that it looks like a selective, high quality, conference), but otherwise, nothing great.

I believe it is indeed a real problem. Seattle SIGCOMM had about 35 papers, out of which 1 (one!) did not have a US affiliated author. Not very open to non-US authors, obviously.

Now, networking research might happen exclusively in the US and China, Europe, Korea or Japan might not have anything interesting to say about it. But somehow, I doubt so.

Compare to the composition of the PC: 44 members, 37 of them from US institutions (the rest: 4 from Germany, 1 from France, 1 from Italy, 1 from Australia).

Further correlation of the PC and the papers: in addition to that single paper with no US affiliated author, there were 3 (three) papers which had at least one non-US affiliated author.

Well, one of these 3 comes from the MPI (which is disproportionally represented in the PC: 2 PC members out of 7 from non-US institution) and another comes from Germany as well (also disproportionally represented vs other countries: 4 German PC members out of the 7 non-US-affiliated PC members).

Correlation is not causation, of course, but it would be nice to have an explanation as to what makes US networking research be represented in SIGCOMM 31-3-1 (US/mixed/nonUS) vs the rest of the world: it does not happen at Infocom for instance; the research budget of the NSF vs the EU for instance, don't bear that discrepency; the industry of networking is not 10 times as big in the US than it is in Europe...

Those numbers do raise some definite questions about openness.

Welcome to readers of Michael's blog) :-)

I hate to inject actual facts into this forum, but...what the hell. So there are a number of different factors that play here. One issue is submissions. For example, in 2008 we had roughly 10% of our submissions with European authors or co-authors. We accepted 3 papers with European authors (4 if you count Temmu's affiliation with HIIT). 10% of 36 papers is 3.6, so we're off by 0.6 of a paper, but that's presumably forgivable no?

Now there's a much bigger difference with Asia. Asia-affiliated authors were listed on a bit under 20% of the submissions, but we only accepted three papers with any Asia-affiliated authors (and only two of these were principally authored by Asian institutions) which is > 2x off expectation. I went back to look at those submissions (predominantly from China and Korea) and I think there's different factor at work in this case. In 2008, these papers appear much more likely to have significant English problems and come from authors with less international publishing experience than the equivalent cohort from Europe or the US. Indeed most of these papers were from institutions that have not published papers in SIGCOMM, INFOCOM or GLOBECOM in the last 3 years (that was as far back as I was willing to look). Given this, I'm less surprised at the outcome. I also expect these authors to improve with experience and become more competitive over time.

Now as another check on the issue of bias, in both the case of European papers and Asian papers, I looked to see if the scores they were awarded by US PC members were substantially different than for non-US PC members. No, it turns out.

Finally, I must admit that I've always found this kind of debate puzzling. This is a research community, not the Olympics right? Indeed, I find the "nationalization" of research to be somewhat ridiculous given how much people move. People always focus on "country of residence" but in fact this frequently differs from country of origin (indeed, the majority of the PC as well as the PC chairs were born outside the US) which in turn frequently differs from country of education. Today many, if not most, of my colleagues in the US were born abroad and our students are increasingly looking for academic jobs outside the US.

Stefan, thanks for the facts. That's always welcome in such a discussion.

So the very low number of papers of, say, European authors in the program, is in line with the very low number of submissions. The question is: why don't they submit there. This is the point being discussed here: is Sigcomm open or not? If Europe's networking research was 10% in size or quality of the US, then the numbers would make sense. But it's hard to make that case.

Second: "nationalization" is an easy way to express some imbalance. To further the Olympic metaphor, if the US had 10 times as many medals as the EU, then people would start questioning the "training" of the US athletes, given the somewhat comparable population numbers and economic development. Further, "nationalization" is a convenient proxy for the diversity of the conversation. The more are welcome at the table, the more interesting the results.

Obviously, the bias is not individual, since PC members even from US institutions come from the world over, but rather, systemic. Not sure what makes it so that European authors won't submit there. This year's PC seems much more balanced. it would be interesting to see if only so few non-US papers were submitted. Maybe the correlation is between PC composition and submissions, since PC members will distribute and be encouraged to submit.

Also, sigcomm can chaperon papers, so language issues (especially for Asian submissions) can be fixed and should not be a focus in the reviewing.

Obviously, the bias is not individual, since PC members even from US institutions come from the world over, but rather, systemic.

I have a Real Problem with this line of argumentation :-) There was a hypothesis advanced -- that SIGCOMM was not open to non-US authors -- that I attempted to rebut and/or explain. In response, rather than consider the possibility that, in fact, there may be no substantive country-of-residence bias (in the literal sense of the word bias) the new hypothesis is that that systemic bias discourages European authors from submitting. It could well be that you're right, but do you see the challenge with this line of reasoning? If we always change the question in response to an answer, then the underlying hypothesis of bias is non-falsifiable. If we're playing this game, I'm happy to simply concede the issue and admit that that the cabal that's been running SIGCOMM for years has consciously and systematically beaten down authors residing outside the US to ensure that they do not submit in the future (indeed, when an author changes residence and moves into the US, we surreptitiously inject them with mood stabilizers to overcome the artificial inhibitions we have previously instilled)

Note that one could draw these divisions arbitrarily and reach similar conclusions. Why are there not more submissions/acceptances from southern institutions in the US? Why aren't there more submissions/acceptances on security? Why aren't there more submissions/acceptances from full professors rather than associate and assistant professors? To say nothing of the obious bias against dark skinned European women and US men of hispanic descent. I simply don't think this kind of stuff is useful... sigh.

Now, wrt the issue of the PC makeup, one thing that is under-appreciated is how many people turn down requests to be on the PC. In the 2008 case we got significantly more turn-downs from Europe than elsewhere (in some cases because they had other commitments, in many other cases because they didn't want to go to Berkeley for the PC meeting). This was difficult to recover from since we have a large number of other constraints as well including new vs experienced PC, area coverage, institutional issues and so on (I recall 8 different constraints when we put the PC together). I continue to contend that the main source of reviewing variability is actually in the area domain and not in the nationality domain.... but this isn't a topic anyone is very interested in.

I expect to see more European submissions this year, just as I expect to see more Asian submissions the year after. Where the conference is held has traditionally played a role in how many submissions are received for obvious reasons. I'm not convinced this matters one way or the other however. SIGCOMM has always tended to publish strong papers and I expect it will continue to do so -- from wherever they originate. In my opinion that is what we should focus on -- the papers -- rather than paper metadata (which is decidedly less interesting)

Jelena said, "I believe this is true and the cause is not intentional wish to reject some work but rather lack of openness toward new ideas." I agree with this statement.

A recent paper of mine had, as its comment to the authors, "I just can't convince myself it works." That's it, no reason for the disbelief, in spite of all the elaboration on the system and the evaluation. I got a score of "1" from the reviewer, I'm sure the paper didn't get a chance for discussion. Good for the PCs of 2009, they had one less paper to discuss, more time to eat roast boars.

If the PC is closed to new solutions, what sort of conference is Sigcomm becoming? The PC quality is horrendous, the terrible ones need to be weeded out.

The result of "opening" up the conference is there for everyone to see in this year reviews. You bring in people who have almost no experience reviewing sigcomm papers and ensure that the majority of the pc/pc chairs is composed of them. One really cannot blame these pc members since they are not exposed enough to understand whats important and it is a pity that they are suddenly entrusted the responsibility of reviewing sigcomm papers. However,the people who are to be blamed are those who want change just for the sake of it. If you really want openness then you need to do it in a more systematic way by first scientifically identifying if there is a need for change and what kind of change is needed. Once that is done, which ever version of "openness" is scientifically proven must be slowly introduced.