A good example of the sort of extra effort made was the use of student
volunteers to meet arriving attendees at Delhi airport. Finding your
way through a strange airport after a long trip is never that much
fun, and being met by a student holding a large "SIGCOMM" sign made
the process of getting out and into a taxi completely painless.
The 5-star Meridien hotel was a very good conference venue, and
certainly presented no great challenges to the conference
attendees. A great deal of work (and some tense negotiations with the
hotel) went into setting up the conference network, with Cisco
donating the services of a larger-than-normal support team to keep it
running, and it seemed to work very smoothly.
There were a few concerns pre-conference as to how the venue would
affect attendance. Compounding the issue of Delhi's distance from the
U.S. (which remains the predominant source of attendees, although not
by much) was the relatively late date of the conference. We try to
rotate among the 3 weeks before the first Monday in September; this
year we were as late as that schedule allows. This led to conflicts
with teaching schedules for some fraction of our attendees. With all
that, we ended up with numbers that were close to last year*: 460
conference attendees plus 40 workshop-only attendees, which is almost
identical to the number in 2009, and within 20% of the last US-located
conference. The geographic distribution was, predictably, rather
different than prior years, notably in the increased attendance from
India (approximately 150). One very impressive achievement from the
organizing committee was the securing of unusually large corporate
support, which enabled them to offer about 180 student travel
grants. In all, 230 students attended the main conference, or about
50%.
At the Community Feedback session, there were some comments about
the difficulty of getting to Delhi from the U.S., and of the high
travel cost affecting the number of students that could affordably
attend. Of course, any location will have high travel costs for some
part of our community, so the sort of rotation that we now do (North America,
Europe, Wild-Card) seems close to the fairest thing possible. The
SIGCOMM community is far more geographically diverse than it was in
the past, as a quick look at author affiliations in the program will
attest. Clearly student travel grants help mitigate the travel cost
issue to some extent and we hope corporate support will continue at
the high level achieved in 2010 to enable similar numbers of travel
grants in the future.
*Figures in this post taken from registration logs are approximate,
rounded to nearest 10.
I hesitate to begin listing highlights of the conference, for fear of offending someone by failing to mention that their paper or contribution, but here goes. I don't think anyone will be offended by my saying that Van Jacobson's talk on content-centric networking was eagerly anticipated and got the technical program off to a flying start. Many of us in the SIGCOMM community have had the chance to hear Van talk about CCN in an informal setting, but this was the first appearance of the project at a formal conference, and I think many were inspired to go and read the full paper.
Prior to Van's talk, Ratul Mahajan, recipient of this year's Rising Star award, gave the keynote. His talk contained a number of thought-provoking insights on how to pick and pursue a productive research agenda, drawing on Ratul's personal experiences. It was striking to hear someone who is in the early stages of his career sound like an elder statesman in the field.
I participated only briefly in the student workshop (I came directly from the airport to the closing panel so I could share my sleep-deprived insights on research careers - a very interesting and interactive session). This workshop has been a fixture of CoNEXT from the beginning, and is an effective way of bringing students into the main conference and giving them a chance to present their work at an early stage. We hope to use travel grant money to increase participation in this workshop next year.
The local arrangements and logistics were excellent. The meeting facilities at the National Research Council were very good, with the auditorium providing great acoustics and visibility. One aspect stood out in particular: the conference banquet. It was a beautiful venue (the Teatro Capranica) and the food was outstanding, but the most impressive thing was that the organizers had only 10 days to line up the venue, when the original venue's booking was pre-empted by the City. The ability to respond and produce such a high quality event under duress was a testament to the skill and dedication of the conference committee.
All in all, I think we have another top quality, broad conference in the SIGCOMM stable. Of course, the quality ultimately depends on the papers, so I hope that our community will increasingly think of CoNEXT as a desirable venue to publish. CoNEXT 2010 will be in Philadelphia, November 30 - December 3. Mark your calendars and start thinking about the papers or posters you might be able to submit.
]]>Technical Steering Committee, Draft Proposal
Charter
The role of the Technical Steering Committee is to oversee the technical aspects of the conference. It selects and oversees the PC chairs, and it serves as a knowledge repository for best practices related to the PC and paper selection process. The TSC is expected to maintain publically documented policies and to develop policies with input from the entire SIGCOMM community.
Composition
The members and chair of the TSC are appointed by the SIGCOMM Chair. There are 6 members, and membership lasts three years. Every year, one of the PC co-chairs is appointed and serves for the next 3 years. In addition, each year the SIGCOMM Chair appoints a representative member of the Sigcomm community to complete the TSC who then serves for 3 years. Such members should be senior people with some history in the SIGCOMM conference community and experience in conference organization
or program committees. (Note: in the start-up year, 6 people will need to be appointed with terms ranging from one to three years; in steady state, 2 members' terms expire and 2 new members are appointed each year.)
Rationale: past experience indicates that a small committee is likely to function more efficiently. One PC chair from each year is sufficient to provide input from that year's process, and the TSC is of course free to consult the other chair (or chairs from prior years). The presence of outside members is intended to ensure that the committee is open to new ideas while the presence of previous years' PC chairs aims to maintain knowledge of what has and hasn't worked well in recent years.
Functions
The TSC has ultimate responsibility for the functions described below (i.e., it is not purely an advisory body). It is expected to gather input from the SIGCOMM community in the process of performing these roles.
1) Select PC chairs.
2) Give guidance and advice to PC chairs on the paper selection process. The TSC is the repository of knowledge about what has happened in previous conferences and PC meetings. (Ideally the TSC will maintain private records to facilitate preserving and transferring knowledge).
3) Provide assistance and recommendations on PC makeup, including PC composition and the pros and cons of specific PC members. PC chairs retain final decision on PC membership but are required to consult with the TSC before making their decision.
4) Set policies and provide advice related to award paper selection, conflict of interest policy, dual-submission and plagiarism.
5) Maintain an active dialog with the SIGCOMM community related to technical aspects of the conference (e.g., topic coverage, paper quality, etc). An open dialog via a website or blog is a good idea.
6) Give guidance/advice on other aspects of the technical program and process. For example, integrating on-line or other possibly-new Internet-based components, panel discussions, new reviewing methods, reviewer feedback, etc.
To complete the archives, please consider loaning your old CCR editions to the ACM DL so that they can capture the early work of the community for the record. Contact Craig Rodkin (rodkin@hq.acm.org) for further details.
]]>I don't think it much matters that a poster presentation has a different format and maybe a shorter representation in the proceedings. What matters is the core technical content, whether some implementation details and graphs are omitted or not.
This discussion is made more complicated by the fact that the treatment of poster submissions, in general, differs greatly between conferences. Some subject such submissions to very limited review, e.g., the student posters at CoNext, while others are reviewed at the same depth as regular papers. For some conferences, some sessions are more or less randomly designated as poster sessions, while others feature oral presentation, with Globecom being one such example. Other conferences move papers that fall just below the cut-off into poster sessions, converting them to short papers. Finally, the proceeding version of a regular and poster paper may be the same or the poster paper may get a one-paragraph abstract - and anything in between. With these distinctions, it is too simple to assume that all posters represent very early work.
One could argue that presentations in multiple venues ensures that good work gets to be known, particularly as the community is increasingly fragmented into ever more narrowly-targeted conferences.
Before considering this specific case, it might be worth going back to the question of why the community, in general, frowns upon double submission. I can think of four reasons:
(1) Avoid wasting reviewer resources, as we don't want to have very similar technical content reviewed six or more times, without the author having had a chance to incorporate the earlier feedback.
(2) Avoid "double credit" - authors shouldn't pad their publication count by republishing essentially the same content except under well-recognized circumstances.
(3) Listener boredom - nobody wants to attend a conference where half the presentations are re-runs.
(4) Wastes trees and library shelf space - why print the same or very similar material twice?
I believe that the last three reasons have become somewhat less important, and listener boredom doesn't factor if one of the formats is a poster, since the attendee can easily skip a poster that represents work they are already familiar with.
This leaves the first reason. Good reviewers remain a scarce resource, and we should husband such resources. Thus, I believe that, in general, such concurrent submissions are something we do not want to encourage or permit. Once it becomes known that such concurrent submissions are acceptable, there will be a strong temptation of more and more authors to submit papers to as many conferences as possible. However, if the poster does not consume any or minimal reviewer resources, the objections seem less strong. We routinely expect work under submission at a conference to be presented as posters at various local events, such as industrial affiliate or grad student research fairs, for example.
Recognizing that this may cause some confusion, one reasonable approach may be to clearly designate minimal-review poster sessions as such and for the poster event to indicate whether material under concurrent review elsewhere is acceptable for submission. The default assumption should be that it is not. Authors should also be encouraged by the conference organizers to cite such "publications" appropriately, such as "Conference X work-in-progress poster session" or "Conference X PhD research overview".
Any such policy should be coordinated with other organizations, such as TCCC. In particular, IEEE Infocom has a long-standing double submissions policy [http://www.comsoc.org/confs/ieee-infocom/policy.html] that is not entirely clear on this particular scenario.
]]>This problem is not unique to reviewing papers. For example, in our discipline, grant agencies face the same problem of receiving far more qualified proposals than can be funded. (There is a difference in that the conference selection problems are largely self-inflicted, as it is generally far easier to increase the number of accepted papers than to increase the NSF budget.)
As with all such processes, the details of the process remain, by necessity, confidential, so it is necessary that authors have some assurance that their labors are evaluated fairly, even if they cannot hear the detailed discussions or weighing that takes place behind closed committee doors.
Thus, the community has developed a set of fairness rules that try to limit both the very real possibility and the perception of favoritism. Such rules include conflict-of-interest stipulations that exclude advisors, collaborators and colleagues from the same institution from reviewing papers. Some conferences also impose special conditions on submissions from technical program committee chairs and even program committee members, from excluding them altogether to raising the bar for such papers. As a note, it is generally much easier with such rules to prevent biased positive reviews than unduly negative ones, a problem that the computer science community has acquired a reputation for.
After excluding reviews based on conflicts-of-interest and perceptions of insider advantage, the third approach to reducing biased reviews is to use double-blind reviews (DBR), where the author identities and affiliations are hidden from the reviewers. Single-blind reviews (SBR), on the other hand, simply hide the reviewer identities from authors, with the goal to allow reviewers to provide frank feedback without worrying about personal or professional repercussions. (Experiments with open reviews, where all identities are disclosed, have been performed at Global Internet 2007, but are beyond the scope of this discussion.)
Below, I will try to highlight why I believe that DBR should be seen, at best, as a tool that mostly maintains the important perception of fairness, rather than dramatically change the review outcome. Thus, similar to the often-derided "security theater" at airport checkpoints, DBR may be partially "review theater" and may not actually help in a scientifically-measurable fashion and is easily circumvented by a determined adversary, but serves an important role in upholding the norms and values of the community for highly-selective conferences. However, as discussed below, DBR can cause serious side effects if it forces or encourages authors to obscure the relationship to their earlier work. Thus, I argue that if community norms and customs call for DBR, it should be handled judiciously, recognizing its limitations and addressing the practical problems.
Many ACM and, more specifically, ACM SIGCOMM conferences use double-blind reviewing (DBR). In double-blind reviewing, the paper submitted for review does not contain author names. As in single-blind reviewing, the authors are not told who reviewed the paper. While not always clearly stated, the goal of double-blind reviewing is to remove bias in the review process, particularly the perception that work by well-known ("prolific") authors or from highly-ranked institutions will be given more credence than work of similar quality from other authors and institutions (Some claim the opposite effect: Reviewers may tire of the work of certain individuals or hold prolific authors to a higher standard.) DBR may also make it less likely that personal likes and dislikes color the review, outside the normal range of conflicts of interest that should prevent review assignments. Some also claim that double-blind reviewing may help reduce gender bias, although there does not appear to be any systematic study of this effect in computer science papers. In general, the quantitative evidence for the impact of double-blind reviewing on paper selection (or on perceived fairness) is not particularly strong and the result may depend on the metrics, as illustrated by the analysis of SIGMOD submissions.
We can view DBR as related to the, relatively recent, custom in the United States to omit a candidate's birthday and picture from a resume,0 to avoid racial and age bias, or a preference for good-looking candidates, during the pre-interview selection process. The evidence for the effect of these appears to be somewhat stronger, however, than for DBR.
DBR may also combat reviewer laziness when reading papers authored by recognized experts. For example, if the author is known to the reviewer, a reviewer may be tempted to skim the analysis since the author presumably knew what he or she was doing, and is likely to be more versed in the mathematical details than the reviewer. This may not only prevent discovering mistakes, but, more likely, has presentation errors, such as missing definitions, truncated equations or inconsistent use of variable names, slip through during the review process.
Different conferences are likely to be affected to a varying degree. For example, the normal "noise" in the review process already makes it difficult to have reproducible acceptance decisions for conferences that only accept one in eight or ten papers. For such highly-selective conferences, reducing the impact of author identities on the fate of borderline papers may be helpful, even if it would make almost no difference for slightly less selective conferences. However, it may also lead to false complacency - "we're doing double-blind reviewing, so the process must be fair.
The concern about the effectiveness of DBR is summarized in a recent SIGCSE bulletin article: "Submitting papers anonymously does not work as well as one would hope. Authors who report on `ongoing work' who have prior publications can be discovered with a quick search on the Web. Papers often include clues such as grants on which the work was supported, or names of proprietary software used for the work, that make such discovery easy if not trivial. In a world where most Computer Science education professionals have web presence, and finding information on the web is increasingly easy, it may be time to reexamine the need for, and the advantages of anonymous submissions."
Even without DBR, it is unlikely that an obviously wrong paper by a "star" author will be accepted in a well-run conference or truly interesting work will not be published in some other conference even if it is rejected at the first conference it is submitted to. There is hardly a dearth of networking conferences to submit papers to, after all. Unfortunately, the stakes for reviewing have gotten higher for authors, as the number of papers accepted at highly-selective conferences has become a metric used to evaluate faculty and tenure candidates for hiring and promotion. Particularly in that context, relying on DBR to give a process that has a fair amount of randomness the sheen of scientific rigor, similar to double-blind drug trials, may well cause significant harm.
As noted, one oft-cited advantage is the perception of fairness that is helped by DBR. However, the SIGCOMM conference fairly regularly attracts criticism that certain authors or institutions have a leg up, despite having used DBR for many years.
The debate about review policies may obscure a larger problem. One could argue that we are probably exceeding the resolution of our evaluation tools, so that the fixation with double-blind reviewing and other procedural considerations, such as author responses, merely obscures the inherent limitations of a process that strongly depends on the choice of the reviewers and the dynamics of a technical program committee meeting, even if there was an objective standard what the "best" papers were in a particular conference year. For example, while not meant to measure such randomness, the SIGCOMM 2005 shadow PC experiment yielded a high variation in the set of accepted papers, with only roughly half the accepted papers appearing both in the set chosen by the shadow TPC and the real TPC.
However, this note is only partially about the benefits of double-blind reviewing, but rather how to implement it effectively without causing undue collateral damage during the review process. It should be noted that there are two stages where anonymization may matter, namely the written reviews and the technical program committee discussion. During the latter, maintaining author anonymity can be particularly challenging if the author is on the TPC or from an institution well-represented on the TPC. If suddenly all Stanford University members of the committee are asked to take a coffee break during the deliberations for a paper, it may not take the rest of the committee all that much guess work to identify the likely authors and institution. No amount of text obfuscation will compensate for the coffee break indicator. Even though it may well remove most of the fairness advantages for the borderline papers that get discussed at TPC meetings, I will focus on the written reviews, simply because there does not appear to be a good solution to the TPC meeting problem, except maybe randomly ejecting TPC members to obscure the author institution.
DBR serves a goal, namely allowing reviewers to focus on the work, not the authors. However, in practice, the goal is mostly to remove the identity of prolific authors as there seems to be little difference in removing the names of authors that few of the reviewers are likely to have heard of. (Removing institutional information may still be helpful, as it may be harder to take papers seriously if they are authored by somebody teaching at a community college.)
Once a particular conference has decided to implement double-blind reviewing, the conference chairs and steering committee have to decide how far to push the removal of author-identifying information. The ACM Transactions on Database Systems provides a sample of such rules. Obviously, the paper under review should not list any authors and institutions, and should omit any "incriminating" acknowledgements and avoid revealing filenames. None of these are necessary for evaluating the paper content and can thus be omitted during the review process. These simple steps alone probably anonymize the paper for a majority of reviewers.
However, self-citations can easily reveal the authors with minimal inspection. Since authors routinely cite their earlier work and prolific authors have no incentive to hide their identity, the simple statement "As we showed in [42]" unmasks at least some of the authors for all but the most careless reviewer. Thus, the next step is to ask authors to convert such citations to the third person.
This is probably where the easy rules and guidelines end. Particularly for highly-selective conferences, many of the competitive papers will contain material that has been published earlier in workshops or been discussed at principal investigator meetings. Among other identifying items, it may contain project, protocol or algorithm acronyms. The current SIGCOMM FAQ encourages such an approach: "When reviewing the subsequent, more mature submissions of such work, reviewers are advised to first assess whether there is an adequate additional contribution over the previous, preliminary version of the paper, if so, then reviewers are advised to measure the full-length submission not just in terms of its additional contribution but on its entire content. This policy aims to encourage the development of work while also encouraging publication of work when it is in its earlier stages.'' However, the reviewer can only identify whether there is substantial additional contribution if they know the earlier work, either because they are personally familiar with it or because the authors cite that work. If the authors obscure the relationship to the earlier work, one of two undesirable outcomes are likely:
* Unjustified suspicion of plagiarism: If the reviewer recognizes the work, but does not recognize the project acronym and the text contains no indication of the earlier work, the reviewer may well get suspicious and search for text snippets online, most likely finding the earlier paper. The reviewer then has to confirm with the TPC chair that the paper under review is indeed original work and not plagiarized. At this point, clearly all author anonymity has been lost, and the reviewer has wasted time on a wild goose chase.
* Undiscovered self-plagiarism: Unfortunately, there are indications that self-plagiarism and double submissions are no longer rare. For example, during this year's IFIP Networking conference, where I serve as TPC co-chair, we have had two papers that were verbatim compilations of earlier work by the authors. While plagiarism detection systems such as docoloc can ferret out some of these cases, they are not perfect, they missed in this case a publication in a very recent conference.
The SIGCOMM FAQ (http://www.sigcomm.org/about/policies/frequently-asked-questions-faq) currently recommends essentially a conditional review with the outcomes accept, accept-only-if-common-authors (i.e., not plagiarized) or reject. In my opinion, suspending suspicion of plagiarism while spending an hour or two on a review and possibly considerable time at the TPC meeting seems beyond what one can require of a reviewer or technical program committee. It is unclear whether this particular approach has ever been exercised.
Simply having the authors indicate that the paper contains material from an earlier publication, but not identifying the precise reference, does not help much since the reviewer can then make no judgement as to whether the incremental contribution is sufficient to merit publication. Alternatively, the conference can set up an incremental novelty examination (ICE) board e.g., consisting of former TPC members, that only investigates such cases and declares papers to be fit for review or too much of a cut-and-paste job. To speed up the process, authors would be asked to declare which earlier papers of theirs the new submission is based on, and outline the major differences to the earlier submissions, with that information visible only to the ICE board. (This review could theoretically be performed by the chairs, but since it has to be done quickly to avoid delaying the review process, a larger committee seems appropriate.) While requiring extra effort and possibly delaying the review process, this seems preferable to having the reviewer do detective work. It also seems likely to increase consistency, so that all papers containing previously published results are held to the same standard of incremental contribution.
Such self-declaration may also make it less likely that an author "forgets" about double submission rules or self-plagiarism considerations, so it may be worthwhile even without DBR.
In general, since the effectiveness of blinding will vary with the paper and the set of TPC reviewers assigned, the fairness effect will also be somewhat random. Some reviewers for some papers will suspect or know the authors, others will not. Since good TPC members should be very familiar with the current work in the paper's technical area, it could almost be considered disqualifying if they do \textbf{not} recognize the work of a prolific author in that area.
Given that trying to obscure technical content by editorial changes is unlikely to be effective for prolific authors with high-visibility projects, heroic obfuscation measures seem to cause more harm and wasted effort than they are likely to increase reviewer objectivity. Thus, DBR may be more of a placebo, working mostly because we believe it does, but that should not distract from the need for harder-to-make changes in our publication process.
]]>I believe that the differences are partially due to the fact that conferences differ in their relative independence in relationship to the related professional society, such as ACM, IEEE or Usenix. In some cases, the conference is just a part of the organization, and has really no independent identity. As an example, the IEEE Communications Society "owns" IEEE Infocom, Globecom and ICC, and it seems highly unlikely that, say, the IEEE Infocom steering committee could just decide to move their conference to full ACM sponsorship. Similarly, the notion that ACM SIGGRAPH would suddenly decide to run its own show seems implausible. As a crude analogy, consider Walmart stores or the company-owned Starbucks outlets, while in the education realm, state universities are probably somewhat similar. In the end, the state government calls the shots and sets the tuition, even if the university president has a fair amount of latitude in running the daily campus affairs.
On the other hand, there are conferences that are much more independent, even if they have been affiliated with the same professional society for years. Such conferences could, and sometimes do, decide to change their affiliation or they might even become their own full-fledged organization. As examples, ACM NOSSDAV and ICNP probably could decide to switch sponsoring society, or at least switch the SIG they ask for sponsorship. These conferences seem much closer to organizations such as private universities or religious congregations, where each such organization is run by self-perpuating group of trustees. These trustees do not own the organization and could not sell it for their own benefit, but they could, for example, decide to disband the organization and sell its assets. Unlike in a business, such trustees are not financially responsible for the organization, but may well face liability for negligence or malfeasance.
Regardless of the structure, I would strongly suggest that long-running conferences that set up a steering committee create at least an informal set of bylaws that addresses a few core issues, such as:
Based on my informal survey among faculty in my department, steering committees across computer science differ dramatically in their responsibilities. In some cases, the steering committee essentially hands the reign to the general chairs for the year, and then does not discuss the conference again until it is time to pick the next chair. In others, the steering committee has much more hands-on involvement.
To further complicate matters, some conferences, such as ICNP, have advisory committees. It is important to distinguish their roles and responsibilities. For example, advisory committees may provide guidance, but may not have direct influence on conference location or chair selection, or may simply provide external "gravitas".
While there is a certain amount of flexibility, the basic conference model, discussed earlier, limits what is feasible. As in all organizations, it is probably a bad idea if a group has responsibility without power, or vice versa. For example, if the SIG budget is on the line if a conference sponsored by the SIG runs a deficit, it is hard to expect the elected officers of the SIG to simply leave the finances completely to the conference organizers or some other organization and hope for the best. Also, given that the SIG cannot simply disown their flagship conference, the SIG leadership has some responsibility to maintain quality and other core attributes of the event year after year. For example, having the organizers dramatically raise registration fees may be considered detrimental to the community represented by the steering committee, and thus, it may have to intervene.
There are many ways to run a successful series of conferences, both technically and logistically, but even successful conferences can do better if all four legs of the conference organization work under a common set of assumptions. As many conferences approach runs of twenty or more years, writing down these assumptions may work better than relying purely on oral traditions.
]]>The current set of SIGCOMM officers took office July 1, 2007. Officers' terms are two years, and ACM bylaws provide for their terms to be extended for another two years if the officers agree and the SIG Governing Board Executive Committee confirms them. However ACM bylaws also state that an election must be held if 1% of the SIG members petition for one (SIGCOMM has just over 1800 members). The current officers have agreed to extend their terms, but a valid petition has been filed with ACM requesting an election. Hence we will hold elections in early 2009 for officers with terms beginning July 1, 2009.
ACM guidelines direct the Chair of the SIG to appoint a Nominations Chair, who appoints a committee to propose at least two candidates for each office by December 15. Self-nominations are also possible via a petition process by 1% of the SIG; these must be received by March 15.
We'd like to hear from the SIGCOMM community on whether this is a real problem, and if so the extent to which it is a problem. If there is a problem, what suggestions do people have as to how it might be addressed?
]]>We formulated these as new guidelines. They're available here and here. The new guidelines don't tie the hands of PC chairs in terms of who is invited to the PC; rather they add an advisory step in the process.
We consider these works-in-progress; we're trying them out with the 2009 conference. We're interested in what SIGCOMM members and conference attendees think about these policies.
]]>