How to Review

Process Considerations
What to Consider in a Submission
Whether to Correct Grammar and Spelling
Editors: What You Can Do in Return for Referees
Sample Referee Forms
Your Comments, Please


This guide concerns refereeing for conference and journal articles. It follows the philosophy that the reviewing process benefits the author as much as the editor. Authors deserve detailed feedback in a non-hostile tone, regardless of whether you recommend rejecting, revising or accepting their work. Do the quality of review that you wish others would do for you. By agreeing to review, you take on the responsibility of doing a thorough job. If you cannot commit to this, you should notify the editor at once and return both the paper and reviewing package. Editors understand that you may not have the time to do a thorough review, but are unforgiving if you commit to a review and then do a poor job. Also, good reviewing does not go unnoticed by editors, and can establish a useful reputation for you over time.

Authors also have a responsibility to reviewers. Reviewers volunteer time from busy schedules to conduct thorough reviews, and are happy to do so for well-thought-through pieces. If you submit premature work, not only will you annoy your colleagues who take the time to referee, but you will gain an unsavory reputation over time. Researchers work in rather small communities and reputations spread.

Process Considerations
Plan to read the submission three times: the first to get a feel for the it, the second reading the paper in depth, and the third to actually mark it up. Fill out the refereeing form right after the third reading, while things are still fresh in memory.

 You can split your review into three sections: (a) the actual refereeing form; (b) general comments on the paper; and (c) specific comments on individual passages in the paper. You could mark up section (c) directly on the paper. Unless disallowed, you can return the editor a copy of the paper with your markup, to be returned the author. Some reviewers make a photocopy of the paper and transcribe their comments clearly on the copy. Otherwise you can type your comments separately, clearly referencing the page, section, paragraph and/or line number.

 You can write the editor a separate, confidential, note if you wish. Be sure to mark it "confidential" so the editor doesn't forward it by mistake. Many refereeing forms have a specific area for confidential comments for the editor. Do not feel obligated to write things to the editor that you do not share with the authors. In general authors benefit from as much as you can tell them.

 Do not be afraid to reject a bad paper; i.e., do not feel obligated to accept a paper which you believe needs more work. Readers don't want to read poor pieces, and authors will benefit from revising and publishing quality work. Papers can be poor for many reasons. Authors may not explain themselves clearly, so the concepts, motivation, background, actual results and contributions do not come across well. The paper may require drastic proofreading. The research may be of low quality or not a significant contribution. The research may be premature to publish. The paper may have been "thrown together" (unsuccessfully) at the last minute to meet a deadline. It may not fit the characteristics of the intended journal or conference. It may too similar to other publications the author has made (though some journals will publish slight modifications of papers that have appeared in conference proceedings---check with the editor if you have any questions). For poor papers you still owe the editor and the author a review (or in the worst cases, a thorough explanation). If the paper is too premature to complete all parts of the review, state this in your comments.

 Be professional and non-hostile in your review. If you get angry with an author (e.g., for doing a poor job or not for proofreading), you can say so -- either directly to the author in your comments or just to the editor.

What to Consider in a Submission
This is a partial list of what to look for when reviewing a submission. The items should help both to guide your evaluation, and form helpful comments and suggestions for the author. Not all are necessary or even desirable for all papers. Be sure to read the refereeing form, as this will give you additional criteria for the particular journal or conference.

 Does the author cover everything promised in the introduction?

 Does the author provide adequate motivation?

 Does the author provide adequate background information?

 Are all descriptions clear? Are the tables and figures clear? Do they make sense on their own or only if one has read the text carefully? Are there too many? Would an additional table or figure help? Would an example help?

 Are the research contributions clear? Are the contributions significant?

 Is the approach clearly explained and well laid out? Does the author justify each of the points made?

 Are the equations, algorithms, methods, experiments, and conclusions: correct, robust comprehensive and sensible?

 Is the research properly ground in the literature?

 Has the author expressed the limitations of the research and the author's approach?

 Has the author performed a complete analysis and drawn insightful conclusions?

 Has the author described his or her future research plans? Is it clear where the research described in this paper will lead and what the next step will be?

 Is the conclusion significant? Is it just a rehash of the paper? Does it provide new synthesis or insights? Does it leave the reader excited about the research, the research domain or the future?

 Does the author use important, current and adequate citations? Are there too many citations? Are any irrelevant or insignificant? Are they at an adequate level for the publication (e.g., research oriented vs. trade journals; technical reports vs. journal articles)? Are there too many citations to the author's own work? Can you suggest any missing citations the author may have overlooked?

 How is the author's writing style? Is it too "dense" to make sense? Does it keep the reader's interest? Is it too informal? Note that an informal style in itself sometimes is very effective in getting a paper's ideas across. Similarly, many authors use humor very effectively in research papers. Only if the informality or humor gets in the way, should it be discouraged. (On the other hand, there are certain fields which enforce very formal writing styles, in which an informal style is deemed inappropriate.)

Whether to Correct Grammar and Spelling
Proofreading includes checking for correct grammar, correct spelling and overall, that a paper "reads well." As you know, spelling checkers check neither grammar or comprehension. Authors should have enough respect for the reviewers and the editors to submit a paper which has been thoroughly proofread. Authors who are not native English speakers (or whatever language the forum allows) are responsible for ensuring that their submission is of the quality a native speaker would submit, even if they must pay someone to help in the editing process.

Nevertheless, as a reviewer you will often find small spelling or grammatical mistakes the author has overlooked (e.g., a typo within a correction made after employing a spell-checker). And of course you may be able to suggest better ways to phrase certain passages in the paper. In all these cases, it is up to you to decide the extent to which you edit the paper. You may decide to correct the first couple of pages, or the first couple of cases of a recurring problem. If the paper requires major corrections and you know a later draft will be reviewed again, you may suggest the author undertakes such proofreading as part of the revision process.


Editors: What You Can Do in Return for Referees
A note for editors. Many referees appreciate feedback on their reviews. Like many other aspects of academia and research, reviewing is a learning process. You may consider sending each reviewer the same package you send the author, i.e., a copy of your correspondence with the paper's author, as well as a copy of each of the paper's reviews (including his or her own, in case the referee didn't make a copy). This makes the reviewer feel more a part of the process and gives valuable feedback.

In addition, many referees are building tenure and promotion files. A written acknowledgment (not email) of the referee's help looks good in these files and is much appreciated.