Bad Referee Reports

Most mathematicians and theoretical computer scientists don't know how to write a referee report. Maybe this is not a surprise, since we don't explicitly teach this in graduate school, and we expect people to pick it up by reading the reports of others. But if most people don't do it well, how do we expect young professors to learn?

Good reports should

1. put the paper in context - is the subject well-studied? Or is it a backwater where people haven't worked in years? Will people want to read it?
2. evaluate the paper - Is it a real breakthrough in the area, or just one in a series of similar results? Does the author introduce some new useful technique?
3. evaluate the writing - is it clear? How could it be improved? Can arguments be restructured to be simpler and clearer? Are too many important subresults left to the reader?
4. evaluate the bibliography - is it complete enough, or (in the other direction) are many irrelevant papers cited?

Good reports should be specific. Don't just say "the writing is bad"; give specific examples of bad writing and how the writing could be improved.

Here is an example of a really bad report:

This paper is of absolutely no interest. I showed it to my colleague, Professor X, and she agrees. I recommend rejection.

A good referee report should be useful to the author. This report doesn't tell the author anything that he/she can use to improve the paper. Is it bad because the problem addressed is too trivial? Or because the results are already known? What is an author expected to do after receiving a report like this? Commit suicide?

Here's another example of a bad report:

Tiling problems have been studied for many years. They are of great interest in combinatorics and logic. This paper is a good contribution to the subject, and I recommend acceptance.

A good referee report should be useful to the editor, too. This report doesn't tell the editor anything useful! Are the results really deep and novel? Or is it just another in a series of similar small results? Not only that, a report like this suggests strongly that the referee didn't really read the paper with care, and just skimmed the paper in a few minutes. Are there really no papers that the author missed citing? Are all the equations really correct in all respects? Is there nothing that could be improved?