Saturday, October 13, 2012

The Experts Disagree

Is it just me, or is the quality and speed of refereeing deteriorating lately?

I know that everybody's busy. I know that for many professors, class sizes are getting bigger and take more time. But I've had three papers out for refereeing now where two have no decisions after 10 months and 12 months, respectively, and the other -- a paper of only 6 pages -- got reports only after 6 months.

Editors should be more diligent about pursuing reports. In the journal I edit, if we don't get a report after 2 months, we send a reminder, and if we don't get any response after 3 months, we start looking for another referee. The result is that it is extremely rare that a paper takes longer than 6 months to get a decision. A year is unheard of.

Speaking of that 6-page paper, we had submitted it earlier to another journal -- I'll call it "Journal A". After 4 ½ months, we got two reports, one of which I excerpt below:

"This trivial observation is claimed to be the main result of the submitted paper..."
"Although the simple argument provided in the paper is wrong... it could be easily corrected."
"However, the "proof" given there does not even contain any correct idea, which could be used to prove this claim..."
"In conclusion, the paper does not contain any original results..."

This report went on to claim that one of our results was a simple consequence of a known theorem, and then proceeded to outline a wildly incorrect argument supporting this.

Needless to say, on this basis of this report, the paper was rejected, even though all the assertions about the proofs being incorrect or that the last result easily followed from known results, were wrong.

In fact, I think this is a good model of how not to write a report. If you, as a referee, claim that an argument is wrong, it is definitely your responsibility to be specific about what is wrong, not make vague assertions like those above. And you also need to be a little more modest! If something appears wrong in a paper, maybe it is wrong. Or maybe, just maybe, you have simply misunderstood it.

We then resubmitted our paper to "Journal B". After 6 months we got two reports, one of which I excerpt below:

"These results are very interesting and the proofs are correct..."
"I think the construction of the Thm 6 is really brilliant..."

The experts disagree, I guess.


katatrepsis said...

I am just pleased that I don't have to deal with varying reviewer opinions in an editorial context (yet!). I have had a number of papers reviewed by experts who differ wildly in their evaluation of the study. You are inevitably sampling from a multi-dimensional distribution of reviewers who vary in quality, expertise, motivation to review, and I always wonder how representative those samples are.

I wonder what your stance is on Peerage of Science (

Anonymous said...

I've had mostly good experience with peer review, though yeah, it can take a while.

Anonymous said...

SICOMP takes about 1.5 years on average to get back to you.

Anonymous said...

Is withdrawing your paper an option, or is that drastic or looked down upon? One and a half years seems intolerable!

Jeffrey Shallit said...

The problem with withdrawing a paper is that students depend on publications to get admitted to good Ph. D. programs and to get jobs. When to withdraw and cut losses is a good question that I am unable to answer definitively.

Gerry said...

You were lucky that Journal B didn't send the paper to the same referee Journal A used.