How to Be a Better Reviewer (JAWS Event Recap)

I was always told that when you receive a set of reviews, you should look at the decision and then close the email for 24 hours.

If it’s a rejection, process those emotions and get the initial sting out before turning to the comments. If it’s an R&R—even more reason to close the email. Celebrate the good news before souring yourself on all of the work you have yet to do.

Of course, I never do this.

I immediately dive into all the gory details and quickly become incredulous that someone, anyone, could think that there was room for improvement on such a perfect paper! If it wasn’t perfect, after all, I never would have submitted it!

Then I wait my 24 hours.

When I come back and read the reviews a second time, after processing, then I realize that, actually, most of the things the reviewers and editor said were right, and addressing the feedback has always made the paper better.

This is a story of receiving reviews. But to receive reviews—especially reviews that improve the paper—there have to be people on the other side of the screen giving their time and labor to produce them. As academics, though, we are rarely taught how to do this act of service.¹ It is often a skill that we have to learn on the job, over time.

To that end, the Junior Americanist Workshop Series (JAWS) hosted a panel featuring Celeste Montoya, Nadia Brown, and Christian Grose about how to be a better reviewer. Below, I paraphrase the panelists’ comments based on my contemporaneous notes and impressions. All errors are my own.

A Review is for the Author AND the Editor

As a reviewer, you have a dual role in helping the editor (as a subject matter expert in your subfield who can critically evaluate the manuscript) and in helping the author (in providing targeted feedback given that subject matter expertise). When writing your review, you need to speak to both audiences. You probably already have a good handle on providing feedback to the author (unless you’re being jerk about it, in which case, stop doing that), but the panelists provided some good insight into what they are looking for in a review as editors.

The biggest insight to come from the discussion was: editors want to know, plainly, whether the article makes a significant contribution. As authors, we all do a bit of puffing ourselves up. We all say “this article is so timely and important.” But of course, if everyone says their article is so timely and important, it’s impossible for an editor (who is probably not familiar with the subfield) to know whether an article truly is timely and important or whether the author is just saying what everyone says. As the subject matter expert, however, you can critically evaluate the validity of these claims. What editors are looking for is clear advice on whether the contribution is significant and why the article merits (or does not merit) publication.

To that end, all three panelists said to stop writing the summary paragraph at the beginning of the review. I guess we have all been told, at some point, to summarize the argument at the top. My understanding is that it signals to the editor you read the article, and it signals to the author that you understand what they’re trying to do. But the editors said they have already read the article, and it will be pretty clear if you didn’t—even without the summary. So this paragraph ends up wasting valuable time and space in a review that is probably already too long. Instead, the panelists suggested writing a paragraph contextualizing the significance and contribution of the article in your own words. Especially if you like the paper and want to see it published, you can use this paragraph to provide insight about why, and justify the authors’ importance claims. As the subject matter expert, you are uniquely qualified to provide this insight in ways editors are not always well equipped to assess.

When You’re Asked to Review a Paper

You should probably just accept.

But what if…

• You’re a bad fit for the topic? Or you’re not a subject matter expert? You should probably just accept. The editor probably had a reason for choosing you, and often, it’s because they’re trying to assemble a team of rivals. The editor felt that you had some unique perspective to bring to the table. You can also plainly state your perspective in the review itself. It’s fine to write “I am a substantive scholar of civil wars, but I am not familiar with the methodological approach, so my review is targeted toward the theory and substantive findings.”
• You’re busy (wink)? You should probably just accept. Reviewers will get very grumpy if you reject a couple of reviewer requests and then submit three articles to their journal with all of the time you saved by not reviewing.
• You’re actually busy? You should probably just accept. Then, tell the editor that you are busy and cannot complete the review in the one month timeline but can in two months. Then, let the editor decide if they want to find someone else or if they are ok waiting the additional month.

If you actually do need to decline, because you’re actually really really busy and do not have the capacity, or because the article was written by your best friend and co-author, or some other reason, then the panelists said the best thing you can do is provide suggestions for alternative reviewers. Ideally, with your rejection, you could provide 3-5 alternates with their email addresses and a sentence about why you think they would be a good fit for the article.

How to Review a Paper

Everyone has their own approach to reviewing a paper, and Celeste provided some insight into her workflow:

First, the whole thing should take a couple of hours. Start by reading the paper through to get a general sense of what you do and don’t like about the paper. Then, read it a second time, getting more into the details. When writing your review, begin with an abstract (primarily targeted toward the editor) that details what is good about the paper, what issues exist, and signals whether your opinion is favorable or unfavorable (without necessarily explicitly saying your editorial decision). Then, focus on the author by providing major feedback and supporting points. Overall, Celeste suggested keeping things concise—a review for a full-length article should be about 1.5 pages single spaced.

This approach generally conforms with my own, although often my first read-through is where I make most of my major notes and comments. On the second read, I look for anything I missed the first time (sometimes, that means I wrote a criticism the authors had actually answered, other times, I did not catch some omission on the first read). I was also writing the summary paragraph, but after this panel, I am going to stop doing that and start writing the significance paragraph instead. You might also look at how Mirya Holman reviews a paper.

Beyond the step-by-step, the panelists had some more general advice:

• Christian noted that he “likes hamburgers and ice cream, not papers.” That is, don’t approach a paper from the perspective of liking or disliking it (or, at least, keep it to yourself). Instead, approach the paper from the perspective of what it does well and where it does not succeed, and point that out to the editor and author. You can dislike a well-executed paper and recommend it for publication.
• Celeste noted that clear accepts and clear rejects tend to be short, while R&Rs tend to be much longer. However, if your review is too short—for example, if you just write “This is an excellent paper and it should be accepted as is”—the editor has nothing substantive on which to base the decision. Again, you should explain the significance of the article and why you feel it should be accepted as is. If you do not provide enough detail, the editor may not even be able to use your review in the decision-making process. On the flip side, if your review is nasty and mean, the editors may simply throw it out.
• Nadia advised that you omit any discussion of your recommendation in the review itself (i.e., do not write “this should be accepted”) and to save that for the dropdown in the editorial manager. If you write that the article should be accepted and put that in your review to the author, the editor is in an awkward spot if they end up rejecting the article.

Finally, what should you put in the “for the editor’s eyes only” box? The panelists advised to use this box sparingly. Reserve it for concerns about ethics, plagiarism, and other sensitive concerns. Use it to contextualize your editorial decision—for example, if you say “major revision,” provide some insight into what you mean by that and what you would be looking for so the editor can decide if they actually think it is feasible for the author to complete. For instance, even if you say the article should be revised, the editor may decide to reject given the amount of work they think needs to be done and the journals submission/acceptance rates.

Do not use the box for negative feedback that you do not wish to share with the author. If you have negative feedback, you should politely share that with the author so they know how to improve their article and to put all your cards on the table. As Celeste said, if you only share supportive comments with the author and only share negative feedback with the editor, the author is going to be very puzzled when their paper is rejected but the review is very supportive and positive.

And remember—you’re anonymous only to the author, not the editor!

Thanks to JAWS for hosting a productive and helpful workshop. Look out for more professional development opportunities and workshops on their website, and sign up for the mailing list!