Tuesday, January 24, 2012

Reflections on open peer review

Peer review Last week I put up a manuscript for open peer review (It's academic publishing Jim, but not as we know it). In that post, I explained my reasons for doing this rather than going down the conventional (journal) academic publishing route. The review process, which I arbitrarily set at 14 days, is still running, but in this post I want to discuss my reflections on the process to date.

As I expected, reviews started to come in rapidly, 7 within the first 48 hours, then stopped equally rapidly. Internet attention is transitory, but in part this is a reflection of the fact that I drew the blog post to the attention of a number of people by email, inviting reviews. However, this pattern is typical for Internet content - a fast decay phase followed by a longer, slower tail (The Spread of Scientific Information: Insights from the Web Usage Statistics in PLoS Article-Level Metrics. (2011) PLoS ONE 6(5): e19917). There were no "spam" comments, which I had anticipated, and even though I had attempted to make clear in the post that anonymous reviews were entirely acceptable, all reviewers chose to identify themselves. Ironically, this is a concern, as while I suspect that reviewers consider named reviews to be somehow more "valid", I am worried that potential negative reviews are simply not posted, rather than being contributed anonymously. Interestingly, relatively few colleagues from my own institution, who I had alerted by email, contributed a review. In part this may be because they were wary of possible conflict of interests. When I repeat this process in future, I will simply post the article and reviewing guidelines online, without individual email notifications. Another concern for the future is the possibility that familiarity may breed indifference, limiting the number of reviews received.

I am grateful to Martin Weller for his additional comment on the review process:

"I tried to put my official reviewer hat on and review it as if I was doing a standard (blind) peer review. It may be that this is an inappropriate transfer of process, and instead I should adopt a different style for open, informal review. But we fall back on what we know. My review may be a bit harsh, but I was conscious that 'asking your mates to review' isn't really comparable to anonymous peer review at all. I might be far less likely to criticise a friend. My colleague Gill Kirkup maintains that anonymity in the peer review process is essential because it protects the reviewer, particularly a young reviewer who is reviewing a paper by someone eminent in the field. Of course, it also allows people to be ruder than they would be otherwise, and often to say incorrect judgements because there is no debate or come back.
So this may be a good way to get feedback on a paper, but would it equate to peer review? I don't think so, but then maybe it's a sufficient filter to allow publication and then post-review. It's also quite a brave thing to do and I suspect many colleagues might be reluctant to go this route. If you write a crap paper that gets rejected by a journal, only a handful of people have seen it - if you do it this way, potentially hundreds will."

A number of people commented on various forums that I was "brave" to expose my work in this way. It doesn't feel brave to me, it feels liberating, although possibly foolish. Specifically, it feels far less brave than exposing my work to non-transparent peer review. Maybe I've just had a run of bad luck, with editors taking capricious cost-based decisions to refuse to even send my work out for review. Entering that lottery - now that's brave (or foolish). Accepting that my peers may tell me that my work is of little or no value (and I have no doubts about the honesty of people who responded, so I feel confident they would), the whole process feels right to me. If some papers are slammed, then I either work on them further or abandon the concepts they contain.

So will I repeat this exercise in future? Most definitely - I already have a manuscript in mind, although this one is perhaps more of a technical report than an investigation. Will this become my sole future publication channel? No, not because I do not believe in it, but there are circumstances (collaboration with junior colleagues for example) where the alleged kudos attaching to publication in conventional journals is important for their careers. Should you repeat my experiment. That's up to you, but if you feel your circumstances permit, I would encourage you to try it for yourself. As I commented on Frances Bell's blog, "... I am not suggesting the approach I have taken is the “best” solution, nor necessarily appropriate for everyone – I have already identified a number of flaws. I do suggest that it is an improvement on the current model of closed, and frequently capricious, peer review. Open is good. If we support open access, why not open peer review?".

Update: Storify capture of Twitter discussion:


  1. I am really glad to see some discussion around peer review. I think there is still a place for closed discussion as part of the development of our work - here are some my earlier thoughts on peer review
    I don't believe that openness, whilst a good destination, is always best for the whole journey. Neither does openness equate with freedom and equality - power relations still exist and can be destructive when we have to pretend they don't exist. I think of this frequently when I see discussion in the blogosphere and on MOOCs.

  2. Thanks Frances. I understand your position, but I wonder if we are hidebound by tradition and expectations. is transparency something new we have yet to get our heads around and derive best practice for? Is being open for part of the review process like being a little bit pregnant?

  3. An interesting analogy Alan;) If someone says they have conceived that is a binary state but pregnancy is a process. So are you seeing openness as a process or as a binary state?
    I don't suppose there is much argument about virginity being a binary state but I remember feeling a bit pregnant and later very pregnant;)

  4. s a biologist I'd argue that virginity is rather less binary than pregnancy!

  5. But is it more or less binary than open?

  6. @Alan @Frances - excuse flippancy but in becoming pregnant somebody *has* to sleep with you, whereas to be invited to peer review for a journal it is merely customary! :-)

    Thanks Alan for sharing this really useful post - I always see the line between commenting on someone's paper and on their blog post as being very permeable - the advantage of open review may be that it may attract the comments of people across more related disciplines who may not be able to comment on an entire paper, but would provide useful feedback on certain aspects. Have you seen any evidence of this?

  7. Amused at your joke :-)

    To a great extent the commenters/reviewers to date have all been pretty much the "usual suspects". This is a potential weakness. Set against this, the manuscript has received much more, more detailed (and possibly better informed) review than it would have in a conventional journal review procedure.

  8. Putting together my role as co-editor of RiLT and the huge number of reviewers we have and thinking I don't remember all that fun -