Thursday, February 09, 2012

More reflections on open peer review

Reflection Earlier this week I published the first product of my open peer review experiment, Student feedback using Google+. So far (after 3 days) the manuscript has been downloaded 74 times (latest figures here). This post is to follow up on my earlier reflection and tidy up a few loose ends concerning my experience of the open peer review process.

Given that this was the first time I undertook the process, in addition to posting the manuscript here for review, I also emailed a number of people I considered qualified to review it and pointed out that the process was under way. Those invitations gave rise to some discussions about "inviting friends to review your work" and consideration of whether this was valid peer review or not. In my opinion it was - open is open to all, friends and foes. However, considering the possible introduction of bias into the review process, when I repeat it in future (damn, given the game away now ;-) I will not issue invitations, only post the manuscript here and publicize the post through the normal channels. If that means the numbers of reviewers is lower, I will extend the review period until an acceptable number of reviewers have commented.

I intended to publish the final version a week or more ago, giving an interval from publishing the preprint to publication of the finished product of less than 21 days. However, personal circumstances and my current teaching load extended this to 23 days, a highly acceptable result in comparison to commercial publishers. Although I consider myself fortunate that most of the reviewers concurred on desirable additions, incorporating the comments of 7 referees is definitely hard work - assuming you attract enough reviews, open peer review is definitely not an easy ride! As hoped for, incorporation of the reviewers comments improved the quality of the publication considerably.

The final published version of this manuscript contain no acknowledgements - this work was a solo effort with no external funding. Since I was not sure about the etiquette of thanking reviewers I did not include them. I hope no-one is offended by that. The other thing I forgot was to add copyright information (CC-BY) to the manuscript itself). I was thinking that this would be covered by the repository page, but I now realize that a) this does not have CC status as I assumed, and b) is easily divorced from the manuscript, so the information must be embedded there. Rookie mistake.

The big question for me is, is this model scalable? If I routinely asked for reviews in this way, would fatigue set in, or would my ‘mates’ become an echo chamber? It is too soon to say, but my concern is that the process I have piloted may not be sustainable because the reviewer ecosystem may not be able to circumvent the Tragedy of the Commons. The only way to tell is further experimentation. Watch this space.

Related posts:


  1. I think fatigue might set in, or the novelty factor wear off. Imagine the number of requests either of us might get if everyone did this. We'd soon start ignoring them as spam. The peer review process when handled by a journal does a number of things: 1) it creates a sense of obligation between the reviewer and the journal 2) it controls that workload by allocating the work around and not overburdening the same reviewers 3) it creates equanimity in that you can be a first time author without a good network but still get reviewed.
    In your model there is a danger it creates an elite which is hard to break into, and it also reduces to sense of duty or ownership of a reviewer.
    Now you know I have issues about the dominance of the peer review model, but it's worth reflecting on what it does give us, as well as all the problems.

  2. There are clear frameworks for building a sustainable reviewer ecosystem, but they do imply a level of organization beyond the individual, see: Governing the Commons

  3. I was fascinated by your post. We are working on a site built exactly on the assumption that peer review is scalable on the net. The site is

  4. "Since I was not sure about the etiquette of thanking reviewers I did not include them. I hope no-one is offended by that."

    How could there be a reason not to thank reviewers? Even if you didn't know which of them wanted to be anonymous -- something that seems a bit silly in an open peer-review situation -- I can't see any reason not to say something like "I thank the seven reviewers whose comments improved this paper".

  5. I had already had some discussions about notifying people the manuscript was available for review. Thanking reviewers could seem incestuous, but you're probably right, and I will than reviewers in future.