Wednesday, October 31, 2012

Coursera Peer Assessment - Writing in the Sciences

MOOC I finally got a chance to experience the Coursera peer assessment model on the Writing in the Sciences course. It works like this:

1. Each student has 7 days to submit a piece of writing (300-500 words in this case) via the website.

2. During the following 7 days, each student must grade 5 (or more) pieces of work from other students. This is done on a 0-3 mark scheme driven by criterion-referenced rubrics covering Clarity and Concision, Language and Style, Focus and Organization, e.g. for Clarity and Concision:
0 points: No score (not explained - nothing submitted?)
1 point: "The writing is difficult to understand throughout; may contain  substantial clutter and serious grammatical problems."
2 points: "The writing is sometimes difficult to understand; may contain needless clutter, unexplained jargon, or grammatical errors."
3 points: "The writing is clear, concise, and easy to understand."
There is also a section for short freetext feedback, and markers are asked to resubmit two versions of the original text, one marked up with suggested changes (strikethrough for removal, bold for addition), the other a final mofidied version.

3. Marks are then returned via the website.

I was quite impressed with the process, which worked well in my case. I am well aware there has been lots of gaming on other Coursera courses. I received four peer assessments with brief but useful feedback. (My mark was 83% - if you care.)

Would it work as well with my students?  I'd like to think so but I'm not sure. For one thing it's not clear that our students are as confident or motivated as the participants in this course. For another, there is the issue of marking cartels as students indulge in the prisoner's dilemma (as they perceive it) with summative assessment. Sadly, I can't see a system like this being a goer for us.

Would I recommend my students to take this course? No, because frankly the course content is not very good. Would I want them to have the experience of having their writing commented on in this way? Absolutely. Have I achieved my personal learning outcomes for this course?
  • To improve my writing (let go of academic writing habits) - No, because I have only participated in a superficial way after becoming disenchanted with the lectures. But two piece of writing simply aren't enough to form new habits.
  • Explore practical strategies of how to teach and assess writing of large groups of students online - Yes, although I don't see myself of being in a position of being able to put a similar strategy in place for summative assessment in the foreseeable future even if a suitable platform was available.

And that's it. I'm taking a break from MOOCs for a while to concentrate on other things, including the #cfhe12 cMOOC which I simply haven't had time to participate in, and which has failed to motivate me because of the excessive North American focus and lack of sufficient structure to make me want to continue. My learning outcomes for this course were:
  • To compare my view of HE with that of others - where does it align? - Fail, because I simply didn't participate intensively enough to achieve this.
  • To experience d2l platform - Shockingly bad instance, although I'm quite happy to believe that d2l can be much better than this if used with more care and thought.


  1. Did you get a sense of the effectiveness of the calibration of graders that Coursera supposedly use? Were you trained and calibrated? (I presume that you gave grades and feedback to others). Do you think that this calibration will overcome the gaming of the system?

  2. There was no training or calibration in this instance, it's difficult to see how that would work in a MOOC setting and it's not a feature of the Coursera platform as far as I know.

  3. This is really interesting, thanks for pointing to me to this at SpotOn. I haven't used the peer assessment system yet as I'm doing a course in programming which is marked automatically - you're right or wrong! However, I'm doing a sociology one next year so perhaps it will be in play for that.

    I also share your view that this would be unlikely to work for traditional HE undergraduate courses. I also wonder whether the system is robust enough - when we mark entries to the sci comm competition at the Biochem Soc, we do the same kind of thing, but then we sit down and discuss the scores as well.