Categories
Edtech MOOC online course

Some (new) observations on peer review

I recently completed a MOOC called Elements of AI. Let me first say that I am privately (and now perhaps not as privately) thrilled to have managed this because I’m highly unlikely to commit to a MOOC if it looks like I might not have time to do it properly (whatever that means) and it often looks that way. I enjoyed the course and definitely learned a bit about AI – robots will not replace teachers anytime soon in case anyone was wondering – but I couldn’t help noticing various aspects of course design along the way. This is what this post is about, in particular the peer review component. 

S B F Ryan: #edcmooc Cuppa Mooc (CC BY 2.0)

Most of my experience with peer review is tied up with Moodle’s workshop activity, which I have written about here, so the way it was set up in this course was a bit of a departure from what I am used to. There are 5 or 6 peer review activities in Elements of AI and they all need to be completed if you want to get the certificate at the end – obviously, I do. *rubs hands in happy anticipation*

Let’s take a look at how these are structured. To begin with, the instructions are really clear and easy to follow – and despite reading them carefully more than once, I still occasionally managed to feel, on submitting the task and reading the sample “correct” answer, that I could have paid closer attention (the “duh, they said that” feeling). The reason I note this is because it’s all too easy to forget about it when you’re the teacher. I often catch myself thinking – well, I did a really detailed job explaining X, so how did the student not get that? 

Before submitting the task, you’re told in no uncertain terms that there’s no resubmitting and which language you’re meant to use (the course is offered in a range of languages). I read my submissions over a couple of times and clicked submit. In the Moodle workshop setup, which I am used to, you can then relax and wait for the assessment stage, which begins at the same time for all the course participants. Elements of AI has no restrictions in terms of when you can sign up (and submit each peer review), so I realized from the start that their setup would have to be different. 

The assessment stage starts as soon as you’ve made your submission. You first read a sample answer, then go on to assess the answers of 3 other course participants. For each of these three you can choose between two random answers you’re shown before you commit to one and assess it on a scale of an intensely frowning face to a radiant smile (there are 5 faces altogether). You are asked to grade the other participants on 4 points:

  1. staying on topic
  2. response is complete/well-rounded
  3. the arguments provided are sound
  4. response is easy to understand

The first time I did this, I read both random responses very carefully and chose the one that seemed more detailed. This was then quickly assessed because the 4 points are quite easy to satisfy if you’ve read the instructions at all carefully. However, I did miss the fact that there was no open-ended answer box where I could justify anything less than a radiant smile. I’m guessing this was intentional so as to prevent people from either submitting overly critical comments or spamming others (or another reason that hasn’t occurred to me) but I often felt an overwhelming urge to say, well, yes, the response was easy to understand, but you might consider improving it further by doing X. Possibly those who aren’t teachers don’t have this problem. 😛

It was also frustrating when I came across an answer that simply said “123” and another that was plagiarized – my guess is that the person who submitted it had a look at someone else’s screen after that other person had already made their submission and could access the sample answer. Or maybe someone copied the sample answer somewhere where others had access to it? The rational part of my brain said, “Who cares? They clearly don’t, so why should you? People could have a million different reasons for signing up for the course.” The teacher part of my brain said, “Jesus. Plagiarizing. Is. Not. Okay. Where do I REPORT this person? They are sadly mistaken if they think they’re getting the certificate.”

Once you’ve assessed the three responses an interesting thing happens. You’ve completed the task and can proceed to the next one, but you still have to wait for someone to assess your work. This, you’re told, will happen regardless, but if you want to speed up the process, you can go ahead and assess some more people. The more responses you assess, the faster your response will come up for assessment. I ended up assessing 9 responses per peer review task, so clearly this incentive worked on me, though I have no idea how much longer I would’ve had to wait for my grades if I had only assessed 3 responses per task. I only know that when I next logged on, usually the following day, my work had already been assessed. 

For a while I was convinced that either whoever had assessed my work had been very lenient or else all responses were automatically awarded four radiant smiles. My work hadn’t been that good, I thought. Then in the very last peer review I got a less than perfect score, so I assume there was at least one other teacher taking the course. 🙂 

In theory then, once your work has been assessed by two of your peers, you’re completely done with the task. However, at the very end of the course, you’re told that in addition to the grades you received from your peers, your work will also be graded by the teaching staff. Happily, your tasks are still marked as complete and you can get your certificate nevertheless. I suspect I’ll be waiting a while for that grade from the teaching staff and it seems a bit irrelevant, to be honest. It would make sense for someone other than other course participants to check the responses if this were done before course completion was officially confirmed (so those who submitted “123” wouldn’t get their certificate, for instance) but now I think of the course as finished and my work as graded, I’m not likely to go back and check whether I received any further feedback, especially if it’s only emoji. 

There were other interesting aspects of the course but I’ll stop here so as not to mess up my chances of posting this soon. In short, the course reminded me of why l like peer review (if everyone participates the way the course designers intended them to) and has given me some new ideas of how similar activities can be set up.

Have you completed any MOOCs or other online courses lately? Did they include peer review? What do you think makes a good peer review activity?

By ven_vve

ELT, elearning, higher ed, teacher training, translation. Partial to the island of Vis since the pre-tourist era.

2 replies on “Some (new) observations on peer review”

Hi, Vedrana,

an interesting read. Just in case you are not familiar with it already, Paul Moss has also shared some interesting content related to peer review on his blog: https://paulgmoss.com/2021/07/06/peer-assessment-and-pragmatics/

Actually, he talks about peer *assessment*, but I think it’s closely related to your field of expertise. 😉 By the way, in the other post of yours (linked above), you say that you always shuffle around and allocate submissions manually, trying for a balance between weaker and stronger students. Does it mean that you pair a stronger student with a weaker one? I’m asking because I’ve recently come across the notion (also on Paul’s blog) that once you get students to read others’ work that is well below their level of understanding, the experience not only becomes redundant but potentially regresses the student in their learning. Also, if you pair a stronger student with a weaker one, their progression is dictated by a movement towards the average between the two peers: perfect for the weaker student who improves, worse for the stronger student whose progress is vitiated. This implication seems highly relevant, especially in the context of placing students into pairs or groups during speaking activities. To be completely honest, I do sometimes ‘sacrifice’ the stronger students but I do so in the hope that they may appreciate the fact that they can help the weaker ones.

Cheers!

Hana

Liked by 1 person

Hi Hana,

Thanks for the thoughtful comment and for introducing me to Paul Moss’ blog. I guess the activity I ask my students to do is peer assessment as well, but of the formative kind, as the feedback the students give their each other doesn’t contribute towards their final grade. The reason it doesn’t is basically down to what Paul discusses in the beginning of his post: some students might feel the grade they received from another student wasn’t fair.

Your question about manual allocation of submissions and the point you raise about stronger students being at a disadvantage if they’re paired with weaker ones is a really interesting one. If this was something I had students do on a regular basis, I would definitely take this potential drawback into account. However, the peer review/assessment activity is complicated to set up (with the two phases) and every semester there are students who claim to feel uncomfortable in the role of teacher, having to assess/give feedback (even if it is anonymous), so we only do it once. One very perceptive student reflected in their learning journal only this semester that they – I’m paraphrasing – didn’t find the activity particularly useful because there could be substantial differences between students and while it might be useful for some, it might also be useless for others. This was a stronger student. So yes, definitely something to consider.

Thanks very much for stopping by – appreciated as always!

Liked by 1 person

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.