Categories
Edtech MOOC online course

Some (new) observations on peer review

I recently completed a MOOC called Elements of AI. Let me first say that I am privately (and now perhaps not as privately) thrilled to have managed this because I’m highly unlikely to commit to a MOOC if it looks like I might not have time to do it properly (whatever that means) and it often looks that way. I enjoyed the course and definitely learned a bit about AI – robots will not replace teachers anytime soon in case anyone was wondering – but I couldn’t help noticing various aspects of course design along the way. This is what this post is about, in particular the peer review component. 

S B F Ryan: #edcmooc Cuppa Mooc (CC BY 2.0)

Most of my experience with peer review is tied up with Moodle’s workshop activity, which I have written about here, so the way it was set up in this course was a bit of a departure from what I am used to. There are 5 or 6 peer review activities in Elements of AI and they all need to be completed if you want to get the certificate at the end – obviously, I do. *rubs hands in happy anticipation*

Let’s take a look at how these are structured. To begin with, the instructions are really clear and easy to follow – and despite reading them carefully more than once, I still occasionally managed to feel, on submitting the task and reading the sample “correct” answer, that I could have paid closer attention (the “duh, they said that” feeling). The reason I note this is because it’s all too easy to forget about it when you’re the teacher. I often catch myself thinking – well, I did a really detailed job explaining X, so how did the student not get that? 

Before submitting the task, you’re told in no uncertain terms that there’s no resubmitting and which language you’re meant to use (the course is offered in a range of languages). I read my submissions over a couple of times and clicked submit. In the Moodle workshop setup, which I am used to, you can then relax and wait for the assessment stage, which begins at the same time for all the course participants. Elements of AI has no restrictions in terms of when you can sign up (and submit each peer review), so I realized from the start that their setup would have to be different. 

The assessment stage starts as soon as you’ve made your submission. You first read a sample answer, then go on to assess the answers of 3 other course participants. For each of these three you can choose between two random answers you’re shown before you commit to one and assess it on a scale of an intensely frowning face to a radiant smile (there are 5 faces altogether). You are asked to grade the other participants on 4 points:

  1. staying on topic
  2. response is complete/well-rounded
  3. the arguments provided are sound
  4. response is easy to understand

The first time I did this, I read both random responses very carefully and chose the one that seemed more detailed. This was then quickly assessed because the 4 points are quite easy to satisfy if you’ve read the instructions at all carefully. However, I did miss the fact that there was no open-ended answer box where I could justify anything less than a radiant smile. I’m guessing this was intentional so as to prevent people from either submitting overly critical comments or spamming others (or another reason that hasn’t occurred to me) but I often felt an overwhelming urge to say, well, yes, the response was easy to understand, but you might consider improving it further by doing X. Possibly those who aren’t teachers don’t have this problem. 😛

It was also frustrating when I came across an answer that simply said “123” and another that was plagiarized – my guess is that the person who submitted it had a look at someone else’s screen after that other person had already made their submission and could access the sample answer. Or maybe someone copied the sample answer somewhere where others had access to it? The rational part of my brain said, “Who cares? They clearly don’t, so why should you? People could have a million different reasons for signing up for the course.” The teacher part of my brain said, “Jesus. Plagiarizing. Is. Not. Okay. Where do I REPORT this person? They are sadly mistaken if they think they’re getting the certificate.”

Once you’ve assessed the three responses an interesting thing happens. You’ve completed the task and can proceed to the next one, but you still have to wait for someone to assess your work. This, you’re told, will happen regardless, but if you want to speed up the process, you can go ahead and assess some more people. The more responses you assess, the faster your response will come up for assessment. I ended up assessing 9 responses per peer review task, so clearly this incentive worked on me, though I have no idea how much longer I would’ve had to wait for my grades if I had only assessed 3 responses per task. I only know that when I next logged on, usually the following day, my work had already been assessed. 

For a while I was convinced that either whoever had assessed my work had been very lenient or else all responses were automatically awarded four radiant smiles. My work hadn’t been that good, I thought. Then in the very last peer review I got a less than perfect score, so I assume there was at least one other teacher taking the course. 🙂 

In theory then, once your work has been assessed by two of your peers, you’re completely done with the task. However, at the very end of the course, you’re told that in addition to the grades you received from your peers, your work will also be graded by the teaching staff. Happily, your tasks are still marked as complete and you can get your certificate nevertheless. I suspect I’ll be waiting a while for that grade from the teaching staff and it seems a bit irrelevant, to be honest. It would make sense for someone other than other course participants to check the responses if this were done before course completion was officially confirmed (so those who submitted “123” wouldn’t get their certificate, for instance) but now I think of the course as finished and my work as graded, I’m not likely to go back and check whether I received any further feedback, especially if it’s only emoji. 

There were other interesting aspects of the course but I’ll stop here so as not to mess up my chances of posting this soon. In short, the course reminded me of why l like peer review (if everyone participates the way the course designers intended them to) and has given me some new ideas of how similar activities can be set up.

Have you completed any MOOCs or other online courses lately? Did they include peer review? What do you think makes a good peer review activity?

Categories
Edtech Moodle online course Tertiary teaching

Practice makes perfect?

This has been sitting in my drafts folder for about a year. If I don’t post it now, I’m not sure I ever will, so here goes. 

danna § curious tangles: words (CC BY-NC-ND 2.0)
March 2020

I recently tweeted this.

It then occurred to me that the tweet could be construed as a promise of a blog post, so I thought I should do something about that.

I’ve previously written about H5P activities in our online course here, here and here. I like the versatility of the tool and the fact that it’s available as a Moodle plugin, so I can use it free of charge. There are many content types (activity types) you can use and so far I’ve only been able to test out a relatively small number: drag and drop, interactive video, audio recorder and course presentation.

I was pretty confident H5P would be useful for tasks more obviously associated with vocabulary learning (probably primarily because I’d noticed a content type called Fill in the blanks), so at some point I thought I should look into it as a possible substitute for Textivate. 

I’ve been using Textivate for several years now and really like it, but if you want to use it in combination with your own resources – to practice specific vocabulary, for instance – you have to get the paid version. Which I completely understand and *have* renewed my subscription a couple of times, but as I’m an adjunct and get paid comparatively little, I’m always on the lookout for free versions of apps and such. 

I decided to revamp our revision unit with the help of H5P content types and made the following changes.

  1. User-defined gapfill (Textivate)Fill in the blanks (H5P). For vocab revision. In the Textivate version, once you define the words students need to add to the text, they conveniently and automatically appear below the text and you drag them to where you think they should go. In the H5P version, you define where the missing words should go and get blank boxes where students need to type these words. It’s not incredibly flashy or exciting in Textivate either, but in H5P it’s really bland, so I decided to jazz it up a little by creating an image in Canva and adding the missing words to it. Then I added the image above the text. When you have a go at the activity in student mode, the image shows up as much smaller than it is, but you can click on it to enlarge it, which I thought could be convenient for practice. If a student was doing the activity for the first time, they could click on the image and view the words, but if they wanted to try it again, they could see if they recalled any of the words without first clicking on the image. Incidentally, you could also add a video instead of an image to the activity, which wasn’t suited to my purpose but could work well in other contexts. 
February 2021

So, here we are, back in the present. It’s a little more difficult now to identify the type of activity I used in Textivate because I can only make a guess based on what the activity looked like in earlier iterations of the course. I knew there was a reason I should have done this sooner. 

  1. Shuffle? Multimatch? (Textivate)Fill in the blanks (H5P) For joining sentences. The point here was to practice joining sentences in a highly controlled way, with relatively little creativity and thus few unexpected outcomes. (No, it’s not one of my favorite activities either, but it’s useful for exam practice.) When I used Shuffle, students were instructed to rewrite the sentences in a separate document (which is not the happiest of solutions and I’m doubtful whether anyone ever actually did this, especially if they revised half an hour before the exam) and then do the Shuffle activity where they matched the two sentence halves and checked if they corresponded (in terms of punctuation, etc.) with what the students thought was correct. I’m happier with how it works with Fill in the blanks (H5P) because you can add hints to the blank boxes. The students have to type out the sentences in order to check if they joined them correctly and if there’s anything you want them to watch out for (something that might cause them to slip up at the exam) you add it as a hint. Now, you might think well, maybe Shuffle/Multimatch wasn’t the best choice of activity for joining sentences and you’d be right, but as I already had a Textivate subscription I sometimes used it in ways that were not ideal. 
  2. User-defined gapfill (Textivate)Mark the words (H5P) For identifying parts of text that need alteration. This was a really convenient change because it made me break the activity up into two steps, which I think is easier to process. Again, in the Textivate version the students were instructed to type the changes they would make in another document and then drag and drop the suggested answers into the correct gap. If they skipped the first step, the activity was deceptively easy compared to what it would be like at the exam. With Mark the words (H5P), they first need to highlight the parts of the text that need to be changed (and can check if they were correct), then there is another Fill in the blanks with hints where they focus on making the actual changes. 

And that’s it as far as H5P in the revision unit goes. There are also a couple of Quizlet sets and Moodle quizzes, so the H5P activities are just part of what students have the opportunity to complete if this is how they wish to practice. The unit is entirely optional, although I sometimes think perhaps it shouldn’t be, but that is material for another post.

If you teach online (synchronously or asynchronously), do you have materials that your students can access in their own time and practice, say, for an exam? What (tools) have you used for this and are these activities optional?

Categories
Moodle online course Tertiary teaching Thoughts and reflections

It’s been a privilege

Those who have been following this blog a little longer (as in 5 years or so) since the dawn of time may remember this post in which I talked about the first semester I had assistant moderators: (mostly) graduate students who helped me moderate forum discussions and comment on student learning journal entries. It was the first time I’d involved students in this capacity in an online course, although, to be fair, I hadn’t been teaching the course for very long at that point. It was in its fifth run. A brief digression right at the start: involving students this way online seems completely natural, yet doing a similar thing in class is much more difficult to imagine, for me at least. 

Photo “Team Work” taken from https://www.flickr.com/photos/jerixthekid/ by mønsterdestrøyer, used under a CC BY-NC-ND 2.0 license.

The original four-mod cast only stayed together that one semester, but running the course with the help of moderators has remained a permanent feature. Twelve semesters on I can say that I have had the privilege of working with twelve incredibly communicative and motivated young people (says she, sounding about ninety-three 😛 ) who I have learned a lot from and who have been hugely helpful. Here I was, all overcome with warm and fuzzy feelings and then it occurred to me that it would be really interesting to do a post in which they would talk about their moderator experience and what it meant to them. 

So I set up a Google doc and added a few questions plus the option that they add their own questions if they felt there was something more they wanted to say. Of course, I told them the answers would be shared on this blog and that they could remain anonymous if they liked. This all happened in the first half of November, so by now I’m feeling guilty for not getting the post out sooner. One lovely (partly) unexpected benefit of the whole endeavor was catching up with some of them and finding out what they were up to professionally. 

Without further ado, I’m adding the questions and answers below. I hope you’ll enjoy reading them! Oh, and if you have any questions for the mods I’ll pass them on.

Q1. How many semesters (roughly) were you an assistant mod for Writing in English? (my comments in italics)

Beatta: 1 semester (Actually, it was two now I’ve checked my records. Then you went on a semester abroad.)

Ivana: Huh, 4 i believe (Three, actually.)

Marija: This would be my 5th, but I am not certain 🙂 (Yes, you’re right!)

Dora: I believe 4 but could be more 🙂 (It was four.)

Q2. What made you decide to “accept the challenge” – it could be argued that being an assistant mod is just more work for students with already busy schedules?

Beatta:  I just really like expressing myself in English so thought this would give me an opportunity to expand my vocabulary and get a better handle on the language. It was not so much about the actual work – as challenging as it may have been sometimes, but rather just talking to other people in English 🙂

Ivana: I liked the concept of online courses, which was completely new for me at the time. Plus I looove writing and expressing myself in that way so this was a perfect way to match my passion for writing, helping students and learning some new english 🙂

Marija: To be completely honest, it just seemed like something I would actually enjoy doing that would look good in my resume 🙂 What is great about this specific course is that all of our work is online, and that I could (and I have) be active anytime, day and night. So, on top of all my other activities, it seemed like a good challenge to take on.

Dora: I always liked being an assistant, helping other students as well as professors. I didn’t mind the additional work, it wasn’t too much for sure. Also, this additional stuff in college always look good in CV and you definitely learn a lot.

Q3. How would you describe your assistant mod experience? Is there anything you’d single out as applicable outside of the course (here I’m not referring to the course content but the work of assistant mods)?

Beatta: I do not remember many details, but I remember having fun. As I said, some tasks were more challenging (i.e. getting the students to “debate” you) or boring than others (i.e. checking their homework) but all in all, I have positive memories regarding it. I really think the assistant mod experience upped my English game – I became more fluent in both speaking and writing, I expressed myself easier and my “ear” and instinct for the language developed further. Regarding some hard skills I may have developed from my mod experience, I think it pushed me to be more/better organised with my private time.

Ivana: It was a long time ago but I remember feeling amused and it really was not a problem for me to work on the tasks we had to fulfill. Sometimes I was looking forward to reading the tasks other students have done or to read about their opinions connected to the subject (and the themes that we were talking about were always rather interesting and current). I also feel like it prepared me for some future obligations that I had (doing some work online). Also I got a job because of the recommendation of prof. Vedrana 🙂

Marija: It is not so hard or too time consuming, but it makes a big difference for the students – I remember having really bad and indifferent assistants in other courses and I felt like I could contribute and make other students’ experiences better. I would like to single out the “leadership” aspect of it. We are only assistants, but we manage student communication, give instructions and directions and provide much needed feedback. It was a new field for me personally and a great practice.. It made me really improve myself and my communicating abilities which I am sure I will use later in life.

Dora: I liked the whole experience and that is why I was an assistant for all those years. We didn’t have many English classes as I would have liked, so it helped me stay fluent and learn even more. Also, it was really interesting to see what other students are thinking, how they do the assignments and how I actually got to know them without ever knowing them 😃 for my future it helped with keeping to the schedule, having obligation to other students to help them when needed and somewhat mentor them.

Q4. How do you see the work of assistant mods as contributing to the course?

Beatta: I think assistants can be of great help, not only to the professor but to the students as well. They can lessen the workload of the professor and help students open up in the debates as well as their assignments (especially blog entries).

Ivana: Sometimes students might feel more open towards the assistant and therefore open themselves in writing also. Plus, I remember that sometimes me or my mods colleagues were needed to direct the debating in a different way that was needed for the course.

Marija: My main task, I believe, is to help the professor manage all the aspects of the course (from portfolio entries to forums and debates), but also to be the link between professor and student – students tend to hesitate in asking for help and directions, but we reassure them and help them realise it is OK, even welcome.

Dora: They can help with work overload for the professor but also students might be “less afraid” to ask assistants some questions.

Q5. Is there anything about being an assistant mod that you found challenging (and how did you address that)?

Beatta: As I already mentioned, it was quite some time ago, so my memories are a bit faded, but I don’t remember it being too challenging. I remember there were lessons where the workload was heavier and/or more demanding (be that in volume or in the type of task – for me the grammar always got me :D). I addressed it by just taking more time to go through it.

Ivana: Nothing challenging about it as far as I am concerned, but it wasn’t boring either. Maybe sometimes I had a lack of time to do some tasks, but then I wrote shorter answers, simple as that. Would recommend this kind of assistance in class anytime because at the end of the day, you do your own schedule.

Marija: Everything was really well organised and I managed to stay on top of things, but sometimes I had too many other responsibilities in order to assist as well as I wanted to. It was such a terrific experience for me because of professor Estatiev too, because every time I felt pressured or thought it was too much, all I had to do was let her know and she would help out – which was greatly appreciated.

Dora: Can’t remember honestly. Just know I enjoyed it!! 🙂

Questions you wish I’d asked (add your suggestions below – possibly to be addressed in another post)

Marija: Would I recommend it and why? Absolutely! Having a great mentor who gives you responsibility and trust you do serious work is such a valuable college experience. It helps you work on yourself, come out of your student comfort zone and makes you work closely with other students and, in the end, do beneficial work for those students who really need assistance in tackling new course concepts. Plus, it sounds really good when you mention it during a job interview (I speak from experience).