Customer satisfaction

I’m a firm believer in student feedback. My institution carries out an instructor evaluation survey each semester, but as it takes a while to process the data, we rarely get the results in time to actually do something about them the following semester. This is why I always ask my students to complete a course satisfaction survey that I can use immediately. (Incidentally, I’ve not noticed before that the names seem to indicate the institution primarily wants to know if the students are happy with me, whereas I’d rather know if they’re happy with the course – or maybe the institution sees us as one and the same?) 

When I first started teaching my writing skills course, I used paper-based feedback forms, then I moved on to Google Forms, while last semester I used the Moodle Feedback activity for the first time and was very pleased with the presentation/analysis of the results. Even though I chose not to record user names when gathering responses, I waited until the exams were over before looking at what the students had to say as I wanted to a) not have anything prejudice the grades, and b) wrap up the course with their comments and suggestions.

The new semester started last week and a new group of students will be taking the course, so it seems to be the perfect time to indulge in a (hopefully) brief analysis of the results and bring some order to my initial impression – enthusiastic, but not particularly insightful – that there is a lot of useful information in there.

A satisfied customer about to take a nap
A satisfied customer about to take a nap

I included 15 questions in the feedback form. My first impulse was to include about three times as many, but I nipped that one in the bud, reasoning that this would probably result in random clicking just to get the questions over with. Besides, the students had been reflecting on each unit of the course in their portfolios as the semester progressed, so I already knew how they felt about some aspects.

I tried to strike a balance between Likert type (7), multiple-choice (3) and open-ended questions (5). Here they are in the order they appeared in the form, each followed by a comment on the answers.

1. The required level of English in the course was too advanced for me to participate easily.

Sure, it stands to reason that I would know this after four months of reading their writing. It’s also a course requirement that students are at least B1 and they were given a placement test at the beginning of the course (which showed that the majority were B2 or even C1). So, yes, I was reasonably sure that nobody would agree completely with this statement, but as the course was entirely in English (we had Erasmus students), and quite a lot of authentic materials were included, I wanted to see hard numbers.

On a scale of 1-5, most chose 1 (to indicate they disagreed completely), but not everyone, which puts me in a bit of a dilemma. If it’s a course requirement that students are B1, surely those who are at that level shouldn’t be made to feel at a disadvantage. On the other hand, if the great majority of students are B2/C1, they won’t be challenged by B1 content.

2. The required level of technical knowledge was too advanced for me to participate easily.

Again, the majority disagreed, but this time only 50% did so completely. Most of the rest opted for “mostly disagree”, which would indicate that the technical difficulties weren’t serious enough to prevent participation, but still, (lack of) technical knowledge was apparently a greater obstacle than the required language level.

This was not entirely unanticipated, as we’d dealt with various technical issues throughout the course, so next came an open-ended question:

3. Which technical problems did you experience and could the instructor have done anything to help you overcome these?

There weren’t many answers to this, possibly also because I’d set it as optional. Most referred to a lack of familiarity with online courses – only two students said they’d taken such a course before – but problems with videos were mentioned. There were a few videos in the course, which worked fine when embedded, but I’ve heard suggestions that it’s better to upload videos directly into Moodle. So I did that, but it didn’t work very well; the videos were viewable, but apparently not without a separate set of instructions. Sorting this out is one of my priorities this semester.

Luckily, the instructor got positive reviews in the second part of this question, which I was particularly pleased with, especially after I saw students in another course were simply instructed to contact the admin in case of technical difficulties.

4. The activities in the course were very time-consuming.

The answers to this were quite diverse. A small number (15%) mostly disagreed – no one disagreed completely – 20% neither agreed nor disagreed, 40% mostly agreed and 20% agreed completely. (The numbers don’t add up to 100; I rounded them off for convenience.) This question was actually intended as a lead-in to a more specific one, namely:

5. On average, I spent the following amount of time on the course each week:

If the class were held on campus, students would spend 3 hours in class weekly, plus any additional time on homework/portfolio assignments. As this is equivalent to a certain number of ECTS points, there wasn’t supposed to be a noticeable discrepancy between the amount of time spent on the course f2f and online. And, in fact, there apparently wasn’t. 50% of the students said they’d spent 1-3 hours working online, and 50% claimed to have spent 3 hours or more. From this perspective, I should have provided the option of saying how much more.

6. If you did not participate regularly in the course – if there was any period apart from the Christmas break when you did not log on for 3 weeks or more – why was this? Could the instructor have done anything to make you participate more actively?

Perhaps not surprisingly, those who took part in the survey had participated regularly, and so, even though I would have very much liked to hear what those who hadn’t would have given as reasons, there wasn’t much in the way of clarification here. Two answers hinted at the difficulty of getting back on track once some activities had not been done on time, but were careful to absolve the instructor of any blame. 🙂

7. The instructions and updates on the course noticeboard were clear and timely.

Happily, most students completely agreed with this (only one mostly agreed). In a previous post I mentioned that I’d written roughly 5800 words on said noticeboard, so I think I wouldn’t have taken kindly to disagreement. 🙂 On a more serious note, I spent a lot of time wording particular posts just so, trying to be as precise and detailed as possible, and predict every possible doubt and query, so it was good to see that this had apparently been achieved.

8. The course content will prove useful in the future.

Another question that I liked the answers to. 60% agreed completely and 40% mostly, although now I wish I’d included the option of saying which parts students felt might not prove so useful. The course is based on selected chapters from two course books, chosen some years ago (not by me). I think I can see which parts might not seem so relevant from the students’ point of view, but I’d like to know for sure as maybe it’s more of an issue of how the content is being presented, rather than the content itself. Then I could do something about it.

9. When I addressed the instructor directly with an issue, my questions were answered quickly and satisfactorily.

Everyone agreed completely with this statement, although purely from the point of view of questionnaire construction I overlooked the possibility that some students didn’t need to contact me directly with an issue, so this question shouldn’t have been compulsory. Anyway, I expected most to agree here because I checked my email regularly and answered queries in the evenings and at the weekends. There really weren’t many of these overall, and I used to do this when I taught f2f as well, so it wasn’t as if it required any extra effort.

10. I found the feedback (from the instructor) in the portfolio relevant and useful.

The instructor is important here because the students also commented on one another’s journal entries in the portfolio. A high rate of agreement (85% agreed completely and 15% mostly) indicates that all the time spent on various feedback modes – error correction for some assignments, and podcasts and screencasts with more general comments – was worth it. However, I felt that despite doing my best to cover as much ground as I could in the feedback, there were still aspects that I wasn’t able to touch upon because it would have meant going into too much detail. I think this could be remedied by using the Moodle gradebook and I plan to do this next semester.

11. I would recommend the course to other students.

This question offered a choice of three answers: ‘yes’, ‘no’ and ‘don’t know’. Looking back, I should have included the option of saying why. Most did opt for ‘yes’, but 20% said they didn’t know, and it would be helpful to know whether their answer would have been the same had this not been an online course. Happily, no one went for ‘no’.

12. I have noticed an improvement in my writing skills in English over the past semester.

This was another multiple choice question, with the same answers as #11. Everyone answered ‘yes’ here, which I’m particularly happy with because I honestly wasn’t sure what to expect. With some people the improvement was obvious, but with some a little less so, especially those who had strong writing skills to begin with (those whom the placement test showed to be at C1, for instance.) I always worry a little, with advanced students, that they might feel they aren’t being challenged enough, particularly as many have been learning English for years.

The last three were open-ended questions and listing all the answers in this post would make it tediously long, so I opted for a random selection which I think illustrates well what the students found important.

13. List three things that you liked about the course and say why.

Not having to attend classes on campus and being able to complete tasks in pajamas was appreciated by a number of people. Students didn’t feel pressured as they progressed through the course and found it interesting to read one another’s reflections on individual units. Interactive activities and online memo boards got a lot of positive feedback as well. Some students particularly liked the fact that the course was online – they described this as modern and innovative – and liked learning about new technologies.

14. List three things that you did not like about the course and say how they could be improved.

I always ask for suggestions as to how something that a student didn’t like could be improved upon, but I don’t often get them. I don’t know whether this is because people feel it wouldn’t be appropriate if they suggested improvements, or simply don’t feel inspired?

Several people noted the course was time-consuming. Reflecting on the work done in individual units was also often seen as a little too much (yes, it is apparently fun reading other people’s reflections, but not writing your own. 🙂 ) Vying for the title of least popular were the unit on punctuation – I’m planning some serious revision of those materials – and the glossary activities. I found it particularly interesting that at least one person was annoyed at being faced with a new online tool in each unit and having to figure out how it worked. I guess I might have overdone it a little.

15. Do you have any other comments, suggestions or any other feedback? Perhaps you wanted to elaborate on an earlier (multiple choice) question?

I thought it was interesting that the majority of the answers here focused on the method of delivery rather than the content. Those who commented on the online aspect said that online learning is the future and that more courses at their institution should incorporate educational technologies to some extent. And it seems fitting to end by quoting the person who said, “It was definitely an experience!”

Do you include any of these questions when asking for feedback? If there is a question you think should be included or if you have picked up on something in the results which I didn’t address, I would love to hear from you in the comments!


Serena goes to Hollywood

This post is about a language teacher’s experience at MachinEVO 2014. Just in case you don’t read to the end – and yes, it’s a longish post – I would like to say straight away that if they hold it next year, sign up! No, this is not a sponsored post.

For those who are thinking about reading to the end, here we go…

I blame Twitter. There I was in the first week of January, minding my own business, scrolling away through my feed, when I came across news of the EVO sessions. Free PD! Topics galore! Moderators I would love to learn from! This was something I would definitely need to sign up for, but what to choose? Okay, let’s go for something I know nothing about – making movies, in Second Life, no less. I like a challenge, me.

Thus it was that I signed up for MachinEVO, a five-week workshop promising to teach participants to “…produce visually appealing videos in virtual worlds. These videos are commonly called machinima and they can be used for language teaching and learning in Virtual Worlds and in the physical classroom.” I could already see my Moodle course getting a whole new dimension.

My SL experience prior to this was limited to a couple of visits about three years ago, and as I recall, I spent most of the time figuring out why my avatar seemed strangely averse to wearing a whole jacket. First one of the sleeves disappeared, and then the other – and I couldn’t get them back on. What with these more pressing worries, I was obviously unable to focus on the educational potential of SL. 🙂

It was made clear that you could attend MachinEVO even with no experience of virtual worlds, as there would be a special fast track for new residents in the first week, when they would be taught all the necessary practicalities before joining their film crew. And so, Serena joined the motion picture industry.

Now, five weeks later, I look back over the whole experience and think – well, I’m still far from those visually appealing videos. That was a somewhat ambitious goal, at least in my case. But I like to think I learned a lot and it was a hugely enjoyable experience overall. And that’s saying something, because there were moments when I truly despaired. I couldn’t possibly do the whole event justice in one post, so I finally decided to focus on the most challenging as well as the most encouraging parts of the ride. Those are the ones that will stay with me.

Clueless is an understatement

During the first week, I was reasonably satisfied with how things were turning out. The new residents all worked together, and in our sessions with the moderators in-world (online, in SL) we learned how to do undemanding yet practical things, like how to sit on objects or turn up someone’s voice. If we missed a session, there were recordings in Adobe Connect to make sure everyone could keep up. So far, so good. I felt quite confident because, unlike some of the others, I didn’t seem to be experiencing many technical difficulties.

In the second week we split up into groups according to the stories that we wanted to work on. I joined a group practically at random, not really hearing what our movie was going to be about, as my avatar didn’t seem to be wearing any trousers. Everything below my waist looked gray to me, but what did the others see? Should I log off? What if I missed something important? Later on, in the recording, I was relieved to see my jeans. 🙂

Group communication took place in Google docs. When I first looked through our document, I could see there were about ten of us, and there was already a detailed storyboard. Group members were invited to share their experience, skills and expectations, and those who had done so clearly had all three. I briefly considered if there was any way I could say that I lacked the first two and was entirely vague on the third, which wouldn’t make it sound as if I my contribution to the group would be minimal, but eventually decided to say nothing.

I downloaded a screen recording and video editing program and experimented with it a little, making a machinima using still photos. These were provided, along with the instructions, by the moderators, while our twice-weekly sessions progressed to dealing first with basic, then advanced filming and editing techniques. In the meantime, between these sessions, our group began meeting in-world to shoot the storyboard sequences.

It soon turned out that there were far fewer people who were actively participating in the making of our movie than had originally signed up to be in the group. And out of these, I was the only complete beginner. I think I must have driven them crazy – in fact, I’m sure I did – especially at the start, when I got the impression that they all knew each other from before, and had worked together at some point. I needed help with everything, asked a million questions, and produced useless footage. They talked in what seemed to be a foreign language – mesh, prim, sim, rez…what??? – while my avatar bumped into walls like a headless chicken.

And the time everything took! A shoot could easily last for three hours, and did. I was convinced that this was because I was new and was slowing everyone down. But that wasn’t all. Video clips then needed to be uploaded to Dropbox. And the others’ clips had to be downloaded so that all of us could experiment with / work on editing. Sometimes it seemed that I was in SL every (real-life) evening, either for a shoot, or a session, or a special guest appearance, because the organizers had also arranged for well-known machinimatographers to speak to the course participants. I’d had no idea it was going to be so time-consuming.

A MachinEVO session
A MachinEVO session

A sharp learning curve

I cannot emphasize enough that my feeling of incompetence was not brought about by the attitude of the others in my group, or of other MachinEVO participants or moderators. In fact, I can’t remember when I last met a friendlier and more supportive group of people, especially when you take into account the fact that we haven’t actually met. Not in person. And yet, after all that time together, I feel like we have. I thought it would be more or less like just another MOOC, but it was much more intense.

Possibly they’ve all worked as kindergarten teachers at some point in their lives? They had boundless reserves of patience. The moderators were constantly active in our Google group, posting content and commenting, which was incredibly useful both in terms of the content itself and in conveying the reassuring impression that the course itself was alive and well. 🙂 I strongly suspect that some of the members of our small movie crew have discovered the secret to extending the standard 24-hour day, because they seemed to devote quite an incredible amount of time to our production.

Over the past couple of weeks I went from zero to the following:

  • gained a whole new insight into how video can be used in (language) teaching / learning – which I think is crucial; if you can’t see a practical application, you might have doubts about justifying the time invested
  • discovered various opportunities for language learning that virtual worlds offer
  • learned how to work with different video editing software, and not just on making machinima
  • opened a YouTube account and can actually see a use for it in future
  • learned / was reminded of some things that make a difference to successful teamwork – and I’m happy to report that our (geographically) quite diverse team made it to the performing stage)
  • met many educators and other machinimatographers from a range of settings
  • experienced different stages of movie production, which also included coming up with and delivering script lines (I’ll be lucky if I get called for another audition 🙂 )

Bonuses (not really educational, but still):

  • I got to wear blue and pink hair (happily not at the same time)
  • people could pronounce my name (the one I’d chosen for my avatar, actually, which made a nice change)
  • now I know what to do if I a jacket sleeve goes missing

All the groups are currently finishing off their movies and getting ready for the film festival at the end of the month, so I expect that the period of intense collaboration has come to an end for now. I might be reporting again after the awards ceremony and, if so, will probably have more reflections to add. And on that note, a very sincere thank you goes out to everyone involved for making MachinEVO possible. Special thanks to all the moderators, and, most importantly, Group 4 members

And yes, I would be very interested to read the impressions of any of the other participants this year; please let me know if you have blogged / decide to blog about MachinEVO too!


Words in the shape of a fruit

I’ve played around with Wordle on a few occasions previously – if I’m honest, not so much because I was interested in finding out frequency of word use, but more because I found it visually appealing. Content was important, but the packaging even more so. No, I’m not really all that shallow; at the time I was playing community manager for the Octopus Facebook page, and discovered that pictures got a lot more likes than just about anything else. My initial reasoning was that those who followed a language school would much prefer useful recommendations to aid them in language learning to pictures of the teachers having a coffee, but that’s a whole other topic.

Anyway. I remembered Wordle while writing a journal entry as part of the online course I’m currently teaching. We use the Mahara e-portfolio, and each unit ends with students reflecting in a journal entry on what they have learned. As this was the first time most of them – possibly all – were using an e-portfolio to keep a learning diary, I thought it might be helpful to record things in my journal as well. I figured it might clarify things in terms of expectations – purpose, tone, entry length, etc., and may also encourage them to be open about what they (dis)liked when they saw that I wasn’t holding back in my reflections. I think that worked, but again, that’s a whole other topic.

So yes, Wordle. I wanted to address the importance of recycling vocabulary in that particular entry – namely, to describe my attempts to get students to revise the 400-odd new vocabulary items over a period of time through interactive activities, as opposed to leaving it up to them to memorize said vocabulary the night before the exam. I ran all the posts from one particular forum through Wordle and ended up with an appealing visual to accompany my Mahara entry. Luckily, some of the key vocabulary from the unit (as opposed to pronouns and conjunctions) made it into the top 100 words – e.g., “dismissal”, “job-seeker”, “welfare” – which meant I was able to work in a little additional revision, at least for those who read the entry. 🙂

Then it occurred to me that I could do the same for the course noticeboard forum. In Moodle, this forum is used only by the instructor in the course, meaning the students can read posts but can’t reply to them. I left a total of 29 posts – on average 200 words in length – in this forum during our three-month course, and suddenly became quite curious about my top 100 words. What had I actually been saying to the students throughout the course?

First I ran all my posts through Wordle and came up with some attractive visuals, but then the next day I came across Tagxedo and, fickle soul that I sometimes am, decided I liked it even more. I won’t go into detail as to why right now, but one of the main reasons was the fact that words could form recognizable shapes, like those of a cat, dolphin or Abraham Lincoln’s face. Ok, sometimes I am that shallow.

This is a visual of the 100 words I used most in the course noticeboard, in the shape of an apple. It turns out these rather more unadventurous shapes are better for presenting the content – if you’re less interested in seeing the actual words clearly, you can go with a reindeer.

taxgedo 101What jumped out at me as I first scanned the apple was the word “don”, just below the stem. I had no idea what it was doing there! If I was urging my students to put on a piece of clothing or warning them about the perils of organized crime, I was clearly inadvertently doing so. What else was I about to find out?! It turned out – perhaps a touch disappointingly – that the program understood “don’t” to be “don”. This would indicate the need to adjust the punctuation settings, something that I can’t remember having to pay any attention to when I used Wordle.

Anyway, the words used most often are “course”, “unit”, “activities”, “topic”, “know”, “feedback”, “posts”, “work”, “page” and “assignment”. I can’t really say that this came as a surprise, because I would have probably listed quite a few of them if I’d had the sense to note down my expectations before Tagxeding the posts. However, it is interesting for me to reflect on the context in which they were used and whether some could/should have made fewer appearances. My post was originally going to include this analysis – it was going to be about the words used, not about e-portfolios or the merits of Wordle vs Tagxedo – but I’ve apparently found it difficult to stick to the topic. 😦 I’m therefore going to stop here and look at the actual word use in another post.