“Incredibly helpful, stressful, time saving, time consuming” are just some of the ways that digital assessment marking has been described.
We have spent the last month exploring digital assessment with some of our members. We wanted to understand their experiences, listen to the challenges, and learn about the different ways institutions are using digital assessments and in particular marking.
Some of our members we spoke to had to move online quickly because of covid and others have been marking and assessing online for multiple years.
As you can see from the first paragraph of this blog, people’s experiences have been very mixed. Below are some of the main insights we discovered.
Onboarding & Training:
This topic came up in every conversation and institutions are tackling it in many ways. Some have a specific team in place that create the assessments for academic staff, these teams are very popular in their institutions. Other members talked about needing to create their own assessment. There are teams to support the educators, but they don’t build the assessment for them. Some members found the creating of assessments quite stressful as they didn’t feel confident with the design aspect and features of their chosen assessment tool. They felt that if the setup is stressful then it has a knock-on effect to the marking as well.
Of course, this is nothing new, but some members wonder whether standardised training for both academic staff and their students could be part of the solution to this challenge. Is this something the provider could offer?
Another point about digital assessment that became clear was that some institutions tend to initially create an online version of a paper exam rather than create a new way of assessing students. If educators were able to have more flexibility in creating new assessments, could this be another, easier way to help solve the training issue? Yet another hypothetical question but one that everyone agrees needs to be explored.
Digital marking:
Saving time
Overall, the educators we spoke to were delighted to be marking online. The majority said it saved time due to things such as students marks automatically being added up, being able to type short notes, multiple choice being automatically marked, the use of digital rubrics which really helps to organise both the markers and students’ thoughts etc.
Many agreed that it was stressful when they initially went over to digital marking but once they became familiar with their chosen tool, marking become much easier.
As well as hearing about the positives, we were interested to learn about the challenges that our members are facing with digital marking.
Organising papers and analysing marking
One member painted a vivid picture by describing how they used to mark and how they had a big pile of papers on their desk. They talked about the motivation they felt when they could physically see the pile getting smaller and smaller. Of course, this doesn’t happen with digital marking, so it is hard to know where one is at in the process especially if there is a large cohort of students’ work to be marked. The time spent staring at screens when marking is also taking its toll on some educators too.
They also described how they would moderate their marks as they progressed going back to change previous marks after correcting a few papers. Although possible using marking software it was more difficult to do. Grouping papers together with similar marks or answers was another approach used when marking, this feature was also missing from digital marking tools.
Having suitable questions to fit into their chosen digital assessment platform. For example, not all digital assessment systems support long essay questions in examinations. Leading to a need to break down the essay question into short answer questions, a feature supported by the software. It was easier to mark, the students were also able to demonstrate the learning outcomes better with a more scaffolded approach.
How do you ensure that the questions you have selected are suitable in a way that assesses what the student has learnt. How do we show critical understanding rather than information that has simply been retained?
Again, does this go back to the point about transferring assessments from paper to digital instead of creating new assessments that could possibly be a better fit through technology?
Marking mathematical subjects
We were told that people have complicated associations with maths and therefore need to be encouraged to play through different assessment styles.
We heard about an assessment where students can essentially have as many goes as they like over a certain period; all inputs are captured, and their highest score is taken. This assessment style means they are repeatedly practising maths which is the whole objective. The institution found that a lot of anxiety is taken away when students can have as many goes as they like. Some students did say it felt a bit like cheating but understood the focus here is practice.
There was also a clear message about the need for digital marking systems to work with emerging tools that support mathematical questions as it is quite a specialist area at present. For instance, when staff are thinking about written feedback, they look to annotate through their chosen assessment platform or VLE. A lot of text is usually lost as the VLE at times doesn’t understand the notations. Symbols would be mistaken, and the wrong ones added. We were told that having the ability to submit flexible documents, such as HTML files, and not only PDFs would make a big difference too as PDFs don’t always lend themselves to maths.
We were told that it would be helpful to have an onboard tech compiler so educators could offer genuine maths feedback. It would also be helpful to expand on the coded languages currently in use on certain assessment platforms.
A big question is how as a sector can we make maths rich content available online so students can access it? We were told that there are things that can be done on the edges such as Vivas but we need the whole sector to ‘grab hold of this’ to really make a difference. It is believed that solutions are getting better all the time but some wonder if there is a risk of having too many options to choose from and the sector misses out on the best way forward.
Providing feedback:
Feedback is a very complicated topic as there are so many layers to be considered. We heard about the challenge they have feeding back to their students who are from a wide range of cultural backgrounds, one size doesn’t fit all for feedback is what we were told. They said some students are very happy to be given direct feedback, others struggle with that approach and prefer a softer way of receiving the feedback. We talked about the importance of being sensitive and caring about the way feedback is delivered and the need to have several different ways to provide this feedback.
Giving feedback in a voice note came up as a way for students to hear the support and kindness in their tutor’s voice and that helps them be more open to the feedback. Interestingly we also learnt that some academics dislike recording their voices when giving feedback in case they inadvertently say something that could be taken in a different way to how it was meant and accidentally offended their student.
Is the answer to type all feedback to students? That of course is an option that many use but as one of our members told us, this takes so much time as they tend to type more than when they were marking by hand and data shows (across most institutions we spoke to) that the students on many occasions don’t read what has been typed. Could this connect in with the standardised training that was mentioned earlier in this blog? Could there be a standardised way to feedback too? One where students and staff both feel confident and understand what is expected of them regarding how the feedback will be delivered.
This takes me back to my opening sentence about feedback… it’s complicated!
Conclusion:
One thing is for sure, we have uncovered more questions than answers, but this is assessment for you and the reason why we will be doing a lot more work in this area to help understand how to best support our members.
We are running a simultaneous project where we are looking at the possible future of assessment, how it could change and what it could look like. Please see some information about it here: https://alternativeassessments.wordpress.com/2022/06/09/what-will-assessment-look-like-in-2035/
We will ensure to share all digital assessment insights we discover, continue to ask more questions and hopefully together with our members start exploring some viable solutions.
For further information contact innovation@jisc.ac.uk