Exploring the challenges and approaches to delivering digital assessments on campus at scale

Background

The pandemic was a driver for many institutions to move towards doing more digital assessments. By necessity these assessments were delivered remotely using a variety of software products that could manage the assessment process and provide different levels of security and checking.

This demonstrated the technical feasibility of delivering remote digital exams at scale.  However, several challenges emerged around academic integrity and student collusion. This has increased since the release of ChatGPT3 in Nov 2022.

Professional bodies such as law, accountancy and medicine (and others) although happy to adopt innovative approaches to assessment and the use of digital exams, still require a process that guarantees the authenticity of the person sitting the exam. For remote assessment many required some form of online proctoring, which many students found intrusive on personal privacy and increased their anxiety. Personal space at home for remote assessments was also often a challenge, along with access to a suitable laptop and reliable wifi connectivity.

As a result, we have seen a move back to exams being implemented in a face to face venue where the identity of the student can be verified and the examination invigilated to stop any cheating.

To meet these requirements we could consider several options including

• Return to traditional paper based exams – losing any advantages and the investment of moving to digital exams.
• Implement a programme of assessment transformation to make assessments more personal, authentic, etc. to address academic integrity concerns. This could include moving away from exams altogether and having continuous assessment. This might be the long-term solution but this option is not easy to implement, especially for exams set by professional bodies.
• Finding a way to deliver digital exams face to face – which is the focus of this post.

Assessment at scale

Before we go any further I should be clear that the real challenge is not implementing a digital examination on campus for groups 50, 100, even 250 students. I remember doing that 20+ years ago, using existing computer labs and cycling students through in groups.

The challenge is doing 10,000s of exams, at scale with groups of maybe a 500+ students at a time. Replicating what most institutions have been doing for years at end of term or year-end exams with large student populations.

Given the choice and more time, the sector may find a better way to assess students. So I’d like to think this was just a detour in the journey towards greater assessment transformation.

The Challenges

We were approached by the University of Warwick who had already done some exploration and research around how to implement large scale digital assessments. As such they proposed there were broadly three approaches to enable large scale digital assessments

1.Bring Your Own Device BYOD: a student brings their own laptop to campus and sits the examination in a controlled campus environment, under examination conditions.

2.University-managed devices: a student takes the examination on a university-managed device in a controlled campus environment, under examination conditions. Using an existing computer lab or temporary space that has been established just for the examinations period.

3.A third party offering: a specialist third party providing equipment and facilities to support an online examination in a controlled environment (which may or may not be strictly ‘on-campus’), to a specification provided by the University.

They had also considered a number of challenges that needed to be considered across all these approaches

Assessment spaces

• Use the same spaces as used for hand-written exams. Adapting them for digital exams.
• Using existing computer labs.
• Finding a suitable external venue.

Access to laptops

• Bring your own device – which stills require an institution to provide spare machines.
• Use existing labs which limits the numbers and has competing access needs around exam time.
• Third party supplier who provides the machines ready configured.

Technical infrastructure requirements

• Wifi or wired connectivity.
• Power – for laptops and machines, power packs, etc.
• Server requirements to meet user demands.
• Technical specification of machines.

Access and software

• Student authentication – logins for assessment.
• Secure software and a secure delivery platform.
• Locked down laptops.

Accessibility and inclusion

• Any additional requirements to meet student needs.
• Noise and distraction – allowing use of noise cancelling headset for example.

Invigilation, staff training

• Technical support staff required.
• Staff need training is how to run digital exams, deal with problems etc.

The issues above were used to create a survey that was sent out to a small list of institutions. I will share some insights from the survey results in a follow-up blog post.

We would welcome your thoughts or comments please contact us on innovation@jisc.ac.uk