Committed to changing the world for the better, University College London (UCL) has long been at the forefront of innovative thinking. Established in 1826, it immediately provided an open education in England to students of any race, class or religion. And in 1878 this was extended to welcome female students on equal terms with men.
It is this culture of pushing forward for the long-term benefit of humanity that has seen the UCL Medical School advance the discovery of adrenaline and understanding of the immune system and autoimmune disease. No less than six former students and staff have been awarded Nobel Prizes.
In 2016, UCL Medical School invested in Speedwell’s eSystem exam software initially to replace their aging question banking software, but also in the knowledge that they were future-proofing their assessments and would be able to make the transition from paper-based to online when the time came. As the new question bank software began to show operational efficiencies and drive improvements, thoughts quickly turned to how Speedwell’s eSystem could deliver the same results for the Medical School’s OSCE exams.
(Read the question banking case study here)
Strategic considerations
The Medical School runs OSCE style exams in all years of the MBBS Programme, with each cohort being around 350 students. In the first year there is a formative OSCE, with all other years having a summative end of year OSCE exam. Historically, the OSCE assessments were run using Speedwell’s CliniQuest, alongside an old question bank database (prior to updating to the eSystem question bank). OMR question papers would then be printed for the exam and scanned for marking post-exam, with the final qualifying exams generating 16,000 individual OMR sheets. The process was quite manual, reliant on a couple of different systems and time consuming with more margin for error than the Medical School were comfortable with.
Gaynor Jones, Head of MBBS Management at UCL and ultimately responsible for delivery of all assessments at the Medical School, had very clear priorities for moving to digital OSCEs.
Primarily she wanted to reduce the administrative burden and stress for all involved. There needed to be a reduction in preparation and processing times – with the old system it would take two people a week to print, pack and check everything required for each OSCE exam, creating a lot of pressure on the team.
Gaynor also saw opportunities to improve and simplify the authoring process and eliminate potential examiner errors such as missing sheets and/ or marks and the stresses on the team in checking and reconciling marks. Finally, another huge driver was to have the capability to be able to provide students with their exam results and feedback in a more timely and efficient manner.
The Challenges
By their nature, OSCE’s are inherently complex with many elements involved in their set-up and delivery. So, it was never going to be an easy decision to change the way they were run, and understandably there was apprehension when considering the move. From an infrastructure perspective Gaynor mentions that initially there were concerns about the strength of the Wi-Fi in off-site exam venues and although there was the option to run OSCEs offline via a dedicated app, the fear was that the local networks might drop out.
Additionally, getting buy-in from key internal stakeholders was another consideration. According to Gaynor “When the idea of moving to digital OSCEs was initially introduced there was an element of fear. With the old system everyone knew what they were doing and were confident with it.” It was clear that this was going to represent a step-change for the Medical School, but one that was ultimately necessary if they were to achieve their goals.
Delivering the first digital OSCE
Prior to running the first exam, the decision was taken to ask Speedwell to Cloud host the software. The team felt this would give greater local control over the speed and ease with which software updates could be completed. Moreover, they could be confident that they were always running the latest version.
For Gaynor and her team, it made sense to pilot the eSystem with the first-year formative assessment, which was an 8 station OSCE. The intention was that any potential issues could be ironed out ahead of the higher stake summative exams later in the year. When it came to running the actual exam in February 2018 it went very well. Everyone involved was delighted and were completely on board to use the software to run the first summative OSCE exam in May 2018.
Gaynor reports that although there were a couple of minor teething issues with the newly released offline OSCE app, they were quickly addressed by Speedwell. Her team was able to provide feedback that allowed new features to be developed to support and improve the app for future examinations. For example, as a result, a review page was added so examiners are able to review all stations at the end of the examination and ensure that nothing has been omitted. It meant that by the time the first summative exam was run, everything had been addressed and it all ran smoothly. And regarding the initial concern around Wi-Fi and networks, Gaynor found that in reality they needn’t have worried; “The fact that you can run them online or offline means that there is always a back-up anyway, and we found that it was fine – we didn’t experience any problems in that regard”
Gaynor continues; “It was amazing, after the success of the first exams all initial apprehension was completely gone. Everyone was delighted and once they had seen it working so well there was pressure to move all of the OSCEs over as soon as possible!” The Medical School ran two more digital OSCE exams in July 2018 and the team’s intention is that for the 2018/2019 academic year all OSCEs will be run this way.
Exceeding expectations
Using the eSystem to run their OSCE assessments, the team have seen huge gains in preparation and processing times, seeing a reduction from 2 weeks to around 2/3 days. Additionally, they have greater flexibility when setting up exams and making late changes.
It has also enabled remote authoring and station access to be set up. Prior to this it would involve exporting questions and physically sending them to authors, who would work on them and then send them back. This would present issues with version control, but since using the software the process is much quicker with greater visibility and certainty in version control.
But perhaps the main benefit has been in the results processing; historically, it would have taken the team several days to check and then process the scanning. Now the results are available immediately and it’s possible to send students examiner feedback on the same day as their results. This feature is something that all at the Medical School feel is a huge benefit, particularly for their students.
Taylor Bennie, MBBS Systems Project Manager and Learning Technologist at UCL was responsible for getting the exams set up and running with Speedwell’s eSystem. He has been pleased with how intuitive the software has been for all involved; “Most, if not all staff found the system easy to use and have enjoyed the process”
For Gaynor and her team all expectations for running digital OSCEs were exceeded. She concludes,“Speedwell’s eSystem has revolutionised our OSCE assessments. The exam leads, examiners and administrators are all delighted and the students feel that we have turned assessment feedback around.”
Looking to the future
The Medical School is now delivering a full academic year of digital OSCEs and has plans in place to pilot online written exams both on fixed PCs and tablets in the near future.
The success of the OSCEs has greatly enhanced confidence in online systems within the Medical School. And aside from the challenge of finding a location with enough PCs, Gaynor claims everyone is much more comfortable with the concept of digital assessments. There is an increased appetite for moving to digital SBAs as soon as possible: “They’ve seen the OSCEs work so well and are asking why can’t we do this with written papers as well? Everyone is thinking there are no barriers anymore we should just get on and do it”
She continues that this increased drive is a good thing, especially in consideration of the forthcoming online national medical exam. (The Medical Licencing Assessment (MLA) is due to be implemented in 2022 and all undergraduate medical students will need to pass it in the future). “We are now very confident that we will be able to achieve this – we didn’t have any confidence before at all!”
And Gaynor’s ambitions don’t end here. Her vision is to eventually integrate Speedwell’s eSystem with their own student record database. The idea is that students will be able to access all of their information via one portal, so they will be able to log in and see their exam results and feedback within their own record, putting them firmly in control.
This will give students an enhanced experience and meet their desire for detailed and timely feedback. Furthermore, it will cut down on administration, ease the pressure and stress associated with results time and drive more process improvements and operational efficiencies within the Medical School.
This can only be beneficial for all involved.
If you would like to find out more about eSystem, any of its features, or arrange a personal demonstration then contact us. Call our sales team on +44 (0) 1223 851703 or email info@speedwellsoftware.com