Best Research

Our research project is about making sure that a student’s results mean the same thing whether they take an exam on paper or on a computer. As Cambridge introduces digital versions of high-stakes exams, we need to check that the digital and paper exams are similar in difficulty and that they test the same knowledge and skills, rather than introducing advantages or disadvantages simply because of the format. For example, an item might feel harder on screen because of scrolling, layout, or how students enter their answer. We use evidence from our Digital Mocks Service and compare it with evidence from past paper exams. We look at overall results to see whether the digital and paper versions perform similarly, and we also look closely at individual questions to spot any that behave differently on screen than on paper. Where we find differences, we investigate what might be causing them and use that learning to improve future digital exams. In short, the aim is simple: whether a student takes the exam on paper or on screen, they should be assessed on the same standards in the same way.

Imagine you’re creating exam questions. AI can help draft them, but sometimes it makes mistakes, like choosing the wrong correct answer, writing unclear wording, or aiming at the wrong age group. In exams, those mistakes are a big problem. Our project makes AI-generated questions safer and more trustworthy by adding a “quality inspector” step. First, an AI writes a draft question. Then, separate AI “checkers” review it against a clear checklist of what a good assessment question looks like: is it correct, clear, the right level, fair, and suitable? If something is wrong, the system flags it so humans don’t waste time on poor questions. We also tested the inspectors by deliberately breaking questions (for example, swapping the correct answer) to see if the checkers caught the errors. Using multiple independent checkers and combining their votes made the system more reliable.

Imagine you're a teacher creating a test. Some questions are easy, some are hard. You want to give hard ones to high-competency students for further practice. Traditionally, you may use your subject-matter expertise to estimate the question difficulty, which is inherently subjective, or you may need to test questions on hundreds of students, which is expensive and time-consuming. Our research uses artificial intelligence, specifically, language models similar to ChatGPT, to predict how difficult a test question will be just by reading the question. We trained AI models to predict question difficulties without collecting student answers to the questions. In training the model, we generated synthetic variations of existing questions to fill the gaps where our data for training the models did not contain enough difficult questions. We also tested whether newer, more powerful AI models (like GPT-4) could do this better than smaller, specialized ones we trained. Our approach outperformed all competitors in an international competition. For testing organizations, this means they can save more items after field-testing and avoid assigning all difficult or easy items to one form, but create more balanced, parallel, and fairer tests more efficiently without needing to test every question on thousands of students.