Most Innovative Use of Technology in Assessment Award

With the rise of online assessments and large language model-driven item generation, efficient item piloting and calibration are critical. To ensure test security and score validity of large-scale Computerized Adaptive Testing (CAT), item banks must be continuously refreshed. However, traditional piloting and item calibration methods are too resource-intensive, creating bottlenecks in test development. To address these challenges, we developed a CAT system integrating three innovative components. Together, these components provide a cost-effective solution for high-stakes testing, supporting CATs with large pools of automatically generated items. We successfully used this system to introduce thousands of new vocabulary items to the Duolingo English Test (DET) while maintaining score validity. Joint calibration of six item types processed over 38 million responses in five hours, demonstrating scalability and readiness for large-scale implementation.

The Singapore Examinations and Assessment Board (SEAB), under the Ministry of Education, oversees national and non-national exams across 400 schools, supporting 80,000 annual registrations. Traditional exam methods that relied solely on internet connectivity posed challenges in ensuring exam integrity and candidate performance. Similarly for delivering national level test across 400 schools traditionally required a solution that mandated local installation and administration at the schools to ensure uninterrupted testing. Additionally, the language proficiency testing required a more flexible and efficient approach to conduct oral exams efficiently in scale. To address these challenges, SEAB undertook a macroscale transformation, transitioning to a web-based, cloud-hosted assessment solution. This innovative system integrates cloud computing with offline functionality, enabling seamless exam delivery across all environments.