Join us for an insightful ALTE webinar titled "Rapid evidence-based iteration in digital-first test development", presented by experts from Duolingo on Monday 13th October, at 3pm Central European Time.
Abstract of the presentation
Since 2019, The Duolingo English Test has made yearly updates centered around the development of new, innovative complex assessment tasks that are automatically generated and scored. These annual updates to the test are underpinned by systems that improve efficiency in test development through novel applications of new and existing technology, such as generative AI for automated item generation, an opt-in experimentation platform for large-scale data collection, and a content management system that supports efficient large-scale human review. These systems support large-scale data collection, rapid iteration, and evidence-based decision-making. This talk will discuss how we have combined assessment best practices with technological innovation to accelerate test development at scale.
We start by identifying gaps in the construct or opportunities to expand the construct coverage that technological advances can help address. To address these, we experiment with different task specifications through a combination of generative AI for item generation and the opt-in experimentation platform that allows us to collect thousands of responses from real test takers for the purpose of preoperational testing. Through these, we are able to collect evidence to support informed decisions about task and item design, such as visual layouts, content requirements, response formats, grading rubrics, and more. To support scalable, long-term item generation processes for these tasks, we develop automated generation pipelines and content guidelines for human experts in their review and revision of generated items. We additionally develop and implement automated measures to ensure the final item bank meets our quality standards.
This process supports a new paradigm in test development that allows for more significant and frequent transformations of assessments that can better adapt to the evolving construct for test takers in a digital world.
Biodata presenters
Yigal Attali is an assessment scientist at Duolingo, specializing in AIG and automatic scoring. He holds a PhD in Cognitive Psychology from the Hebrew University of Jerusalem.
Yena Park is an assessment scientist at Duolingo, working in test development for the Duolingo English Test. She holds a PhD in second language studies specializing in language assessment.
Andrew Runge is an AI Research Engineer at Duolingo. He works in test development for the Duolingo English Test, where he specializes in automatic item generation. He holds a master’s in Language Technologies from Carnegie Mellon University
The webinar will be delivered via Microsoft Teams and the link to access the event will be sent before the event.
The webinar will be video-recorded.
The deadline for the registration is Friday 10th October.