M.A. Anisia Katinskaia defends her doctoral thesis "Assessing Learner Answers in Computer-Aided Language Learning" on Friday the 25th of April 2025 at 10 o'clock in the University of Helsinki Unioninkatu 35 building, Auditorium 116 (Unioninkatu 35, 1st floor). Her opponent is Associate Professor Helen Yannakoudakis (King's College London, United Kingdom) and custos Professor Petri Myllymäki (University of Helsinki). The defence will be held in English.
The thesis of Anisia Katinskaia is a part of research done in the Department of Computer Science and in the Department of Digital Humanities at the University of Helsinki. Her supervisor has been Professor Roman Yangarber (University of Helsinki).
Assessing Learner Answers in Computer-Aided Language Learning
With the growth of language technology, remote learning, and student mobility, interest in Computer-Aided Language Learning (CALL) has increased. Revita is a CALL system with a particular focus on the needs of advanced learners. This thesis investigates how to assess learners' answers to grammar exercises generated by Revita based on user-selected texts. The flexibility of this exercise generation approach often results in multiple grammatically and semantically valid answers---referred to as alternative-correct (AC) answers.
The thesis provides an overview of CALL systems, their structure, and their functions, with an emphasis on the ``computer-as-a-tutor'' modality. We then outline the main principles and components of Revita, focusing on the system of linguistic constructs that affect exercise generation, feedback, and student modeling. Following that, the thesis describes the collection and annotation of the ReLCo learner corpus, which is created based on Revita's database of answers, and presents a typology of the most frequent AC answers identified during the annotation process.
We approach the assessment of learner answers using three methods. First, sequence classification predicts whether the context with a given answer is grammatically correct. The second approach involves grammatical error detection (GED), where tokens are classified as grammatical or erroneous. The GED-based approach allows for the simultaneous assessment of multiple answers and provides an estimate of predictive uncertainty. Third, we apply grammatical error correction (GEC) to assess learner answers, based on the hypothesis that correct answers should not be edited by the GEC model. We experiment with a T5-based GEC model, fine-tuned on both synthetic and real learner data, with and without re-ranking of correction hypotheses. Evaluation—both automatic and manual—shows that assessing AC answers is more challenging for GEC models than identifying clear errors.
We then focus on a specific type of AC answers provided in exercises on verbal aspect. We explore how several Transformer models encode verbal aspect using behavioral and causal probing methods. To probe aspect, we propose interventions into the semantics of boundedness within the context. Our findings indicate that the category of aspect is encoded in ways that align with linguistic theory. However, contexts that allow alternative-correct aspect forms are especially challenging for both neural models and humans, due to the absence of explicit cues indicating whether the described action is bounded or unbounded. These probing experiments demonstrate that different types of AC answers require separate investigations.
Availability of the dissertation
An electronic version of the doctoral dissertation will be available in the University of Helsinki open repository Helda at http://urn.fi/URN:ISBN:978-952-84-1303-5.
Printed copies will be available on request from Anisia Katinskaia: anisia.katinskaia@helsinki.fi