The DigiTala Moodle plugin was tested in upper secondary schools and universities in spring 2023. The purpose of the testing was to find out the strengths and weaknesses of the DigiTala tool and assessment models. Nine different student groups were involved in the testing, of which five were Finnish language students and four Swedish students. The groups variously consisted of about 5-20 students.
During the spring, a research assistant from the DigiTala research project visited schools to guide the student groups with testing. At first, the groups were told about the DigiTala research project in general and then instructed in more detail on how to use the tool. The students completed two DigiTala assignments in Moodle: one read-aloud and one freeform assignment. After submitting the performances, the students received an evaluation report produced by the machine, which scored and verbally evaluated different dimensions of speech, such as fluency and pronunciation. In addition, freeform assignments also receive a holistic assessment, which assesses the student's competence at skill levels A1–C2 of the CEFR.
After testing, the students responded to a Webropol questionnaire asking for the testers' opinions on different aspects of the tool: user experience, technical aspects and evaluation report. In addition the students were allowed to present their own suggestions for improving the tool. The survey received 80 responses from students.
Feedback in all areas of the survey was very positive. This is despite the fact that the Swedish language groups had certain difficulties with the interaction between the tool and the assessment server, so that not all students received comprehensive assessments of all aspects of their speech performance. The students felt that the use of the tool was mostly clear and fluent. Majority of the students said the evaluation report was received quickly enough and its content was considered at least fairly useful.
The biggest difference between the Finnish and Swedish groups was whether the automatic assessment corresponded to the perception of one's own language skills and whether the student still missed a human assessment in addition to the machine assessment. Around three-quarters of Finnish learners considered automatic assessment to be in line with their own language proficiency assessment, and nearly as many considered automatic assessment sufficient and did not need a human assessment in addition. On the other hand, only half of the Swedish learners said that the automatic assessment corresponded to their own perception of the level of their language skills, and more than four fifths also missed a human assessment. This difference may well be explained by technical problems encountered in the automatic assessment specifically in the Swedish language groups.
User tests show that the attitude towards automatic speech assessment and the DigiTala plugin is very positive, but there is still room for improvement. Known issues that also occurred during testing include unsuitability of the plugin for certain browsers and mobile devices. In addition, some of the student testers found some elements of the user interface unpleasant: the buttons were too close to each other in places, the transitions between different sections were unclear, and the coloring of the tool could still be fine-tuned. All in all, however, the tests left a positive impression on both the testers and student groups. Users provided valuable perspectives for the further development of the tool.