Our publication about „On Using an LLM-Based Approach for Automatic Evaluation of Programming Tasks in MOOC Videos“ has been published.
Abstract:
Interactive videos promote learner engagement by transforming the passive activity of watching videos into an active learning process. LIVE, a web platform developed in previous research that allows for the integration of interactive elements into videos, also supports embedding programming tasks. However, at the beginning of our research, the output of the submitted solution was automatically compared to the reference output, but the code itself still had to be checked manually, as no automatic process was available to verify the solution and provide feedback. To address this, a stand-alone application was developed that automatically evaluates students’ solutions to Python programming tasks using a self-hosted large language model. This application was then integrated into the existing platform. To test this approach, a massive open online course containing four interactive videos with several programming tasks and quizzes was created. Students’ performance was analyzed using quiz results, task submissions, self-assessments, and user statistics provided by the course platform and LIVE. The results indicate that utilizing a large language model for automated feedback can positively impact learning outcomes. However, some limitations must be taken into account, such as the need to correct some of the model’s generated outputs manually due to insufficient feedback quality for certain tasks. This suggests that the combination of automated and manual evaluation seems to be the most effective approach for now.
[full article @ publisher’s homepage]
[draft @ ResearchGate]
Reference: Kindlhofer, C., Wachtler, J., Ebner, M. (2026). On Using an LLM-Based Approach for Automatic Evaluation of Programming Tasks in MOOC Videos. In: Smith, B.K., Borge, M. (eds) Learning and Collaboration Technologies. HCII 2026. Lecture Notes in Computer Science, vol 16733. Springer, Cham. https://doi.org/10.1007/978-3-032-30784-2_13