A human-in-the-loop system for labeling knowledge components in Japanese mathematics exercises

Abstract

Many modern learning systems rely on a data representation of the knowledge that is to be learned to estimate a learner’s mastery state and recommend appropriate learning tasks to further improve their acquisition of knowledge and skills. In particular, the rapid development of intelligent tutoring systems (ITS) and standardized curricula has increased the need for information on knowledge structures and their links to learning materials and tasks. However, manually labeling educational data has traditionally been a time-consuming, labor-intensive task, and thus has limited its use by time-constrained teachers and practitioners. In previous research, a range of machine-learning methods have been proposed to address this problem, with only a few of them focusing on Japanese educational datasets from secondary schools. In this paper, to support the labeling of Japanese mathematics exercises by teachers and other domain experts, we apply natural language processing techniques including word-embedding and key-phrase-based exercise-to-exercise similarity methods. We evaluated the proposed method by both the performance of the models when compared to several state-of-the-art methods, and also its effectiveness in supporting humans in the task of labeling educational materials. Through this two-phase evaluation, we found that the proposed method outperformed other methods, and when implemented in a human-in-the-loop system it achieved significantly more accuracy and consumed less time for the task of labeling mathematics exercises.