YINSIGHT: Supporting data-informed competency assessment with customizable indicators in a self-regulated learning context

Abstract

Recently, educational assessment has shifted focus toward evaluating not only performance but also learners' attitudes and behaviors toward learning, known as competency assessment. Traditional methods, such as self-report sheets and teacher observations, are limited by bias and reliability. With the rise of ICT tools, learning trace data offer a promising solution for assessing learning processes more reliably. However, existing frameworks for competency assessment based on trace data lack flexibility in real-world applications, prompting the need for customization of the framework according to user needs. To address this gap, this study introduces YINSIGHT, a system that allows users to customize competency assessment indicators according to specific contexts and needs. We outlined the framework for creating these indicators, implemented the YINSIGHT system, and evaluated its effectiveness through semi-structured interviews within a scenario of self-regulated learning. The participants were two English and one math teacher from a high school in Japan. The thematic analysis of interviews revealed that while traditional competency assessments rely heavily on performance-based methods, teachers expressed expectations for YINSIGHT's ability to capture self-directed learning activities, particularly in extensive reading contexts. However, significant concerns emerged regarding system usability problems and compatibility with current practices. Teachers also provided constructive suggestions for gradual implementation and system improvements to address these barriers. This study thus contributes to the continuous improvement of learning and teaching from multiple perspectives on the activities that use the system.