Proficiency informed knowledge tracing: Leveraging large language models to enhance predictive methods in mathematics learning

Abstract

Predictive methods play a key role in learning systems to enable effective interventions. However, most existing approaches emphasize concept-level suggestions and pay limited attention to cognitive constructs aligned with instructional goals. For instance, in mathematics, mastery depends on the integrated and balanced development of multiple interwoven proficiency strands. When predictive outputs do not reflect instructional objectives, educators may struggle to trust or utilize the insights generated by these models. To address this, we propose a predictive framework that incorporates cognitive construct-aligned features into the modeling process. The research focuses on elementary mathematics, where proficiency strands represent fundamental cognitive dimensions of learning. Since manually labeling these strands is time-intensive for experts, large language models (LLMs) are used to classify semantic features. These features are integrated into the proposed Proficiency-informed Knowledge Tracing (PKT) model as auxiliary signals to enhance mastery estimation. PKT is evaluated on the XES3G5M dataset, which contains elementary mathematics problems, and is compared with established deep learning models. The results indicate that PKT achieves overall competitive performance and surpasses base models, demonstrating a capacity to capture latent mastery patterns. Further experiments reveal that integrating LLM-generated features improves model calibration, as measured by Mean Absolute Error (MAE) and Root Mean Square Error (RMSE), compared to base models. PKT ranked first overall when predictive performance and calibration were considered together, followed by AKT and sparseKT. Additionally, the model generates interpretable diagnostic profiles at both cognitive and concept levels. These findings underscore the value of incorporating cognitively relevant constructs into knowledge tracing.