Using face, gaze, and code features detected with off-the-shelf equipment to derive student emotion in an introduction to programming context

Abstract

Students in an online Introduction to Programming (CS 1) class suffer a severe disadvantage. In a face-to-face learning environment, teachers can see if students struggle with the assigned programming task through their body language or overall demeanor. In online learning, all that the teachers see is a small image of their student, if at all, and are thus unable to provide the needed intervention to help the student overcome their stumbling block. Our research investigates extrapolating student emotion through a web camera and code logs gathered while students work on programming exercises, with a hardware-based gaze tracker to serve as ground truth for software-based gaze tracking. Code, face, and gaze data, together with annotated emotion data gathered from our experiments, were used to train learning models via XGBoost. Our best model can predict a student’s emotional state at a precision, recall, and F1-score of 72.7%, 77.3%, and 74.9%, respectively. We achieved these results from analyzing 8 hours and 20 minutes of experimental data from 26 participants using only software-based gaze tracking, with statistically similar results to hardware-based gaze tracking to within a ±8% equivalence margin.