The potential of large language models for the assessment of cognitive engagement in informal online communities

Abstract

Studying the cognitive engagement shown in forum messages of informal online communities can help to associate discussion behaviour with underlying learning processes. The ICAP framework can serve as a theoretical lens for labelling forum messages by distinguishing different levels of cognitive engagement, yet requires time-consuming, high-inference human coding, which is not feasible at scale. Although this can be mitigated by supervised classification models, these rely on large amounts of labelled training data, and their decision-making often remains opaque. In contrast to this, Large Language Models (LLMs) work “out-of-the-box” for qualitative coding tasks via natural language instruction and can produce reasoning traces and explanations for their judgments. In this study, we explore this potential by integrating the ICAP framework into natural-language prompts for few-shot in-context classification, comparing LLM judgments and explanations with those of human raters. Results indicate that LLM judgments generally align with the human ratings, while their reasoning traces and variations offer additional insight into where the coding scheme invites ambiguous interpretations. This suggests that LLMs could serve as cognitive proxies that help diagnose and improve high-inference coding processes. Finally, we apply our approach to large-scale forum data, revealing cross-platform differences in expressed cognitive engagement.