Automatic Rating of Students’ Discussion Forum Posts based on Cognitive Presence Component in the Community of Inquiry Model

Abstract

Abstract The Community of Inquiry (CoI) framework has emerged as a significant tool for evaluating learning dynamics within online discussion forums, particularly in fostering critical thinking and knowledge construction. Cognitive presence, a key component of the CoI framework, reflects the thoughtful reconstruction of knowledge and problem-solving processes inherent in the learning journey. Researchers utilizing machine learning techniques for text analysis have shown considerable promise in automatically analyzing phases of cognitive presence in online discussions. Therefore, this study was started by exploring and analyzing those existing automated methods used in online discussion forums to identify indicators of cognitive presence within discussion forum posts. However, none of the existing methods has attempted to implement a rating mechanism that provides consistent ratings, improving the efficiency and scalability of assessing cognitive presence, allowing for large-scale analysis of online discussion forums. Therefore, this study aims to develop an automatic rating model that utilizes cognitive presence in students' posts and evaluate its performance and accuracy by comparing it with manual coding. Our model employed a novel approach that integrated machine learning techniques, specifically random forest (RF) classification coupled with TF-IDF feature extraction, Support Vector Machine (SVM) classification with Word2Vec embedding, and a rule-based classifier built on indicators mapping, to evaluate cognitive presence within a collaborative learning environment's online discussion forum. Additionally, we combined the predictions of these models into a weighted voting classifier. The model was trained and tested on a dataset comprising 781 messages containing 47,592 words. Our findings indicated that our classifiers achieved an impressive accuracy rate of 69%, underscoring their potential efficacy in assessing the quality of student learning in online discussion forums. This highlights the robustness of the combined approach in enhancing classification performance. Furthermore, we developed a rating model using the weighted voting classifier, assigning ratings to each student's posts based on the cognitive presence category they fall into. This model facilitated the calculation of individual student scores based on their contributions to the discussion, providing a comprehensive assessment of each student's engagement and participation level. Unlike existing literature, our approach integrated the domain knowledge alongside machine learning techniques, allowing us to accurately classify student posts that may not be captured by machine learning alone. This integration allowed us to accurately classify student posts that may not be fully captured by machine learning techniques alone. By leveraging domain knowledge, our approach enhanced the classification process, ensuring a more comprehensive assessment of student contributions within online discussion forums. This research contributed to the ongoing conversation on leveraging machine learning for cognitive analysis in online learning environments, highlighting the importance of context-specific methodologies in advancing educational assessment practices.