Estimating Story Points in Scrum: Balancing Accuracy and Interpretability with Explainable AI

Abstract

Abstract This study addresses the challenge of accurate Story Points (SP) estimation in agile software development. SP, a unit for measuring development effort, are crucial for project planning and resource allocation. However, manual SP estimation is critical yet a tedious and error-prone process, often causing delays and exceeding project budgets. This highlights the pressing need for automated and accurate SP prediction in the software development industry. This study addresses this need by proposing a novel data preprocessing approach for story point estimation. It involves removing similar user stories, data augmentation and description segmentation. Furthermore, the study contributes 3 new datasets to the public domain specifically designed for story point estimation research. This enhanced data richness and diversity are shown to significantly improve model performance. The study leverages these 3 datasets and the Choetkiertikul dataset to train various traditional and transformer models, including Support Vector Machines (SVM), Random Forest, Recurrent Neural Network (RNN), Bidirectional Encoder Representations from Transformers (BERT), DistilBERT and RoBERTa. Among the traditional approaches, SVM achieved the highest accuracy (46.12%). BERT outperformed other transformer models (44.58%) but fell slightly short of SVM’s performance. To enhance model transparency and interpretability, the study employed Local Interpretable Model-agnostic Explanations (LIME), Shapley Additive explanations (SHAP), and Transformer Interpret libraries. These techniques offer explanations by highlighting keywords influential in the model’s predictions. Additionally, a human-based evaluation involving 7 industry professionals was conducted to assess both model performance and the reliability of the Explainable Artificial Intelligence (XAI) methods.