Logo PTI Logo FedCSIS

Proceedings of the 18th Conference on Computer Science and Intelligence Systems

Annals of Computer Science and Information Systems, Volume 35

Defect Backlog Size Prediction for Open-Source Projects with the Autoregressive Moving Average and Exponential Smoothing Models

, , ,

DOI: http://dx.doi.org/10.15439/2023F5474

Citation: Proceedings of the 18th Conference on Computer Science and Intelligence Systems, M. Ganzha, L. Maciaszek, M. Paprzycki, D. Ślęzak (eds). ACSIS, Vol. 35, pages 83–92 ()

Full text

Abstract. Context: predicting the number of defects in a defect backlog in a given time horizon can help allocate project resources and organize software development. Goal: to compare the accuracy of three defect backlog prediction methods in the context of large open-source (OSS) projects, i.e., ARIMA, Exponential Smoothing (ETS), and the state-of-the-art method developed at Ericsson AB (SM). Method: we perform a simulation study on a sample of 20 open-source projects to compare the prediction accuracy of the methods. Also, we use the Na\"{\i}ve prediction method as a baseline for sanity check. We use statistical inference tests and effect size coefficients to compare the prediction errors. Results: ARIMA, ETS, and SM were more accurate than the Na\"{\i}ve method. Also, the prediction errors were statistically lower for ETS than for SM (however, the effect size was negligible). Conclusions: ETS seems slightly more accurate than SM when predicting defect backlog size of OSS projects.

References

  1. P. Sielicka, “Defect backlog size prediction with the autoregressive moving average and exponential smoothing models,” Master’s thesis, Poznan University of Technology, Poland, 2019.
  2. W. Meding, “Effective monitoring of progress of agile software development teams in modern software companies: an industrial case study,” in Proceedings of the 27th International Workshop on Software Measurement and 12th International Conference on Software Process and Product Measurement, 2017, pp. 23–32.
  3. M. Staron and W. Meding, “A method for forecasting defect backlog in large streamline software development projects and its industrial evaluation,” Information and Software Technology, vol. 52, no. 10, pp. 1069–1079, 2010.
  4. ——, “Defect inflow prediction in large software project,” e-Informatica Software Engineering Journal, vol. 4, no. 1, pp. 89–107, 2010.
  5. ——, “Predicting weekly defect inflow in large software projects based on project planning and test status,” Information and Software Technology, vol. 50, no. 7, pp. 782–796, 2008.
  6. R. Rana, M. Staron, C. Berger, and J. Hansson, “Analysing defect inflow distribution of automotive software projects.” PROMISE ’14’ - 10th International Conference on Predictive Models in Software Engineering, 2013.
  7. G. E. Box, G. M. Jenkins, G. C. Reinsel, and G. M. Ljung, Time series analysis: forecasting and control. John Wiley & Sons, 2015.
  8. C. C. Holt, “Forecasting seasonals and trends by exponentially weighted moving averages,” International journal of forecasting, vol. 20, no. 1, pp. 5–10, 2004.
  9. P. R. Winters, “Forecasting sales by exponentially weighted moving averages,” Management science, vol. 6, no. 3, pp. 324–342, 1960.
  10. R. J. Hyndman and G. Athanasopoulos, Forecasting: principles and practice, 2nd edition. OTexts: Melbourne, Australia, 2013, oTexts.com/fpp2. Accessed on 03.06.2019.
  11. R. J. Hyndman and Y. Khandakar, “Automatic time series forecasting: the forecast package for R,” Journal of Statistical Software, vol. 26, no. 3, pp. 1–22, 2008.
  12. R. Rana, “Software defect prediction techniques in automotive domain: Evaluation, selection and adoption,” Ph.D. dissertation, University of Gothenburg, 2015.
  13. H. D. T.J. Yu, V.Y. Shen, “An analysis of several software defect models,” IEEE Transactions on Software Engineering, vol. 14, no. 9, pp. 1261 – 1270, 1998.
  14. B. Bernard Nicolau de França and G. Horta Travassos, “Simulation based studies in software engineering: A matter of validity,” CLEI electronic journal, vol. 18, no. 1, pp. 5–5, 2015.
  15. M. Shepperd and S. MacDonell, “Evaluating prediction systems in software project estimation,” Information and Software Technology, vol. 54, no. 8, pp. 820–827, 2012.
  16. B. Kitchenham, L. Madeyski, D. Budgen, J. Keung, P. Brereton, S. Charters, S. Gibbs, and A. Pohthong, “Robust statistical methods for empirical software engineering,” Empirical Software Engineering, vol. 22, no. 2, 2017.
  17. C. Wohlin and et al, Experimentation in Software Engineering: An Introduction. Kluwer Academic Publisher, Boston, MA, 2000.