Logo PTI Logo FedCSIS

Proceedings of the 21st Conference on Computer Science and Intelligence Systems (FedCSIS)

Annals of Computer Science and Information Systems, Volume 47

Automated Testing Tools and Software Reliability: A Systematic Mapping

, ,

DOI: http://dx.doi.org/10.15439/2026F0950

Citation: Artur Santos de Farias, ,

Full text

Abstract. Modern software systems operate in increasingly complex and heterogeneous environments, intensifying challenges related to reliability. Automated software testing (AST) helps address these demands, yet existing evidence on how widely used tools support reliability-oriented attributes remains dispersed. A consolidated view is needed to understand their contributions to coverage, fault detection, robustness, and execution reliability. This study conducts a systematic mapping of 184 primary works to identify the most frequently cited AST tools and to examine empirical evidence of their reliability-related capabilities. The analysis synthesizes findings on structural coverage, fault revelation, robustness under stress, and cross-platform consistency, revealing complementary strengths and gaps in current tool support. The results offer a structured perspective on how automation contributes to reliability evaluation, supporting software practitioners in selecting tools aligned with reliability goals and assisting researchers in identifying opportunities for advancing reliability-aware testing. In addition to the empirical synthesis, this study produces an evidence-based artifact in the form of a reliability-oriented attribute matrix, explicitly linking automated testing tools to traceable primary-study evidence and supporting informed, practical tool selection.

References

  1. Bhagat and Shreya, “Review on mobile application development based on flutter platform,” International Journal for Research in Applied Science and Engineering Technology, vol. 10, pp. 803–809, 01 2022.
  2. J. Petrić, T. Hall, and D. Bowes, “Which software faults are tests not detecting?” in Proceedings of the 24th International Conference on Evaluation and Assessment in Software Engineering, ser. EASE ’20. New York, NY, USA: Association for Computing Machinery, 2020. https://dx.doi.org/10.1145/3383219.3383236. ISBN 9781450377317 pp. 160–169.
  3. F. Gurcan, G. G. M. Dalveren, N. E. Cagiltay, D. Roman, and A. Soylu, “Evolution of software testing strategies and trends: Semantic content analysis of software research corpus of the last 40 years,” IEEE Access, vol. 10, pp. 106 093–106 109, 2022. https://dx.doi.org/10.1109/ACCESS.2022.3211949
  4. F. Pudlitz, F. Brokhausen, and A. Vogelsang, “What am i testing and where? comparing testing procedures based on lightweight requirements annotations,” Empirical Software Engineering, vol. 25, no. 4, pp. 2809– 2843, Jul 2020. https://dx.doi.org/10.1007/s10664-020-09815-w
  5. K. L. Petry, E. O. Jr, and A. F. Zorzo, “Model-based testing of software product lines: Mapping study and research roadmap,” Journal of Systems and Software, vol. 167, p. 110608, 2020. https://dx.doi.org/10.1016/j.jss.2020.110608
  6. ISO/IEC/IEEE, International Standard—Software and Systems Engineering Software Testing–Part 1: General Concepts, ISO/IEC/IEEE Std. 29 119-1:2022(E), 2022.
  7. V. Garousi, A. B. Keleş, Y. Balaman, Z. Özdemir Güler, and A. Arcuri, “Model-based testing in practice: An experience report from the web applications domain,” Journal of Systems and Software, vol. 180, p. 111032, 2021. https://dx.doi.org/10.1016/j.jss.2021.111032
  8. V. Stray, R. Florea, and L. Paruch, “Exploring human factors of the agile software tester,” Software Quality Journal, vol. 30, no. 2, pp. 455–481, Jun 2022. https://dx.doi.org/10.1007/s11219-021-09561-2
  9. Peischl, Bernhard, Tazl, Oliver, Wotawa, and Franz, “Testing anticipatory systems: A systematic mapping study on the state of the art,” Journal of Systems and Software, vol. 192, p. 111387, 06 2022. https://dx.doi.org/10.1016/j.jss.2022.111387
  10. M. Swillus and A. Zaidman, “Sentiment overflow in the testing stack: Analyzing software testing posts on stack overflow,” Journal of Systems and Software, vol. 205, p. 111804, 2023. https://dx.doi.org/10.1016/j.jss.2023.111804
  11. J. Lojda, R. Panek, J. Podivinsky, O. Cekan, M. Krcma, and Z. Kotasek, “Testing embedded software through fault injection: Case study on smart lock,” in 2021 IEEE 22nd Latin American Test Symposium (LATS), 2021. https://dx.doi.org/10.1109/LATS53581.2021.9651770 pp. 1–6.
  12. D. R. Falcó, A. Serrano-Cases, A. Martinez-Alvarez, and S. CuencaAsensi, “Soft error reliability predictor based on a deep feedforward neural network,” in 2020 IEEE Latin-American Test Symposium (LATS), 2020. https://dx.doi.org/10.1109/LATS49555.2020.9093688 pp. 1–5.
  13. J. Podivinsky, J. Lojda, R. Panek, O. Cekan, M. Krcma, and Z. Kotasek, “Evaluation platform for testing fault tolerance: Testing reliability of smart electronic locks,” in 2020 IEEE 11th Latin American Symposium on Circuits Systems (LASCAS), 2020. https://dx.doi.org/10.1109/LASCAS45839.2020.9068977 pp. 1–4.
  14. R. Natella, D. Cotroneo, and H. S. Madeira, “Assessing dependability with software fault injection: A survey,” ACM Comput. Surv., vol. 48, no. 3, Feb. 2016. https://dx.doi.org/10.1145/2841425. [Online]. Available: https://doi.org/10.1145/2841425
  15. P. Kong, L. Li, J. Gao, K. Liu, T. F. Bissyandé, and J. Klein, “Automated testing of android apps: A systematic literature review,” IEEE Transactions on Reliability, vol. 68, pp. 45–66, 03 2019. https://dx.doi.org/10.1109/TR.2018.2865733
  16. V. Garousi, M. Felderer, M. Kuhrmann, K. Herkiloğlu, and S. Eldh, “Exploring the industry’s challenges in software testing: An empirical study,” J. Softw. Evol. Process, vol. 32, no. 8, Aug. 2020. https://dx.doi.org/10.1002/smr.2251
  17. M. Waseem, P. Liang, M. Shahin, A. D. Salle, and G. Márquez, “Design, monitoring, and testing of microservices systems: The practitioners’ perspective,” Journal of Systems and Software, vol. 182, p. 111061, 08 2021. https://dx.doi.org/10.1016/j.jss.2021.111061
  18. J. B. Minani, F. Sabir, N. Moha, and Y.-G. Guéhéneuc, “A systematic review of iot systems testing: Objectives, approaches, tools, and challenges,” IEEE Transactions on Software Engineering, vol. 50, pp. 785–815, 2024. https://dx.doi.org/10.1109/TSE.2024.3363611
  19. M. Dadkhah, S. Araban, and S. Paydar, “A systematic literature review on semantic web enabled software testing,” Journal of Systems and Software, vol. 162, p. 110485, 2020. https://dx.doi.org/10.1016/j.jss.2019.110485
  20. S. Piparia, D. Adamo, R. Bryce, H. Do, and B. Bryant, “Combinatorial testing of context aware android applications,” in 2021 16th Conference on Computer Science and Intelligence Systems (FedCSIS), 2021. https://dx.doi.org/110.15439/2021F003
  21. J. Bicevskis, Z. Bicevska, A. Nikiforova, and I. Oditis, “Data quality model-based testing of information systems,” in 2020 15th Conference on Computer Science and Information Systems (FedCSIS), 2020. https://dx.doi.org/10.15439/2020F253 pp. 488–499.
  22. I. Santos, E. F. Coutinho, and S. R. S. Souza, “Software testing ecosystems insights and research opportunities,” in SBES ’20: Proceedings of the XXXIV Brazilian Symposium on Software Engineering, 2020. https://dx.doi.org/10.1145/3422392.3422458 pp. 421–426.
  23. M. Nass, E. Alégroth, and R. Feldt, “Why many challenges with gui test automation (will) remain,” Information and Software Technology, vol. 138, p. 106625, 2021. https://dx.doi.org/10.1016/j.infsof.2021.106625
  24. I. V. Kovalev, M. V. Saramud, V. V. Losev, N. A. Testoyedov, and M. V. Karaseva, “On the problem of predicting meantime between failures on the basis of data from automated testing of on-board software components,” Journal of Physics: Conference Series, vol. 1679, p. 052037, 2020. https://dx.doi.org/10.1088/1742-6596/1679/5/052037
  25. O. A. Lazzarini Lemos, F. Fagundes Silveira, F. Cutigi Ferrari, and A. Garcia, “The impact of software testing education on code reliability: An empirical assessment,” Journal of Systems and Software, vol. 137, pp. 497–511, 2018. https://dx.doi.org/10.1016/j.jss.2017.02.042. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0164121217300419
  26. I. Sommerville, Software Enginerring, 10th ed. São Paulo: Pearson Universidades, 2019.
  27. V. Pradhan, J. Dhar, and A. Kumar, “Testing coverage-based software reliability growth model considering uncertainty of operating environment,” Systems Engineering, vol. 26, no. 4, pp. 449– 462, 2023. https://doi.org/10.1002/sys.21671. [Online]. Available: https://incose.onlinelibrary.wiley.com/doi/abs/10.1002/sys.21671
  28. P. Raulamo-Jurvanen, S. Hosio, and M. V. Mäntylä, “Practitioner evaluations on software testing tools,” in Proceedings of the 23rd International Conference on Evaluation and Assessment in Software Engineering, ser. EASE ’19. New York, NY, USA: Association for Computing Machinery, 2019. https://dx.doi.org/10.1145/3319008.3319018. ISBN 9781450371452 p. 57–66. [Online]. Available: https://doi.org/10.1145/ 3319008.3319018
  29. A. Bertolino, B. Miranda, R. Pietrantuono, and S. Russo, “Hybrid is better: Why and how test coverage and software reliability can benefit each other,” in Web Information Systems and Technologies. Cham: Springer International Publishing, 2019. ISBN 978-3-030-35330-8 pp. 25–38.
  30. S. Oveisi, A. Moeini, S. Mirzaei, and M. A. Farsi, “Software reliability prediction: A survey,” Quality and Reliability Engineering International, vol. 39, pp. 412–453, 12 2022. https://dx.doi.org/10.1002/qre.3220
  31. T. Bach, A. Andrzejak, R. Pannemans, and D. Lo, “The impact of coverage on bug density in a large industrial software project,” in 2017 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), 2017. https://dx.doi.org/10.1109/ESEM.2017.44 pp. 307– 313.
  32. C. Diwaker, P. Tomar, A. Solanki, A. Nayyar, N. Z. Jhanjhi, and A. Abdullah, “A new model for predicting component-based software reliability using soft computing,” IEEE Access, vol. 7, pp. 147 191–147 203, 2019. https://dx.doi.org/10.1109/ACCESS.2019.2946862
  33. R. Pietrantuono, A. Bertolino, G. D. Angelis, B. Miranda, and S. Russo, “Towards continuous software reliability testing in devops,” in 2019 IEEE/ACM 14th International Workshop on Automation of Software Test (AST), 2019. https://dx.doi.org/10.1109/AST.2019.00009 pp. 21–27.
  34. Y. Yang, Y. Zhou, H. Sun, Z. Su, Z. Zuo, and L. Xu, “Hunting for bugs in code coverage tools via randomized differential testing,” in 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE), 2019. https://dx.doi.org/10.1109/ICSE.2019.00061 pp. 488–499.
  35. B. Kitchenham, O. P. Brereton, D. Budgen, and M. Turner, “Guidelines for performing systematic literature reviews in software engineering,” Keele University and Durham University Joint Report, Keele University, Tech. Rep. EBSE 2007-001, 2007.
  36. A. Farias, R. Rocha, I. Vanderlei, J. Araujo, A. Araújo, and J. Dantas, “Software testing evidence: Results from a systematic mapping,” in Proceedings of the 21st International Conference on Web Information Systems and Technologies - Volume 1: WEBIST, INSTICC. SciTePress, 2025. https://dx.doi.org/10.5220/0013739100003985. ISBN 978-989-758-772-6 pp. 386–393.
  37. SCIMAGO, “Scimago journal & country rank,” 2024, accessed on: November 16, 2024. [Online]. Available: https://www.scimagojr.com/