Logo PTI Logo FedCSIS

Proceedings of the 21st Conference on Computer Science and Intelligence Systems (FedCSIS)

Annals of Computer Science and Information Systems, Volume 47

Assessing Static Analysis Tools for Security Vulnerability Detection: An Empirical Study

, ,

DOI: http://dx.doi.org/10.15439/2026F8556

Citation: Artur Santos de Farias, ,

Full text

Abstract. Static analysis tools are widely adopted to support security vulnerability detection in modern software development, particularly when integrated into continuous integration (CI) pipelines. However, the increasing number of available tools and the operational constraints imposed by CI environments complicate tool selection, while empirical evidence supporting informed adoption decisions in realistic CI settings remains limited. This study addresses this gap by empirically assessing static analysis tools for security vulnerability detection in Java projects executed within GitHub Actions pipelines. The evaluation considers a selected set of representative open-source tools and analyzes their behavior across repositories containing documented, real-world vulnerabilities. The assessment focuses on detection effectiveness, coverage of vulnerability categories, consistency and complementarity across tools, and operational behavior in CI, including execution overhead and stability. The results reveal substantial variability among the evaluated tools, demonstrating that no single solution simultaneously optimizes detection capability, execution efficiency, and CI robustness. These findings expose explicit trade-offs that directly affect the practical adoption of static analysis in CI-driven development workflows. By providing CI-grounded empirical evidence, this study supports more informed tool selection and configuration decisions, benefiting researchers, practitioners, and organizations seeking to strengthen security-oriented practices in continuous integration environments.

References

  1. Capgemini. (2024) World quality report 16th edition | 2024-25. Acesso em: 31 jan. 2025. [Online]. Available: https://ebook.capgemini.com/ world-quality-report-2024-25/
  2. P. Wadhwani and A. Ambekar. (2025) Software testing market size. Acesso em: 31 jan. 2025. [Online]. Available: https://www.gminsights. com/industry-analysis/software-testing-market
  3. Statista, “Cost of a data breach in the u.s. 2024,” Statista Research Department, 2024, accessed: 2025-0305. [Online]. Available: https://www.statista.com/statistics/273575/ us-average-cost-incurred-by-a-data-breach/
  4. ISOIEC27000, Information technology — Security techniques — Information security management systems — Overview and vocabulary, INTERNATIONAL STANDARD Std. ISO/IEC 27 000:2018(E), 2018.
  5. J. Holdsworth and M. Kosinski. (2024) What is information security? Acesso em: 5 mar. 2025. [Online]. Available: https: //www.ibm.com/think/topics/information-security
  6. ISO/IEC/IEEE., International Standard—Software and Systems Engineering Software Testing–Part 1: General Concepts, ISO/IEC/IEEE Std. 29 119-1:2022(E), 2022.
  7. P. Ammann and J. Offutt, Introduction to Software Testing, 2nd ed. USA: Cambridge University Press, 2016.
  8. Y. Li, P. Yao, K. Yu, C. Wang, Y. Ye, S. Li, M. Luo, Y. Liu, and K. Ren, “Understanding industry perspectives of static application security testing (sast) evaluation,” Proc. ACM Softw. Eng., vol. 2, no. FSE, Jun. 2025. https://dx.doi.org/10.1145/3729404. [Online]. Available: https://doi.org/10.1145/3729404
  9. P. Raulamo-Jurvanen, M. Mäntylä, and V. Garousi, “Choosing the right test automation tool: a grey literature review of practitioner sources,” in Proceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering, 2017. https://dx.doi.org/10.1145/3084226.3084252 pp. 21–30.
  10. B. Aloraini, M. Nagappan, D. M. German, S. Hayashi, and Y. Higo, “An empirical study of security warnings from static application security testing tools,” in Journal of Systems and Software, 2019. https://dx.doi.org/10.1016/j.jss.2019.110427 p. 110427.
  11. J. Yang, L. Tan, J. Peyton, and K. A. Duer, “Towards better utilizing static application security testing,” in 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), 2019. https://dx.doi.org/10.1109/ICSE-SEIP.2019.00014 pp. 51–60.
  12. K. Li, S. Chen, L. Fan, R. Feng, H. Liu, C. Liu, Y. Liu, and Y. Chen, “Comparison and evaluation on static application security testing (sast) tools for java,” in Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2023. https://dx.doi.org/10.1145/3611643.361626 pp. 921–933.
  13. A. Decan, T. Mens, P. R. Mazrae, and M. Golzadeh, “On the use of github actions in software development repositories,” in 2022 IEEE International Conference on Software Maintenance and Evolution, 2022. https://dx.doi.org/10.1109/ICSME55016.2022.00029 pp. 235–245.
  14. W. Charoenwet, P. Thongtanunam, V.-T. Pham, and C. Treude, “An empirical study of static analysis tools for secure code review,” in the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, 2024. https://dx.doi.org/10.1145/3650212.3680313 pp. 691 – 703.
  15. M. Esposito, V. Falaschi, and D. Falessi, “An extensive comparison of static application security testing tools,” in Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering, ser. EASE ’24. New York, NY, USA: Association for Computing Machinery, 2024. https://dx.doi.org/10.1145/3661167.3661199. ISBN 9798400717017 p. 69–78. [Online]. Available: https://doi.org/10.1145/ 3661167.3661199
  16. D. Dalaq, K. F. Daya, A. Dalaq, M. N. Arefin, and M. K. Niazi, “A systematic literature review on static application security testing (sast) tools: Evaluation, benchmarks, challenges, and future directions,” in Proceedings of the 2025 29th International Conference on Evaluation and Assessment in Software Engineering Companion, ser. EASE Companion ’25. New York, NY, USA: Association for Computing Machinery, 2025. https://dx.doi.org/10.1145/3727967.3756838. ISBN 9798400718328 p. 162–168. [Online]. Available: https://doi.org/10. 1145/3727967.3756838
  17. A. S. Ami, K. Moran, D. Poshyvanyk, and A. Nadkarni, “"false negative - that one is going to kill you": Understanding industry perspectives of static analysis based security testing,” in 2024 IEEE Symposium on Security and Privacy (SP), 2024. https://dx.doi.org/10.1109/SP54263.2024.00019 pp. 3979–3997.
  18. M. Kassab, J. F. DeFranco, and P. A. Laplante, “Software testing: State of the practice,” IEEE Software, vol. 34, pp. 46–52, 2017. https://dx.doi.org/10.1109/MS.2017.3571582
  19. G. Weidman, Penetration Testing: A Hands-On Introduction to Hacking, 1st ed. No Starch Press, 2014.
  20. I. Sommerville, Software Enginerring, 10th ed. São Paulo: Pearson Universidades, 2019.
  21. S. Romano, F. Zampetti, M. T. Baldassarre, M. D. Penta, and G. Scanniello, “Do static analysis tools affect software quality when using testdriven development?” in ESEM ’22: Proceedings of the 16th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement, 2022. https://dx.doi.org/10.1145/3544902.3546233 pp. 80–91.
  22. J. Santos, D. A. da Costa, and U. Kulesza, “Investigating the impact of continuous integration practices on the productivity and quality of open-source projects,” in ESEM ’22: Proceedings of the 16th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement, 2022. https://dx.doi.org/10.1145/3544902.3546244 pp. 137–147.
  23. B. Kitchenham and S. M. Charters, “Guidelines for performing systematic literature reviews in software engineering,” Keele University and Durham University, Tech. Rep. EBSE-2007-001, January 2007. [Online]. Available: https://www.elsevier.com/__data/promis_ misc/525444systematicreviewsguide.pdf
  24. F. Pudlitz, F. Brokhausen, and A. Vogelsang, “What am i testing and where? comparing testing procedures based on lightweight requirements annotations,” Empirical Software Engineering, vol. 24, no. 4, pp. 2809– 2843, 2020. https://dx.doi.org/10.1007/s10664-020-09815-w
  25. A. Farias, R. Rocha, I. Vanderlei, J. Araujo, A. Araújo, and J. Dantas, “Software testing evidence: Results from a systematic mapping,” in Proceedings of the 21st International Conference on Web Information Systems and Technologies - Volume 1: WEBIST, INSTICC. SciTePress, 2025. https://dx.doi.org/10.5220/0013739100003985. ISBN 978-989-758-772-6 pp. 386–393.