Logo PTI Logo FedCSIS

Proceedings of the 21st Conference on Computer Science and Intelligence Systems (FedCSIS)

Annals of Computer Science and Information Systems, Volume 47

LLMs across the Software Development Life Cycle in Project-Based Software Engineering Courses: an Empirical Study

, , ,

DOI: http://dx.doi.org/10.15439/2026F9512

Citation: Christos Hadjichristofi, , ,

Full text

Abstract. Large Language Models (LLMs) are increasingly used in software development and are becoming part of software engineering education. However, there is limited empirical evidence on how students use AI tools across development tasks in project-based Software Engineering courses. This paper presents an empirical study based on approximately 8,500 validated student-submitted AI usage logs collected over five semesters between 2022 and 2026 from two undergraduate courses at the National Technical University of Athens. We examine in which phases of the Software Development Life Cycle (SDLC) students use AI tools, how they evaluate AI assistance, and how their perceptions evolve over time. The results show that AI usage is mainly concentrated in coding and testing, with lower usage in requirements, architecture, and design. Students rate quality higher than knowledge gained, and those with higher task experience report higher quality and knowledge gains. Overall, students view AI positively, report low perceived threat, and tend to use it more for task completion than for learning.

References

  1. A. Fan, B. Gokkaya, M. Harman, M. Lyubarskiy, S. Sengupta, S. Yoo, and J. M. Zhang, “Large language models for software engineering: Survey and open problems,” in 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE), 2023, pp. 31–53.
  2. X. Hou, Y. Zhao, Y. Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang, “Large language models for software engineering: A systematic literature review,” ACM Trans. Softw. Eng. Methodol., vol. 33, no. 8, Dec. 2024. [Online]. Available: https://doi.org/10.1145/3695988
  3. J. Finnie-Ansley, P. Denny, B. A. Becker, A. Luxton-Reilly, and J. Prather, “The robots are coming: Exploring the implications of openai codex on introductory programming,” in Proceedings of the 24th Australasian Computing Education Conference, ser. ACE ’22. New York, NY, USA: Association for Computing Machinery, 2022, p. 10–19. [Online]. Available: https://doi.org/10.1145/3511861.3511863
  4. B. A. Becker, P. Denny, J. Finnie-Ansley, A. Luxton-Reilly, J. Prather, and E. A. Santos, “Programming is hard - or at least it used to be: Educational opportunities and challenges of ai code generation,” in Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1, ser. SIGCSE 2023. New York, NY, USA: Association for Computing Machinery, 2023, p. 500–506. [Online]. Available: https://doi.org/10.1145/3545945.3569759
  5. E. L. Ouh, B. K. S. Gan, K. Jin Shim, and S. Wlodkowski, “Chatgpt, can you generate solutions for my coding exercises? an evaluation on its effectiveness in an undergraduate java programming course.” in Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1, ser. ITiCSE 2023. New York, NY, USA: Association for Computing Machinery, 2023, p. 54–60. [Online]. Available: https://doi.org/10.1145/3587102.3588794
  6. P. Denny, V. Kumar, and N. Giacaman, “Conversing with copilot: Exploring prompt engineering for solving cs1 problems using natural language,” in Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1, ser. SIGCSE 2023. New York, NY, USA: Association for Computing Machinery, 2023, p. 1136–1142. [Online]. Available: https://doi.org/10.1145/3545945.3569823
  7. M. Kazemitabaar, J. Chow, C. K. T. Ma, B. J. Ericson, D. Weintrop, and T. Grossman, “Studying the effect of ai code generators on supporting novice learners in introductory programming,” in Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, ser. CHI ’23. New York, NY, USA: Association for Computing Machinery, 2023. [Online]. Available: https://doi.org/10.1145/3544548.3580919
  8. J. Leinonen, P. Denny, S. MacNeil, S. Sarsa, S. Bernstein, J. Kim, A. Tran, and A. Hellas, “Comparing code explanations created by students and large language models,” in Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1, ser. ITiCSE 2023. New York, NY, USA: Association for Computing Machinery, 2023, p. 124–130. [Online]. Available: https://doi.org/10.1145/3587102.3588785
  9. J. Savelka, A. Agarwal, C. Bogart, Y. Song, and M. Sakr, “Can generative pre-trained transformers (gpt) pass assessments in higher education programming courses?” in Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1, ser. ITiCSE 2023. New York, NY, USA: Association for Computing Machinery, 2023, p. 117–123. [Online]. Available: https://doi.org/10.1145/3587102.3588792
  10. A. Shoufan, “Exploring students’ perceptions of chatgpt: Thematic analysis and follow-up survey,” IEEE Access, vol. 11, pp. 38 805–38 818, 2023.
  11. P. Haindl and G. Weinberger, “Students’ experiences of using chatgpt in an undergraduate programming course,” IEEE Access, vol. 12, pp. 43 519–43 529, 2024.
  12. R. Budhiraja, I. Joshi, J. S. Challa, H. D. Akolekar, and D. Kumar, ““it’s not like jarvis, but it’s pretty close!” - examining chatgpt’s usage among undergraduate students in computer science,” in Proceedings of the 26th Australasian Computing Education Conference, ser. ACE ’24. New York, NY, USA: Association for Computing Machinery, 2024, p. 124–133. [Online]. Available: https://doi.org/10.1145/3636243.3636257
  13. C. Rahe and W. Maalej, “How do programming students use generative ai?” Proc. ACM Softw. Eng., vol. 2, no. FSE, Jun. 2025. [Online]. Available: https://doi.org/10.1145/3715762
  14. R. Chen, S. Jiang, J. Shen, A. J. Moon, and L. Wei, “Examining the usage of generative ai models in student learning activities for software programming,” in 2025 2nd IEEE/ACM International Conference on AIpowered Software (AIware), 2025, pp. 76–86.
  15. H. Alquran and S. Banitaan, “Human–ai collaboration in programming education: Student perspectives on llm-based coding assistants,” Computers, vol. 15, no. 3, 2026. [Online]. Available: https://www.mdpi. com/2073-431X/15/3/154
  16. D. Stoyanova, S. Stoyanova-Petrova, S. Shotarova, S. Lyubomirov, and N. Mileva, “Chatgpt in programming education: An empirical study on its impact on student performance, creativity, and teamwork,” Education Sciences, vol. 16, no. 1, 2026. [Online]. Available: https://www.mdpi.com/2227-7102/16/1/19
  17. S. Rasnayaka, G. Wang, R. Shariffdeen, and G. N. Iyer, “An empirical study on usage and perceptions of llms in a software engineering project,” in Proceedings of the 1st International Workshop on Large Language Models for Code, ser. LLM4Code ’24. New York, NY, USA: Association for Computing Machinery, 2024, p. 111–118. [Online]. Available: https://doi.org/10.1145/3643795.3648379
  18. A. Naman, R. Shariffdeen, G. Wang, S. Rasnayaka, and G. N. Iyer, “Analysis of student-llm interaction in a software engineering project,” in 2025 IEEE/ACM International Workshop on Large Language Models for Code (LLM4Code), 2025, pp. 112–119.
  19. P. Vaithilingam, T. Zhang, and E. L. Glassman, “Expectation vs.nbsp;experience: Evaluating the usability of code generation tools powered by large language models,” in Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems, ser. CHI EA ’22. New York, NY, USA: Association for Computing Machinery, 2022. [Online]. Available: https://doi.org/10.1145/3491101.3519665
  20. A. Moradi Dakhel, V. Majdinasab, A. Nikanjam, F. Khomh, M. C. Desmarais, and Z. M. J. Jiang, “Github copilot ai pair programmer: Asset or liability?” J. Syst. Softw., vol. 203, no. C, Sep. 2023. [Online]. Available: https://doi.org/10.1016/j.jss.2023.111734
  21. J. Prather, B. N. Reeves, P. Denny, B. A. Becker, J. Leinonen, A. Luxton-Reilly, G. Powell, J. Finnie-Ansley, and E. A. Santos, ““it’s weird that it knows what i want”: Usability and interactions with copilot for novice programmers,” ACM Trans. Comput.-Hum. Interact., vol. 31, no. 1, Nov. 2023. [Online]. Available: https://doi.org/10.1145/3617367
  22. I. Ozkaya, “Application of large language models to software engineering tasks: Opportunities, risks, and implications,” IEEE Software, vol. 40, no. 3, pp. 4–8, 2023.
  23. S. Christodoulakis, “Design and deployment of a multi-ai model online wrapper with integrated data collection,” Master’s thesis, School of Electrical and Computer Engineering, National Technical University of Athens, Athens, Greece, 2025. [Online]. Available: http://artemis.cslab.ece.ntua.gr:8080/jspui/handle/123456789/20048
  24. C. Hadjichristofi, M. Tsilimigkounakis, G. Sotiropoulos, and V. Vescoukis, “An empirical evaluation of large language models applying software architectural patterns,” AI, vol. 7, no. 6, 2026. [Online]. Available: https://www.mdpi.com/2673-2688/7/6/195