Logo PTI Logo FedCSIS

Communication Papers of the 18th Conference on Computer Science and Intelligence Systems

Annals of Computer Science and Information Systems, Volume 37

Can ChatGPT Replace a Template-based Code Generator?

DOI: http://dx.doi.org/10.15439/2023F5691

Citation: Communication Papers of the 18th Conference on Computer Science and Intelligence Systems, M. Ganzha, L. Maciaszek, M. Paprzycki, D. Ślęzak (eds). ACSIS, Vol. 37, pages 43–50 ()

Full text

Abstract. This article examines whether a large language model (LLM) tool, such as ChatGPT, can replace a template-based source code generator. To this end, we conducted an experiment in which we attempted to replace an existing template-based DAO class generator (which creates entity classes and a repository for a specified database table) with a solution in which templates of target classes were presented to ChatGPT alongside the source model. We then instructed ChatGPT to generate new classes. A novelty in this work is an attempt at two-stage cooperation with ChatGPT: first we provide the pattern, then we fill it. The experiment proved that, at present, such a solution yields results that are neither predictable nor replicable, and successive attempts to execute the same commands returned wildly varying results. ChatGPT randomly recognises the rules that are present in templates, and complex instructions impact the generated results negatively. At present, classic code generation methods yield markedly superior results.

References

  1. Steven Kelly and Juha-Pekka Tolvanen, “Domain-Specific Modeling: Enabling Full Code Generation,” John Wiley & Sons, 2008.
  2. Sven Jörges, “Construction and Evolution of Code Generators,” Springer-Verlag Berlin Heidelberg, 2013.
  3. Priyan Vaithilingam, Tianyi Zhang, and Elena L. Glassman, “Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models,” In Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems, CHI EA ’22, New York, NY, USA, 2022, Association for Computing Machinery. DOI https://doi.org/10.1145/3491101.3519665
  4. Hammond A. Pearce, Baleegh Ahmad, Benjamin Tan, Brendan DolanGavitt, and Ramesh Karri, “Asleep at the keyboard? assessing the security of github copilot’s code contributions,” 2022 IEEE Symposium on Security and Privacy (SP), pages 754–768, 2021. DOI https://doi.org/10.48550/arXiv.2108.09293
  5. Rajeev Alur, Rastislav Bodík, Garvit Juniwal, Milo M. K. Martin, Mukund Raghothaman, Sanjit A. Seshia, Rishabh Singh, Armando Solar-Lezama, Emina Torlak, and Abhishek Udupa, “Syntax-guided synthesis,” 2013 Formal Methods in Computer-Aided Design, pages 1–8, 2013. DOI 10.1109/FMCAD.2013.6679385
  6. Allen Cypher, “Eager: programming repetitive tasks by example,” Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 1991. DOI https://dl.acm.org/doi/10.1145/108844.108850
  7. Sumit Gulwani, “Automating string processing in spreadsheets using input-output examples,” In ACM-SIGACT Symposium on Principles of Programming Languages, 2011. DOI https://doi.org/10.1145/1925844.1926423
  8. Vu Le and Sumit Gulwani, “Flashextract: a framework for data extraction by examples,” Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation, 2014. https://doi.org/10.1145/2666356.2594333
  9. Matej Balog, Alexander L. Gaunt, Marc Brockschmidt, Sebastian Nowozin, and Daniel Tarlow, “Deepcoder: Learning to write programs,” ArXiv, abs/1611.01989, 2016. DOI https://doi.org/10.48550/arXiv.1611.01989
  10. Tonglei Guo and Huilin Gao, “Content enhanced bert-based text-to-sql generation,” ArXiv, abs/1910.07179, 2019. DOI https://doi.org/10.48550/arXiv.1910.07179
  11. Shirley Anugrah Hayati, Raphaël Olivier, Pravalika Avvaru, Pengcheng Yin, Anthony Tomasic, and Graham Neubig, “Retrieval-based neural code generation,” In Conference on Empirical Methods in Natural Language Processing, 2018. DOI https://doi.org/10.48550/arXiv.1808.10025
  12. Matteo Ciniselli, Nathan Cooper, Luca Pascarella, Denys Poshyvanyk, Massimiliano Di Penta, and Gabriele Bavota, “An empirical study on the usage of bert models for code completion,” 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR), pages 108–119, 2021. DOI https://doi.org/10.48550/arXiv.2103.07115
  13. Antonio Mastropaolo, Simone Scalabrino, Nathan Cooper, David NaderPalacio, Denys Poshyvanyk, Rocco Oliveto, and Gabriele Bavota, “Studying the usage of text-to-text transfer transformer to support code-related tasks,” 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE), pages 336–347, 2021. DOI https://doi.org/10.48550/arXiv.2102.02017
  14. Eugene Syriani, Lechanceux Luhunu, and Houari A. Sahraoui, “Systematic mapping study of template-based code generation,” Comput. Lang. Syst. Struct., 52:43–62, 2017. DOI https://doi.org/10.48550/arXiv.1703.06353