Proceedings of the 20th Conference on Computer Science and Intelligence Systems (FedCSIS)

DOI: http://dx.doi.org/10.15439/2025F5613

Citation: Proceedings of the 20th Conference on Computer Science and Intelligence Systems (FedCSIS), M. Bolanowski, M. Ganzha, L. Maciaszek, M. Paprzycki, D. Ślęzak (eds). ACSIS, Vol. 43, pages 531–542 (2025)

Full text

Abstract. We propose a meta-language-based approach enabling Large Language Models (LLMs) to reliably generate structured, machine-readable outputs (MLDS) adapted to domain requirements, without adhering strictly to standard formats like JSON or XML. By embedding explicit schema instructions within prompts, we evaluated the method across diverse use cases, including automated Virtual Reality environment generation and automotive security modeling. Our experiments demonstrate that the meta-language approach significantly improves LLM-generated structure compliance, with an 88 \% validation rate across 132 test scenarios. Compared to traditional methods using LangChain and Pydantic, our MLDS method reduces setup complexity by approximately 80 \%, despite a marginally higher error rate. Furthermore, the MLDS artifacts produced were easily editable, enabling rapid iterative refinement. This flexibility greatly alleviates the ``blank page syndrome'' by providing structured initial artifacts suitable for immediate use or further human enhancement, making our approach highly practical for rapid prototyping and integration into complex industrial workflows.

References

1. C. K. Joyce, “The blank page: Effects of constraint on creativity,” Social Science Research Network (SSRN), 2009.
2. A. Fischer, L. Burk, R. Tavakoli Kolagari, and U. Wienkop, “Conalade mlds research artifacts,” https://github.com/CoNaLaDe-MLDS/Artifacts.
3. J. A. Díaz-Pace, A. Tommasel, and R. Capilla, “Helping novice architects to make quality design decisions using an llm-based assistant,” in Software Architecture, M. Galster, P. Scandurra, T. Mikkonen, P. Oliveira Antonino, E. Y. Nakagawa, and E. Navarro, Eds. Cham: Springer Nature Switzerland, 2024, pp. 324–332.
4. S. Colvin, “Pydantic: Data validation and settings management using python type annotations.” [Online]. Available: https://pydantic.dev/
5. H. Chase, “Langchain: Building applications with large language models,” 2023. [Online]. Available: https://github.com/langchain-ai/langchain
6. E. Filipovska, A. Mladenovska, M. Bajrami, J. Dobreva, V. Hillman, P. Lameski, and E. Zdravevski, “Benchmarking openai’s apis and other large language models for repeatable and efficient question answering across multiple documents,” in 2024 19th Conference on Computer Science and Intelligence Systems (FedCSIS), 2024, pp. 107–117.
7. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe, “Training language models to follow instructions with human feedback,” in Proceedings of the 36th International Conference on Neural Information Processing Systems, ser. NIPS ’22. Red Hook, NY, USA: Curran Associates Inc., 2022.
8. W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Dong, Y. Du, C. Yang, Y. Chen, Z. Chen, J. Jiang, R. Ren, Y. Li, X. Tang, Z. Liu, P. Liu, J.-Y. Nie, and J.-R. Wen, “A survey of large language models,” 2025. [Online]. Available: https://arxiv.org/abs/2303.18223
9. S. Geng, H. Cooper, M. Moskal, S. Jenkins, J. Berman, N. Ranchin, R. West, E. Horvitz, and H. Nori, “JSON-SchemaBench: A rigorous benchmark of structured outputs for language models,” arXiv preprint https://arxiv.org/abs/2501.10868, 2025.
10. Object Management Group (OMG), “Meta object facility (MOF) 2.0 core specification,” 2006. [Online]. Available: http://www.omg.org/spec/MOF/2.0
11. Object Management Group (OMG), “Unified modeling language (UML) specification version 2.5.1,” 2017. [Online]. Available: https://www.omg.org/spec/UML/2.5.1/
12. B. Wang, Z. Wang, X. Wang, Y. Cao, R. A Saurous, and Y. Kim, “Grammar prompting for domain-specific language generation with large language models,” Advances in Neural Information Processing Systems, vol. 36, pp. 65 030–65 055, 2023.
13. T. Scholak, N. Schucher, and D. Bahdanau, “Picard: Parsing incrementally for constrained auto-regressive decoding from language models,” in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021, pp. 9895–9901.
14. N. Bassamzadeh and C. Methani, “A Comparative Study of DSL Code Generation: Fine-Tuning vs. Optimized Retrieval Augmentation,” arXiv preprint https://arxiv.org/abs/2407.02742, 2024.
15. L. Netz, J. Michael, and B. Rumpe, “From Natural Language to Web Applications: Using Large Language Models for Model-Driven Software Engineering,” in Modellierung, 2024.
16. A. Fischer, J.-P. Tolvanen, and R. Tavakoli Kolagari, “Automotive cybersecurity engineering with modeling support,” in 2024 19th Conference on Computer Science and Intelligence Systems (FedCSIS), 2024, pp. 319–329.