Logo PTI Logo FedCSIS

Proceedings of the 20th Conference on Computer Science and Intelligence Systems (FedCSIS)

Annals of Computer Science and Information Systems, Volume 43

Leveraging Large Language Models for Automated Export Control Screening: Evaluating LLMs Framework

, , ,

DOI: http://dx.doi.org/10.15439/2025F2314

Citation: Proceedings of the 20th Conference on Computer Science and Intelligence Systems (FedCSIS), M. Bolanowski, M. Ganzha, L. Maciaszek, M. Paprzycki, D. Ślęzak (eds). ACSIS, Vol. 43, pages 441–449 ()

Full text

Abstract. Export control (EC) compliance is a critical yet labour-intensive process within research institutions, where the classification of sensitive technologies and cross-border disclosures often depends on expert interpretation of complex legal frameworks. This paper investigates the potential of large language models (LLMs), specifically in this study ChatGPT-4o and LLaMA-3.3, to support EC screening through a multi-stage, expert-in-the-loop framework. The methodology includes prompt variation, regulatory conditioning, reflective reasoning, and expert-informed evaluation to simulate real-world compliance workflows. Using a curated dataset of UK research project descriptions and the UK Strategic Export Control List, we assess model performance across over 1,400 outputs. Results show that while both models benefit from domain-specific grounding, ChatGPT-4o consistently produces more stable and interpretable classifications. Prompt sensitivity, bias behaviour, and ambiguity handling are also examined to highlight model limitations. The findings suggest that LLMs can support early stage EC assessment but require structured prompting and human oversight to ensure regulatory alignment.

References

  1. Export Control Joint Unit, Department for International Trade, and Department for Business and Trade, “Open general export licences (ogels),” 2025, last updated: 6 March 2025. [Online]. Available: https://www.gov.uk/government/collections/open-general-export-licences-ogels
  2. UK Department for International Trade and Export Control Joint Unit, “Do i need an export licence?” 2019, last updated: 13 August 2019. [Online]. Available: https://www.gov.uk/guidance/beginners-guide-to-export-controls
  3. UK Department for Business and Trade, “Spire export control system,” 2024. [Online]. Available: https://www.spire.trade.gov.uk
  4. U.S. Department of Commerce, Bureau of Industry and Security, “Export administration regulations (ear),” 2024, accessed: May 2024. [Online]. Available: https://www.bis.gov/regulations/ear
  5. U.S. Department of State, Directorate of Defense Trade Controls, “International traffic in arms regulations (itar),” 2024. [Online]. Available: https://www.pmddtc.state.gov/ddtc_public?id=ddtc_public_portal_itar_landing
  6. UK Government, “The export control order 2008,” 2008, statutory Instrument No. 3231. [Online]. Available: https://www.legislation.gov.uk/uksi/2008/3231/contents/made
  7. European Parliament and Council, “Regulation (eu) 2021/821 setting up a union regime for the control of exports, transfer, brokering and transit of dual-use items,” 2021. [Online]. Available: https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A32021R0821
  8. Export Control Joint Unit, Department for International Trade, and Department for Business and Trade, “Export controls: Military goods, software and technology,” 2024, last updated: 10 April 2024. [Online]. Available: https://www.gov.uk/guidance/export-controls-military-goods-software-and-technology
  9. S. Hassani, M. Sabetzadeh, D. Amyot, and J. Liao, “Rethinking legal compliance automation: Opportunities with large language models,” in 2024 IEEE International Requirements Engineering Conference (RE), 2024, pp. 432–440. [Online]. Available: https://doi.org/10.1109/RE59067.2024.00051
  10. J. Ioannidis, J. Harper, M. S. Quah, and D. Hunter, “Gracenote.ai: Legal generative ai for regulatory compliance,” in Proceedings of the 3rd International Workshop on AI for Legal Professionals (LegalAIIA), 2023. [Online]. Available: https://doi.org/10.2139/ssrn.4494272
  11. H. Zhang, N. Jethani, S. Jones, N. Genes, V. J. Major, I. S. Jaffe, and et al., “Evaluating large language models in extracting cognitive exam dates and scores,” PLOS Digit Health, vol. 3, no. 12, p. e0000685, 2024. https://dx.doi.org/10.1371/journal.pdig.0000685. [Online]. Available: https://doi.org/10.1371/journal.pdig.0000685
  12. S. Hassani, “Enhancing legal compliance and regulation analysis with large language models,” in 2024 IEEE International Requirements Engineering Conference (RE), 2024. https://dx.doi.org/10.1109/RE59067.2024.00065 pp. 507–511. [Online]. Available: https://doi.org/10.1109/RE59067.2024.00065
  13. N. Chen, X. Lin, H. Jiang, and Y. An, “Automated building information modeling compliance check through a large language model combined with deep learning and ontology,” Buildings, vol. 14, no. 7, p. 1983, 2024. [Online]. Available: https://doi.org/10.3390/buildings14071983
  14. A. Berger, L. Hillebrand, D. Leonhard, T. Deußer, T. B. F. D. Oliveira, T. Dilmaghani, and R. Sifa, “Towards automated regulatory compliance verification in financial auditing with large language models,” in 2023 IEEE International Conference on Big Data (BigData), 2023, pp. 4626–4635. [Online]. Available: https://doi.org/10.1109/BigData59044.2023.10386518
  15. S. Wilson, F. Schaub, Y. Agarwal, A. Acquisti, L. Cranor, and N. Sadeh, “The creation and analysis of a website privacy policy corpus,” in Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), vol. 1, 2016, pp. 1330–1340. [Online]. Available: https://doi.org/10.18653/v1/P16-1126
  16. O. Amaral, S. Abualhaija, D. Torre, M. Sabetzadeh, and L. C. Briand, “AI-enabled automation for completeness checking of privacy policies,” IEEE Transactions on Software Engineering, vol. 48, no. 11, pp. 4647–4674, 2022. https://dx.doi.org/10.1109/TSE.2021.3124332. [Online]. Available: https://doi.org/10.1109/TSE.2021.3124332
  17. K. M. Sathyendra, S. Wilson, F. Schaub, S. Zimmeck, and N. Sadeh, “Identifying the provision of choices in privacy policy text,” in Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, 2017. https://dx.doi.org/10.18653/v1/D17-1294 pp. 2774–2779. [Online]. Available: https://doi.org/10.18653/v1/D17-1294
  18. S. R. Bogireddy and N. Dasari, “Comparative analysis of chatgpt-4 and llama: Performance evaluation on text summarization, data analysis, and question answering,” in 2024 15th International Conference on Computing Communication and Networking Technologies (ICCCNT), 2024. https://dx.doi.org/10.1109/ICCCNT61001.2024.10725662 pp. 1–7. [Online]. Available: https://doi.org/10.1109/ICCCNT61001.2024.10725662
  19. S. Pandya, T. E. Bresler, T. Wilson, Z. Htway, and M. Fujita, “Decoding the nccn guidelines with ai: A comparative evaluation of chatgpt-4.0 and llama 2 in the management of thyroid carcinoma,” The American Surgeon, vol. 91, no. 1, pp. 94–98, 2025. [Online]. Available: https://doi.org/10.1177/00031348241269430
  20. H. Yin, A. Aryani, and N. Nambiar, “Evaluating the performance of large language models for sdg mapping,” arXiv, techreport https://arxiv.org/abs/2408.02201, 2024. [Online]. Available: https://doi.org/10.48550/arXiv.2408.02201
  21. S. Fantechi, L. Gnesi, L. Passaro, and L. Semini, “Inconsistency detection in natural language requirements using chatgpt: A preliminary evaluation,” in Proceedings of the 2023 IEEE 31st International Requirements Engineering Conference (RE), 2023. [Online]. Available: https://doi.org/10.1109/RE57278.2023.00045
  22. D. Al-Turki, H. Hettiarachchi, M. M. Gaber, M. M. Abdelsamea, S. Basurra, S. Iranmanesh, H. Saadany, and E. Vakaj, “Human-in-the-loop learning with llms for efficient rase tagging in building compliance regulations,” IEEE Access, 2024. https://dx.doi.org/10.1109/ACCESS.2024.3512434 Early Access. [Online]. Available: https://doi.org/10.1109/ACCESS.2024.3512434
  23. Łukasz Kurant, “Mechanism for detecting cause-and-effect relationships in court judgments,” in Annals of Computer Science and Information Systems, vol. 35, 2023. https://dx.doi.org/10.15439/2023F4827 pp. 1041–1046. [Online]. Available: https://doi.org/10.15439/2023F4827
  24. M. Esche, L. Ho, M. Nischwitz, and R. Meyer, “Risk-based continuous quality control for software in legal metrology,” in Annals of Computer Science and Information Systems, vol. 35, 2023. https://dx.doi.org/10.15439/2023F6171 pp. 451–461. [Online]. Available: https://doi.org/10.15439/2023F6171
  25. S. Lawal, X. Zhao, A. Rios, R. Krishnan, and D. Ferraiolo, “Translating natural language specifications into access control policies by leveraging large language models,” in 2024 IEEE 6th International Conference on Trust, Privacy and Security in Intelligent Systems, and Applications (TPS-ISA), 2024. https://dx.doi.org/10.1109/TPS-ISA62245.2024.00048 pp. 361–370. [Online]. Available: https://doi.org/10.1109/TPS-ISA62245.2024.00048
  26. University of Liverpool Legal and Compliance, “Export controls: How export control legislation applies to collaborating internationally,” 2024, accessed: May 2024. [Online]. Available: https://www.liverpool.ac.uk/legal/exportcontrols/
  27. P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W. Y. Chang, T. Rocktäschel, S. Riedel, and D. Kiela, “Retrieval-augmented generation for knowledge-intensive nlp tasks,” in Advances in Neural Information Processing Systems, vol. 33, 2020, pp. 9459–9474. [Online]. Available: https://doi.org/10.48550/arXiv.2005.11401