Logo PTI Logo FedCSIS

Position Papers of the 19th Conference on Computer Science and Intelligence Systems

Annals of Computer Science and Information Systems, Volume 40

No Train, No Pain? Assessing the Ability of LLMs for Text Classification with no Finetuning

,

DOI: http://dx.doi.org/10.15439/2024F9098

Citation: Position Papers of the 19th Conference on Computer Science and Intelligence Systems, M. Bolanowski, M. Ganzha, L. Maciaszek, M. Paprzycki, D. Ślęzak (eds). ACSIS, Vol. 40, pages 9–16 ()

Full text

Abstract. Modern SotA Text Classification algorithms depend heavily on well annotated and diverse data capturing the intricacies of the unknown data distribution. What options do we have when labeled data is sparse or annotation is expensive and time consuming? With the advent of strong LLM backbones, we have another option at our disposal: Text Classification by making use of the reasoning ability and the strong general prior of contemporary foundation models. In this work we assess the ability of cutting edge LLMs for Text Classification and find that for the right combination of backbone and prompt strategy we're able to near-rival trained baselines for the advanced task of mapping job-postings to a taxonomy of industrial sectors without any finetuning. All our code is made publicly available at our github repository

References

  1. R. Fechner, J. Dörpinghaus, and A. Firll, “Classifying industrial sectors from german textual data with a domain adapted transformer,” in 2023 18th Conference on Computer Science and Intelligence Systems (FedCSIS). IEEE, 2023, pp. 463–470.
  2. S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural computation, vol. 9, no. 8, pp. 1735–1780, 1997.
  3. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems, vol. 30, 2017.
  4. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint https://arxiv.org/abs/1810.04805, 2018.
  5. Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized bert pretraining approach,” arXiv preprint https://arxiv.org/abs/1907.11692, 2019.
  6. N. Reimers and I. Gurevych, “Sentence-bert: Sentence embeddings using siamese bert-networks,” arXiv preprint https://arxiv.org/abs/1908.10084, 2019.
  7. V. Sanh, L. Debut, J. Chaumond, and T. Wolf, “Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter,” arXiv preprint https://arxiv.org/abs/1910.01108, 2019.
  8. M. M. Mirończuk and J. Protasiewicz, “A recent overview of the state-of-the-art elements of text classification,” Expert Systems with Applications, vol. 106, pp. 36–54, 2018.
  9. Z. Dai, Z. Yang, Y. Yang, J. Carbonell, Q. V. Le, and R. Salakhutdinov, “Transformer-xl: Attentive language models beyond a fixed-length context,” arXiv preprint https://arxiv.org/abs/1901.02860, 2019.
  10. I. Beltagy, M. E. Peters, and A. Cohan, “Longformer: The long-document transformer,” arXiv preprint https://arxiv.org/abs/2004.05150, 2020.
  11. J.-J. Decorte, J. Van Hautte, T. Demeester, and C. Develder, “Jobbert: Understanding job titles through skills,” arXiv preprint https://arxiv.org/abs/2109.09605, 2021.
  12. H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar et al., “Llama: Open and efficient foundation language models,” arXiv preprint https://arxiv.org/abs/2302.13971, 2023.
  13. Y. Liu, G. Deng, Y. Li, K. Wang, T. Zhang, Y. Liu, H. Wang, Y. Zheng, and Y. Liu, “Prompt injection attack against llm-integrated applications,” arXiv preprint https://arxiv.org/abs/2306.05499, 2023.
  14. J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou et al., “Chain-of-thought prompting elicits reasoning in large language models,” Advances in Neural Information Processing Systems, vol. 35, pp. 24 824–24 837, 2022.
  15. S. Yao, D. Yu, J. Zhao, I. Shafran, T. L. Griffiths, Y. Cao, and K. Narasimhan, “Tree of thoughts: Deliberate problem solving with large language models,” arXiv preprint https://arxiv.org/abs/2305.10601, 2023.
  16. P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel et al., “Retrieval-augmented generation for knowledge-intensive nlp tasks,” Advances in Neural Information Processing Systems, vol. 33, pp. 9459–9474, 2020.
  17. H. Chase, “LangChain,” Oct. 2022. [Online]. Available: https://github.com/langchain-ai/langchain
  18. M. Pejic-Bach, T. Bertoncel, M. Meško, and Ž. Krstić, “Text mining of industry 4.0 job advertisements,” International journal of information management, vol. 50, pp. 416–431, 2020.
  19. R. Chaisricharoen, W. Srimaharaj, S. Chaising, and K. Pamanee, “Classification approach for industry standards categorization,” in 2022 Joint International Conference on Digital Arts, Media and Technology with ECTI Northern Section Conference on Electrical, Electronics, Computer and Telecommunications Engineering (ECTI DAMT & NCON). IEEE, 2022, pp. 308–313.
  20. A. McCallum, K. Nigam et al., “A comparison of event models for naive bayes text classification,” in AAAI-98 workshop on learning for text categorization, vol. 752, no. 1. Madison, WI, 1998, pp. 41–48.
  21. A. M. Kibriya, E. Frank, B. Pfahringer, and G. Holmes, “Multinomial naive bayes for text categorization revisited,” in AI 2004: Advances in Artificial Intelligence: 17th Australian Joint Conference on Artificial Intelligence, Cairns, Australia, December 4-6, 2004. Proceedings 17. Springer, 2005, pp. 488–499.
  22. H. Hayashi and Q. Zhao, “Quick induction of nntrees for text categorization based on discriminative multiple centroid approach,” in 2010 IEEE International Conference on Systems, Man and Cybernetics. IEEE, 2010, pp. 705–712.
  23. K. Kowsari, K. Jafari Meimandi, M. Heidarysafa, S. Mendu, L. Barnes, and D. Brown, “Text classification algorithms: A survey,” Information, vol. 10, no. 4, p. 150, 2019.
  24. C. Ospino, “Occupations: Labor market classifications, taxonomies, and ontologies in the 21st century,” Inter-American Development Bank, 2018.
  25. M. Rodrigues, Fernández-Macı́as, and Enrique, Sostero, Matteo, “A unified conceptual framework of tasks, skills and competences,” Seville, 2021. [Online]. Available: https://joint-research-centre.ec.europa.eu/publications/unified-conceptual-framework-tasks-skills-and-competences en
  26. A.-S. Gnehm, E. Bühlmann, and S. Clematide, “Evaluation of transfer learning and domain adaptation for analyzing german-speaking job advertisements,” in Proceedings of the Thirteenth Language Resources and Evaluation Conference, 2022, pp. 3892–3901.
  27. A.-S. Gnehm, E. Bühlmann, H. Buchs, and S. Clematide, “Fine-grained extraction and classification of skill requirements in german-speaking job ads.” Association for Computational Linguistics, 2022.
  28. J. Büchel, J. Engler, and A. Mertens, “The demand for data skills in german companies: Evidence from online job advertisements,” How to Reconstruct Ukraine? Challenges, Plans and the Role of the EU, p. 56, 2023.
  29. B. Gehrke, H. Legler, M. Leidmann, and K. Hippe, “Forschungsund wissensintensive wirtschaftszweige: Produktion, wertschöpfung und beschäftigung in deutschland sowie qualifikationserfordernisse im europäischen vergleich,” Studien zum deutschen Innovationssystem, Tech. Rep., 2009.
  30. N. Gillmann and V. Hassler, “Coronabetroffenheit der wirtschaftszweige in gesamt-und ostdeutschland,” ifo Dresden berichtet, vol. 27, no. 04, pp. 03–05, 2020.
  31. U. Kies, D. Klein, and A. Schulte, “Cluster wald und holz Deutschland: Makroökonomische bedeutung, regionale zentren und strukturwandel der beschäftigung in holzbasierten wirtschaftszweigen,” Cluster in Mitteldeutschland–Strukturen, Potenziale, Förderung, p. 103, 2012.
  32. V.-P. Niitamo, “Berufs-und qualifikationsanforderungen im ikt-bereich in europa erkennen und messen,” Schmidt, SL; Strietska-Ilina, O.; Dworschak, B, pp. 194–201, 2005.
  33. J. Hartmann and G. Schütz, “Die klassifizierung der berufe und der wirtschaftszweige im sozio-oekonomischen panel-neuvercodung der daten 1984-2001,” SOEP Survey Papers, Tech. Rep., 2017.
  34. M. Titze, M. Brachert, and A. Kubis, “The identification of regional industrial clusters using qualitative input–output analysis (qioa),” Regional Studies, vol. 45, no. 1, pp. 89–102, 2011.
  35. U. Kies, T. Mrosek, and A. Schulte, “Spatial analysis of regional industrial clusters in the german forest sector,” International Forestry Review, vol. 11, no. 1, pp. 38–51, 2009.
  36. Ollama, “Ollama software repository,” 2024. [Online]. Available: https://github.com/ollama/ollama
  37. distilbert, “distilbert-base-german-cased software repository,” 2024. [Online]. Available: https://huggingface.co/distilbert/distilbert-base-german-cased
  38. A.-S. “Gnehm, E. Bühlmann, and S. Clematide, “”evaluation of transfer learning and domain adaptation for analyzing german-speaking job advertisements”,” in “Proceedings of the 13th Language Resources and Evaluation Conference”. “Marseille, France”: “European Language Resources Association”, june “2022”.
  39. T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, “Focal loss for dense object detection,” in Proceedings of the IEEE international conference on computer vision, 2017, pp. 2980–2988.
  40. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, “Scikit-learn: Machine learning in Python,” Journal of Machine Learning Research, vol. 12, pp. 2825–2830, 2011.
  41. Statistisches Bundesamt, “Klassifikation der Wirtschaftszweige,” Wiesbaden, 2008. [Online]. Available: https://www.destatis.de/static/DE/dokumente/klassifikation-wz-2008-3100100089004.pdf