Logo PTI Logo FedCSIS

Proceedings of the 21st Conference on Computer Science and Intelligence Systems (FedCSIS)

Annals of Computer Science and Information Systems, Volume 47

Two-Step Occupation Coding

,

DOI: http://dx.doi.org/10.15439/2026F3041

Citation: Alexander Esser,

Full text

Abstract. Occupation coding links job titles in free text to occupational taxonomies and is a core task in labor market research. Existing approaches typically address this problem in a single end-to-end step, jointly identifying job titles and assigning occupational codes. This paper presents a novel two-step approach that separates these tasks. In the first step, a domain-specific Named Entity Recognition (NER) model identifies occupational titles in continuous text, even under noise such as OCR errors. In the second step, the extracted job titles are mapped to a taxonomy, enabling the classifier to focus exclusively on this mapping. We demonstrate that this separation improves accuracy, robustness, and interpretability compared to single-step approaches. The method has been developed for German documents but is transferable to arbitrary languages. We further introduce a margin-based confidence criterion for occupation coding, replacing common absolute thresholds. To support reproducibility, we publish the source code and evaluation scripts.

References

  1. R. Dorau, K. Hein, J. Dörpinghaus, and M. Tiemann, “Automated classification of German job titles according to KldB: Challenges and novel methods,” in INFORMATIK, 2025. DOI : 10.18420/inf2025_86
  2. H. Gweon, M. Schonlau, L. Kaczmirek, M. Blohm, and S. Steiner, “Three methods for occupation coding based on statistical learning,” Journal of Official Statistics, vol. 33, no. 1, 2017. DOI : 10.1515/jos-2017-0006
  3. M. Schierholz, “Automating survey coding for occupation,” M.S. thesis, Ludwig-Maximilians-Universität München, 2014.
  4. M. Schierholz and M. Schonlau, “Machine learning for occupation coding – a comparison study,” JSSAM, vol. 9, no. 5, 2021. DOI: 10.1093/jssam/smaa023
  5. J.-J. Decorte, J. Van Hautte, T. Demeester, and C. Develder, “JobBERT: Understanding job titles through skills,” arXiv preprint https://arxiv.org/abs/2109.09605, 2021.
  6. J. Van Hautte, V. Schelstraete, and M. Wornoo, “Leveraging the inherent hierarchy of vacancy titles for automated job ontology expansion,” arXiv preprint https://arxiv.org/abs/2004.02814, 2020.
  7. A. M. Esser and J. Dörpinghaus, “Noise-aware named entity recognition for historical VET documents,” in VISAPP, 2026.
  8. A. Fischer and J. Dörpinghaus, “Web mining of online resources for German labor market research and education: Finding the ground truth?” Knowledge, vol. 4, no. 1, 2024. DOI: 10.3390/knowledge4010003
  9. J. Dörpinghaus, J. Binnewitt, S. Winnige, K. Hein, and K. Krüger, “Towards a German labor market ontology: Challenges and applications,” Applied Ontology, vol. 18, no. 4, 2023. DOI: 10.3233/AO-230027
  10. T. Reiser, J. Dörpinghaus, P. Steiner, and M. Tiemann, “Towards a dataset of digitalized historical German VET and CVET regulations,” Data, vol. 9, no. 11, 2024. DOI: 10.3390/data9110128
  11. R. Dorau and K. Hein, “Towards the automated classification of German job titles according to KldB,” in FedCSIS, 2025. DOI : 10.15439/2025F4081
  12. Bundesagentur für Arbeit, Klassifikation der Berufe 2010 – überarbeitete Fassung 2020, 2020.
  13. D. Nadeau and S. Sekine, “A survey of named entity recognition and classification,” Lingvisticae Investigationes, vol. 30, no. 1, 2007. DOI : 10.1075/li.30.1.03nad
  14. E. F. Tjong Kim Sang and F. De Meulder, “Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition,” in CoNLL, 2003. DOI: 10.3115/1119176.1119195
  15. T. Eftimov, B. Koroušić Seljak, and P. Korošec, “A rule-based named-entity recognition method for knowledge extraction of evidence-based dietary recommendations,” PLOS ONE, vol. 12, no. 6, 2017. DOI: 10.1371/journal.pone.0179488
  16. D. M. Bikel, R. Schwartz, and R. M. Weischedel, “An algorithm that learns what’s in a name,” Machine Learning, vol. 34, no. 1, 1999. DOI : 10.1023/A:1007558221122
  17. A. McCallum and W. Li, “Early results for named entity recognition with conditional random fields, feature induction and web-enhanced lexicons,” in CoNLL, 2003. DOI: 10.3115/1119176.1119206
  18. D. Jurafsky and J. H. Martin, Speech and Language Processing. 2025.
  19. L. R. Rabiner, “A tutorial on hidden markov models and selected applications in speech recognition,” IEEE, vol. 77, no. 2, 1989. DOI: 10.1109/5.18626
  20. J. D. Lafferty, A. McCallum, and F. C. N. Pereira, “Conditional random fields: Probabilistic models for segmenting and labeling sequence data,” in ICML, 2001.
  21. A. Viterbi, “Error bounds for convolutional codes and an asymptotically optimum decoding algorithm,” IEEE Transactions on Information Theory, vol. 13, no. 2, 1967. DOI: 10.1109/TIT.1967.1054010
  22. Z. Huang, W. Xu, and K. Yu, “Bidirectional LSTM-CRF models for sequence tagging,” arXiv preprint https://arxiv.org/abs/1508.01991, 2015.
  23. G. Lample, M. Ballesteros, S. Subramanian, K. Kawakami, and C. Dyer, “Neural architectures for Named Entity Recognition,” in NAACL-HLT, 2016. DOI: 10.18653/v1/N16-1030
  24. J. Pennington, R. Socher, and C. Manning, “GloVe: Global vectors for word representation,” in EMNLP, 2014. DOI: 10.3115/v1/D14-1162
  25. P. Bojanowski, E. Grave, A. Joulin, and T. Mikolov, “Enriching word vectors with subword information,” Transactions of the Association for Computational Linguistics, vol. 5, 2017. DOI : 10.1162/tacl_a_00051
  26. E. Grave, P. Bojanowski, P. Gupta, A. Joulin, and T. Mikolov, “Learning word vectors for 157 languages,” arXiv preprint https://arxiv.org/abs/1802.06893, 2018.
  27. A. Akbik, D. Blythe, and R. Vollgraf, “Contextual string embeddings for sequence labeling,” in COLING, 2018.
  28. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in NAACL-HLT, 2019. DOI:10.18653/v1/N19-1423
  29. A. Vaswani et al., “Attention is all you need,” NeurIPS, 2017.
  30. T. Reiser, J. Dörpinghaus, and M. Tiemann, “Detecting occupations in German texts: Challenges and data,” in AISEER, 2025.
  31. P. Safikhani, H. Avetisyan, D. Föste-Eggers, and D. Broneske, “Automated occupation coding with hierarchical features: A data-centric approach to classification with pre-trained language models,” Discover Artificial Intelligence, vol. 3, no. 1, 2023. DOI: 10.1007/s44163-023-00050-y
  32. National Research Council, Work, Jobs, and Occupations: A Critical Review of the Dictionary of Occupational Titles. 1980.
  33. M. Schierholz, “New methods for job and occupation classification,” Ph.D. dissertation, Universität Mannheim, 2019.
  34. M. Schierholz, M. Gensicke, N. Tschersich, and F. Kreuter, “Occupation coding during the interview,” J. R. Stat. Soc. Ser. A, vol. 181, no. 2, 2018. DOI : 10.1111/rssa.12297
  35. M. Schierholz, “Eine Hilfsklassifikation mit Tätigkeitsbeschreibungen für Zwecke der Berufskodierung,” AStA Wirtschafts- und Sozialstatistisches Archiv, vol. 12, no. 3, 2018. DOI : 10.1007/s11943-018-0231-2
  36. A. Bethmann, M. Schierholz, K. Wenzig, and M. Zielonka, “Automatic coding of occupations using machine learning algorithms for occupation coding in several German panel surveys,” in Statistics Canada Symposium, 2014.
  37. R. H. Creecy, B. M. Masand, S. J. Smith, and D. L. Waltz, “Trading MIPS and memory for knowledge engineering,” Communications of the ACM, vol. 35, no. 8, 1992. DOI : 10.1145/135226.135228
  38. D. E. Russ et al., “Computer-based coding of free-text job descriptions to efficiently identify occupations in epidemiological studies,” Occupational and Environmental Medicine, vol. 73, no. 6, 2016. DOI: 10.1136/oemed-2015-103152
  39. K. Takahashi, H. Taki, S. Tanabe, and W. Li, “An automatic coding system with a three-grade confidence level corresponding to the national/international occupation and industry standard: Open to the public on the web,” in KEOD, 2014. DOI : 10.5220/0005131703690375
  40. A. Ikudo, J. I. Lane, J. Staudt, and B. A. Weinberg, “Occupational classifications: A machine learning approach,” Journal of Economic and Social Measurement, vol. 44, no. 2-3, 2019. DOI: 10.3233/JEM-190463
  41. F. Javed, Q. Luo, M. McNair, F. Jacob, M. Zhao, and T. Kang, “Carotene: A job title classification system for the online recruitment domain,” in BigDataService, IEEE, 2015. DOI : 10.1109/BigDataService.2015.61
  42. J. Wang, K. Abdelfatah, M. Korayem, and J. Balaji, “DeepCarotene: Job title classification with multi-stream convolutional neural network,” in Big Data, IEEE, 2019. DOI: 10.1109/BigData47090.2019.9005673
  43. A. M. Esser and J. Dörpinghaus, “Noise-aware Named Entity Recognition for historical VET documents,” 2026, Extended version of the VISAPP 2026 paper. https://arxiv.org/abs/ 2601.00488 [cs.CL].
  44. K. Hein, M. Tiemann, and J. Dörpinghaus, “Talking about tasks or just sharing job offers? a case study on job-related tweets,” in INFORMATIK, 2025. DOI: 10.18420/inf2025_88
  45. M. Tiemann and J. Dörpinghaus, “Occupations and education in X data: How representative is the data?” In INFORMATIK, 2025. DOI: 10.18420/inf2025_90
  46. Bundesagentur für Arbeit, Klassifikation der Berufe, 1988.
  47. M. Namysł and I. Konya, “Efficient, lexicon-free OCR using deep learning,” in ICDAR, 2019. DOI : 10.1109/ICDAR.2019.00055
  48. M. Namysł, “Robust information extraction from unstructured documents,” Ph.D. dissertation, Rheinische Friedrich-Wilhelms-Universität Bonn, 2023.
  49. M. Namysł, S. Behnke, and J. Köhler, “NAT: Noiseaware training for robust neural sequence labeling,” in ACL, 2020. DOI: 10.18653/v1/2020.acl-main.138
  50. G. Xu, W. Ding, W. Fu, Z. Wu, and Z. Liu, “Robust learning for text classification with multi-source noise simulation and hard example mining,” in ECML PKDD, 2021. DOI: 10.1007/978-3-030-86517-7_18
  51. M. Namysł, S. Behnke, and J. Köhler, “Empirical error modeling improves robustness of noisy neural sequence labeling,” in ACL-IJCNLP Findings, 2021. DOI: 10.18653/v1/2021.findings-acl.27
  52. S. Gururangan et al., “Don’t stop pretraining: Adapt language models to domains and tasks,” in ACL, 2020. DOI: 10.18653/v1/2020.acl-main.740
  53. A. Paszke et al., “PyTorch: An imperative style, high-performance deep learning library,” NeurIPS, vol. 32, 2019.
  54. T. Wolf et al., Transformers, 2022.
  55. A. Wang and O. Russakovsky, “Overwriting pretrained bias with finetuning data,” in ICCV, 2023. DOI: 10.1109/ICCV51070.2023.00366
  56. F. Pedregosa et al., “Scikit-learn: Machine learning in Python,” Journal of Machine Learning Research, vol. 12, 2011.
  57. N. Reimers and I. Gurevych, “Sentence-BERT: Sentence embeddings using siamese BERT-networks,” in EMNLP-IJCNLP, Association for Computational Linguistics, 2019. DOI: 10.18653/v1/D19-1410
  58. A. M. Esser and T. Reiser, “Processing and linking vocational documents,” mimeo, 2026.