Proceedings of the 17th Conference on Computer Science and Intelligence Systems

DOI: http://dx.doi.org/10.15439/2022F294

Citation: Proceedings of the 17th Conference on Computer Science and Intelligence Systems, M. Ganzha, L. Maciaszek, M. Paprzycki, D. Ślęzak (eds). ACSIS, Vol. 30, pages 417–420 (2022)

Full text

Abstract. XGBoost and other gradient boosting frameworksare usually the default choice for solving classification and regression problems for tabular data, especially in data science competitions, as they often, combined with proper data preprocessing and feature engineering, supply high accuracy of predictions. They are also fast to learn, easy to tune, and can supply a ranking of variables, making interpretation of learned models easier. On the other hand, deep networks are the top choice for complex data, such as text, audio, or images. However, despite the many successful applications of deep networks, they are not yet prevalent on tabular ones. It may be related to difficulties in the choice of the proper architecture and its parameters. A solution to this problem may be found in recent works on deep architectures dedicated to tabular data, such as TabNet, which has recently been reported to achieve comparable or even better accuracy than XGBoost on some tabular datasets. In this paper, we compare XGBoost with TabNet in the context of the FedCSIS 2022 challenge, aimed at predicting forwarding contracts based on contract data and planned routes. The data has a typical tabular form, described by a multidimensional vector of numeric and nominal features. Of particular interest is investigating whether aggregation of predictions derived from XGBoost and TabNet could produce better results than either algorithm alone. The paper discusses the competition solution and shows some added experiments comparing XGBoost with TabNet on competition data, including incremental model re-building and parameter tuning. The experiments showed that the XGBoost and TabNet ensemble is a promising solution for building predictive models for tabular data. In the tests conducted, such an ensemble achieved a lower prediction error than each of the algorithms individually.

References

1. A. Janusz, A. Jamiołkowski, and M. Okulewicz, “Predicting the costs of forwarding contracts: Analysis of data mining competition results,” in Proceedings of the 17th Conference on Computer Science and Intelligence Systems, FedCSIS 2022, Sofia, Bulgaria, September 4-7, 2022. IEEE, 2022.
2. E. W. Ngai, L. Xiu, and D. C. Chau, “Application of data mining techniques in customer relationship management: A literature review and classification,” Expert systems with applications, vol. 36, no. 2, pp. 2592–2602, 2009. [Online]. Available: http://dx.doi.org/10.1016/j.eswa.2008.02.021
3. J. Bobadilla, F. Ortega, A. Hernando, and A. Gutiérrez, “Recommender systems survey,” Knowledge-based systems, vol. 46, pp. 109–132, 2013. [Online]. Available: http://dx.doi.org/10.1016/j.knosys.2013.03.012
4. I. Portugal, P. Alencar, and D. Cowan, “The use of machine learning algorithms in recommender systems: A systematic review,” Expert Systems with Applications, vol. 97, pp. 205–227, 2018.
5. S. Gupta, D. Hanssens, B. Hardie, W. Kahn, V. Kumar, N. Lin, N. Ravishanker, and S. Sriram, “Modeling customer lifetime value,” Journal of service research, vol. 9, no. 2, pp. 139–155, 2006. [Online]. Available: http://dx.doi.org/10.1177/1094670506293810
6. J. Ahn, J. Hwang, D. Kim, H. Choi, and S. Kang, “A survey on churn analysis in various business domains,” IEEE Access, vol. 8, pp. 220 816–220 839, 2020. [Online]. Available: http://dx.doi.org/10.1109/ ACCESS.2020.3042657
7. D. L. García, À. Nebot, and A. Vellido, “Intelligent data analysis approaches to churn as a business problem: a survey,” Knowledge and Information Systems, vol. 51, no. 3, pp. 719–774, 2017. [Online]. Available: http://dx.doi.org/10.1007/s10115-016-0995-z
8. S. Ö. Arik and T. Pfister, “Tabnet: Attentive interpretable tabular learning,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 8, 2021, pp. 6679–6687.
9. L. Katzir, G. Elidan, and R. El-Yaniv, “Net-dnf: Effective deep modeling of tabular data,” in International Conference on Learning Representations, 2020.
10. S. Popov, S. Morozov, and A. Babenko, “Neural oblivious decision ensembles for deep learning on tabular data,” arXiv preprint https://arxiv.org/abs/1909.06312, 2019.
11. X. Huang, A. Khetan, M. Cvitkovic, and Z. Karnin, “Tabtransformer: Tabular data modeling using contextual embeddings,” arXiv preprint https://arxiv.org/abs/2012.06678, 2020.
12. T. Chen and C. Guestrin, “XGBoost: A scalable tree boosting system,” in Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ser. KDD ’16. New York, NY, USA: ACM, 2016, pp. 785–794. [Online]. Available: http://doi.acm.org/10.1145/2939672.2939785