Logo PTI Logo FedCSIS

Proceedings of the 17th Conference on Computer Science and Intelligence Systems

Annals of Computer Science and Information Systems, Volume 30

Considering various aspects of models’ quality in the ML pipeline - application in the logistics sector

, ,

DOI: http://dx.doi.org/10.15439/2022F296

Citation: Proceedings of the 17th Conference on Computer Science and Intelligence Systems, M. Ganzha, L. Maciaszek, M. Paprzycki, D. Ślęzak (eds). ACSIS, Vol. 30, pages 403–412 ()

Full text

Abstract. The industrial machine learning applications todayinvolve developing and deploying MLOps pipelines to ensure the versatile quality of forecasting models over an extended period, simultaneously assuring the model's accuracy, stability, short training time, and resilience. In this study, we present the ML pipeline conforming to all the abovementioned aspects of models' quality formulated as a constrained multi-objective optimization problem. We also provide the reference implementation on state-of-the-art methods for data preprocessing, feature extraction, dimensionality reduction, feature and instance selection, model fitting, and ensemble blending. The experimental study on the real data set from the logistics industry confirmed the qualities of the proposed approach, as the successful participation in an international data competition did.

References

  1. E. Zdravevski, P. Lameski, C. Apanowicz, and D. Śl ̨ezak, “From big data to business analytics: The case study of churn prediction,” Appl. Soft Comput., vol. 90, p. 106164, 2020. http://dx.doi.org/10.1016/j.asoc.2020.106164
  2. E. Kannout, “Context Clustering-based Recommender Systems,” in 2020 15th Conference on Computer Science and Information Systems (FedCSIS), 2020. http://dx.doi.org/10.15439/2020F54 pp. 85–91.
  3. M. Grzegorowski, A. Janusz, S. Lazewski, M. Swiechowski, and M. Jankowska, “Prescriptive analytics for optimization of fmcg delivery plans,” in Proceedings of IPMU’22, 2022.
  4. Y. Li, Y. Yang, K. Zhu, and J. Zhang, “Clothing sale forecasting by a composite gruprophet model with an attention mechanism,” IEEE Transactions on Industrial Informatics, vol. 17, no. 12, pp. 8335–8344, 2021. http://dx.doi.org/10.1109/TII.2021.3057922
  5. M. Grzegorowski, J. Litwin, M. Wnuk, M. Pabis, and L. Marcinowski, “Survival-based feature extraction - application in supply management for dispersed vending machines,” IEEE Transactions on Industrial Informatics, 2022. http://dx.doi.org/10.1109/TII.2022.3178547
  6. D. Ślęzak, M. Grzegorowski, A. Janusz, M. Kozielski, S. H. Nguyen, M. Sikora, S. Stawicki, and L. Wrobel, “A Framework for Learning and Embedding Multi-Sensor Forecasting Models into a Decision Support System: A Case Study of Methane Concentration in Coal Mines,” Information Sciences, vol. 451-452, pp. 112–133, 2018.
  7. C. Renggli, L. Rimanic, N. M. Gürel, B. Karlas, W. Wu, and C. Zhang, “A Data Quality-Driven View of MLOps,” CoRR, vol. abs/2102.07750, 2021. [Online]. Available: https://arxiv.org/abs/2102.07750
  8. Y. Zhou, Y. Yu, and B. Ding, “Towards MLOps: A Case Study of ML Pipeline Platform,” in 2020 International Conference on Artificial Intelligence and Computer Engineering (ICAICE), 2020. http://dx.doi.org/10.1109/I-CAICE51518.2020.00102 pp. 494–500.
  9. A. Subbaswamy, R. Adams, and S. Saria, “ Evaluating Model Robustness and Stability to Dataset Shift ,” in Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, ser. Proceedings of Machine Learning Research, A. Banerjee and K. Fukumizu, Eds., vol. 130. PMLR, 13–15 Apr 2021, pp. 2611–2619. [Online]. Available: https://proceedings.mlr.press/v130/subbaswamy21a.html
  10. M. Grzegorowski and D. Ślęzak, “On resilient feature selection: Computational foundations of r-C-reducts,” Inf. Sci., vol. 499, pp. 25–44, 2019. http://dx.doi.org/10.1016/j.ins.2019.05.041
  11. C. Rudin, “Please Stop Explaining Black Box Models for High Stakes Decisions,” CoRR, vol. abs/1811.10154, 2018.
  12. X. He, K. Zhao, and X. Chu, “Automl: A survey of the state-of-the-art,” Knowl. Based Syst., vol. 212, p. 106622, 2021. http://dx.doi.org/10.1016/j.knosys.2020.106622
  13. J. Blank and K. Deb, “Pymoo: Multi-Objective Optimization in Python,” IEEE Access, vol. 8, pp. 89 497–89 509, 2020. http://dx.doi.org/10.1109/ACCESS.2020.2990567
  14. H. M. Ridha, C. Gomes, H. Hizam, M. Ahmadipour, A. A. Heidari, and H. Chen, “Multi-objective optimization and multi-criteria decision-making methods for optimal design of standalone photovoltaic system: A comprehensive review,” Renewable and Sustainable Energy Reviews, vol. 135, p. 110202, 2021. http://dx.doi.org/10.1016/j.rser.2020.110202
  15. M. Grzegorowski, E. Zdravevski, A. Janusz, P. Lameski, C. Apanowicz, and D. Śl ̨ezak, “Cost Optimization for Big Data Workloads Based on Dynamic Scheduling and Cluster-Size Tuning,” Big Data Research, vol. 25, p. 100203, 2021. http://dx.doi.org/10.1016/j.bdr.2021.100203
  16. N. Verbiest, J. Derrac, C. Cornelis, S. García, and F. Herrera, “Evolutionary wrapper approaches for training set selection as preprocessing mechanism for support vector machines: Experimental evaluation and support vector analysis,” Applied Soft Computing, vol. 38, pp. 10–22, 2016. http://dx.doi.org/10.1016/j.asoc.2015.09.006
  17. A. Janusz, M. Grzegorowski, M. Michalak, Ł. Wróbel, M. Sikora, and D. Ślęzak, “Predicting Seismic Events in Coal Mines Based on Underground Sensor Measurements,” Engineering Applications of Artificial Intelligence, vol. 64, pp. 83–94, 2017.
  18. M. Grzegorowski, “Massively Parallel Feature Extraction Framework Application in Predicting Dangerous Seismic Events,” in Proceedings of the 2016 Federated Conference on Computer Science and Information Systems, FedCSIS 2016, Gdańsk, Poland, September 11-14, 2016, ser. Annals of Computer Science and Information Systems, M. Ganzha, L. A. Maciaszek, and M. Paprzycki, Eds., vol. 8. IEEE, 2016. http://dx.doi.org/10.15439/2016F90 pp. 225–229.
  19. A. Janusz, D. Ślęzak, S. Stawicki, and M. Rosiak, “Knowledge Pit - A Data Challenge Platform,” in Proceedings of the 24th International Workshop on Concurrency, Specification and Programming, Rzeszow, Poland, September 28-30, 2015, ser. CEUR Workshop Proceedings, Z. Suraj and L. Czaja, Eds., vol. 1492. CEUR-WS.org, 2015, pp. 191–195.
  20. G. F. Frederico, “From Supply Chain 4.0 to Supply Chain 5.0: Findings from a Systematic Literature Review and Research Directions,” Logistics, vol. 5, no. 3, 2021. http://dx.doi.org/10.3390/logistics5030049
  21. L. Barua, B. Zou, and Y. Zhou, “Machine learning for international freight transportation management: A comprehensive review,” Research in Transportation Business & Management, vol. 34, p. 100453, 2020. http://dx.doi.org/10.1016/j.rtbm.2020.100453 Data analytics for international transportation management.
  22. N. Servos, X. Liu, M. Teucke, and M. Freitag, “Travel Time Prediction in a Multimodal Freight Transport Relation Using Machine Learning Algorithms,” Logistics, vol. 4, no. 1, 2020. http://dx.doi.org/10.3390/logistics4010001
  23. S.-H. Chung, “Applications of smart technologies in logistics and transport: A review,” Transportation Research Part E: Logistics and Transportation Review, vol. 153, p. 102455, 2021. http://dx.doi.org/10.1016/j.tre.2021.102455
  24. J. Nobre and R. F. Neves, “Combining Principal Component Analysis, Discrete Wavelet Transform and XGBoost to trade in the financial markets,” Expert Systems with Applications, vol. 125, pp. 181–194, 2019. http://dx.doi.org/10.1016/j.eswa.2019.01.083
  25. R. Kasimbeyli, Z. Kamisli Ozturk, N. Kasimbeyli, G. Dinc Yalcin, and B. çmen Erdem, “Comparison of Some Scalarization Methods in Multiobjective Optimization,” Bull. Malays. Math. Sci. Soc., vol. 42, p. 18751905, 09 2019. http://dx.doi.org/10.1007/s40840-017-0579-4
  26. M. Grzegorowski, “Selected aspects of interactive feature extraction,” Ph.D. dissertation, University of Warsaw, 2021.
  27. D. Granato, J. S. Santos, G. B. Escher, B. L. Ferreira, and R. M. Maggio, “Use of principal component analysis (PCA) and hierarchical cluster analysis (HCA) for multivariate association between bioactive compounds and functional properties in foods: A critical perspective,” Trends in Food Science & Technology, vol. 72, pp. 83–90, 2018. doi: 10.1016/j.tifs.2017.12.006
  28. M. Grzegorowski and S. Stawicki, “Window-based feature extraction framework for multi-sensor data: A posture recognition case study,” in 2015 Federated Conference on Computer Science and Information Systems, FedCSIS 2015, Lódz, Poland, September 13-16, 2015, ser. Annals of Computer Science and Information Systems, M. Ganzha, L. A. Maciaszek, and M. Paprzycki, Eds., vol. 5. IEEE, 2015. http://dx.doi.org/10.15439/2015F425 pp. 397–405. [Online]. Available: https://doi.org/10.15439/2015F425
  29. E. Zdravevski, P. Lameski, R. Mingov, A. Kulakov, and D. Gjorgjevikj, “Robust histogram-based feature engineering of time series data,” in 2015 Federated Conference on Computer Science and Information Systems, FedCSIS 2015, Lódz, Poland, September 13-16, 2015, ser. Annals of Computer Science and Information Systems, M. Ganzha, L. A. Maciaszek, and M. Paprzycki, Eds., vol. 5. IEEE, 2015. http://dx.doi.org/10.15439/2015F420 pp. 381–388. [Online]. Available: https://doi.org/10.15439/2015F420
  30. B. Petrovska, E. Zdravevski, P. Lameski, R. Corizzo, I. Stajduhar, and J. Lerga, “Deep Learning for Feature Extraction in Remote Sensing: A Case-Study of Aerial Scene Classification,” Sensors, vol. 20, no. 14, p. 3906, 2020. http://dx.doi.org/10.3390/s20143906. [Online]. Available: https://doi.org/10.3390/s20143906
  31. D. Ślęzak, A. Chadzynska-Krasowska, J. Holland, P. Synak, R. Glick, and M. Perkowski, “Scalable cyber-security analytics with a new summary-based approximate query engine,” in 2017 IEEE International Conference on Big Data (IEEE BigData 2017), Boston, MA, USA, December 11-14, 2017, J. Nie, Z. Obradovic, T. Suzumura, R. Ghosh, R. Nambiar, C. Wang, H. Zang, R. Baeza-Yates, X. Hu, J. Kepner, A. Cuzzocrea, J. Tang, and M. Toyoda, Eds. IEEE Computer Society, 2017. http://dx.doi.org/10.1109/BigData.2017.8258128 pp. 1840–1849. [Online]. Available: https://doi.org/10.1109/BigData.2017.8258128
  32. D. Ślęzak, R. Glick, P. Betlinski, and P. Synak, “A new approximate query engine based on intelligent capture and fast transformations of granulated data summaries,” J. Intell. Inf. Syst., vol. 50, no. 2, pp. 385–414, 2018. http://dx.doi.org/10.1007/s10844-017-0471-6. [Online]. Available: https://doi.org/10.1007/s10844-017-0471-6
  33. M. Muniswamaiah, T. Agerwala, and C. C. Tappert, “Approximate Query Processing for Big Data in Heterogeneous Databases,” in 2020 IEEE International Conference on Big Data (IEEE BigData 2020), Atlanta, GA, USA, December 10-13, 2020, X. Wu, C. Jermaine, L. Xiong, X. Hu, O. Kotevska, S. Lu, W. Xu, S. Aluru, C. Zhai, E. Al-Masri, Z. Chen, and J. Saltz, Eds. IEEE, 2020. http://dx.doi.org/10.1109/BigData50022.2020.9378310 pp. 5765–5767. [Online]. Available: https://doi.org/10.1109/BigData50022.2020.9378310
  34. G. Ke, Q. Meng, T. Finley, T. Wang, W. Chen, W. Ma, Q. Ye, and T.-Y. Liu, “LightGBM: A Highly Efficient Gradient Boosting Decision Tree,” in Advances in Neural Information Processing Systems, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds., vol. 30. Curran Associates, Inc., 2017. [Online]. Available: https://proceedings.neurips.cc/paper/2017/file/6449f44a102fde848669bdd9eb6b76fa-Paper.pdf
  35. E. Wari and W. Zhu, “A survey on metaheuristics for optimization in food manufacturing industry,” Applied Soft Computing, vol. 46, pp. 328– 343, 2016. http://dx.doi.org/10.1016/j.asoc.2016.04.034
  36. M. Okulewicz and J. Mandziuk, “A metaheuristic approach to solve Dynamic Vehicle Routing Problem in continuous search space,” Swarm Evol. Comput., vol. 48, pp. 44–61, 2019. doi: 10.1016/j.swevo.2019.03.008. [Online]. Available: https://doi.org/10.1016/j.swevo.2019.03.008
  37. M. Ulinski, A. Zychowski, M. Okulewicz, M. Zaborski, and H. Kordulewski, “Generalized Self-adapting Particle Swarm Optimization Algorithm,” in Parallel Problem Solving from Nature - PPSN XV - 15th International Conference, Coimbra, Portugal, September 8-12, 2018, Proceedings, Part I, ser. Lecture Notes in Computer Science, A. Auger, C. M. Fonseca, N. Lourenço, P. Machado, L. Paquete, and L. D. Whitley, Eds., vol. 11101. Springer, 2018. http://dx.doi.org/10.1007/978-3-319-99253-2_3 pp. 29–40. [Online]. Available: https://doi.org/10.1007/978-3-319-99253-2_3
  38. M. Grzegorowski, A. Janusz, D. Śl ̨ezak, and M. S. Szczuka, “On the Role of Feature Space Granulation in Feature Selection Processes,” in 2017 IEEE International Conference on Big Data, BigData 2017, Boston, MA, USA, December 11-14, 2017, J. Nie, Z. Obradovic, T. Suzumura, R. Ghosh, R. Nambiar, C. Wang, H. Zang, R. Baeza-Yates, X. Hu, J. Kepner, A. Cuzzocrea, J. Tang, and M. Toyoda, Eds. IEEE Computer Society, 2017. http://dx.doi.org/10.1109/BigData.2017.8258124 pp. 1806–1815.
  39. S. Stawicki, D. Ślęzak, A. Janusz, and S. Widz, “Decision Bireducts and Decision Reducts – A Comparison,” International Journal of Approximate Reasoning, vol. 84, pp. 75–109, 2017.
  40. J. G. Bazan, A. Skowron, and P. Synak, “Dynamic Reducts as a Tool for Extracting Laws from Decisions Tables,” in Methodologies for Intelligent Systems, 8th International Symposium, ISMIS ’94, Charlotte, North Carolina, USA, October 16-19, 1994, Proceedings, ser. Lecture Notes in Computer Science, Z. W. Ras and M. Zemankova, Eds., vol. 869. Springer, 1994. http://dx.doi.org/10.1007/3-540-58495-1_35 pp. 346–355.
  41. S. H. Nguyen and M. S. Szczuka, “Feature Selection in Decision Systems with Constraints,” in Rough Sets - International Joint Conference, IJCRS 2016, Santiago de Chile, Chile, October 7-11, 2016, Proceedings, ser. Lecture Notes in Computer Science, V. Flores, F. A. C. Gomide, A. Janusz, C. Meneses, D. Miao, G. Peters, D. Śl ̨ezak, G. Wang, R. Weber, and Y. Yao, Eds., vol. 9920, 2016. http://dx.doi.org/10.1007/978-3-319-47160-0_49 pp. 537–547. [Online]. Available: https://doi.org/10.1007/978-3-319-47160-0_49
  42. K. Killamsetty, D. Sivasubramanian, G. Ramakrishnan, and R. Iyer, “Glister: Generalization based data subset selection for efficient and robust learning,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 9, pp. 8110–8118, May 2021. [Online]. Available: https://ojs.aaai.org/index.php/AAAI/article/view/16988
  43. N. Zhai, P. Yao, and X. Zhou, “Multivariate Time Series Forecast in Industrial Process Based on XGBoost and GRU,” in 2020 IEEE 9th Joint International Information Technology and Artificial Intelligence Conference (ITAIC), vol. 9, 2020. http://dx.doi.org/10.1109/ITAIC49862.2020.9338878 pp. 1397–1400.
  44. Y. Wang and X. Sherry Ni, “A XGBoost risk model via feature selection and Bayesian hyper-parameter optimization,” International Journal of Database Management Systems, vol. 11, no. 01, p. 0117, Feb 2019. http://dx.doi.org/10.5121/ijdms.2019.11101
  45. A. Janusz, A. Jamiołkowski, and M. Okulewicz, “Predicting the Costs of Forwarding Contracts: Analysis of Data Mining Competition Results,” in Proceedings of the 17th Conference on Computer Science and Intelligence Systems, FedCSIS 2022, Sofia, Bulgaria, September 4-7, 2022. IEEE, 2022.
  46. J. G. Bazan, S. Bazan-Socha, S. Buregwa-Czuma, Ł. Dydo, W. Rzasa, ̨ and A. Skowron, “A Classifier Based on a Decision Tree with Verifying Cuts,” Fundam. Informaticae, vol. 143, no. 1-2, pp. 1–18, 2016. http://dx.doi.org/10.3233/FI-2016-1300
  47. D. Ślęzak, M. Grzegorowski, A. Janusz, and S. Stawicki, “Toward interactive attribute selection with infolattices - A position paper,” in Rough Sets - International Joint Conference, IJCRS 2017, Olsztyn, Poland, July 3-7, 2017, Proceedings, Part II, ser. Lecture Notes in Computer Science, L. Polkowski, Y. Yao, P. Artiemjew, D. Ciucci, D. Liu, D. Śl ̨ezak, and B. Zielosko, Eds., vol. 10314. Springer, 2017. http://dx.doi.org/10.1007/978-3-319-60840-2_38 pp. 526–539. [Online]. Available: https://doi.org/10.1007/978-3-319-60840-2_38
  48. A. Janusz, G. Hao, D. Kaluza, T. Li, R. Wojciechowski, and D. Ślęzak, “Predicting escalations in customer support: Analysis of data mining challenge results,” in 2020 IEEE International Conference on Big Data (IEEE BigData 2020), Atlanta, GA, USA, December 10-13, 2020, X. Wu, C. Jermaine, L. Xiong, X. Hu, O. Kotevska, S. Lu, W. Xu, S. Aluru, C. Zhai, E. Al-Masri, Z. Chen, and J. Saltz, Eds. IEEE, 2020. http://dx.doi.org/10.1109/BigData50022.2020.9378024 pp. 5519–5526. [Online]. Available: https://doi.org/10.1109/BigData50022.2020.9378024