Logo PTI Logo FedCSIS

Proceedings of the 20th Conference on Computer Science and Intelligence Systems (FedCSIS)

Annals of Computer Science and Information Systems, Volume 43

Out-Of-Distribution Is Not Magic: The Clash Between Rejection Rate and Model Success

, , ,

DOI: http://dx.doi.org/10.15439/2025F5708

Citation: Proceedings of the 20th Conference on Computer Science and Intelligence Systems (FedCSIS), M. Bolanowski, M. Ganzha, L. Maciaszek, M. Paprzycki, D. Ślęzak (eds). ACSIS, Vol. 43, pages 345–350 ()

Full text

Abstract. Recent advancements in Internet protocols, including DNS over HTTPS (DoH) and Encrypted Service Name Indicators (ESNI), are making traditional Deep Packet Inspection (DPI) engines obsolete. Consequently, there is a growing need for next generation traffic classification-based artificial intelligence (AI). While DPI automatically categorizes unknown traffic as 'other,' AI-based models cannot automatically handle unknown or Out-of-Distribution (OOD) traffic. AI models must effectively detect and classify OOD traffic to ensure robustness, reliability, and accuracy in real-world applications; however, current research often fails to address the challenges of OOD detection. In this paper, we evaluate various state-of-the-art OOD detection techniques for internet traffic classification and explore the drawbacks and advantages of using different levels of thresholds for the model's tolerance for OOD. Our findings reveal that varying rejection rates have distinct effects on OOD techniques, leading to a change in the optimal strategy for achieving dependable and precise detection across diverse OOD scenarios. We demonstrate that adjusting rejection rates from 10\% to 30\% can significantly improve the True Detection Rate (TDR) by up to 60\%, while the False Detection Rate (FDR) may increase by less than 10\%. Moreover, we emphasize that rejection-rate-based evaluation is pivotal for next-generation flow classification, promising a substantial reduction in FDR through correct methodological assessment.

References

  1. D. Moore, K. Keys, R. Koga, E. Lagache, and K. C. Claffy, “The {CoralReef} software suite as a tool for system and network administrators,” in LISA, 2001.
  2. S. Xu, J. Han, Y. Liu, H. Liu, and Y. Bai, “Few-shot traffic classification based on autoencoder and deep graph convolutional networks,” Scientific Reports, vol. 15, no. 1, p. 8995, 2025.
  3. P. Tang, Y. Dong, S. Mao, H.-L. Wei, and J. Jin, “Online classification of network traffic based on granular computing,” IEEE Transactions on Systems, Man, and Cybernetics: Systems, vol. 53, no. 8, pp. 5199–5211, 2023.
  4. L. Yang, A. Finamore, F. Jun, and D. Rossi, “Deep Learning and Traffic Classification: Lessons learned from a commercial-grade dataset with hundreds of encrypted and zero-day applications,” arXiv preprint https://arxiv.org/abs/2104.03182, 2021.
  5. A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet Classification with Deep Convolutional Neural Networks,” in Advances in Neural Information Processing Systems, vol. 25, 2012.
  6. X. Wang, Y. Wang, Y. Lai, Z. Hao, and A. X. Liu, “Reliable openset network traffic classification,” IEEE Transactions on Information Forensics and Security, 2025.
  7. S. Vaze, K. Han, A. Vedaldi, and A. Zisserman, “Open-Set Recognition: A good closed-set classifier is all you need,” CoRR, vol. abs/2110.06207, 2021. [Online]. Available: https://arxiv.org/abs/2110.06207
  8. V. Franc, D. Průša, and V. Voracek, “Optimal strategies for reject option classifiers,” CoRR, vol. abs/2101.12523, 2021. [Online]. Available: https://arxiv.org/abs/2101.12523
  9. Z. Fang, J. Lu, and G. Zhang, “Out-of-distribution detection with nonsemantic exploration,” Information Sciences, vol. 705, p. 121989, 2025.
  10. S. Thulasidasan, S. Thapa, S. Dhaubhadel, G. Chennupati, T. Bhattacharya, and J. Bilmes, “An effective baseline for robustness to distributional shift,” in ICMLA. IEEE, 2021, pp. 278–285.
  11. T. DeVries and G. W. Taylor, “Learning confidence for out-of-distribution detection in neural networks,” arXiv preprint https://arxiv.org/abs/1802.04865, 2018.
  12. Y. Chen, Z. Li, J. Shi, G. Gou, C. Liu, and G. Xiong, “Not Afraid of the Unseen: a Siamese Network based Scheme for Unknown Traffic Discovery,” in ISCC, 2020, pp. 1–7.
  13. H. He, Y. Lai, Y. Wang, S. Le, and Z. Zhao, “A data skew-based unknown traffic classification approach for TLS applications,” Future Generation Computer Systems, 2022.
  14. Z. Yang and W. Lin, “Unknown traffic identification based on deep adaptation networks,” in 2020 IEEE 45th LCN Symposium on Emerging Topics in Networking (LCN Symposium). IEEE, 2020, pp. 10–18.
  15. S. Liang, Y. Li, and R. Srikant, “Enhancing the reliability of out-of-distribution image detection in neural networks,” arXiv preprint https://arxiv.org/abs/1706.02690, 2017.
  16. L. Neal, M. Olson, X. Fern, W.-K. Wong, and F. Li, “Open Set Learning with Counterfactual Images,” in European Conference, Munich, Germany, September 8–14, 2018, 2018, pp. 620–635.
  17. S. Rezaei and X. Liu, “Deep Learning for Encrypted Traffic Classification: An Overview,” IEEE Communications Magazine, vol. 57, no. 5, pp. 76–81, may 2019.
  18. G. Draper-Gil, A. H. Lashkari, M. S. I. Mamun, and A. A. Ghorbani, “Characterization of Encrypted and VPN traffic using time-related features,” in ICISSP, Rome, Italy, February 19-21, 2016, pp. 407–414.
  19. Stratosphere, “Stratosphere Laboratory Datasets,” 2015, retrieved March 13, 2020, from https://www.stratosphereips.org/datasets-overview.
  20. R. Dubin, A. Dvir, O. Pele, J. Muehlstein, Y. Zion, M. Bahumi, and I. Kirshenboim, “Analyzing HTTPS Encrypted Traffic to Identify User’s Operating System, Browser and Application,” in IEEE CCNC, June 2017.
  21. A. Lichy, O. Bader, R. Dubin, A. Dvir, and C. Hajaj, “When a rf beats a cnn and gru, together—a comparison of deep learning and classical machine learning approaches for encrypted malware traffic classification,” Computers & Security, vol. 124, p. 103000, 2023.
  22. S. Jorgensen, J. Holodnak, J. Dempsey, K. de Souza, A. Raghunath, V. Rivet, N. DeMoes, A. Alejos, and A. Wollaber, “Extensible Machine Learning for Encrypted Network Traffic Application Labeling via Uncertainty Quantification,” arXiv preprint https://arxiv.org/abs/2205.05628, 2022.
  23. I. Meiri, “Ood not magic - github repository,” 2025, available at: https://github.com/ArielCyber/OOD_Is_Not_Magic.
  24. D. Hendrycks and K. Gimpel, “A baseline for detecting misclassified and out-of-distribution examples in neural networks,” arXiv preprint https://arxiv.org/abs/1610.02136, 2016.
  25. J. Ren, P. J. Liu, E. Fertig, J. Snoek, R. Poplin, M. Depristo, J. Dillon, and B. Lakshminarayanan, “Likelihood ratios for out-of-distribution detection,” in Advances in Neural Information Processing Systems, vol. 32. Curran Associates, Inc., 2019.