Logo PTI Logo FedCSIS

Proceedings of the 21st Conference on Computer Science and Intelligence Systems (FedCSIS)

Annals of Computer Science and Information Systems, Volume 47

Intelligent Communication Channel Selection for Heterogeneous Maritime Networks: A Q-Learning and Hysteresis-Based Approach

, , , , ,

DOI: http://dx.doi.org/10.15439/2026F5513

Citation: Shoaib Elahi, , , , ,

Full text

Abstract. The rapid digitization of the maritime industry has exponentially increased the demand for reliable, high-speed, and cost-effective communication. Modern autonomous and semi-autonomous vessels rely on a heterogeneous aggregation of network interfaces, including Geostationary Earth Orbit (GEO) satellites, Low Earth Orbit (LEO) satellites, 4G/5G cellular networks, Wi-Fi, and short-range radio frequencies. However, the maritime environment is inherently volatile, leading to unpredictable fluctuations in network latency, bandwidth availability, and signal stability. Traditional static routing and multi-criteria decision-making algorithms often fail to adapt to these rapid environmental changes, resulting in suboptimal data transmission, exorbitant satellite communication costs, and potentially dangerous delays in critical telemetry delivery. To address these critical challenges, this paper proposes an intelligent, autonomous communication channel selection system powered by Reinforcement Learning (RL). Specifically, a Q-Learning agent is designed to dynamically evaluate real-time state parameters (latency, bandwidth) and classify incoming data packets into critical, normal, and background priorities. The agent continuously learns to map specific data profiles to the most optimal network interface, penalizing high costs and extreme delays while rewarding connection stability. Furthermore, a Hysteresis Decision Engine is integrated as a stabilizing threshold layer. Results demonstrate that the proposed dual-layer architecture not only minimizes transmission delays for high-priority navigation data.

References

  1. G. Paragliola and A. Coronato, “Definition of a novel federated learning approach to reduce communication costs,” Expert Systems With Applications, vol. 189, p. 116109, 2022.
  2. T. Nguyen, N. Tran, L. L. Minh, S. H. Dang, and M. N. Vu, “Machine Learning for Dynamic Resource Allocation in Satellite Networks,” IEEE Transactions on Communications, vol. 68, no. 5, pp. 2901-2915, 2020.
  3. M. Sheng, J. Li, and Y. Shi, “Intelligent Network Selection in Heterogeneous Wireless Networks,” IEEE Wireless Communications, vol. 25, no. 1, pp. 44-50, 2018.
  4. R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed. Cambridge, MA, USA: MIT Press, 2018.
  5. Naeem, M., Bashir, S., Khan, M.U. and Syed, A.A., 2016, January. Performance comparison of scheduling algorithms for MU-MIMO systems. In 2016 13th International Bhurban Conference on Applied Sciences and Technology (IBCAST) (pp. 601-606). IEEE.
  6. X. Wang, L. Liu, and S. Li, “Q-Learning-Based Dynamic Channel Access for Maritime Cognitive Radio Networks,” IEEE Internet of Things Journal, vol. 7, no. 8, pp. 7192-7203, 2020.
  7. J. Hu et al., “Maritime Communications: A Survey on Trends, Standards, and Challenges in the 5G/6G Era,” IEEE Communications Surveys & Tutorials, vol. 24, no. 2, pp. 1024-1051, 2022.
  8. Y. Zhang, M. Zhou, and H. Zhao, “QoS Provisioning in Maritime Internet of Things: Architecture and Routing,” IEEE Network, vol. 34, no. 5, pp. 82-88, 2020.
  9. Rizvi, S.T.H., Kanwal, N. and Naeem, M., 2025. Bridging simplicity and sophistication using glinear: A novel architecture for enhanced time series prediction. Digital Signal Processing, p.105702.
  10. K. Lee, S. Kim, and J. Park, “Multi-Criteria Network Selection for Autonomous Ships in Heterogeneous Maritime Networks,” IEEE Access, vol. 9, pp. 15678-15689, 2021.
  11. Naeem, M., Coronato, A., Ullah, Z., Bashir, S. and Paragliola, G., 2022. Optimal user scheduling in multi antenna system using multi agent reinforcement learning. Sensors, 22(21), p.8278.
  12. X. Chen, Y. Wang, and Z. Liu, “Mobility Management and PingPong Mitigation in 5G Ultra-Dense Networks,” IEEE Transactions on Vehicular Technology, vol. 69, no. 7, pp. 7812-7825, 2020.
  13. Jamal, M., Ullah, Z., Naeem, M., Abbas, M. and Coronato, A., 2024. A hybrid multi-agent reinforcement learning approach for spectrum sharing in vehicular networks. Future Internet, 16(5), p.152.
  14. M. Ali, A. Khan, and S. Ahmed, “Hysteresis-based Handover Algorithms for Heterogeneous Wireless Networks: A Review,” Journal of Network and Computer Applications, vol. 175, p. 102905, 2021.
  15. Khan, A., Irfan, M., Ullah, Y., Ahmad, S., Ullah, S. and Hayat, B., 2019, December. Pilot contamination mitigation for high and low interference users in multi-cell massive MIMO systems. In 2019 15th International Conference on Emerging Technologies (ICET) (pp. 1-5). IEEE.
  16. R. Silva et al., “Reinforcement Learning for Autonomous Routing in Unpredictable Environments,” IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 3, pp. 2105-2118, 2022.
  17. Khan, A., Hayat, B., Ahmad, S., Karim, F.K., Liu, X., Khan, W.U. and Mostafa, S.M., 2025. AI-Empowered Multi-UAV and IRS Collaboration for Spectrum and Energy Optimization in B5G Networks. IEEE Internet of Things Journal.
  18. Naeem, Muddasar and Bashir, Sajid and Ullah, Zaib and Syed, Aqeel A syed, A near optimal scheduling algorithm for efficient radio resource management in multi-user MIMO systems, Wireless Personal Communications, 1411–1427, 2019, Springer
  19. T. L. Saaty, “Decision making with the analytic hierarchy process,” International Journal of Services Sciences, vol. 1, no. 1, pp. 83-98, 2008.
  20. C. L. Hwang and K. Yoon, Multiple Attribute Decision Making: Methods and Applications. Berlin, Heidelberg: Springer, 1981.
  21. Ismail, A., Naeem, M., Khalid, U.B. and Abbas, M., 2025. Improving adherence to medication in an intelligent environment using reinforcement learning. Journal of Reliable Intelligent Environments, 11(1), p.3.