Logo PTI Logo FedCSIS

Proceedings of the 18th Conference on Computer Science and Intelligence Systems

Annals of Computer Science and Information Systems, Volume 35

Multi-queue service for task scheduling based on data availability

,

DOI: http://dx.doi.org/10.15439/2023F8957

Citation: Proceedings of the 18th Conference on Computer Science and Intelligence Systems, M. Ganzha, L. Maciaszek, M. Paprzycki, D. Ślęzak (eds). ACSIS, Vol. 35, pages 699–709 ()

Full text

Abstract. Large-scale computation (LSC) systems are often performed in distributed environments where message passing is the key to orchestrating computations. In this paper we present a new message queue concept developed within the context of an LSC system (BalticLSC). The concept consists in proposing a multi-queue, where queues are grouped into families. A queue family can be used to distribute messages of the same kind to multiple computation modules distributed between various nodes. Such message families can be synchronised to implement a mechanism for initiating computation jobs based on multiple data inputs. Moreover, the proposed multi-queue has built-in mechanisms for controlling message sequences in applications where complex data set splitting is necessary. The presented multi-queue concept was implemented and applied with success in a working LSC system.

References

  1. Adam Barker, Paolo Besana, David Robertson, and Jon B. Weissman. The benefits of service choreography for data-intensive computing. In Proceedings of the 7th International Workshop on Challenges of Large Applications in Distributed Environments, CLADE ’09, page 1–10, New York, NY, USA, 2009. Association for Computing Machinery.
  2. Li Chunlin, Tang Jianhang, and Luo Youlong. Multi-queue scheduling of heterogeneous jobs in hybrid geo-distributed cloud environment. The Journal of Supercomputing, 74:5263–5292, 2018.
  3. Philippe Dobbelaere and Kyumars Sheykh Esmaili. Kafka versus RabbitMQ: A comparative study of two industry reference publish/subscribe implementations: Industry paper. In Proceedings of the 11th ACM International Conference on Distributed and Event-Based Systems, DEBS ’17, page 227–238, New York, NY, USA, 2017. Association for Computing Machinery.
  4. Nishant Garg. Apache Kafka. Packt Publishing Birmingham, UK, 2013.
  5. Mohammad Hedayati, Kai Shen, Michael L Scott, and Mike Marty. Multi-queue fair queuing. In USENIX Annual Technical Conference, pages 301–314, 2019.
  6. Péter Kacsuk, editor. Science Gateways for Distributed Computing Infrastructures. Springer International Publishing, 2014.
  7. Peter Kacsuk, Zoltan Farkas, Miklos Kozlovszky, Gabor Hermann, Akos Balasko, Krisztian Karoczkai, and Istvan Marton. WS-PGRADE/gUSE generic DCI gateway framework for a large variety of user communities. Journal of Grid Computing, 10(4):601–630, nov 2012.
  8. Peter Kacsuk, József Kovács, and Zoltán Farkas. The Flowbster cloud-oriented workflow system to process large scientific data sets. Journal of Grid Computing, 16(1):55–83, jan 2018.
  9. A V Karthick, E Ramaraj, and R Ganapathy Subramanian. An efficient multi queue job scheduling for cloud computing. In 2014 World Congress on Computing and Communication Technologies, pages 164–166, 2014.
  10. Haopeng Li and Hui Li. A scheduling strategy based on multi-queues of cassandra. In 2017 IEEE International Conference on Big Data (Big Data), pages 2664–2669, 2017.
  11. Ji Liu, Esther Pacitti, Patrick Valduriez, and Marta Mattoso. A survey of data-intensive scientific workflow management. Journal of Grid Computing, 13:457–493, 2015.
  12. Pedro García López, Aitor Arjona, Josep Sampé, Aleksander Slominski, and Lionel Villard. Triggerflow: Trigger-based orchestration of serverless workflows. In Proceedings of the 14th ACM International Conference on Distributed and Event-Based Systems, DEBS ’20, page 3–14, New York, NY, USA, 2020. Association for Computing Machinery.
  13. Krzysztof Marek, Michał Śmiałek, Kamil Rybiński, Radosław Roszczyk, and Marek Wdowiak. BalticLSC: Low-code software development platform for large scale computations. Computing and Informatics, 40(4):734–753, 2021.
  14. Chris Peltz. Web services orchestration and choreography. Computer, 36(10):46–52, 2003.
  15. Anastasiia Postnikova, Nikita Koval, Giorgi Nadiradze, and Dan Alistarh. Multi-queues can be state-of-the-art priority schedulers. In Proceedings of the 27th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, pages 353–367, 2022.
  16. Stephen Ross-Talbot. Orchestration and choreography: Standards, tools and technologies for distributed workflows. In NETTAB Workshop-Workflows management: new abilities for the biological information overflow, volume 1, page 8, Naples, Italy, 2005.
  17. Maciej Rostanski, Krzysztof Grochla, and Aleksander Seman. Evaluation of highly available and fault-tolerant middleware clustered architectures using RabbitMQ. In 2014 Federated Conference on Computer Science and Information Systems, pages 879–884, 2014.
  18. Radoslaw Roszczyk, Marek Wdowiak, Michal Smialek, Kamil Rybinski, and Krzysztof Marek. BalticLSC: A low-code HPC platform for small and medium research teams. In 2021 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). IEEE, oct 2021.
  19. Kamil Rybiński, Michał Śmiałek, Agris Sostaks, Krzysztof Marek, Radosław Roszczyk, and Marek Wdowiak. Visual low-code language for orchestrating large-scale distributed computing. Journal of Grid Computing, 21(3), jul 2023.
  20. Gaurav Sharma, Neha Miglani, and Ajay Kumar. PLB: a resilient and adaptive task scheduling scheme based on multi-queues for cloud environment. Cluster Computing, 24(3):2615–2637, 2021.
  21. T Sharvari and Nag K Sowmya. A study on modern messaging systems-Kafka, RabbitMQ and NATS streaming. 2019.
  22. Jaspreet Singh and Deepali Gupta. An smarter multi queue job scheduling policy for cloud computing. International Journal of Applied Engineering Research, 12(9):1929–1934, 2017.
  23. John Vineet and Liu Xia. A survey of distributed message broker queues. 2017.
  24. Steve Vinoski. Advanced Message Queuing Protocol. IEEE Internet Computing, 10(6):87–89, 2006.
  25. Guozhang Wang, Joel Koshy, Sriram Subramanian, Kartik Paramasivam, Mammad Zadeh, Neha Narkhede, Jun Rao, Jay Kreps, and Joe Stein. Building a replicated logging system with Apache Kafka. Proceedings of the VLDB Endowment, 8(12):1654–1655, aug 2015.