Logo PTI
Polish Information Processing Society
Logo FedCSIS

Annals of Computer Science and Information Systems, Volume 11

Proceedings of the 2017 Federated Conference on Computer Science and Information Systems

CloudLightning: a Self-Organized Self-Managed Heterogeneous Cloud

, , , , , ,

DOI: http://dx.doi.org/10.15439/2017F274

Citation: Proceedings of the 2017 Federated Conference on Computer Science and Information Systems, M. Ganzha, L. Maciaszek, M. Paprzycki (eds). ACSIS, Vol. 11, pages 749–758 ()

Full text

Abstract. The increasing heterogeneity of cloud resources, and the increasing diversity of services being deployed in cloud environments are leading to significant increases in the complexities of cloud resource management. This paper presents an architecture to manage heterogeneous resources and to improve service delivery in cloud environments. A loosely-coupled, hierarchical, self-adapting management model, deployed across multiple layers, is used for heterogeneous resource management. Moreover, a service-specific coalition formation mechanism is employed to identify appropriate resources to support the process parallelism associated with high performance services. Finally, a proof-of-concept of the proposed hierarchical cloud architecture, as realized in CloudLightning project, is presented.

References

  1. Microsoft, “Steve Ballmer: Worldwide partner conference 2013 keynote,” Press Release, Houston, Texas, Jul. 2013.
  2. L. A. Barroso and U. Hölzle, “The case for energy-proportional computing,” Computer, no. 12, pp. 33–37, 2007.
  3. D. C. Marinescu, A. Paya, J. P. Morrison, and P. Healy, “Distributed hierarchical control versus an economic model for cloud resource management,” arXiv preprint https://arxiv.org/abs/1503.01061, 2015.
  4. L. A. Barroso, J. Clidaras, and U. Hölzle, “The datacenter as a computer: An introduction to the design of warehouse-scale machines,” Synthesis lectures on computer architecture, vol. 8, no. 3, pp. 1–154, 2013.
  5. L. Tang, J. Mars, X. Zhang, R. Hagmann, R. Hundt, and E. Tune, “Optimizing google’s warehouse scale computers: The numa experience,” in High Performance Computer Architecture (HPCA2013), 2013 IEEE 19th International Symposium on. IEEE, 2013, pp. 188–197.
  6. M. Ahuja, C. C. Chen, R. Gottapu, J. Hallmann, W. Hasan, R. Johnson, M. Kozyrczak, R. Pabbati, N. Pandit, S. Pokuri et al., “Peta-scale data warehousing at yahoo!” in Proceedings of the 2009 ACM SIGMOD International Conference on Management of data. ACM, 2009, pp. 855–862.
  7. J. Hauswald, M. A. Laurenzano, Y. Zhang, C. Li, A. Rovinski, A. Khurana, R. G. Dreslinski, T. Mudge, V. Petrucci, L. Tang et al., “Sirius: An open end-to-end voice and vision personal assistant and its implications for future warehouse scale computers,” in Proceedings of the Twentieth International Conference on Architectural Support for Programming Languages and Operating Systems. ACM, 2015, pp. 223–238.
  8. J. C. Doyle, B. A. Francis, and A. R. Tannenbaum, Feedback control theory. Courier Corporation, 2013.
  9. Y. Lu, T. Abdelzaher, C. Lu, L. Sha, and X. Liu, “Feedback control with queueing-theoretic prediction for relative delay guarantees in web servers,” in Real-Time and Embedded Technology and Applications Symposium, 2003. Proceedings. The 9th IEEE. IEEE, 2003, pp. 208–217.
  10. T. F. Abdelzaher, K. G. Shin, and N. Bhatti, “Performance guarantees for web server end-systems: A control-theoretical approach,” IEEE transactions on parallel and distributed systems, vol. 13, no. 1, pp. 80–96, 2002.
  11. X. Wang and Y. Wang, “Coordinating power control and performance management for virtualized server clusters,” IEEE Transactions on Parallel and Distributed Systems, vol. 22, no. 2, pp. 245–259, 2011.
  12. X. Wang, M. Chen, C. Lefurgy, and T. W. Keller, “Ship: A scalable hierarchical power control architecture for large-scale data centers,” IEEE Transactions on Parallel and Distributed Systems, vol. 23, no. 1, pp. 168–176, 2012.
  13. S. Crago, K. Dunn, P. Eads, L. Hochstein, D.-I. Kang, M. Kang, D. Modium, K. Singh, J. Suh, and J. P. Walters, “Heterogeneous cloud computing,” in 2011 IEEE International Conference on Cluster Computing. IEEE, 2011, pp. 378–385.
  14. T. R. Scogland, C. P. Steffen, T. Wilde, F. Parent, S. Coghlan, N. Bates, W.-c. Feng, and E. Strohmaier, “A power-measurement methodology for large-scale, high-performance computing,” in Proceedings of the 5th ACM/SPEC international conference on Performance engineering. ACM, 2014, pp. 149–159.
  15. G. Lee and R. H. Katz, “Heterogeneity-aware resource allocation and scheduling in the cloud.” in HotCloud, 2011.
  16. J. Novet. (2016, November) Aws launches elastic gpus for ec2, fpga-backed f1 instances, r4 and refreshed t2, c5 and i3 coming in q1. [Online]. Available: http://venturebeat.com/2016/11/30/aws-launches-elastic-gpus-for-ec2-fpga-backed-f1-instances-r4-and-refreshed-t2-c5-and-i3-coming-in-q1/
  17. OpenStack Heterogeneous Accelerator Support, https://wiki.openstack.org/wiki/HeterogeneousInstanceTypes.
  18. S. Conway, C. Dekate, and E. Joseph, “Worldwide highperformance data analysis 2014–2018 forecast,” IDC, Doc, vol. 248789, 2014.
  19. J. G. F. Coutinho, O. Pell, E. ONeill, P. Sanders, J. McGlone, P. Grigoras, W. Luk, and C. Ragusa, “Harness project: Managing heterogeneous computing resources for a cloud platform,” in International Symposium on Applied Reconfigurable Computing. Springer, 2014, pp. 324–329.
  20. L. López, F. J. Nieto, T.-H. Velivassaki, S. Kosta, C.-H. Hong, R. Montella, I. Mavroidis, and C. Fernández, “Heterogeneous secure multi-level remote acceleration service for low-power integrated systems and devices,” Procedia Computer Science, vol. 97, pp. 118–121, 2016.
  21. B. Hindman, A. Konwinski, M. Zaharia, A. Ghodsi, A. D. Joseph, R. H. Katz, S. Shenker, and I. Stoica, “Mesos: A platform for fine-grained resource sharing in the data center.” in NSDI, vol. 11, 2011, pp. 22–22.
  22. A. Verma, L. Pedrosa, M. Korupolu, D. Oppenheimer, E. Tune, and J. Wilkes, “Large-scale cluster management at google with borg,” in Proceedings of the Tenth European Conference on Computer Systems. ACM, 2015, p. 18.
  23. M. Schwarzkopf, A. Konwinski, M. Abd-El-Malek, and J. Wilkes, “Omega: flexible, scalable schedulers for large compute clusters,” in Proceedings of the 8th ACM European Conference on Computer Systems. ACM, 2013, pp. 351–364.
  24. D. Bernstein, “Containers and cloud: From lxc to docker to kubernetes,” IEEE Cloud Computing, no. 3, pp. 81–84, 2014.
  25. “Simplicity and complexity in the description of nature,” Engineering and Science, vol. 57, no. 3, pp. 2–9, 1988.
  26. P. Schuster, “Nonlinear dynamics from physics to biology,” Complexity, vol. 12, no. 4, pp. 9–11, 2007.
  27. Y.-Y. Liu, J.-J. Slotine, and A.-L. Barabasi, “Controllability of complex networks,” Nature, vol. 473, no. 7346, pp. 167–173, May 2011.
  28. P.-A. Noël, C. D. Brummitt, and R. M. D’Souza, “Controlling self-organizing dynamics on networks using models that self-organize,” Phys. Rev. Lett., vol. 111, p. 078701, Aug 2013.
  29. F. Heylighen and C. Gershenson, “The meaning of self-organization in computing,” in IEEE Intelligent Systems, Section Trends & Controversies - Sel-Organization and Information Systems, 2003.
  30. A. M. Turing, “The chemical basis of morphogenesis,” Philosophical Transactions of the Royal Society of London B: Biological Sciences, vol. 237, no. 641, pp. 37–72, 1952.
  31. F. Heylighen et al., “The science of self-organization and adaptivity,” The encyclopedia of life support systems, vol. 5, no. 3, pp. 253–280, 2001.
  32. J. Kramer and J. Magee, “Self-managed systems: an architectural challenge,” in Future of Software Engineering, 2007. FOSE’07. IEEE, 2007, pp. 259–268.
  33. M. Puviani and R. Frei, “Self-management for cloud computing,” in Science and Information Conference (SAI), 2013. IEEE, 2013, pp. 940–946.
  34. Q. Zhang, L. Cheng, and R. Boutaba, “Cloud computing: state-of-the-art and research challenges,” Journal of internet services and applications, vol. 1, no. 1, pp. 7–18, 2010.
  35. M. Parashar and S. Hariri, “Autonomic computing: An overview,” in Unconventional Programming Paradigms. Springer, 2005, pp. 257–269.
  36. D. C. Marinescu, A. Paya, J. P. Morrison, and P. Healy, “An auction-driven self-organizing cloud delivery model,” arXiv preprint https://arxiv.org/abs/1312.2998, 2013.
  37. I. Brandic, “Towards self-manageable cloud services,” in 2009 33rd Annual IEEE International Computer Software and Applications Conference, vol. 2. IEEE, 2009, pp. 128–133.
  38. D. Dong, H. Xiong, P. Stack and J. P. Morrison, “Managing and Unifying Heterogeneous Resources in Cloud Environments,” The 7th International Conference on Cloud Computing and Services Science (CLOSER 2017), 24-26 April 2017, Porto, Portugal.
  39. OpenStack Nova, http://docs.openstack.org/developer/nova/.
  40. Kubernetes, http://kubernetes.io/.
  41. B. Hindman, A. Konwinski, M. Zaharia, A. Ghodsi, A. D. Joseph, R. Katz, S. Shenker, and I. Stoica, “Mesos: A platform for fine-grained resource sharing in the data center,” in Proceedings of the 8th USENIX Conference on Networked Systems Design and Implementation (NSDI 2011), 2011, pp. 295–308.
  42. Docker Swarm, https://github.com/docker/swarm.
  43. OpenStack Ironic, http://docs.openstack.org/developer/ironic/deploy/user-guide.html.
  44. Z. Wang and X. Su, “Dynamically hierarchical resource-allocation algorithm in cloud computing environment,” The Journal of Supercomputing, vol. 71, no. 7, pp. 2748–2766, 2015.
  45. A. Konak, D. W. Coit, and A. E. Smith, “Multi-objective optimization using genetic algorithms: A tutorial,” Reliability Engineering & System Safety, vol. 91, no. 9, pp. 992–1007, 2006.
  46. T. Saber, A. Ventresque, J. Marques-Silva, J. Thorburn, and L. Murphy, “Milp for the multi-objective vm reassignment problem,” in Tools with Artificial Intelligence (ICTAI), 2015 IEEE 27th International Conference on. IEEE, 2015, pp. 41–48.
  47. E. Zitzler and L. Thiele, “Multiobjective optimization using evolutionary algorithmsa comparative case study,” in International Conference on Parallel Problem Solving from Nature. Springer, 1998, pp. 292–301.
  48. A. Beloglazov and R. Buyya, “Energy efficient resource management in virtualized cloud data centers,” in Proceedings of the 2010 10th IEEE/ACM international conference on cluster, cloud and grid computing. IEEE Computer Society, 2010, pp. 826–831.
  49. X. Li, Z. Qian, S. Lu, and J. Wu, “Energy efficient virtual machine placement algorithm with balanced and improved resource utilization in a data center,” Mathematical and Computer Modelling, vol. 58, no. 5, pp. 1222–1235, 2013.
  50. J. Dong, X. Jin, H. Wang, Y. Li, P. Zhang, and S. Cheng, “Energy-saving virtual machine placement in cloud data centers,” in Cluster, Cloud and Grid Computing (CCGrid), 2013 13th IEEE/ACM International Symposium on. IEEE, 2013, pp. 618–624.
  51. S. Srikantaiah, A. Kansal, and F. Zhao, “Energy aware consolidation for cloud computing,” in Proceedings of the 2008 conference on Power aware computing and systems, vol. 10. San Diego, California, 2008, pp. 1–5.
  52. D. C. Marinescu, A. Paya, and J. P. Morrison, “Coalition formation and combinatorial auctions; applications to self-organization and self-management in utility computing,” arXiv preprint https://arxiv.org/abs/1406.7487, 2014.
  53. C. Filelis-Papadopoulos, H. Xiong, A. Spataru, G. Castane, D. Dong, G. Gravvanis and J. P. Morrison, “A Generic Framework Supporting Self-organisation and Self-management in Hierarchical Systems,” The 16th International Symposium on Parallel and Distributed Computing (ISPDC 2017), 3-6 July 2017, Innsbruck, Austria.