Logo PTI
Polish Information Processing Society
Logo FedCSIS

Annals of Computer Science and Information Systems, Volume 18

Proceedings of the 2019 Federated Conference on Computer Science and Information Systems

Big Data Platform for Smart Grids Power Consumption Anomaly Detection

, ,

DOI: http://dx.doi.org/10.15439/2019F210

Citation: Proceedings of the 2019 Federated Conference on Computer Science and Information Systems, M. Ganzha, L. Maciaszek, M. Paprzycki (eds). ACSIS, Vol. 18, pages 771–780 ()

Full text

Abstract. Big data processing in the Smart Grid context has many large-scale applications that require real-time data analysis (e.g., intrusion and data injection attacks detection, electric device health monitoring). In this paper, we present a big data platform for anomaly detection of power consumption data. The platform is based on an ingestion layer with data densification options, Apache Flink as part of the speed layer and HDFS/KairosDB as data storage layers. We showcase the application of the platform to a scenario of power consumption anomaly detection, benchmarking different alternative frameworks used at the speed layer level (Flink, Storm, Spark).

References

  1. Y. Demchenko, P. Grosso, C. De Laat, and P. Membrey, “Addressing big data issues in scientific data infrastructure,” in 2013 International Conference on Collaboration Technologies and Systems (CTS). IEEE, 2013. http://dx.doi.org/10.1109/CTS.2013.6567203 pp. 48–55.
  2. A. Radenski, T. Gurov, K. Kaloyanova, N. Kirov, M. Nisheva, P. Stanchev, and E. Stoimenova, “Big data techniques, systems, applications, and platforms: Case studies from academia,” in 2016 Federated Conference on Computer Science and Information Systems (FedCSIS). IEEE, 2016. http://dx.doi.org/10.15439/2016F91 pp. 883–888.
  3. Y. Zhang, T. Huang, and E. F. Bompard, “Big data analytics in smart grids: A review,” Energy Informatics, vol. 1, no. 1, p. 8, 2018. http://dx.doi.org/10.1186/s42162-018-0007-5
  4. K. Zhou, C. Fu, and S. Yang, “Big data driven smart energy management: From big data to big insights,” Renewable and Sustainable Energy Reviews, vol. 56, pp. 215–225, 2016. http://dx.doi.org/10.1016/j.rser.2015.11.050
  5. B. Rossi and S. Chren, “Smart grids data analysis: A systematic mapping study,” arXiv preprint https://arxiv.org/abs/1808.00156, 2018. [Online]. Available: https://arxiv.org/abs/1808.00156
  6. M. Ge, H. Bangui, and B. Buhnova, “Big data for internet of things: a survey,” Future Generation Computer Systems, vol. 87, pp. 601–614, 2018. http://dx.doi.org/10.1016/j.future.2018.04.053
  7. X. Liu and P. S. Nielsen, “Regression-based online anomaly detection for smart grid data,” arXiv preprint https://arxiv.org/abs/1606.05781, 2016.
  8. B. Rossi, S. Chren, B. Buhnova, and T. Pitner, “Anomaly detection in smart grid data: An experience report,” in 2016 IEEE International Conference on Systems, Man, and Cybernetics (SMC). IEEE, 2016. http://dx.doi.org/10.1109/SMC.2016.7844583 pp. 2313–2318.
  9. Z.-H. Yu and W.-L. Chin, “Blind False Data Injection Attack Using PCA Approximation Method in Smart Grid,” IEEE Transactions on Smart Grid, vol. 6, no. 3, pp. 1219–1226, 2015. http://dx.doi.org/10.1109/TSG.2014.2382714
  10. J. Lee, Y.-c. Kim, and G.-L. Park, “An Analysis of Smart Meter Readings Using Artificial Neural Networks,” in Convergence and Hybrid Information Technology. Berlin, Heidelberg: Springer Berlin Heidelberg, 2012, vol. 7425, pp. 182–188.
  11. F. McLoughlin, A. Duffy, and M. Conlon, “A clustering approach to domestic electricity load profile characterisation using smart metering data,” Applied energy, vol. 141, pp. 190–199, 2015. http://dx.doi.org/10.1016/j.apenergy.2014.12.039
  12. P. Pääkkönen and D. Pakkala, “Reference architecture and classification of technologies, products and services for big data systems,” Big data research, vol. 2, no. 4, pp. 166–186, 2015. http://dx.doi.org/10.1016/j.bdr.2015.01.001
  13. N. Marz and J. Warren, Big Data: Principles and best practices of scalable real-time data systems. New York; Manning Publications Co., 2015.
  14. J. Lin, “The lambda and the kappa,” IEEE Internet Computing, vol. 21, no. 5, pp. 60–66, 2017. http://dx.doi.org/10.1109/MIC.2017.3481351
  15. S. Shahrivari, “Beyond batch processing: towards real-time and stream- ing big data,” Computers, vol. 3, no. 4, pp. 117–129, 2014. http://dx.doi.org/10.3390/computers3040117
  16. J. Kreps, “Questioning the lambda architecture,” Online article, July, 2014. [Online]. Available: https://www.oreilly.com/ideas/questioning-the-lambda-architecture
  17. M. Kiran, P. Murphy, I. Monga, J. Dugan, and S. S. Baveja, “Lambda architecture for cost-effective batch and speed big data processing,” in 2015 IEEE International Conference on Big Data (Big Data). IEEE, 2015. http://dx.doi.org/10.1109/BigData.2015.7364082 pp. 2785–2792.
  18. J. van Rooij, V. Gulisano, and M. Papatriantafilou, “Locovolt: Distributed detection of broken meters in smart grids through stream processing,” in Proceedings of the 12th ACM International Conference on Distributed and Event-based Systems. ACM, 2018. http://dx.doi.org/10.1145/3210284.3210298 pp. 171–182.
  19. H. Sequeira, P. Carreira, T. Goldschmidt, and P. Vorst, “Energy cloud: Real-time cloud-native energy management system to monitor and analyze energy consumption in multiple industrial sites,” in 2014 IEEE/ACM 7th International Conference on Utility and Cloud Computing. IEEE, 2014. http://dx.doi.org/10.1109/UCC.2014.79 pp. 529–534.
  20. M. Mayilvaganan and M. Sabitha, “A cloud-based architecture for big-data analytics in smart grid: A proposal,” in 2013 IEEE International Conference on Computational Intelligence and Computing Research, Dec 2013. http://dx.doi.org/10.1109/ICCIC.2013.6724168 pp. 1–4.
  21. A. A. Munshi and Y. A. I. Mohamed, “Data lake lambda architecture for smart grids big data analytics,” IEEE Access, vol. 6, pp. 40 463–40 471, 2018. http://dx.doi.org/10.1109/ACCESS.2018.2858256
  22. X. Liu and P. Nielsen, “Streamlining smart meter data analytics,” in Proceedings of the 10th Conference on Sustainable Development of Energy, Water and Environment Systems. International Centre for Sustainable Development of Energy, Water and Environment Systems, 2015.
  23. M. Rodríguez Fernández, A. Cortés García, I. González Alonso, and E. Zalama Casanova, “Using the big data generated by the smart home to improve energy efficiency management,” Energy Efficiency, vol. 9, no. 1, pp. 249–260, Jan 2016. http://dx.doi.org/10.1007/s12053-015-9361-3. [Online]. Available: https://doi.org/10.1007/s12053-015-9361-3
  24. N. Balac, T. Sipes, N. Wolter, K. Nunes, B. Sinkovits, and H. Karimabadi, “Large scale predictive analytics for real-time energy management,” in 2013 IEEE International Conference on Big Data, Oct 2013. http://dx.doi.org/10.1109/BigData.2013.6691635 pp. 657–664.
  25. A. R. Al-Ali, I. A. Zualkernan, M. Rashid, R. Gupta, and M. Alikarar, “A smart home energy management system using iot and big data analytics approach,” IEEE Transactions on Consumer Electronics, vol. 63, no. 4, pp. 426–434, November 2017. http://dx.doi.org/10.1109/TCE.2017.015014
  26. H. Daki, A. El Hannani, A. Aqqal, A. Haidine, and A. Dahbi, “Big data management in smart grid: concepts, requirements and implementation,” Journal of Big Data, vol. 4, 12 2017. http://dx.doi.org/10.1186/s40537-017-0070-y
  27. C. Yang, W. Chen, K. Huang, J. Liu, W. Hsu, and C. Hsu, “Implementation of smart power management and service system on cloud computing,” in 2012 9th International Conference on Ubiquitous Intelligence and Computing and 9th International Conference on Autonomic and Trusted Computing, Sep. 2012. http://dx.doi.org/10.1109/UIC-ATC.2012.160 pp. 924–929.
  28. M. Ali, Z. Y. Dong, X. Li, and P. Zhang, “Rsa-grid: a grid computing based framework for power system reliability and security analysis,” in 2006 IEEE Power Engineering Society General Meeting, June 2006. http://dx.doi.org/10.1109/PES.2006.1709374. ISSN 1932-5517
  29. T. Rajeev and S. Ashok, “A cloud computing approach for power management of microgrids,” in ISGT2011-India, Dec 2011. http://dx.doi.org/10.1109/ISET-India.2011.6145354 pp. 49–52.
  30. S. Chren, B. Rossi, and T. Pitner, “Smart grids deployments within eu projects: The role of smart meters,” in 2016 Smart cities symposium Prague (SCSP). IEEE, 2016. http://dx.doi.org/10.1109/SCSP.2016.7501033 pp. 1–5.
  31. M. Schvarcbacher, K. Hrabovská, B. Rossi, and T. Pitner, “Smart grid testing management platform (sgtmp),” Applied Sciences, vol. 8, no. 11, p. 2278, 2018. http://dx.doi.org/10.3390/app8112278
  32. K. Hrabovská, N. Šimková, B. Rossi, and T. Pitner, “Smart grids and software testing process models,” in 2019 Smart cities symposium Prague (SCSP). IEEE, 2019, pp. 1–5.
  33. J. Peppanen, X. Zhang, S. Grijalva, and M. J. Reno, “Handling bad or missing smart meter data through advanced data imputation,” in IEEE Innovative Smart Grid Technologies Conference. IEEE, 2016. http://dx.doi.org/10.1109/ISGT.2016.7781213 pp. 1–5.
  34. X. Liu, N. Iftikhar, H. Huo, R. Li, and P. S. Nielsen, “Two approaches for synthesizing scalable residential energy consumption data,” Future Generation Computer Systems, vol. 95, pp. 586–600, 2019. http://dx.doi.org/10.1016/j.future.2019.01.045
  35. M. Zaharia, R. S. Xin, P. Wendell, T. Das, M. Armbrust, A. Dave, X. Meng, J. Rosen, S. Venkataraman, M. J. Franklin et al., “Apache spark: a unified engine for big data processing,” Communications of the ACM, vol. 59, no. 11, pp. 56–65, 2016. http://dx.doi.org/10.1145/2934664
  36. P. Carbone, A. Katsifodimos, S. Ewen, V. Markl, S. Haridi, and K. Tzoumas, “Apache flink: Stream and batch processing in a single engine,” Bulletin of the IEEE Computer Society Technical Committee on Data Engineering, vol. 36, no. 4, 2015.
  37. S. T. Allen, M. Jankowski, and P. Pathirana, Storm Applied: Strategies for real-time event processing. Manning Publications Co., 2015.
  38. J. Karimov, T. Rabl, A. Katsifodimos, R. Samarev, H. Heiskanen, and V. Markl, “Benchmarking distributed stream processing engines,” arXiv preprint https://arxiv.org/abs/1802.08496, 2018.
  39. Y. Wang, “Stream processing systems benchmark: Streambench,” G2 Pro gradu, diplomityö, Aalto University, Finland, 2016-06-13. [Online]. Available: http://urn.fi/URN:NBN:fi:aalto-201606172599
  40. M. A. Lopez, A. G. P. Lobato, and O. C. M. Duarte, “A performance comparison of open-source stream processing platforms,” in 2016 IEEE Global Communications Conference (GLOBECOM). IEEE, 2016. http://dx.doi.org/10.1109/GLOCOM.2016.7841533 pp. 1–6.
  41. S. Chintapalli, D. Dagit, B. Evans, R. Farivar, T. Graves, M. Holderbaugh, Z. Liu, K. Nusbaum, K. Patil, B. J. Peng et al., “Benchmarking streaming computation engines: Storm, flink and spark streaming,” in 2016 IEEE international parallel and distributed processing symposium workshops (IPDPSW). IEEE, 2016. http://dx.doi.org/10.1109/IPDPSW.2016.138 pp. 1789–1792.
  42. Smart* data set for sustainability. [Online]. Available: http://traces.cs.umass.edu/index.php/Smart/Smart