Increasing data availability and fault tolerance for decentralized collaborative data-sharing systems
Kamil Jarosz, Łukasz Opioła, Łukasz Dutka, Renata G. Słota, Jacek Kitowski
Citation: Proceedings of the 17th Conference on Computer Science and Intelligence Systems, M. Ganzha, L. Maciaszek, M. Paprzycki, D. Ślęzak (eds). ACSIS, Vol. 30, pages 563–566 (2022)
Abstract. In order to realize collaboration on a global scale, academic research requires often large quantities of data to be shared between geographically dispersed organizations. The requirement to protect and govern data in a network of loosely coupled, autonomous institutions is an incentive for decentralized solutions, where the participants are in full control of their data without trusting a third-party provider to store and process the data. In order to increase data availability and fault tolerance in decentralized collaborative systems, we propose a layer, which is based on replication and decentralized authority over the data. The solution consists of an idea of peer-sets, which are groups of peers implementing collective data management, a consensus protocol which synchronizes a distributed ledger between peers, and an atomic commitment protocol used to implement optional two-way references between documents. This architecture may be utilized in various decentralized collaborative data-sharing systems, such as Onedata.
- D. Cesini et al., “The eXtreme-DataCloud project solutions for data management services in distributed e-infrastructures,” EPJ Web of Conferences, vol. 245, p. 04010, 01 2020. http://dx.doi.org/10.1051/epj-conf/202024504010
- ScienceMesh. [Online]. Available: https://sciencemesh.io/
- M. Wrzeszcz, Ł. Dutka, R. G. Słota, and J. Kitowski, “New approach to global data access in computational infrastructures,” Future Generation Computer Systems, vol. 125, pp. 575–589, 2021. http://dx.doi.org/10.1016/j.future.2021.06.054
- L. Opioła, L. Dutka, R. G. Słota, and J. Kitowski, “Trust-driven, decentralized data access control for open network of autonomous data providers,” in 2018 16th Annual Conference on Privacy, Security and Trust (PST), 2018. http://dx.doi.org/10.1109/PST.2018.8514209 pp. 1–10.
- Arora, Ishank, Alfageme Sainz, Samuel, Ferreira, Pedro, Gonzalez Labrador, Hugo, and Moscicki, Jakub, “Enabling interoperable data and application services in a federated sciencemesh,” EPJ Web Conf., vol. 251, p. 02041, 2021. http://dx.doi.org/10.1051/epjconf/202125102041
- T. T. A. Dinh, R. Liu, M. Zhang, G. Chen, B. C. Ooi, and J. Wang, “Untangling blockchain: A data processing view of blockchain systems,” IEEE Transactions on Knowledge and Data Engineering, vol. 30, no. 7, pp. 1366–1385, 2018. http://dx.doi.org/10.1109/TKDE.2017.2781227
- J. Benet, “IPFS - content addressed, versioned, P2P file system,” arXiv preprint https://arxiv.org/abs/1407.3561, 2014. http://dx.doi.org/10.48550/arXiv.1407.3561
- D. Ulybyshev, M. Villarreal-Vasquez, B. Bhargava, G. Mani, S. Seaberg, P. Conoval, R. Pike, and J. Kobes, “(WIP) Blockhub: Blockchain-based software development system for untrusted environments,” in 2018 IEEE 11th International Conference on Cloud Computing (CLOUD), 2018. http://dx.doi.org/10.1109/CLOUD.2018.00081 pp. 582–585.
- M. Pease, R. Shostak, and L. Lamport, “Reaching agreement in the presence of faults,” J. ACM, vol. 27, no. 2, p. 228–234, apr 1980. http://dx.doi.org/10.1145/322186.322188
- L. Lamport, “Paxos made simple,” ACM SIGACT News (Distributed Computing Column) 32, 4 (Whole Number 121, December 2001), pp. 51–58, 2001.
- I. Moraru, D. G. Andersen, and M. Kaminsky, “There is more consensus in egalitarian parliaments,” in Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles, ser. SOSP ’13. New York, NY, USA: Association for Computing Machinery, 2013. http://dx.doi.org/10.1145/2517349.2517350. ISBN 9781450323888 p. 358–372.
- D. Ongaro and J. Ousterhout, “In search of an understandable consensus algorithm,” in Proceedings of the 2014 USENIX Conference on USENIX Annual Technical Conference, ser. USENIX ATC’14. USA: USENIX Association, 2014. ISBN 9781931971102 p. 305–320.
- F. P. Junqueira, B. C. Reed, and M. Serafini, “Zab: High-performance broadcast for primary-backup systems,” in 2011 IEEE/IFIP 41st International Conference on Dependable Systems Networks (DSN), 2011. http://dx.doi.org/10.1109/DSN.2011.5958223 pp. 245–256.
- P. R. Nair and D. R. Dorai, “Evaluation of performance and security of proof of work and proof of stake using blockchain,” in 2021 Third International Conference on Intelligent Communication Tech nologies and Virtual Mobile Networks (ICICV), 2021. http://dx.doi.org/10.1109/ICICV50876.2021.9388487 pp. 279–283.
- M. Castro and B. Liskov, “Practical byzantine fault tolerance,” in Proceedings of the Third Symposium on Operating Systems Design and Implementation, ser. OSDI ’99. USA: USENIX Association, 1999. ISBN 1880446391 p. 173–186.
- Amazon Quantum Ledger Database (QLDB). [Online]. Available: https://aws.amazon.com/qldb/
- The BitTorrent protocol specification. [Online]. Available: https://www.bittorrent.org/beps/bep_0003.html
- Git. [Online]. Available: https://git-scm.com/
- Handle.Net Registry. [Online]. Available: https://www.handle.net/
- World Wide Web Consortium. Decentralized identifiers (DIDs) v1.0. [Online]. Available: https://www.w3.org/TR/did-core/
- S. Maiyya, F. Nawab, D. Agrawal, and A. E. Abbadi, “Unifying consensus and atomic commitment for effective cloud data management,” Proc. VLDB Endow., vol. 12, no. 5, p. 611–623, jan 2019. http://dx.doi.org/10.14778/3303753.3303765
- P. Fan, J. Liu, W. Yin, H. Wang, X. Chen, and H. Sun, “2PC*: A distributed transaction concurrency control protocol of multi-microservice based on cloud computing platform,” J. Cloud Comput., vol. 9, no. 1, jul 2020. http://dx.doi.org/10.1186/s13677-020-00183-w