Design Guidelines for Correct, Efficient, and Scalable Synchronization using One-Sided RDMA

Tobias Ziegler; Jacob Nelson-Slivon; Viktor Leis; Carsten Binnig

doi:10.1145/3589276

journal article Jun 13, 2023

Design Guidelines for Correct, Efficient, and Scalable Synchronization using One-Sided RDMA

Tobias Ziegler

Jacob Nelson-Slivon

Viktor Leis

Carsten Binnig

Proceedings of the ACM on Management of Data Vol. 1 No. 2 pp. 1-26 · Association for Computing Machinery (ACM)

View at Publisher Save 10.1145/3589276

Abstract

Remote data structures built with one-sided Remote Direct Memory Access (RDMA) are at the heart of many disaggregated database management systems today. Concurrent access to these data structures by thousands of remote workers necessitates a highly efficient synchronization scheme. Remarkably, our investigation reveals that existing synchronization schemes display substantial variations in performance and scalability. Even worse, some schemes do not correctly synchronize, resulting in rare and hard-to-detect data corruption. Motivated by these observations, we conduct the first comprehensive analysis of one-sided synchronization techniques and provide general principles for correct synchronization using one-sided RDMA. Our research demonstrates that adherence to these principles not only guarantees correctness but also results in substantial performance enhancements.

Topics

No keywords indexed for this article. Browse by subject →

References

60

[1]

ARM. 2018. Arm CoreLink CCI-550 Cache Coherent Interconnect Technical Reference Manual. https://developer.arm. com/documentation/100282/0100/?lang=en. https://developer.arm.com/documentation/100282/0100/?lang=en

[2]

ARM. 2021. Introducing the AMBA Coherent Hub Interface. https://developer.arm.com/documentation/102407/0100

[3]

10.1145/3448016.3457560

[4]

10.1109/tpds.2017.2729545

[5]

Yeounoh Chung and Erfan Zamanian. 2015. Using RDMA for Lock Management. CoRR abs/1507.03274 (2015). arXiv:1507.03274 http://arxiv.org/abs/1507.03274

[6]

NVIDIA Coporation. 2021. NVIDIA InfiniBand Adaptive Routing Technology. Whitepaper WP-10326-001_v01.

[7]

10.1145/2983990.2984033

[8]

Aleksandar Dragojevic Dushyanth Narayanan Miguel Castro and Orion Hodson. 2014. FaRM: Fast Remote Memory. In NSDI.

[9]

Aleksandar Dragojevic Dushyanth Narayanan Edmund B. Nightingale Matthew Renzelmann Alex Shamis Anirudh Badam and Miguel Castro. 2015. No compromises: distributed transactions with consistency availability and performance. In SOSP. 10.1145/2815400.2815425

[10]

10.1145/3587096

[11]

Philipp Fent Alexander van Renen Andreas Kipf Viktor Leis Thomas Neumann and Alfons Kemper. 2020. Low- Latency Communication for Fast DBMS Using RDMA and Shared Memory. In ICDE. 10.1109/icde48307.2020.00131

[12]

Torsten Hoefler Duncan Roweth Keith Underwood Bob Alverson Mark Griswold Vahid Tabatabaee Mohan Kalkunte Surendra Anubolu Siyan Shen Abdul Kabbani Moray McLaren and Steve Scott. 2023. Datacenter Ethernet and RDMA: Issues at Hyperscale. arXiv:2302.03337 [cs.NI]

[13]

10.1007/s11704-

[14]

10.1109/isca.2005.23

[15]

InfiniBand Trade Association 2007. InfiniBand Architecture Specification Volume 1. InfiniBand Trade Association. Release 1.2.1.

[16]

InfiniBand Trade Association. 2010. RDMA Over Converged Ethernet (RoCE). https://cw.infinibandta.org/document/ dl/7148.

[17]

Intel. 2012. Intel Data Direct I/O Technology (Intel DDIO): A P rimer. https://www.intel.com/content/dam/www/ public/us/en/documents/technology-briefs/data-direct-i-o-technology-brief.pdf

[18]

Anuj Kalia, Michael Kaminsky, and David G. Andersen. 2014. Using RDMA efficiently for key-value services. In SIGCOMM.

[19]

Anuj Kalia, Michael Kaminsky, and David G. Andersen. 2016. Design Guidelines for High Performance RDMA Systems. login Usenix Mag. 41, 3 (2016).

[20]

Anuj Kalia, Michael Kaminsky, and David G. Andersen. 2016. FaSST: Fast, Scalable and Simple Distributed Transactions with Two-Sided (RDMA) Datagram RPCs. In OSDI.

[21]

Tejas Karmarkar. 2015. Availability of Linux RDMA on Microsoft Azure. Online. https://azure.microsoft.com/enus/ blog/azure-linux-rdma-hpc-available/

[22]

Dario Korolija, Dimitrios Koutsoukos, Kimberly Keeton, Konstantin Taranov, Dejan S. Milojicic, and Gustavo Alonso. 2022. Farview: Disaggregated Memory with Operator Off-loading for Database Engines. In 12th Conference on Innovative Data Systems Research, CIDR 2022 , Chaminade, CA, USA, January 9--12, 2022. www.cidrdb.org. https://www.cidrdb.org/cidr2022/papers/p11-korolija.pdf

[23]

Viktor Leis, Michael Haubenschild, and Thomas Neumann. 2019. Optimistic Lock Coupling: A Scalable and Efficient General-Purpose Synchronization Method. IEEE Data Eng. Bull. 42 (2019), 73--84.

[24]

10.1145/2933349.2933352

[25]

10.1109/hoti.2007.8

[26]

Jiuxing Liu, Jiesheng Wu, Sushmitha P. Kini, Pete Wyckoff, and Dhabaleswar K. Panda. 2003. High performance RDMA-based MPI implementation over InfiniBand. In ICS.

[27]

Simon Loesing Markus Pilman Thomas Etter and Donald Kossmann. 2015. On the Design and Scalability of Distributed Shared-Data Databases. In SIGMOD. 10.1145/2723372.2751519

[28]

10.1109/cluster48925.2021.00033

[29]

10.1109/compsac51774.2021.00021

[30]

Christopher Mitchell, Yifeng Geng, and Jinyang Li. 2013. Using One-Sided RDMA Reads to Build a Fast, CPU-Efficient Key-Value Store. In 2013 USENIX Annual Technical Conference, San Jose, CA, USA, June 26--28, 2013, Andrew Birrell and Emin Gün Sirer (Eds.). USENIX Association, 103--114. https://www.usenix.org/conference/atc13/technicalsessions/ presentation/mitchell

[31]

Christopher Mitchell Yifeng Geng and Jinyang Li. 2013. Using One-Sided RDMA Reads to Build a Fast CPU-Efficient Key-Value Store. In USENIX ATC.

[32]

Christopher Mitchell Kate Montgomery Lamont Nelson Siddhartha Sen and Jinyang Li. 2016. Balancing CPU and Network in the Cell Distributed B-Tree Store. In USENIX ATC.

[33]

10.1109/ccgrid.2007.58

[34]

Jacob Nelson and Roberto Palmieri. 2020. Performance Evaluation of the Impact of NUMA on One-sided RDMA Interactions. In SRDS. 10.1109/srds51746.2020.00036

[35]

PCI-SIG. 2014. PCI Express Base Specification Revision 4.0. (2014).

[36]

R. Recio B. Metzler P. Culley J. Hilland and D. Garcia. 2007. A Remote Direct Memory Access Protocol Specification. Technical Report. https://doi.org/10.17487/rfc5040 10.17487/rfc5040

[37]

Yufei Ren, Tan Li, Dantong Yu, Shudong Jin, and Thomas G. Robertazzi. 2013. Design and performance evaluation of NUMA-aware RDMA-based end-to-end data transfer systems. In HiPC.

[38]

Zhenyuan Ruan, Malte Schwarzkopf, Marcos K. Aguilera, and Adam Belay. 2020. AIFM: High-Performance, Application- Integrated Far Memory. In 14th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2020, Virtual Event, November 4--6, 2020. USENIX Association, 315--332. https://www.usenix.org/conference/osdi20/presentation/ruan

[39]

H. Shah F. Marti W. Noureddine A. Eiriksson and R. Sharp. 2014. Remote Direct Memory Access (RDMA) Protocol Extensions. Technical Report. https://doi.org/10.17487/rfc7306 10.17487/rfc7306

[40]

10.1145/3452296.3472934

[41]

10.1109/hpca.2010.5416638

[42]

Konstantin Taranov Fabian Fischer and Torsten Hoefler. 2022. Efficient RDMA Communication Protocols. arXiv:2212.09134 [cs.NI]

[43]

10.1145/3448016.3452817

[44]

Shin-Yeh Tsai, Yizhou Shan, and Yiying Zhang. 2020. Disaggregating Persistent Memory and Controlling Them Remotely: An Exploration of Passive Disaggregated Key-Value Stores. In 2020 USENIX Annual Technical Conference, USENIX ATC 2020, July 15--17, 2020, Ada Gavrilovska and Erez Zadok (Eds.). USENIX Association, 33--48. https://www.usenix.org/conference/atc20/presentation/tsai

[45]

10.1109/tcc.2021.3116516

[46]

10.1145/3514221.3517824

[47]

10.48550/arxiv.2207.03027

[48]

Tinggang Wang, Shuo Yang, Hideaki Kimura, Garret Swart, and Spyros Blanas. 2020. Efficient Usage of One-Sided RDMA for Linear Probing. In International Workshop on Accelerating Analytics and Data Management Systems Using Modern Processor and Storage Architectures, ADMS@VLDB 2020, Tokyo, Japan, August 31, 2020, Rajesh Bordawekar and Tirthankar Lahiri (Eds.). 1--13. http://www.adms-conf.org/2020-camera-ready/ADMS20_06.pdf

[49]

10.1145/2807591.2807614

[50]

10.1145/3468520

Showing 50 of 60 references

Cited By

28

From Scale-Up to Scale-Out: PolarDB's Journey to Achieving 2 Billion tpmC

Xinjun Yang, Feifei Li · 2025

Proceedings of the VLDB Endowment

Metrics

28

Citations

60

References

Details

Published: Jun 13, 2023
Vol/Issue: 1(2)
Pages: 1-26
License: View

Authors

T

Tobias Ziegler

Technische Universität Darmstadt, Darmstadt, Germany

J

Jacob Nelson-Slivon

Lehigh University, Bethlehem, Germany

V

Viktor Leis

Technische Universität München, München, Germany

C

Carsten Binnig

Technische Universität Darmstadt, Darmstadt, Germany

Funding

Deutsche Forschungsgemeinschaft Award: BI2011/1,BI2011/2,SFB1053

Cite This Article

Tobias Ziegler, Jacob Nelson-Slivon, Viktor Leis, et al. (2023). Design Guidelines for Correct, Efficient, and Scalable Synchronization using One-Sided RDMA. Proceedings of the ACM on Management of Data, 1(2), 1-26. https://doi.org/10.1145/3589276

Design Guidelines for Correct, Efficient, and Scalable Synchronization using One-Sided RDMA

You May Also Like