What is RoCE(RDMA over Converged Ethernet)？

What is RoCE

RoCE (RDMA over Converged Ethernet) is an RDMA-based technology. Designed for low-latency and high-throughput data transmission over lossless Ethernet. Based on the advantages of Ethernet speed, RDMA technology can achieve high-speed data access between servers with minimal CPU usage. Provides remote memory access with high bandwidth and low latency. It is suitable for new business scenarios such as AI intelligent computing, high-performance computing, and high-speed storage.

The RDMA protocol contains: Infiniband (IB), Internet Wide Area RDMA Protocol (iWARP), and RDMA over Converged Ethernet (RoCE)：

InfiniBand: RDMA was designed with the RDMA in mind. The physical link layer, network layer, and transport layer have been redesigned. From the hardware level, reliable transmission is guaranteed, providing higher bandwidth and lower latency. However, the cost is high, and it needs to support IB NICs and switches.

iWARP: The TCP-based RDMA network uses TCP to achieve reliable transmission. Compared with RoCE, in the case of large-scale networking, a large number of TCP connections in iWARP will occupy a large amount of memory resources and require higher system specifications. You can use a common Ethernet switch, but you need an iWARP-capable network card.

RoCE: Based on RDMA over Ethernet, RoCEv1 is based on the network link layer and cannot cross network segments, so there is basically no application.

RoCEv2 is based on UDP and can have good scalability across network segments. Moreover, it can achieve throughput and relatively good latency. So it’s a program that has been adopted on a large scale.

RoCE consumes fewer resources than iWARP and supports more features than iWARP. You can use a common Ethernet switch, but you need a RoCE-capable NIC card.

Types of RoCE

RoCE innovatively selects Ethernet as the network carrier to carry RDMA, and has launched two versions:

RoCEv1 works at the link layer and is limited by the broadcast domain and does not support cross-domain communication. RoCEv2, on the other hand, is a network-layer protocol with routing capabilities and breaks geographical restrictions.

RoCEv1

In April 2010, IBTA released RoCE, which is also known as IBoE (InfiniBand over Ethernet) as an add-on to the Infiniband Architecture Specification. At this time, the RoCE standard replaces the TCP/IP network layer with the IB network layer on top of the Ethernet link layer, so it does not support the IP routing function. The typeID of the RoCE V1 protocol at the Ethernet layer is 0x8915.

RoCE v1 (Layer 2) runs on the Ehternet Link Layer (Layer 2), so the typeID of the RoCE v1 protocol at the Ethernet layer is 0x8915. RoCE v1 is an RDMA protocol implemented at the Ethernet link layer (switches need to support flow control technologies such as PFC to ensure reliable transmission at the physical layer), allowing two hosts in the same VLAN to communicate.

RoCEv2

Since the data frames of RoCEv1 do not have IP headers, they can only communicate within the L2 subnet. In order to solve this problem, IBTA proposed RoCE V2 in 2014, which extends RoCEv1 and replaces GRH (Global Routing Header) with UDP header + IP header, and the extended frame structure is shown in the following figure.

For RoCE v1 and RoCE v2, the following two points are worth noting:

RoCE v1 (Layer 2) runs on the Ehternet Link Layer (Layer 2) so the Ethertype is 0x8915, so the normal frame size is 1500 bytes, while the Jumbo Frame is 9000 bytes.

RoCE v2 (Layer 3) runs on UDP/IPv4 or UDP/IPv6 (Layer 3) and uses UDP Port 4791 for transmission. Because RoCE v2 packets are routeable on Layer 3, they are sometimes referred to as Routable RoCE or RRoCE for short.

Advantages of RoCE:

High performance: RoCE provides remote memory access with high bandwidth and low latency. It is suitable for scenarios such as high-performance computing, AI intelligent computing, and high-speed storage.

RDMA support: RoCE is a protocol that supports RDMA. It allows data to be transferred directly between memories without involving the host’s CPU. This reduces the processing overhead of data transfer and increases efficiency.

Protocol stack: RoCE uses the UDP/IP protocol stack, while traditional networks use the TCP/IP protocol stack. Since RoCE runs directly on Ethernet, it avoids some of the overhead of the TCP/IP protocol and improves performance.

Open ecosystem: RoCE is based on a mature Ethernet technology system. There are many supporting vendors in the industry, and no dedicated hardware is required. It can be deployed based on multi-vendor open hardware NICs/switches.

Low cost: After years of technology development and accumulation, the unit bandwidth cost of Ethernet high-end chips is more competitive, and the overall cost of switches is lower.

Applicable scenario: Typically used for communication between high-performance computing, storage, and networking devices within large-scale data centers that are required. Traditional networks are more suitable for general enterprise networks and the Internet.

Application scenarios of RoCE

The efficient data transmission characteristics of RoCE have made it widely used in several fields, especially in large-scale data processing and computationally intensive tasks.

Data Centers

Modern data centers are faced with the storage and processing requirements of massive amounts of data, and RoCE can effectively improve the communication efficiency between data storage and computing nodes by providing low-latency and high-bandwidth network communication solutions. This is especially important for virtualized environments, distributed databases, and containerized applications.

Storage Area Network (SAN)

RoCE plays a key role in storage area networks (SANs) by providing efficient data transmission and supporting fast data reads and writes for large-scale distributed storage systems. Especially in big data and cloud storage platforms, the performance advantages of RoCE make data management and backup more efficient.

High Performance Computing (HPC)

In high-performance computing fields such as scientific research, climate simulation, and genetic analysis, RoCE’s low latency and high throughput provide strong support for data exchange between multiple computing nodes. RoCE helps accelerate the completion of high-performance computing tasks and shorten the return time of calculation results.

Artificial Intelligence and Machine Learning

With the increasing requirements for computing performance in artificial intelligence and machine learning, RoCE plays an important role in large-scale distributed training. It can quickly transfer data between multiple servers, greatly improving the efficiency of deep learning model training and reducing data transmission bottlenecks.

Conclusion

As an Ethernet-based RDMA technology, RoCE plays an important role in data centers, high-performance computing, storage networks, and other fields due to its advantages of low latency, high bandwidth, and reduced CPU burden. As technology continues to evolve, RoCE will continue to be a central force driving innovation in modern computing and networking.