What is InfiniBand?

In recent years, the rise of large-scale AI models represented by ChatGPT has further increased the attention of InfiniBand. For those interested in cutting-edge networking solutions, it’s crucial to understand its technical principles, benefits, products, and prospects. This article will cover what InfiniBand is and how it works.

InfiniBand:

InfiniBand(IB) is a high-performance, low-latency computer network architecture. It is designed to meet the needs of large-scale, high-bandwidth, low-latency applications. Originally launched in 1999 by the InfiniBand Trade Association (IBTA). The aim is to provide a more efficient connectivity solution than traditional network protocols such as Ethernet. Especially in high-performance computing (HPC), data centers, large-scale storage systems, and hyperscale distributed computing environments.

InfiniBand Technical Principle

Introducing the introduction of InfiniBand. Next, let’s take a look at how it works, and why it will be stronger than traditional Ethernet. How exactly is its low latency and high performance achieved?

RDMA: Basic Functions

One of the most prominent advantages of InfiniBand is that it was the first to introduce the RDMA (Remote Direct Memory Access)  protocol.

In traditional TCP/IP, data from the NIC is copied to core memory and then to application storage. Or copy data from the application space to the core memory, and then send it to the Internet through the network card.

This type of I/O operation needs to go through the core memory conversion. It increases the length of the data stream path, the burden on the CPU, and the transmission latency.

However, RDMA allows data to be transferred directly between application memory spaces without CPU involvement. This feature significantly reduces latency and improves overall system efficiency.

InfiniBand Networks and Its Architecture

InfiniBand is a channel-based architecture with four main types of components:

HCA (Host Channel Adapter):

HCA is the host interface in the Infiniband network. It is the hardware interface between the computer (host) and the Infiniband network. HCA provides an efficient communication interface and is responsible for data transfer between the host and the network.

TCA (Target Channel Adapter):

A TCA is a network interface used to connect to a storage device or other target device. It is an interface for data interaction with external devices of a computer system.

InfiniBand Link:

Cables and connectors in an Infiniband network are used to physically connect individual network devices. There are two common connection methods: optical fiber and cable. Fiber optics are suitable for long-distance transmission, while cables are used for short-distance connections.

InfiniBand Switches: 

Switches in the Infiniband network are responsible for routing and forwarding data between devices in the network. Switches function in a network similar to traditional Ethernet switches, supporting multipoint-to-multipoint communication.

InfiniBand Protocol Stack

InfiniBand’s protocol stack consists of multiple layers. It includes the physical layer, link layer, network layer, transport layer, and application layer. Each layer is responsible for different functions to ensure the efficiency and reliability of network communication. As shown in the figure below:

Advantages of InfiniBand

High Bandwidth

InfiniBand offers very high bandwidth, typically higher than traditional Ethernet. For example, InfiniBand EDR (Enhanced Data Rate) provides data transfer rates of up to 100 Gbps, and HDR (High Data Rate) versions can reach 200 Gbps or even higher. As technology advances, so does InfiniBand’s data rate. Far exceeds most Ethernet and Fibre Channel standards.

Low Latency

The InfiniBand network is designed with a strong focus on reducing latency. Especially in high-performance computing (HPC), low latency is critical. Latency is typically less than 1 microsecond, which is much lower than most legacy networking technologies such as Ethernet. Low latency gives it advantages in parallel computing, data storage, and processing.

Scalability

InfiniBand supports massive scaling and is capable of connecting thousands of devices. It uses switches and routers to build a hierarchical network architecture. Supports connections from a few to thousands of devices for large-scale data centers and supercomputers.

Low Power Consumption

InfiniBand offers lower power consumption than some traditional networking technologies. This is especially true for scenarios with high bandwidth and low latency. It can reduce energy consumption while maintaining performance.

High Reliability and Availability

The InfiniBand network is redundant and supports multipath redundancy and failback mechanisms. The failure of any one link in the network usually does not result in an overall system downtime. It automatically adjusts to an alternate path, ensuring continuous data transfer. This provides high reliability and high availability.

MVSLINK: InfiniBnad Series Products:

Conclusion

InfiniBand is a networking technology designed to meet the needs of high-performance computing, large-scale data transmission, and low-latency applications. The high bandwidth, low latency, and RDMA capabilities it offers make it ideal for supercomputers, AI training, big data analytics, and finance.