RDMA, InfiniBand and RoCE¶
This note explains the network technology underneath an AI or HPC cluster for a reader who works in computing but not in interconnect design. It introduces remote direct memory access, then the two fabrics that carry it, and it ends at the point where the two fabrics genuinely differ.
examples/rdma/msgstream.c is the worked example referred to throughout.
It is a single program that runs unmodified on both fabrics.
1. The problem RDMA solves¶
A conventional network transfer between two machines copies the payload several times. The sending application writes into a buffer, the kernel copies that buffer into socket memory, the network stack builds packet headers around it, and the network card reads the result. The receiver performs the same sequence in reverse. Each copy costs memory bandwidth and each transition between application and kernel costs a context switch.
For a web server this is unimportant, because the network is slow compared with the copies. For a machine learning cluster it is the dominant cost. Training a large model requires every accelerator to exchange gradients with every other accelerator after each step. The accelerators are idle while that exchange runs, and the exchange is large, so the cluster spends a significant fraction of its life waiting on the network. The copies and the context switches are pure overhead in that wait.
Remote direct memory access removes them. The network adapter reads directly from the sending application's memory and writes directly into the receiving application's memory. The kernel is not on the data path at all. It sets up the permissions once, at connection time, and is then out of the way.
Two consequences follow, and they matter more than the absence of copying.
The first is that the application talks to the adapter directly. It posts a work request onto a queue that the adapter reads, and later collects a completion from a queue the adapter writes. No system call is involved in either step.
The second is that the remote CPU need not participate. An adapter can be told to write into a peer's memory without the peer's software being involved or even aware. This is what makes collective operations across thousands of accelerators tractable.
2. Queue pairs, the unit of connection¶
An RDMA connection is a queue pair, which is a send queue and a receive queue owned by one process and bound to a queue pair on the far side.
Establishing one requires each side to learn how to reach the other, and
that information cannot be obtained over RDMA because no connection exists
yet. Every RDMA program therefore begins with an exchange over some
ordinary channel. msgstream uses a TCP socket for this and nothing else.
Production systems use a connection manager, which is the same idea with
more machinery.
The queue pair is then walked through three states.
INIT grants the adapter access to a region of the application's memory
and sets the port and partition it will use.
RTR, ready to receive, is where the local queue pair is told the address
of the remote one. This is the only place in the program where the two
fabrics are treated differently, and section 4 returns to it.
RTS, ready to send, completes the transition, and from that point the
application posts work requests and reaps completions.
3. Two-sided and one-sided operations¶
RDMA offers two families of operation and the difference between them is the more important of the two ideas in this note.
A two-sided operation is a send matched by a receive. The receiver must have posted a receive work request, nominating a buffer, before the send arrives. Both sides see a completion. This is a message passing model and it resembles sockets, minus the copying. It is the right choice when the receiver does not know in advance what it is about to be told.
A one-sided operation is an RDMA read or an RDMA write. The initiator names a remote address and a key authorising access to it, and the remote adapter performs the access. The remote application posts nothing, sees no completion, and is not scheduled. The data appears in its memory.
For this to be possible the target must first have published two values,
the virtual address of its buffer and a remote key issued by its adapter
when the memory was registered. msgstream carries both in the same TCP
bootstrap that carries the queue pair address. The remote key is what
makes the operation safe. Without it an adapter will refuse the access, so
a peer can only reach memory that was deliberately offered to it.
One-sided operations are why RDMA scales. A machine that has published a buffer imposes no per-transfer cost on itself when its peers use it.
4. Where InfiniBand and RoCE differ¶
Both fabrics carry the same verbs interface, and an application written
against that interface is portable between them. msgstream demonstrates
this literally, since the same binary runs on both. What differs is how a
peer is named and what supplies the properties RDMA depends on.
4.1 InfiniBand¶
InfiniBand is a purpose-built fabric, defined from the physical layer up for this workload. It is not Ethernet and it does not carry IP.
Addressing is by local identifier, a 16 bit LID, and a port does not
have one when it starts. A LID is assigned by a subnet manager, a
process that discovers the topology, allocates identifiers, programs the
forwarding tables in every switch, and continues to monitor the fabric.
Until a subnet manager runs, a cabled and electrically healthy InfiniBand
port sits in the INIT state with lid 0 and cannot pass traffic.
This is easy to observe. ibstat on a fabric with no subnet manager
reports State: Initializing and Base lid: 0. Starting opensm moves
the same ports to State: Active with a LID, and neither the cable nor
the adapter changed.
InfiniBand also provides credit-based flow control in hardware. A sender may not transmit unless the receiver has advertised buffer space for it. Congestion therefore does not produce loss, because a packet is never sent to a place that cannot hold it.
4.2 RoCE¶
RDMA over Converged Ethernet carries the same operations across an ordinary Ethernet network. In its second version, the one in current use, an RDMA packet is placed inside a UDP datagram addressed to port 4791.
Addressing is by global identifier, a 128 bit GID derived from the port's IP configuration. There is no subnet manager, because Ethernet already has ARP and IP routing. A RoCE port is reachable as soon as the network beneath it works.
The property that InfiniBand gets from credit flow control has to be recreated, because Ethernet drops packets when congested and RDMA performs poorly on a lossy path. It is supplied by priority flow control, in which a switch tells its upstream neighbour to pause one traffic class, and by explicit congestion notification, in which a switch marks packets as congestion approaches so senders slow down before loss occurs. Both must be configured consistently on every switch in the path. A RoCE fabric with this configuration wrong will work in testing and collapse under load, which is the usual failure mode in practice.
4.3 The difference stated plainly¶
InfiniBand provides losslessness and fabric management as properties of the technology. RoCE provides them as configuration of a general purpose network. The trade is that InfiniBand needs its own switches, cables and management, while RoCE runs on the Ethernet an organisation already operates and already knows how to run.
The following table is the same contrast in short form.
| InfiniBand | RoCEv2 | |
|---|---|---|
| Address | LID, assigned by a subnet manager | GID, derived from IP |
| Fabric management | subnet manager, required | ARP and IP routing |
| Encapsulation | native, no IP | UDP port 4791 |
| Losslessness | credit flow control in hardware | PFC and ECN, configured |
| Switches | InfiniBand only | Ethernet |
| Verbs interface | identical | identical |
4.4 The difference in the source¶
In msgstream.c the entire divergence is one branch in qp_to_rtr. On
Ethernet the address handle is marked global and carries the remote GID.
On InfiniBand it is not global and carries the remote LID.
if (link_layer == IBV_LINK_LAYER_ETHERNET) {
attr.ah_attr.is_global = 1;
attr.ah_attr.grh.dgid = remote->gid;
attr.ah_attr.grh.sgid_index = gid_index;
attr.ah_attr.grh.hop_limit = 255;
} else {
attr.ah_attr.is_global = 0;
attr.ah_attr.dlid = remote->lid;
}
Everything else, the memory registration, the work requests, the completion handling and the one-sided write, is shared.
5. Observing the difference¶
RoCE traffic is IP traffic, so ordinary tools see it.
tcpdump -i eno1 -nn 'udp port 4791'
192.168.1.49.49552 > 192.168.1.52.4791: UDP, length 128
192.168.1.52.49553 > 192.168.1.49.4791: UDP, length 20
The 128 byte datagrams are the messages and the 20 byte datagrams are acknowledgements. A packet capture, a firewall rule and a routing table all apply, because this is Ethernet carrying UDP.
The same capture on InfiniBand yields nothing, and not because the traffic is hidden. There is no IP layer to capture. InfiniBand diagnostics work on the fabric's own terms instead.
ibstat adapter and port state, LID, rate
ibnetdiscover the topology as the subnet manager sees it
perfquery port counters
That asymmetry is the practical consequence of the architectural choice. On RoCE the operator brings existing Ethernet knowledge and existing Ethernet tools. On InfiniBand the operator learns a second set of both, and receives a fabric that does not need to be configured into losslessness.
6. Where Cnuas fits¶
Cnuas emulates this equipment in software. A developer can bring up an InfiniBand fabric, a RoCE fabric, or a hybrid of the two, with no interconnect hardware present, and run the same verbs applications against it. The rack around that fabric is emulated to the same standard, with Open Rack v3 power shelves reporting over Modbus RTU on an RS-485 segment to a BMC running a genuine OpenBMC port.
The purpose is access. The equipment described in this note is expensive and is concentrated in a small number of organisations, which places the systems layer of AI infrastructure out of reach of most people who would otherwise study it.
7. Further reading¶
- InfiniBand Architecture Specification, InfiniBand Trade Association
- Annex A17, RoCEv2, InfiniBand Trade Association
- IEEE 802.1Qbb, Priority-based Flow Control
- RFC 3168, Explicit Congestion Notification
rdma-core, the userspace verbs libraries and diagnostics