HiCAIN System Topology¶
1. Overview¶
The Cnuas Virtual AI HPC Infrastructure is a layered, 100% software-emulated data center stack. Five components cooperate at three planes:
| Plane | Component | Role |
|---|---|---|
| Host userspace daemons | hicain-vswitchd (HiSwitch) |
RoCE/IB fabric switch |
higpu-link-switchd (HiLink) |
GPU-to-GPU interconnect (NVSwitch-equivalent) | |
| QEMU device models | hicain_vnic |
Virtual RoCE-IB NIC PCIe device |
higpu |
Virtual GPU PCIe device | |
| Guest | Kernel modules + libhicain-rdmav34.so provider + libhicart.so runtime |
Verbs and runtime exposed to applications |
Two independent fabrics carry distinct kinds of traffic:
- The HiSwitch fabric carries every Ethernet frame including
RoCEv2 (UDP/4791) RDMA traffic. This is the data network and
is what
ibv_rc_pingpong/perftest/ NCCL's IB transport flow over. - The HiLink fabric is independent. It carries only HiGPU ↔ HiGPU peer traffic, the equivalent of NVIDIA NVLink. A VM with a HiGPU but no HiNIC can still do GPU-to-GPU; a VM with HiNIC but no HiGPU never touches HiLink.
A real VM typically has both
(-device hicain-vnic ... -device higpu ...) and therefore wires
into both daemons via separate UNIX-domain sockets.
2. Physical Rack Reference (OCP Open Rack v3)¶
HiCAIN's physical-equivalent model is the Open Compute Project Open Rack v3 (ORv3), the 2023+ revision used by every major hyperscaler for AI/HPC builds (Meta, Microsoft Azure Maia, Google's TPU pods, NVIDIA HGX/B200 reference designs). ORv3 is not just a deeper EIA-310 rack; it's a different rack with different mechanical, electrical, and serviceability properties:
| ORv3 property | Value | Why it matters for HiCAIN |
|---|---|---|
| Rack-unit height (OU) | 48 mm (vs 44.5 mm for 19″ U) | All our rack diagrams are sized in OU |
| Equipment bay width | 21″ (537 mm), 20 % wider than 19″ | Gives room for 8x GPU sleds + dual-NIC headers |
| Power | 48 V DC busbars at the rear, fed by a power shelf | Hot-pluggable PSUs; no per-server AC inlets |
| Servicing | Front-only; cables and connectors on the cold aisle | Switch + GPU + compute all share the cold-aisle service model |
| Modular zones | power / equipment / cable | Maps 1:1 to our TOR-row / blade-rows / cable-bay layout |
Why ORv3 and not ORv2? ORv2 (2014) targeted general-purpose cloud workloads; ORv3 (2023) is the rack the AI buildout was designed around, wider 21″ bays, denser busbars, and explicit support for top-of-rack accelerator-optimised switches like the ones HiSwitch + HiLink emulate. An ORv2-style 19″ rack would not fit the dual-fabric (RoCE TOR + GPU peer switch) header we ship one rack-unit apart in every HiCAIN rack.
2.1 Component-to-ORv3 Slot Mapping¶
| ORv3 slot (top-down) | HiCAIN component | OU height | Function |
|---|---|---|---|
| 1OU | hicain-vswitchd (HiSwitch TOR) |
1 | RoCEv2 + InfiniBand fabric, 10 ports, DCB |
| 1OU | higpu-link-switchd (HiLink spine) |
1 | GPU peer fabric, 8 ports, NVSwitch-equivalent |
| 1OU | Management host | 1 | Bare-metal Ubuntu; runs the two daemons + real GPU |
| 8x 2OU | VM "blades" (QEMU guests) | 16 | Each carries HiNIC + HiGPU + the full guest stack |
| 4OU | Cable management | 4 | UDS files in /var/run/hicain/ and /var/run/hilink/ |
| 2OU | Power shelf (1+1 redundant PSUs) | 2 | 48 V DC to rear busbar |
| Total | 24 OU | Half-height ORv3 rack; ORv3 racks ship in 41 OU and 44 OU heights |
In the emulator the "U-height" is purely visual, components are
processes and .qcow2 images, not metal, but the mapping makes
it easy to point at a real ORv3 photo and explain what each
HiCAIN process would be if it were physical.
2.2 Single Rack (front view)¶
This is the basic deployment: one rack, two switches at the top (HiSwitch + HiLink), one management host, eight VM slots, power at the bottom.
Green LED = active, grey LED = idle. Blue port stubs are the
RoCE/IB fabric; pink port stubs are the HiLink GPU fabric. Every
"cable" you see is a SOCK_SEQPACKET UNIX-domain socket on the
host, the emulator preserves every interface and connector you'd
find in a real ORv3 rack, just transported over UDS instead of
copper/fibre.
2.3 Two Racks Aggregated¶
Scale-out: two ORv3 racks adjacent in a hot-aisle / cold-aisle row, peered via a single yellow inter-TOR cable. HiLink stays rack-local (NVSwitch domains do not span racks); cross-rack GPU traffic falls back to RoCEv2 through the two TORs, the same path NCCL would take in a real cluster.
The wire-level mechanics of the inter-TOR link (FDB learning via gratuitous ARP, IB Subnet-Administration traffic, etc.) are covered in HiCAIN Virtual Switch Design §1.2.
2.4 Logical View (same topology, no chassis)¶
If you want to look past the rack and just see who talks to whom, this is the same two-rack layout as a labelled flowchart.
Note the dotted lines: HiLink connections never leave the rack.
3. Host-Level Topology¶
This is the canonical two-VM development layout used by the
tests/test_rocev2_e2e.py gate.
Legend. Solid lines (===) carry RoCE/IB traffic over the
HiSwitch fabric. Dotted lines (-.->) carry GPU-to-GPU traffic
over the HiLink fabric. Each connection is a SOCK_SEQPACKET UDS
between the QEMU device backend and the corresponding switch
daemon.
4. Per-VM Internal Topology¶
Inside a single VM, the kernel and userspace stack stratifies as follows. The arrows show call direction, not data direction (RDMA data motion is reflected by what each verb does, not by the ABI boundary).
* NCCL runs only on the host today, not in VMs, the guests
have no CUDA driver and no CUDA-capable PCI device. Pulling NCCL
into the guest is tracked by the hinic-nccl L2/L3 backlog and
requires either GPU passthrough into the VM or our patched kernel
loaded on the bare-metal host.
4.1 ABI Boundaries¶
| Boundary | Direction | Mechanism |
|---|---|---|
| App ↔ libibverbs | function calls | C ABI, stock rdma-core |
| libibverbs ↔ libhicain provider | function-pointer table (verbs_context_ops) |
resolved at dlopen time when driver_id == RDMA_DRIVER_HICAIN |
| libibverbs ↔ kernel | ioctl on /dev/infiniband/uverbsN |
uverbs cmd mask + UAPI structs |
kernel verbs ↔ hicain_ib.ko |
function-pointer table (struct ib_device_ops) |
resolved at ib_register_device() |
hicain_ib.ko ↔ hicain_net.ko |
per-frame notifier chain + dev_queue_xmit |
RoCEv2 framing happens here |
hicain_net.ko ↔ QEMU hicain_vnic |
MMIO BAR + IRQ doorbell | virtio-like rings on a memory window |
QEMU hicain_vnic ↔ HiSwitch |
SOCK_SEQPACKET UDS |
one connection per port |
App ↔ libhicart.so |
function calls | C ABI |
libhicart.so ↔ higpu.ko |
ioctl on /dev/higpu0 |
custom UAPI |
higpu.ko ↔ QEMU higpu |
MMIO BAR + IRQ | command/completion rings |
QEMU higpu ↔ HiLink |
SOCK_SEQPACKET UDS |
one connection per port |
5. Fabric Walks¶
5.1 RoCEv2 frame end-to-end (vm-a → vm-b)¶
This is what ibv_rc_pingpong and the e2e gate exercise.
Per-frame cost is dominated by two UDS hops and the kernel-side
ICRC compute on TX + validation on RX. Multi-packet SEND/WRITE
(>MTU) fragments into FIRST + N×MIDDLE + LAST sequences; ACKs
coalesce by setting ackreq=1 only on LAST/ONLY.
5.2 HiLink (GPU-to-GPU) frame walk¶
For comparison: the HiLink fabric uses a much smaller wire header because it's an interconnect, not a fabric, and ignores Ethernet framing entirely.
HiLink has no IBA framing, no ICRC, no PSN/ACK reliability, it mirrors the NVLink/NVSwitch programming model where the interconnect is assumed lossless and ordered.
6. Packaging and Deployment Topology¶
How the artifacts produced by hicain-tools pkg build map to
where they install:
The split is deliberate:
- The switch and link daemons are host-side processes; they never ship into the VM.
- The kernel modules + provider + GPU runtime ship inside the VM image because they have to be loaded by the guest kernel.
hicain-nccl-testsis host-only because NCCL needs real CUDA.
7. Address-space Summary¶
| Identifier | Space | Allocator | Example |
|---|---|---|---|
| Ethernet MAC | 48 bits | static, encoded in QEMU args (52:54:00:42:00:NN) |
52:54:00:42:00:01 |
| IPv4 (data) | private /24 | static at boot | 10.42.0.1, 10.42.0.2 |
| IPv4 (mgmt SSH via SLIRP) | localhost loopback | qemu user-net | localhost:2222, localhost:2223 |
| IB GID | 128 bits | derived: ::ffff:<data IPv4> |
::ffff:10.42.0.1 |
| QPN | 24 bits | per-device counter | 1..N |
| HiLink GPU ID | 8 bits | static, from -device higpu,gpu_id=N |
0..7 |
| HiSwitch port | 4 bits | static, 10-port fixed layout | 0..9 |
| HiLink port | 3 bits | static, 8-port fixed | 0..7 |
8. Code-to-Concept Cross-Reference¶
| Concept | Code home | Key file |
|---|---|---|
| HiSwitch daemon | src/hiswitch/switch/ |
src/main.c, src/eth_pipeline.c, src/ib_pipeline.c |
| HiSwitch CLI | tools/src/hicain_tools/ |
pkg/, deploy/ |
| HiLink daemon | src/hilink/switch/ |
higpu-link-switchd source |
| HiLink CLI | src/hilink/cli/ |
Python, Typer + Rich |
| HiNIC QEMU device | qemu/hw/net/hicain_vnic.c |
full PCIe + MMIO + UDS bridge |
| HiNIC kernel module | src/hinic/kernel/ |
hicain_net.c, hicain_ib.c, hicain_proto.h |
| HiNIC userspace provider | src/hinic/rdma-core/hicain.c |
the real one, not userspace/libhicain.c |
| HiGPU QEMU device | qemu/hw/misc/higpu/ |
PCIe + HiLink endpoint |
| HiGPU kernel module | src/higpu/kernel/ |
higpu.c |
| HiGPU runtime | src/higpu/userspace/libhicart/ |
libhicart.so |
| VM image build | tools/src/hicain_tools/image/ |
image build Typer command |
| Packaging | tools/src/hicain_tools/pkg/targets/ |
one module per .deb target |
| E2E gate | tests/test_rocev2_e2e.py |
15 cases: pingpong, perftest, SRQ, atomics, UD |
| NCCL L1 gate | tests/test_nccl_link.py |
7 cases: deb build, ldd, dlopen, init banner |
9. Live Traffic Flow (animated)¶
The SVG below is animated SMIL, it loops continuously while the page is open and behaves like a silent embedded video. Two flows are shown concurrently against the same physical rack pair from §2:
- Blue, A RoCEv2 SEND from
vm-a0tovm-b0(cross-rack): the packet rises out of the VM, hits TOR A, crosses the inter-TOR cable, drops into TOR B, then lands invm-b0. - Pink, A HiLink
hicartMemcpyPeerfromvm-a1's GPU tovm-a2's GPU (rack-local): up to HiLink A, back down to the peer VM. Never touches the TOR.
The diagram above is an embedded animated SVG. If you see a still
image, your browser is rendering it through the fallback <img>
tag which may suppress SMIL animation in some Chrome versions; open
the SVG directly and the blue +
pink packet dots will loop continuously.
If your browser has prefers-reduced-motion set the animation
auto-disables (the SVG falls back to the static layout).
9.1 Recording a real screen-capture video¶
The animated SVG is good for documentation and is built-in to the
docs. If you want an actual MP4/WebM (e.g. to embed in a slide
deck or share standalone), capture the rendered SVG with ffmpeg:
# 1. Serve the docs locally so the SVG renders with its animations.
mkdocs serve --dev-addr 127.0.0.1:8765 &
# 2. Capture the SVG region for one full animation period (12 s = 2
# RoCEv2 loops + 3 HiLink loops; pick a multiple of LCM(6,4)=12
# so the loop closes cleanly).
ffmpeg -y -f x11grab -framerate 30 -video_size 900x540 \
-i :0.0+0,0 -t 12 \
-vf "scale=900:540,format=yuv420p" \
hicain-vdc-flow.mp4
For headless capture (no X server), render frames with
chromium --headless --screenshot, or use Puppeteer / Playwright
to drive a real browser and pipe MediaRecorder output to disk.
For terminal demos (CLI walkthroughs of hicain-tools deploy,
ibv_devinfo, pingpong sessions, etc.) prefer asciinema: it
captures the terminal at typing-speed, plays back at any size,
embeds as a <script> tag, and the recording itself is text so
it diffs cleanly in git: