Skip to content

CnuasLink Switch Design

1. Overview

CnuasLink is the inter-GPU fabric in the Cnuas, mirroring NVIDIA NVLink and NVSwitch. The CnuasLink Switch is a separate daemon from the Cnuas Switch (which carries RoCE/IB traffic) so that students see the same topological separation as in real datacentres: NIC fabric and GPU fabric are independent.

Real World Cnuas
NVLink CnuasLink (per-GPU endpoint protocol)
NVSwitch CnuasLink Switch (cnuasgpu-link-switchd)
nvidia-fabricmanager hi-fabricmgr

2. Design Goals

  1. Independent of the Cnuas Switch, separate process, separate visualisation
  2. Same 10-port chassis convention as the Cnuas Switch (consistency)
  3. Hybrid datapath: ivshmem for same-host GPU-to-GPU pairs (mimics NVLink bandwidth), UNIX socket through the switch for everything else
  4. SIMT-aware routing, collective primitives (AllReduce, AllGather) can be accelerated by switch-resident reduction (mirrors NVSwitch SHARP)
  5. Ports are GPU-facing, no Ethernet or InfiniBand classification logic

3. Identifiers

Field Value
Daemon cnuasgpu-link-switchd
Run dir default /var/run/cnuaslink
Mgmt socket /var/run/cnuaslink/mgmt.sock
Port sockets /var/run/cnuaslink/port_0.sock through port_9.sock
Wire protocol CnuasLink (raw frames over SOCK_SEQPACKET)

4. Port Layout

Port range Role Notes
0, 7 Fabric (GPU connections) Each port serves one CnuasGPU endpoint
8 Inter-switch link Connect to peer CnuasLink Switch in second rack (when present)
9 Console / OTel Telemetry export, no data plane

Same role concept as the Cnuas Switch, different protocol on the wire.

5.1 Frame Format

flowchart LR M["Magic<br/>2 B"] --> V["Ver<br/>1 B"] --> T["Type<br/>1 B"] --> S["Src GPU<br/>2 B"] --> D["Dst GPU<br/>2 B"] --> L["Length<br/>4 B"] --> P["Payload<br/>N B"] classDef hdr fill:#dbeafe,stroke:#1e40af,stroke-width:2px,color:#1e3a8a; classDef pay fill:#fef3c7,stroke:#b45309,stroke-width:2px,color:#78350f; class M,V,T,S,D,L hdr; class P pay;
Field Size Value
Magic 2 B 0x484C ("HL")
Version 1 B 0x01
Type 1 B See type table below
Src GPU ID 2 B Sender GPU global ID
Dst GPU ID 2 B Receiver GPU global ID (or 0xFFFF broadcast)
Length 4 B Payload bytes
Payload N B Type-dependent

5.2 Frame Types

Code Type Payload
0x01 DATA Raw bytes (used by CnuasSHMEM put/get and CnuasCCL stages)
0x02 DOORBELL 64-bit doorbell value
0x03 ATOMIC Atomic op descriptor (op, addr, value)
0x04 COLLECTIVE_REDUCE Reduction stage (op, dtype, count, partial)
0x05 COLLECTIVE_GATHER Gather stage
0x06 DISCOVERY GPU announces itself on link bring-up
0x07 LINK_KEEPALIVE Periodic heartbeat

5.3 GPU IDs

Each CnuasGPU has a globally unique 16-bit ID derived from (rack_id << 8) | slot_id. The CnuasLink Switch keeps a forwarding table from GPU ID to port, analogous to the L2 FDB in the Cnuas Switch.

6. Datapath

6.1 Hybrid Transport

Two transports, chosen automatically by the GPU driver based on peer location:

Path Transport Used When
Fast path ivshmem (POSIX shared memory + UNIX socket for signalling) Two CnuasGPUs are in QEMU instances on the same host
Switch path UNIX SOCK_SEQPACKET to CnuasLink Switch Cross-host or multi-hop

The CnuasLink Switch maintains a peer registry. GPU driver queries the switch on cnuaslink_open() to learn whether a direct ivshmem channel exists for a given peer.

6.2 ivshmem Fast Path

When two CnuasGPUs are on the same host:

  1. QEMU launched with -object memory-backend-file,id=cnuaslink-AB,... shared between the two QEMU processes
  2. Each CnuasGPU sees the shared region in BAR1 of the peer (or a dedicated BAR3 for inter-GPU memory)
  3. Direct loads/stores to peer memory; doorbells go through a QEMU-instantiated ivshmem-doorbell device
  4. No frame encapsulation, raw memory access

This is roughly how NVLink P2P works on real hardware.

6.3 Switch Path

When direct ivshmem isn't available:

  1. GPU driver writes CnuasLink frame to its endpoint TX queue
  2. QEMU emulator picks up the frame, sends over UNIX socket to switch
  3. Switch routes by Dst GPU ID, forwards to destination port
  4. Destination QEMU receives frame, writes payload to GPU's RX buffer
  5. RX MSI raised in destination guest

Higher latency than ivshmem but works across racks and hosts.

6.4 Collective Acceleration

The switch can fold reductions in flight:

  1. CnuasCCL launches AllReduce(sum, X)
  2. Each GPU sends COLLECTIVE_REDUCE to switch
  3. Switch sums incoming partials in real time
  4. When all GPUs have contributed, switch broadcasts the sum back
  5. This mirrors NVSwitch SHARP

This is a v2 feature; v1 implements collectives as N-stage point-to-point.

7. Switch Daemon

7.1 Architecture

Same single-threaded epoll model as the Cnuas Switch. Reuses the same patterns:

  • SOCK_SEQPACKET listeners on ports 0, 8
  • SOCK_STREAM listener on management socket
  • Single-threaded event loop

7.2 Forwarding Tables

Table Key Value
GPU FDB 16-bit GPU ID Egress port
Group table Group ID List of GPU IDs (used by collectives)

GPU FDB populated automatically via DISCOVERY frames on link up.

7.3 Management API

JSON over UNIX socket, same pattern as the Cnuas Switch:

Command Effect
port_status Per-port state, counters, peer GPU ID
gpu_fdb_dump List of (GPU ID → port) mappings
group_create Create a collective group
group_dump List groups
set_link_state Up / down a port
telemetry_dump Frame counters per port and per type
telemetry_clear Reset counters

8. WebUI Visualisation

The Cnuas rack view gains a second top-of-rack switch slot:

flowchart TB S1["Cnuas Switch (RoCE/IB) &nbsp;&mdash;&nbsp; ports 0-7 to NICs"] S2["CnuasLink Switch &nbsp;&mdash;&nbsp; ports 0-7 to CnuasGPU endpoints"] C0["Sled chassis 0 &nbsp;&mdash;&nbsp; VMs, each VM may host one CnuasGPU"] C1["Sled chassis 1"] PSU["PSU"] BBU["BBU"] S1 --- S2 --- C0 --- C1 --- PSU --- BBU classDef sw fill:#dbeafe,stroke:#1e40af,stroke-width:2px,color:#1e3a8a; classDef link fill:#e0e7ff,stroke:#3730a3,stroke-width:2px,color:#1e1b4b; classDef sled fill:#dcfce7,stroke:#166534,stroke-width:2px,color:#14532d; classDef pwr fill:#fef3c7,stroke:#b45309,stroke-width:2px,color:#78350f; class S1 sw; class S2 link; class C0,C1 sled; class PSU,BBU pwr;

The CnuasLink Switch chassis is rendered with the same visual style as the Cnuas Switch (port LEDs, vendor strip) but with a different colour accent to distinguish it.

9. CLI

cnuaslink-cli, analogous to cnuas-cli:

cnuaslink-cli port status
cnuaslink-cli port status 0
cnuaslink-cli gpu-fdb show
cnuaslink-cli group create --name allred-group --gpus 0,1,2,3
cnuaslink-cli telemetry show

Reuses the same Typer + Rich + Pydantic stack as cnuas-cli.

10. Telemetry

Per-port counters:

Counter
rx_frames
tx_frames
rx_bytes
tx_bytes
rx_drops
tx_drops
collective_reduce_count
collective_gather_count
latency_us_p50
latency_us_p99

Exported via OTel on port 9 (same pattern as Cnuas Switch).

11. Phasing

Phase Scope
1 Daemon scaffold, port sockets, mgmt socket, JSON API
2 DATA frame forwarding by Dst GPU ID, FDB learning via DISCOVERY
3 DOORBELL and ATOMIC frame types
4 cnuaslink-cli
5 ivshmem fast-path: switch advertises same-host peers
6 WebUI rack rendering with CnuasLink Switch slot
7 COLLECTIVE_REDUCE, COLLECTIVE_GATHER stages (v1: passthrough)
8 OTel telemetry export
9 Switch-side collective folding (SHARP-style), v2

12. Relationship to Cnuas Switch

Aspect Cnuas Switch CnuasLink Switch
Process cnuas-vswitchd cnuasgpu-link-switchd
Wire protocol RoCEv2 / InfiniBand frames CnuasLink frames
Endpoint device RoCE-IB-vNIC (cnuas-vnic) CnuasGPU (cnuasgpu)
Routing key MAC / LID GPU ID
Use case NIC traffic, storage, SSH, CnuasCCL fallback GPU-to-GPU compute traffic
Run dir /var/run/cnuas /var/run/cnuaslink

The two switches are completely independent processes. A typical rack runs both side by side. CnuasCCL can fall back to using RoCE through the Cnuas Switch if CnuasLink is unavailable, but that is a software policy decision in CnuasCCL, not a coupling between the daemons.

13. Directory Structure (planned)

src/
  cnuaslink-switch/             # daemon source
    src/
    include/
    BUILD.bazel
  cnuaslink-cli/                # CLI
    cnuaslink_cli/
    BUILD.bazel
docs/
  CnuasLink_Switch_Design.md

14. Open Questions

  1. GPU ID assignment: static at QEMU launch via property, or learned dynamically from a discovery handshake?
  2. ivshmem device naming: one shared region per pair (n^2 regions), or one big region and a routing layer? Pair-wise scales worse but is simpler.
  3. Frame size: 9 KB jumbo (matching Cnuas Switch) or larger (e.g. 64 KB to match NVLink burst behaviour)?
  4. Authentication: do we need any peer authentication on link bring-up, or trust the local socket namespace?