CnuasNIC, Datasheet

Item	Value
Part	cnuas-vnic
Type	RoCEv2 + native InfiniBand virtual RDMA NIC
Version	v0.1.0-26-gbc3a3c9
Repo	PacketFive/CnuasNIC

1. Overview

CnuasNIC is the per-VM RDMA network adapter of the Cnuas fabric. The cnuas-vnic QEMU PCIe device, together with two kernel drivers and a libibverbs provider, presents a fully functional RDMA NIC that speaks both RoCEv2 and native InfiniBand to the CnuasSwitch fabric. Standard verbs applications (perftest, ibv_rc_pingpong, MPI, NCCL, UCX) run unmodified against it.

Key features

• Dual link layer: RoCEv2 (UDP/IPv4 encapsulation) and native InfiniBand (LRH+BTH).
• Reliable Connected (RC), Unreliable Datagram (UD), plus SMI/GSI management QPs.
• Full RDMA verb set: SEND/RECV, RDMA WRITE, RDMA READ, and atomics.
• Shared Receive Queue (SRQ) and InfiniBand multicast (attach/detach).
• Hardware-accurate ICRC compute on TX and validation on RX.
• MAD processing with a Subnet Management Agent (SMA) on QP0 and GSI on QP1.
• Drop-in libibverbs provider (cnuas, rdmav34 ABI), no app changes.

2. Driver stack

flowchart TB APP["Verbs app<br/>perftest / pingpong / MPI / NCCL"] VERBS["libibverbs.so.1<br/>(stock rdma-core)"] PROV["libcnuas-rdmav34.so<br/>(CnuasNIC userspace provider)"] IB["cnuas_ib.ko<br/>ib_device driver (L3 / RDMA)"] NET["cnuas_net.ko<br/>netdev driver (L2)"] DEV["QEMU cnuas-vnic<br/>PCIe + MMIO + MSI-X"] SW["CnuasSwitch fabric<br/>(UDS SEQPACKET)"] APP --> VERBS --> PROV --> IB VERBS --> IB IB --> NET --> DEV --> SW classDef app fill:#fee2e2,stroke:#b91c1c,color:#7f1d1d classDef lib fill:#e0e7ff,stroke:#3730a3,color:#1e1b4b classDef k fill:#fef3c7,stroke:#92400e,color:#78350f classDef q fill:#dcfce7,stroke:#166534,color:#14532d class APP app class VERBS,PROV lib class IB,NET k class DEV,SW q

3. Functional specifications

Parameter	Value
PCI vendor : device ID	0x1AF4 : 0x10F0
Link layers	RoCEv2 (UDP dst 4791), native InfiniBand (LRH+BTH)
Ports per device	1 (dual port_immutable personality)
QP types	RC, UD, SMI (QP0), GSI (QP1)
RDMA operations	SEND, RECV, RDMA WRITE, RDMA READ
Atomic operations	CMP_AND_SWP, FETCH_AND_ADD
Receive model	Per-QP receive queue and Shared Receive Queue (SRQ)
Multicast	InfiniBand multicast (attach_mcast / detach_mcast)
Max path MTU	Up to 4096 B
Integrity	Real ICRC compute on TX, validation on RX
Management	MAD post/recv, SMA GET/SET, directed-route SMPs
Multi-packet	Segmented WRITE/READ honouring negotiated path MTU

4. Kernel driver features

Two kernel modules make up the guest-side device.

4.1 cnuas_net.ko, netdev driver (L2)

• Implements net_device_ops: ndo_open, ndo_stop, ndo_start_xmit, ndo_get_stats64.
• Owns the PCIe device, MMIO doorbell ring, and MSI-X interrupts.
• NAPI receive path; feeds RoCEv2/IB frames to/from the QEMU vNIC over UDS.

4.2 cnuas_ib.ko, InfiniBand ib_device driver (L3 / RDMA)

Implements the ib_device_ops verb set (registered against ib-core):

Category	ib_device_ops entry points
Device / port	query_device, query_port, query_gid, query_pkey, get_port_immutable, get_link_layer
Protection domain	alloc_pd, dealloc_pd
User context	alloc_ucontext, dealloc_ucontext
Memory region	reg_user_mr, dereg_mr
Completion queue	create_cq, destroy_cq, poll_cq, req_notify_cq
Queue pair	create_qp, modify_qp, destroy_qp
Shared receive queue	create_srq, modify_srq, query_srq, destroy_srq, post_srq_recv
Address handle	create_ah, create_user_ah, destroy_ah
Data path	post_send, post_recv
Multicast	attach_mcast, detach_mcast
Management	process_mad (MAD/SMA on QP0/QP1)

Additional in-driver logic: RC reliability (PSN/ACK/NAK), RDMA READ response generation, atomics execution, ICRC compute/validate, LRH-aware RX dispatch, and SMA attribute handlers.

5. Userspace provider features

• Provider name cnuas, against the rdma-core rdmav34 ABI.
• Two libraries ship: libcnuas-rdmav34.so, a drop-in for the rdma-core tree, and a standalone libcnuas.so built without it.
• Installed to /usr/lib/x86_64-linux-gnu/libibverbs/ and dlopen()ed by libibverbs.
• Implements the userspace verbs_context_ops mirror of the kernel verbs above (QP/CQ/SRQ/MR/AH create + post/poll fast paths via mapped doorbells).
• Kernel↔userspace ABI shared through cnuas-abi.h (command/response structs).

6. RDMA data flow (RoCEv2 SEND)

sequenceDiagram autonumber participant App_A as App (vm-a) participant Prov_A as libcnuas (vm-a) participant Kern_A as cnuas_ib+net (vm-a) participant CnuasSw as cnuas-vswitchd participant Kern_B as cnuas_ib+net (vm-b) participant App_B as App (vm-b) App_A->>Prov_A: ibv_post_send(SEND) Prov_A->>Kern_A: write() doorbell Kern_A->>Kern_A: build [eth][ip4][udp][bth][payload][icrc] Kern_A->>CnuasSw: SEQPACKET frame (port_0) CnuasSw->>CnuasSw: classify + FDB lookup + DCB CnuasSw->>Kern_B: SEQPACKET frame (port_1) Kern_B->>Kern_B: validate ICRC, parse BTH/RETH/AETH, copy payload Kern_B-->>CnuasSw: ACK frame CnuasSw-->>Kern_A: ACK frame Kern_A-->>Prov_A: completion CQE Prov_A-->>App_A: ibv_poll_cq returns Kern_B-->>App_B: receive CQE

7. Interfaces

Interface	Description
Host bus	QEMU PCIe device (cnuas-vnic), advertises PCIe Gen5 x16
Fabric transport	UNIX domain socket to CnuasSwitch (shared fabric)
Kernel netdev	cnuas_net.ko (net_device_ops)
Kernel RDMA	cnuas_ib.ko (ib_device)
Verbs provider	cnuas (rdmav34), libcnuas-rdmav34.so
ABI header	cnuas-abi.h (kernel ↔ userspace)

8. Verified consumers

ibv_rc_pingpong, perftest (ib_send_bw/ib_write_bw/ib_read_bw), ibstat / ibv_devinfo, and NCCL/UCX build+link gates.

9. Typical usage

# In a guest VM with the cnuas-vnic PCIe device
sudo insmod kernel/cnuas_net.ko
sudo insmod kernel/cnuas_ib.ko
ibv_devices        # lists cnuas_0
ibv_devinfo
ib_send_bw         # perftest

10. Ordering / integration information

Item	Value
Repo	PacketFive/CnuasNIC
Submodule path	src/cnuasnic (in PacketFive/cnuas)
Version	v0.1.0
Language	C (GNU C, kernel + userspace)
License	GPL-2.0-or-later OR MIT (kernel modules GPL-only)

11. Revision history

Revision	Date	Notes
A	2026-07-05	Initial datasheet
B	2026-07-05	Added driver-stack diagram, full verbs table, kernel/userspace feature detail (through I5f)