HiNIC, Datasheet¶

Part: hicain-vnic · Type: RoCEv2 + native InfiniBand virtual RDMA NIC · Version: `v0.1.0-23-gd97b1c9` · Repo: PacketFive/HiNIC

1. Overview¶

HiNIC is the per-VM RDMA network adapter of the HiCAIN fabric. The hicain-vnic QEMU PCIe device, together with two kernel drivers and a libibverbs provider, presents a fully functional RDMA NIC that speaks both RoCEv2 and native InfiniBand to the HiSwitch fabric. Standard verbs applications (perftest, ibv_rc_pingpong, MPI, NCCL, UCX) run unmodified against it.

Key features¶

Dual link layer: RoCEv2 (UDP/IPv4 encapsulation) and native InfiniBand (LRH+BTH).
Reliable Connected (RC), Unreliable Datagram (UD), plus SMI/GSI management QPs.
Full RDMA verb set: SEND/RECV, RDMA WRITE, RDMA READ, and atomics.
Shared Receive Queue (SRQ) and InfiniBand multicast (attach/detach).
Hardware-accurate ICRC compute on TX and validation on RX.
MAD processing with a Subnet Management Agent (SMA) on QP0 and GSI on QP1.
Drop-in libibverbs provider (hicain, rdmav34 ABI), no app changes.

2. Driver stack¶

flowchart TB APP["Verbs app perftest / pingpong / MPI / NCCL"] VERBS["libibverbs.so.1 (stock rdma-core)"] PROV["libhicain-rdmav34.so (HiNIC userspace provider)"] IB["hicain_ib.ko ib_device driver (L3 / RDMA)"] NET["hicain_net.ko netdev driver (L2)"] DEV["QEMU hicain-vnic PCIe + MMIO + MSI-X"] SW["HiSwitch fabric (UDS SEQPACKET)"] APP --> VERBS --> PROV --> IB VERBS --> IB IB --> NET --> DEV --> SW classDef app fill:#fee2e2,stroke:#b91c1c,color:#7f1d1d classDef lib fill:#e0e7ff,stroke:#3730a3,color:#1e1b4b classDef k fill:#fef3c7,stroke:#92400e,color:#78350f classDef q fill:#dcfce7,stroke:#166534,color:#14532d class APP app class VERBS,PROV lib class IB,NET k class DEV,SW q

3. Functional specifications¶

Parameter	Value
PCI vendor : device ID	`0x1ED5` : `0xCA10`
Link layers	RoCEv2 (UDP dst 4791), native InfiniBand (LRH+BTH)
Ports per device	1 (dual `port_immutable` personality)
QP types	RC, UD, SMI (QP0), GSI (QP1)
RDMA operations	SEND, RECV, RDMA WRITE, RDMA READ
Atomic operations	CMP_AND_SWP, FETCH_AND_ADD
Receive model	Per-QP receive queue and Shared Receive Queue (SRQ)
Multicast	InfiniBand multicast (`attach_mcast` / `detach_mcast`)
Max path MTU	Up to 4096 B
Integrity	Real ICRC compute on TX, validation on RX
Management	MAD post/recv, SMA GET/SET, directed-route SMPs
Multi-packet	Segmented WRITE/READ honouring negotiated path MTU

4. Kernel driver features¶

Two kernel modules make up the guest-side device.

4.1 `hicain_net.ko`, netdev driver (L2)¶

Implements net_device_ops: ndo_open, ndo_stop, ndo_start_xmit, ndo_get_stats64.
Owns the PCIe device, MMIO doorbell ring, and MSI-X interrupts.
NAPI receive path; feeds RoCEv2/IB frames to/from the QEMU vNIC over UDS.

4.2 `hicain_ib.ko`, InfiniBand `ib_device` driver (L3 / RDMA)¶

Implements the ib_device_ops verb set (registered against ib-core):

Category	`ib_device_ops` entry points
Device / port	`query_device`, `query_port`, `query_gid`, `query_pkey`, `get_port_immutable`, `get_link_layer`
Protection domain	`alloc_pd`, `dealloc_pd`
User context	`alloc_ucontext`, `dealloc_ucontext`
Memory region	`reg_user_mr`, `dereg_mr`
Completion queue	`create_cq`, `destroy_cq`, `poll_cq`, `req_notify_cq`
Queue pair	`create_qp`, `modify_qp`, `destroy_qp`
Shared receive queue	`create_srq`, `modify_srq`, `query_srq`, `destroy_srq`, `post_srq_recv`
Address handle	`create_ah`, `create_user_ah`, `destroy_ah`
Data path	`post_send`, `post_recv`
Multicast	`attach_mcast`, `detach_mcast`
Management	`process_mad` (MAD/SMA on QP0/QP1)

Additional in-driver logic: RC reliability (PSN/ACK/NAK), RDMA READ response generation, atomics execution, ICRC compute/validate, LRH-aware RX dispatch, and SMA attribute handlers.

5. Userspace provider features¶

Provider name: hicain (rdma-core rdmav34 ABI).
Libraries: libhicain-rdmav34.so (rdma-core drop-in) and a standalone libhicain.so (built without the rdma-core tree).
Installed to /usr/lib/x86_64-linux-gnu/libibverbs/ and dlopen()ed by libibverbs.
Implements the userspace verbs_context_ops mirror of the kernel verbs above (QP/CQ/SRQ/MR/AH create + post/poll fast paths via mapped doorbells).
Kernel↔userspace ABI shared through hicain-abi.h (command/response structs).

6. RDMA data flow (RoCEv2 SEND)¶

sequenceDiagram autonumber participant App_A as App (vm-a) participant Prov_A as libhicain (vm-a) participant Kern_A as hicain_ib+net (vm-a) participant HiSw as hicain-vswitchd participant Kern_B as hicain_ib+net (vm-b) participant App_B as App (vm-b) App_A->>Prov_A: ibv_post_send(SEND) Prov_A->>Kern_A: write() doorbell Kern_A->>Kern_A: build [eth][ip4][udp][bth][payload][icrc] Kern_A->>HiSw: SEQPACKET frame (port_0) HiSw->>HiSw: classify + FDB lookup + DCB HiSw->>Kern_B: SEQPACKET frame (port_1) Kern_B->>Kern_B: validate ICRC, parse BTH/RETH/AETH, copy payload Kern_B-->>HiSw: ACK frame HiSw-->>Kern_A: ACK frame Kern_A-->>Prov_A: completion CQE Prov_A-->>App_A: ibv_poll_cq returns Kern_B-->>App_B: receive CQE

7. Interfaces¶

Interface	Description
Host bus	QEMU PCIe device (`hicain-vnic`)
Fabric transport	UNIX domain socket to HiSwitch (shared fabric)
Kernel netdev	`hicain_net.ko` (`net_device_ops`)
Kernel RDMA	`hicain_ib.ko` (`ib_device`)
Verbs provider	`hicain` (rdmav34), `libhicain-rdmav34.so`
ABI header	`hicain-abi.h` (kernel ↔ userspace)

8. Verified consumers¶

ibv_rc_pingpong, perftest (ib_send_bw/ib_write_bw/ib_read_bw), ibstat / ibv_devinfo, and NCCL/UCX build+link gates.

9. Typical usage¶

# In a guest VM with the hicain-vnic PCIe device
sudo insmod kernel/hicain_net.ko
sudo insmod kernel/hicain_ib.ko
ibv_devices        # lists hicain_0
ibv_devinfo
ib_send_bw         # perftest

10. Ordering / integration information¶

Item	Value
Repo	`PacketFive/HiNIC`
Submodule path	`src/hinic` (in `PacketFive/vdc`)
Version	v0.1.0
Language	C (GNU C, kernel + userspace)
License	GPL-2.0-or-later OR MIT (kernel modules GPL-only)

11. Revision history¶

Revision	Date	Notes
A	2026-07-05	Initial datasheet
B	2026-07-05	Added driver-stack diagram, full verbs table, kernel/userspace feature detail (through I5f)