Cnuas Verbs Provider, Datasheet
| Item |
Value |
| Part |
libcnuas-rdmav34.so |
| Type |
rdma-core user space provider |
| Provider name |
cnuas |
| Package |
cnuas-libcnuas-provider |
| Version |
v0.1.0-26-gbc3a3c9 |
| Repo |
PacketFive/CnuasNIC |
1. Overview
The Cnuas verbs provider is the user space half of the RDMA stack. It plugs into
the standard rdma-core library so that libibverbs applications, perftest,
rdma_cm users and NCCL reach a CnuasNIC through the
same calls they use for physical adapters. No application changes and no
LD_PRELOAD shims are involved: rdma-core discovers the provider from a
driver description file at startup and matches it to the cnuas_ib kernel
device.
Key features
- Discovered by stock
rdma-core, no patched libibverbs required.
- Complete queue pair, completion queue and memory region life cycle.
- Shared receive queue and multicast attachment.
- Address handle creation for datagram traffic.
- Versioned kernel to user ABI, refused on mismatch.
2. Discovery and loading
| Item |
Value |
| Driver description file |
cnuas.driver, containing driver cnuas |
| Installed location |
The rdma-core provider directory, typically /etc/libibverbs.d/ |
| Shared object |
libcnuas-rdmav34.so |
| Registration macro |
PROVIDER_DRIVER(cnuas, cnuas_dev_ops) |
| Kernel ABI version |
1, accepted as both minimum and maximum |
| Kernel counterpart |
cnuas_ib.ko, see Kernel Modules |
The rdmav34 suffix follows the rdma-core provider naming convention and
records the provider ABI generation the library is built for.
ABI mismatch is a refusal, not a fault
The provider declares the same value as its minimum and maximum accepted
kernel ABI. If the loaded cnuas_ib.ko reports a different version the
provider declines the device and ibv_devinfo shows no adapter, rather than
binding and failing later in an unclear way. Keep the module and the
provider from the same release.
3. Verbs implemented
| Group |
Operations |
| Device and port |
query_device_ex, query_port |
| Protection domain |
alloc_pd, dealloc_pd |
| Memory region |
reg_mr, dereg_mr |
| Completion queue |
create_cq, destroy_cq, poll_cq, req_notify_cq |
| Queue pair |
create_qp, modify_qp, destroy_qp |
| Data path |
post_send, post_recv |
| Shared receive queue |
create_srq, modify_srq, query_srq, destroy_srq, post_srq_recv |
| Address handle |
create_ah, destroy_ah |
| Multicast |
attach_mcast, detach_mcast |
Device limits that applications read through query_device_ex come from the
kernel module and are tabulated in
Kernel Modules section 4.1.
4. Application compatibility
| Consumer |
Status |
ibv_devinfo, ibv_devices |
Works |
ib_send_bw, ib_write_bw, ib_read_bw, ib_send_lat |
Works |
rdma and ip tooling |
Works |
| NCCL over the network transport |
Works, see tests/test_nccl_link.py |
opensm |
Not supported, no user space management datagram device exists |
| Vendor specific verbs extensions |
Not supported |
5. Validation
| Feature |
Test |
| RoCEv2 end to end, two guests through the switch |
tests/test_rocev2_e2e.py |
| Native InfiniBand end to end |
tests/test_ib_e2e.py |
| Collective transport over the fabric |
tests/test_nccl_link.py |
These are integration tests: a meaningful run needs the switch daemon, two guest
virtual machines and the provider at once, which is why they live in the
superproject rather than in PacketFive/CnuasNIC. Current results are recorded
in the Validation Matrix.
| Item |
Value |
| Repo |
PacketFive/CnuasNIC |
| Submodule path |
src/cnuasnic |
| Source |
src/cnuasnic/userspace/, src/cnuasnic/rdma-core/ |
| Build |
Make for the standalone object, CMake through rdma_provider() when built inside an rdma-core tree |
| Debian package |
cnuas-libcnuas-provider |
| Depends on |
libibverbs, cnuas_ib.ko |
| Language |
C |
7. Revision history
| Revision |
Notes |
| A |
First publication. Operation list read from the provider operations table in the source. |