CnuasDev, Datasheet¶
| Item | Value |
|---|---|
| Part | libcnuasdev.so |
| Type | Device access library |
| SONAME | libcnuasdev.so.0, current build libcnuasdev.so.0.1.0 |
| Package | cnuas-libcnuasrt |
| Version | v0.2.0-14-ga157b80-dirty |
| Repo | PacketFive/CnuasGPU |
1. Overview¶
CnuasDev is the thin layer between CnuasRT and the
cnuasgpu.ko character device. It turns the ioctl and mmap protocol of the
driver into a small C interface, so the runtime never issues a raw ioctl and
tools that need device facts without the full runtime have somewhere lighter to
link against.
This datasheet also specifies the compute backend ABI, because the backend shared objects are built and shipped alongside the device library and share its release cadence.
Key features¶
- Device enumeration and per device information without opening the runtime.
- Explicit allocate, free, and copy against device memory.
- Direct memory mapping of device memory into the host address space.
- A published backend ABI so a new instruction set is a new shared object rather than a rebuild of the runtime.
- Backend selection at first use with an environment variable override.
- A CnuasLink transport that speaks frames rather than raw ioctls, with the wire format available as a standalone unit that needs no device.
2. Application programming interface¶
Every symbol below is exported by libcnuasdev.so.0.1.0.
| Group | Function | Purpose |
|---|---|---|
| Discovery | cnuasdev_get_device_count |
Count the /dev/cnuasgpuN nodes present |
| Handle | cnuasdev_open |
Open a device by index and return a handle |
| Handle | cnuasdev_close |
Release a handle and any allocations tied to it |
| Information | cnuasdev_query_info |
Identity, geometry, and memory figures for the device |
| Memory | cnuasdev_alloc |
Reserve a device memory range |
| Memory | cnuasdev_free |
Release a device memory range |
| Memory | cnuasdev_memcpy |
Copy between host and device memory |
| Mapping | cnuasdev_mmap |
Map device memory into the calling address space |
| Mapping | cnuasdev_munmap |
Undo a mapping |
| Diagnostics | cnuasdev_status_str |
Text for a cnuasdev_status_t code |
| CnuasLink | cnuasdev_link_id |
Local fabric identifier, cached after first use |
| CnuasLink | cnuasdev_link_status |
Whether the switch is reachable and a frame is staged |
| CnuasLink | cnuasdev_link_send |
Encode and send a frame, filling in the local source |
| CnuasLink | cnuasdev_link_recv |
Receive one frame and decode it |
| CnuasLink | cnuasdev_link_send_raw |
Send bytes that are already a frame |
| CnuasLink | cnuasdev_link_recv_raw |
Receive bytes without decoding |
| CnuasLink | cnuaslink_frame_encode |
Build a frame in a caller buffer |
| CnuasLink | cnuaslink_frame_decode |
Validate and parse a received frame |
| CnuasLink | cnuaslink_frame_type_str |
Text for a frame type |
2a. CnuasLink transport¶
The transport is a synchronous wrapper over the three LINK ioctls. It is the
lowest layer that speaks frames rather than bytes and is the foundation CnuasCCL
is expected to build on.
| Parameter | Value |
|---|---|
| Header | cnuaslink_frame.h |
| Frame header size | 12 bytes, fixed |
| Maximum frame | 65536 bytes |
| Maximum payload | 65524 bytes |
| Byte order | Little endian for every multi byte field |
| Identifier range | 1 to 65534; 0 and 65535 are reserved |
| Receive timeout | 0 polls, 4294967295 waits without limit, otherwise milliseconds |
Two properties of the fabric shape what can be built on top and are stated here so callers do not have to rediscover them. Receive is not demultiplexed by peer, so a frame from any sender satisfies any waiting receive and a caller expecting a specific peer must check the source itself. Discovery and keepalive frames train the forwarding table but are never forwarded, so neither can carry a message to another rank. Both points are treated at length in the CnuasCCL bootstrap design.
The wire format is defined normatively in the CnuasLink repository. The copy in
cnuaslink_frame.h is a restatement, because the two live in separate
repositories, and lib/test/cnuaslink_frame_smoke asserts the encoded bytes
against literals so that a drift between them fails a test rather than producing
frames the switch discards.
3. Device memory model¶
| Parameter | Value |
|---|---|
| Backing | PCI BAR1 of the CnuasGPU device, RAM backed in the emulator |
| Default size | 256 MiB, set by the devmem_size device property |
| Allocator | Bump pointer, page aligned, allocated in the kernel driver |
| Reclaim | On explicit free and on close of the file descriptor |
| Host mapping | Write combining, through mmap on the character device |
| Accounting | devmem_size and devmem_used are published in sysfs |
The allocator is deliberately simple
Allocation advances a pointer and does not coalesce freed ranges. A process that allocates and frees in a long loop will exhaust the device memory window even though its live footprint is small. Close and reopen the handle to reset. A free list allocator is roadmap work.
4. Character device interface¶
| Item | Value |
|---|---|
| Node | /dev/cnuasgpuN, one per device |
| Class | /sys/class/cnuasgpu/ |
| Sysfs attributes | gpu_id, sm_count, devmem_size, devmem_used, link_up, link_rx_ready |
| Operations | ioctl for information, allocate, free, copy, and the three CnuasLink calls; mmap for direct access |
5. Compute backend ABI¶
Backends are standalone shared objects named libcnuasrt-${name}.so. Each one
exports a single vtable symbol, cnuas_compute_backend, that carries a name, an
ABI version pair, a capability mask, and the kernel function pointers.
| Parameter | Value |
|---|---|
| Vtable symbol | cnuas_compute_backend |
| ABI version | Major 0, minor 1 |
| Shipped backends | libcnuasrt-scalar.so, libcnuasrt-avx2.so, libcnuasrt-avx512.so |
| Selection | Highest capability backend the host processor supports |
| Discovery | dlopen at first compute call |
Capability bits¶
| Bit | Meaning | Implemented |
|---|---|---|
CNUAS_CCAP_VEC_F32 |
32 bit float vector kernels | Yes |
CNUAS_CCAP_GEMM_F32 |
32 bit float matrix multiply | Yes |
CNUAS_CCAP_X86_AVX2 |
Host uses AVX2 code paths | Yes |
CNUAS_CCAP_X86_AVX512 |
Host uses AVX-512 code paths | Yes |
CNUAS_CCAP_GEMM_F16, CNUAS_CCAP_GEMM_BF16 |
Reduced precision matrix multiply | Reserved |
CNUAS_CCAP_CONV2D_F32 |
Convolution | Reserved |
CNUAS_CCAP_FFT_C32 |
Complex transform | Reserved |
CNUAS_CCAP_TENSOR_CORE |
Tensor tile unit | Reserved |
CNUAS_CCAP_CNUASIR_EXEC |
Executes CnuasIR | Reserved |
CNUAS_CCAP_REDUCE |
Reduction primitives | Reserved |
CNUAS_CCAP_ARM_NEON, CNUAS_CCAP_ARM_SVE, CNUAS_CCAP_RISCV_RVV, CNUAS_CCAP_FPGA |
Non x86 hosts and offload | Reserved |
Reserved bits are defined in the header so that the ABI does not have to change when the corresponding kernels land. No shipped backend sets them.
Selection controls¶
| Environment variable | Effect |
|---|---|
CNUAS_COMPUTE_BACKEND |
Load the named backend, bypassing detection |
CNUAS_BACKEND_DIR |
Override the backend search directory |
CNUAS_COMPUTE_LIST |
List discovered backends |
CNUAS_COMPUTE_VERBOSE |
Explain the selection decision |
6. Validation¶
| Feature | Test | Device required |
|---|---|---|
| Information ioctl, allocate, copy round trip, free | driver/test/cnuasgpu_smoke.c |
Yes |
| Backend vtable numerics per instruction set | lib/test/compute_backend_smoke.c |
No |
| Backend shared object ABI, capabilities, and numerics | lib/test/compute_so_smoke.c |
No |
| CnuasLink wire format, golden bytes, round trip, and every rejection | lib/test/cnuaslink_frame_smoke.c |
No |
| CnuasLink send and receive against a live switch | driver/test/cnuasgpu_link_test.c |
Yes |
The three tests that need no emulated device make up the fastest regression
signal in the accelerator stack, and all of them run under make -C lib check.
7. Integration information¶
| Item | Value |
|---|---|
| Repo | PacketFive/CnuasGPU |
| Submodule path | src/cnuasgpu |
| Source | lib/libcnuasdev/, lib/compute/ |
| Headers | cnuasdev.h, cnuaslink_frame.h, cnuas_compute_ops.h |
| Debian package | cnuas-libcnuasrt |
| Depends on | cnuasgpu.ko and a CnuasGPU PCI device for the device path |
| Language | C |
| Build | Make |
8. Revision history¶
| Revision | Notes |
|---|---|
| A | First publication. Symbol list taken from the dynamic symbol table of libcnuasdev.so.0.1.0, capability bits from cnuas_compute_ops.h. |
| B | Added the CnuasLink transport and wire format of roadmap milestone 8.2a, section 2a and the new symbols in section 2. Frame parameters taken from cnuaslink_frame.h and cross checked against the switch header. |