CnuasGPU Host Character-Device Backend

cnuasgpu_host.ko provides a Linux CnuasGPU device without QEMU, a guest, a PCI function, or BARs. It creates /dev/cnuasgpu_hostN from host pages and preserves the existing node-less Soft-GPU as a separate lower-overhead mode.

The authoritative detailed design is src/cnuasgpu/docs/CnuasGPU_Host_CharDev.md.

1. The three supported backends

Backend	Selection	Node	Memory
Node-less Soft-GPU	CNUAS_DEVICE_BACKEND=host	None	Arena in the calling process
Host character device	CNUAS_DEVICE_BACKEND=hostdev	/dev/cnuasgpu_hostN	Kernel-owned host pages
QEMU PCIe device	CNUAS_DEVICE_BACKEND=device	/dev/cnuasgpuN	Emulated BAR1

Automatic discovery prefers /dev/cnuasgpuN, then /dev/cnuasgpu_hostN, then the node-less arena. A host node therefore never hides a QEMU PCIe CnuasGPU.

The host and PCI modules have independent majors, minor spaces, classes, and lifetimes. They can load or unload in either order.

2. Build and load

Build both modules against the target host kernel:

make -C src/cnuasgpu/driver \
  KDIR=/lib/modules/$(uname -r)/build

Load the host module with its defaults:

sudo insmod src/cnuasgpu/driver/cnuasgpu_host.ko
ls -l /dev/cnuasgpu_host0

export CNUAS_DEVICE_BACKEND=hostdev
src/cnuasgpu/tools/cnuassmi/cnuassmi

Unload only after clients, mappings, exports, and queued work have closed:

sudo rmmod cnuasgpu_host

Secure Boot hosts require a trusted module signature. The Debian GPU-module package includes cnuasgpu_host.ko, but does not auto-load it because memory, permissions, and queue policy are deployment decisions.

3. Configuration

Every feature is independently configurable at module load and reported through a capability ioctl and sysfs. Parameters are read-only after load so a queried capability cannot disappear while a client uses it.

Parameter	Default	Range	Meaning
devices	1	1–8	Host nodes to create
devmem_mb	64	1–65536	Arena size per node
pinned_max_mb	32	0–65536 and no more than arena	Per-device pinned allocation ceiling
zerocopy	Y	Boolean	Allow arena mmap
pinned	Y	Boolean	Allow committed page-backed allocations
dmabuf	Y	Boolean	Allow pinned allocation export
async	Y	Boolean	Enable batched queue and completions
queue_depth	64	1–4096	Outstanding operations per open descriptor
batch_max	32	1–1024	Operations accepted in one submission
devnode_mode	0600	Permission mask	Created node permissions

Example: two 16 GiB host devices, 8 GiB pinned ceiling, a larger asynchronous queue, and DMA-BUF disabled:

sudo insmod src/cnuasgpu/driver/cnuasgpu_host.ko \
  devices=2 \
  devmem_mb=16384 \
  pinned_max_mb=8192 \
  zerocopy=Y \
  pinned=Y \
  dmabuf=N \
  async=Y \
  queue_depth=1024 \
  batch_max=128 \
  devnode_mode=0660

Invalid combinations refuse module load rather than being clamped. In particular, DMA-BUF is not advertised when pinning is disabled.

4. Zero-copy mapping

With zerocopy=Y, mmap maps the same arena pages used by synchronous and asynchronous copy operations. Pages are populated on first fault, so mapping a large arena does not immediately commit all of it.

This is zero-copy access between one process and the kernel-owned arena. It is not cross-process sharing. Cross-process or device sharing uses a DMA-BUF fd. With zero-copy disabled, mapping returns unsupported and callers must use copy operations.

5. Pinned allocations

cnuasdev_alloc_flags(..., CNUASDEV_ALLOC_F_PINNED, ...) commits every page at allocation time and holds those pages for the allocation lifetime. This means:

• allocation fails immediately if pages cannot be provided;
• mappings and DMA-BUF scatterlists have stable backing; and
• the allocation is charged against pinned_max_mb.

This does not pin arbitrary caller-owned user pages and does not use SetPageReserved. Kernel-allocated pages are non-swappable while owned by the module. Exceeding the configured ceiling returns a quota error distinct from machine memory exhaustion.

6. DMA-BUF sharing

cnuasdev_export_dmabuf() exports a page-aligned range of a pinned allocation. The fd can be sent over a Unix socket or imported by a kernel subsystem.

The fd outlives the CnuasGPU device descriptor that created it. Explicit free is refused while exports exist; orphaned memory is reclaimed after the last export closes. Read-only exports reject writable mappings and writing CPU access directions.

This is host-RAM sharing, not PCI peer memory. The host module reports its DMA-BUF capability through the host capability ABI rather than CNUASGPU_CAP_PEER_RDMA.

7. Batched asynchronous queue

One open descriptor owns one ordered workqueue and completion stream. A batch can contain:

• host-to-device, device-to-host, and device-to-device copies;
• byte fill;
• barrier; and
• no-op.

Accepted operations execute in order and each produces exactly one completion with a sequence, opcode, caller cookie, and result. Submission reports how many operations were accepted and stops at the first invalid/unavailable entry. There is no silent synchronous fallback.

CnuasDev exposes:

cnuasdev_query_caps(...);
cnuasdev_async_submit(...);
cnuasdev_async_poll(...);
cnuasdev_async_wait(...);
cnuasdev_synchronize(...);
cnuasdev_async_fd(...);

The completion fd is pollable for integration with an application event loop. Closing a descriptor cancels or drains work through defined lifetime rules; no accepted operation is reported complete before execution.

8. Security and resource model

• Nodes default to mode 0600 because one node represents one shared arena.
• Allocations are owned by the creating file descriptor.
• Cross-process access requires explicit fd transfer.
• Every offset, size, alignment, flag, count, and arithmetic operation is bounds/overflow checked.
• Arena, pinned memory, device count, queue depth, and batch size are bounded.
• DMA-BUF exports and queued operations retain backing objects until safe release.
• Module removal cannot invalidate live mappings or exports.

For multi-tenant use, keep 0600 or install an explicit group/udev policy. Changing devnode_mode to 0666 gives every local user access to the shared arena and is not a safe general-purpose default.

9. Performance interpretation

The node-less backend normally has lower control-path overhead because allocation, mapping, copy, and dispatch are userspace function calls. A character device introduces syscalls and synchronization.

The host device can win when batching amortizes those transitions, mappings remove copies, or DMA-BUF avoids process-to-process copies. A device node alone does not accelerate AVX2/AVX-512 arithmetic. No performance ordering is claimed until both modes are benchmarked on identical hosts and workloads.

Compare allocation latency, first-touch and warm mapping cost, copy bandwidth, batch size versus operation rate, launch latency, kernel throughput, concurrent clients, CPU use, and memory footprint.

10. Validation

Host-only ABI and userspace tests:

make -C src/cnuasgpu check

Loaded-module tests run in a throwaway VM, avoiding Secure Boot and host kernel changes:

KDIR=out/work/kbuild619 \
KERNEL=out/work/kbuild619/arch/x86/boot/bzImage \
  src/cnuasgpu/scripts/run-hostdev-vm.sh

The gate exercises default features, each feature disabled independently, minimal mode, multiple nodes, invalid parameters, mapping coherence, pinned limits, DMA-BUF lifetime, asynchronous order/completions/errors, close with work in flight, unload, KASAN, and lock debugging.

11. Limits

• CnuasRT remains synchronous; new asynchronous operations are currently a CnuasDev API.
• The host module provides memory and queue semantics, not a kernel compute engine. CPU compute still runs through the userspace compute backends.
• Allocator offsets are monotonic and not reused, matching the PCI driver.
• CnuasLink ioctls are unsupported because no accelerator fabric endpoint is attached to the host module.