CnuasGPU as a Standalone Soft-GPU¶

This page is published from docs/CnuasGPU_SoftGPU.md in the PacketFive/CnuasGPU submodule, which is the authoritative copy. Edit it there and refresh this one, rather than editing this file alone.

CnuasGPU runs as a software accelerator on an ordinary host, outside QEMU and without a guest. This document describes that form, what it provides, and how an inference runtime uses it.

It also sets out how CnuasGPU maps onto a tile-based accelerator architecture (TTU / TCP / CCP / TVP).

1. Standalone operation¶

CnuasGPU runs outside QEMU through the host device backend. It needs no kernel module, no /dev/cnuasgpuN, no guest, and no configuration. The host arena is selected automatically when no character device node is present. CNUAS_DEVICE_BACKEND forces the choice, host or device, and section 3.2.1 describes the integration that automatic selection serves.

The backend is implemented in lib/libcnuasdev/cnuasdev_host.c and is covered by tests that fail if it is removed:

Evidence	What it shows
`lib/test/cnuasdev_host_smoke`, 29 checks	The backend itself: enumeration, alloc alignment, H2D/D2H/D2D, bounds rejection, arena exhaustion, and that the pointer from `cnuasdev_mmap` survives `cnuasdev_munmap`.
`lib/test/cnuasrt_activation_smoke`, 19 checks	The full public runtime path, `cnuasInit` through `cnuasMalloc`, `cnuasMemcpy` and `cnuasReluF32`. With the backend deselected it reports skip, so it cannot pass for an unrelated reason.
ThreadSanitizer, 8 threads	Concurrent allocation is race free and hands out no duplicate offsets; removing the mutex reports races, so the test has detection power.

1.1 Why there is no `/dev/cnuasgpuN` in the host form¶

The node is created inside cnuasgpu_probe() in driver/cnuasgpu.c, and that function runs only when the PCI bus matches a function against cnuasgpu_pci_ids. No emulated function means no probe, so no node. The character device region is allocated at module init, but nodes appear per probe.

Nothing about the ioctl surface requires PCI. CNUASGPU_IOC_INFO, ALLOC, FREE, MEMCPY and the three link calls are an ordinary character device ABI. What is PCI-coupled is the implementation beneath them: registers are readl and writel against BAR0, device memory is memcpy_toio and memcpy_fromio against BAR1, and mmap is remap_pfn_range over bar1_phys.

A host character device is therefore possible, and it would need no PCI ID, because a character device is named by the major and minor from alloc_chrdev_region rather than by a config-space vendor and device pair. /dev/kvm and /dev/fuse are the familiar examples of a device node with no bus behind it. It would require a small host kernel module that keeps the ioctl ABI byte for byte and swaps the implementation underneath: a register file in kernel memory for BAR0, allocated pages for BAR1, and remap_vmalloc_range or vm_insert_page for mmap.

The value of a host node is not realism. The device ioctl path is the only part of the accelerator that cannot be exercised without a guest, which is why two rows of the validation matrix are marked gated. A host node would let the same ABI the guest uses run with no VM.

CUSE does not work here. Its cuse_frontend_fops in fs/fuse/cuse.c provides read, write, open, release, ioctl and poll and no mmap at all, so a device whose purpose is mapping device memory cannot be built on it.

A host character device is not part of the current implementation. It is carried in the Cnuas Roadmap.

The layers that run with no device are:

Layer	Runs with no device?
Compute backends (`cnuas_compute_ops`, including the activations)	Yes. `compute_backend_smoke` links the backend archive directly and needs neither a device nor a guest.
CnuasBLAS	Yes, through the test stub and through the host backend.
libcnuasrt public API (`cnuasMalloc`, `cnuasSgemm`, `cnuasReluF32`, ...)	Yes, on any host with no device node, without being asked. Where a node is present the character device is used, and `CNUAS_DEVICE_BACKEND=host` overrides that.

The host test suite runs with 0 skipped.

Two limits apply. The host backend emulates the interface of the device, not its timing or its memory system, so it serves correctness and bring-up rather than performance claims. Its allocator is a bump pointer that reclaims only the most recent block, matching the kernel driver rather than improving on it, so that code which works here also works against the emulated device.

The host backend is what allows a ggml backend to allocate and hand back buffers without a guest.

2. Mapping onto a tile accelerator (TTU / TCP / CCP / TVP)¶

A conventional tile-based accelerator is usually described in four parts. Measured against that vocabulary, CnuasGPU stands as follows.

Unit	Role	State in CnuasGPU
TTU, Tile Tensor Unit	Matrix multiply with direct accumulation	Partial. `sgemm` computes `alpha·AB + beta·C`, so accumulation into a destination tile is present, and the AVX-512 backend already blocks internally with a 16×16 `C` tile and MC/KC/NC panels. What is absent is an exposed tile abstraction: callers cannot address a tile register file or issue an accumulate-into-tile primitive. `cnuasdev_device_info.tensor_size` exists and is 0 when there is no such unit, which is what the host backend reports.
TCP, Tile Control Processor	Pointer arithmetic and loop control feeding the tensor unit	Effectively present. This is what the CnuasIR scalar subset does, and what the CnuasCC code generator emits: induction variables, address computation, bounded loops.
CCP, Cluster Control Processor	Higher-level sequencing across several tiles/cores	Absent. There is no cluster-level scheduler; dispatch is one call, one backend, one thread team.
TVP, Tile Vector Processor	Elementwise/activation functions	Present (ABI 0.2). `relu`, `gelu`, `silu`, `sigmoid`, `tanh` and row-wise `softmax`, vectorised for AVX2 and AVX-512 by `lib/compute/common/cnuas_activations_simd.h`. See `CnuasGPU_Compute_Backends.md` §3.2.1.

In summary, CnuasGPU has a competent TCP, a usable but unexposed TTU, a TVP, and no CCP.

3. What an inference runtime actually needs¶

3.1 ggml / llama.cpp¶

Verified against llama.cpp at commit 0b1bad14ff20.

A backend is a shared object exporting ggml_backend_init() (via GGML_BACKEND_DL_IMPL) and optionally ggml_backend_score(). The loader dlopens it, checks an integer api_version, and registers its devices (ggml/src/ggml-backend-reg.cpp). ggml_backend_load_all_from_path() honours the GGML_BACKEND_PATH environment variable, which the code documents as the out-of-tree path.

Two consequences matter:

A partial backend is legitimate. There is no fixed minimum operation list; supports_op() is consulted per graph node. The ggml-blas backend supports only MUL_MAT and OUT_PROD under narrow layout and type conditions, plus view/no-op nodes, and returns false for everything else. ggml_backend_sched then splits the graph and routes the remainder to CPU, which is assumed to be the last backend in priority order.
Claiming an operation that graph_compute cannot execute is fatal, so supports_op() must be conservative rather than optimistic.

ggml-blas is therefore the right template: it reuses ggml_backend_cpu_buffer_type() and imports host pointers with ggml_backend_cpu_buffer_from_ptr(), which matches CnuasGPU's host backend exactly. ggml-cann and ggml-opencl assume a real device runtime and separate device buffers; ggml-rpc is a proxy/transport backend, not a kernel backend, and would only add serialisation overhead in the same-process case. (ggml-rpc becomes interesting only if the QEMU/VM boundary is the point - architecturally that resembles ggml-virtgpu, which remotes ggml through virtio_gpu.)

The binding constraint is ABI, not design: the plugin must be built against the same ggml revision as the host application, and dynamic loading requires GGML_BACKEND_DL with shared libraries.

3.2 Ollama¶

Ollama does not vendor a full llama.cpp fork; it fetches a pinned upstream revision with CMake FetchContent, applies a compatibility patch, and runs llama-server as a subprocess. Its default preset sets BUILD_SHARED_LIBS=ON and GGML_BACKEND_DL=ON, so on current Linux and Windows packages an out-of-tree backend can be loaded without rebuilding or patching Ollama. macOS packages set both options off and statically embed Metal, so this does not apply there.

This has been run rather than reasoned about. Ollama 0.32.9, which ships ggml 0.19.0, was installed from the upstream release archive and started as

$ GGML_BACKEND_PATH=/abs/path/libggml-cnuas.so ollama serve
$ ollama run qwen3:0.6b "..."

with no Cnuas variable set anywhere. The model answers correctly, and 41 of its matrix multiplications were measured executing on CnuasGPU during prompt processing, at shapes 1024x510x1024 and 1024x510x3072, for 71.7 GFLOP in total.

3.2.1 The environment is not propagated¶

The first attempt did not work, and the reason is worth recording because it is invisible from reading the code. It aborted in the device sgemm, and the debug log showed why. Ollama does not hand its own environment to the runner. It passes exactly three variables:

GGML_BACKEND_PATH=...  PATH=...  LD_LIBRARY_PATH=...

CNUAS_DEVICE_BACKEND=host, set in the parent, therefore never arrived. The device layer looked for /dev/cnuasgpuN, which cannot exist on a host with no kernel module, reported zero device memory, and failed on the first multiply.

An earlier revision of this document asserted that Ollama propagates os.Environ() into the runner. That is not what it does, and the assertion is withdrawn.

The fix belonged in this project rather than in Ollama. A Soft-GPU whose only route to being a Soft-GPU is an environment variable that the embedding application refuses to forward is not usable, so cnuasdev_host_enabled now selects the host arena automatically when no character device node is present. CNUAS_DEVICE_BACKEND still forces the question either way, host or device, and the automatic case announces itself once on stderr so that a caller which believed it was driving real hardware can discover from the log that it was not. Automatic selection cannot mask a missing kernel module inside a guest, because there the node exists and is therefore chosen.

3.2.2 What runs on the accelerator and what does not¶

Only the prompt is processed on CnuasGPU under stock settings. The backend declines any multiply whose batch dimension is below CNUAS_GGML_MIN_BATCH, which defaults to 32, the same shape test ggml-blas applies. Autoregressive decode is a sequence of n=1 multiplies and so stays on the CPU. Since that variable is filtered out too, only the default is reachable from Ollama, and the split is therefore fixed at prompt on the accelerator, decode on the host. This is the same division of labour ggml-blas produces, and it is a deliberate consequence of the round trip through device memory rather than an oversight.

3.3 vLLM¶

vLLM has a genuine out-of-tree plugin mechanism (the vllm.platform_plugins entry point, used by vllm-ascend, vllm-spyre, vllm-gaudi, vllm-openvino), and a PyTorch PrivateUse1 device is not formally required. Platform.dispatch_key defaults to "CPU", and vllm-openvino declares device_type = "cpu".

But a platform must supply attention backends, worker and model-runner integration, and collectives, all built around torch.Tensor. With an fp32-only operation set and no quantised or attention kernels, a CnuasGPU platform would execute nearly everything on Torch CPU and intercept a few fp32 linear layers. That is substantial integration risk for little benefit.

Recommendation: ggml/llama.cpp first, vLLM not yet.

4. The performance question¶

A soft GPU running on the same host CPU cannot beat that CPU's native inference path. It adds indirection over the same silicon.

Against native llama.cpp CPU inference:

ggml has optimised quantised matrix-vector and matrix-matrix kernels with AVX2, AVX-512, VNNI, BF16 and AMX paths. CnuasGPU has fp32 SGEMM.
Typical GGUF models are quantised; serving them from CnuasGPU means dequantising or keeping fp32 copies, which multiplies memory traffic.
Autoregressive decode is dominated by low-batch matrix-vector work. Even llama.cpp's own BLAS backend refuses small matrices, because BLAS wins only on large ones.
Graph splitting adds synchronisation boundaries around each offloaded region.

Expect at best parity on large prefill GEMMs, and materially slower token decode. If an SGEMM path does match, the credit belongs to the underlying microkernel, not to emulation.

There is no precedent to appeal to. Searches of current llama.cpp and vLLM for gem5, verilator or simulator return no backend or platform integration whose purpose is to execute through a hardware simulator.

The value of a CnuasGPU ggml backend is not speed. It is:

bringing up and testing an accelerator software stack before silicon exists;
conformance-testing a backend implementation against a reference, in CI, with no hardware;
giving a reproducible, inspectable target for teaching and for research on accelerator software.