Cnuas Installation Guide¶
This guide installs Cnuas on a Linux host. It covers two paths that are useful on their own and can be installed independently.
| Path | Installs | Needs a guest kernel |
|---|---|---|
| Host only | Soft-GPU, soft switch, control plane, facility twin | No |
| Rack scale | The emulated ORV3 rack under QEMU and Renode | Yes |
Read Minimum System Requirements first. The commonest installation failure is a host with too little RAM for the declared GPU device memory, which surfaces much later as a device that will not open.
1. Host prerequisites¶
Cnuas is developed and validated on Ubuntu 24.04. Other distributions work if the equivalent packages are present.
sudo apt update
sudo apt install -y build-essential git git-lfs pkg-config \
libjansson-dev libcurl4-openssl-dev libglib2.0-dev libpixman-1-dev \
ninja-build python3-venv python3-pip
Confirm the compiler and Python are new enough. Cnuas C code is C99 with GNU extensions and the Python packages need 3.11 or newer.
2. Getting the source¶
Cnuas is a superproject with submodules. The accelerator lives in its own repository and a clone without submodules will not build.
If a checkout already exists and the accelerator directory looks empty, the submodules were skipped. The same command repairs it.
3. Python environment¶
The control plane, the facility twin and the tooling package are Python. Use one virtual environment for all of them so that versions stay consistent.
Install the packages in editable mode from the repository.
Check that the entry points resolve. Each prints its own help.
The virtual environment must be active in every shell that runs these commands. Activating it in one terminal does not affect another.
4. Host-only installation¶
This is the whole of the Soft-GPU path. No QEMU, no guest, no root.
4.1 Build the accelerator libraries and tools¶
This produces the device library, the runtime, the BLAS and collective libraries, the compiler and the management tools.
| Component | Role |
|---|---|
libcnuasdev |
Device access, host arena and emulated device backends |
libcnuasrt |
Runtime, kernel launch and memory management |
libcnuasblas |
Dense linear algebra used by model execution |
libcnuasccl |
Collectives across devices |
cnuascc |
CnuasIR compiler |
cnuassmi |
Device status tool, the analogue of a vendor SMI tool |
4.2 Confirm the Soft-GPU works¶
cnuassmi opens the device and prints what it found. On a host with no
character device it announces that it is using the host arena, which is the
expected message for a host-only installation.
Expect a notice on stderr that no /dev/cnuasgpuN is present, followed by a
table showing the device and its memory usage. A table of zeroes means the
device failed to open.
4.3 Set the device memory¶
The default arena is 512 MiB, which is enough for the test suite and for small models. Larger models need a larger arena.
Accepted values are 1 to 1048576 MiB. A rejected value prints a notice and the default is used, so check stderr rather than assuming the setting took.
To require the emulated character device and refuse the host fallback, set the backend explicitly. This is useful in tests that must not silently pass on the wrong backend.
4.4 Build the soft switch and fabric¶
These are host daemons. They present a RoCE and InfiniBand switch, and the GPU-to-GPU fabric, without any kernel component on the host.
4.5 Run the host test suites¶
Everything above is covered by tests that need no guest.
make -C src/cnuasgpu/lib/test check
make -C src/cnuasgpu check
source .venv-cnuas/bin/activate && pytest tests
5. Rack-scale installation¶
The rack-scale path adds the emulated sled hardware, which needs a patched QEMU and a guest kernel with the Cnuas drivers.
5.1 Build QEMU¶
The Cnuas devices live in this repository's QEMU fork and are not in upstream QEMU. A distribution QEMU will not have them.
cd qemu
mkdir -p build && cd build
../configure --target-list=x86_64-softmmu --enable-kvm --enable-slirp \
--disable-docs --disable-werror
ninja
cd ../..
Confirm the devices are present. If they are missing the build did not pick up the Cnuas configuration.
If the binary predates a change to the device sources, rerun ninja. An
incremental rebuild takes about a minute.
5.2 Build the guest kernel¶
5.3 Build the guest kernel modules¶
These modules load inside the guest, not on the host. There is no host kernel module in Cnuas.
5.4 Build the golden VM image¶
The tooling builds a reproducible guest image containing the kernel, the modules and the userspace.
The image is what the VM lifecycle commands boot. Without it the guest tests skip rather than fail.
5.5 Start a sled¶
To attach an accelerator with a specific device memory size, pass the property through to QEMU. The size must be a power of two between 16 MiB and 1 TiB.
Remember that the guest needs its own RAM in addition to this. An 8 GiB device on a 4 GiB guest asks the host for roughly 14 GiB once overheads are counted.
6. ORV3 power shelf¶
The power shelf is emulated in Renode rather than QEMU, because the shelf is an RS-485 multi-drop segment and QEMU has no model for one.
6.1 Install Renode¶
Install Renode from the upstream project release for the host distribution. Renode needs Mono or .NET.
6.2 Build the satellite firmware¶
The firmware is shared by the PSU and BBU personalities and runs on Cortex-M.
Cross-building needs arm-none-eabi-gcc.
The same sources build a host test binary that needs no cross toolchain, which is what CI runs.
6.3 Start the shelf¶
This brings up six PSUs at Modbus unit addresses 0xC0 to 0xC5 and six BBUs at 0x40 to 0x45, all on one RS-485 segment, with a leg of that segment terminated in a TCP socket on port 3485 for the rack management controller to attach to.
6.4 Attach the OpenBMC rack management controller¶
The management controller runs OpenBMC on an emulated AST2600 under QEMU and reaches the shelf by pointing a serial port at the RS-485 bridge socket.
From inside OpenBMC the shelf is read by rackmon over Modbus, exactly as it
would be on real ORV3 hardware.
7. Datacentre simulation¶
The facility twin generates an OpenUSD campus and can be driven from live power measurements. It is part of the host-only path and needs no guest.
7.1 Generate the campus¶
rooms prints the room programme, which is the list of data halls, telecom
rooms, electrical rooms and support space. power prints the power roll-up
from the rack count and per-rack draw.
7.2 Drive it from the running shelf¶
With the ORV3 shelf running, the twin can consume real measurements rather than a static estimate.
This is the path that connects a rack drawing emulated watts to the facility power figure, so the campus number moves when the emulated load moves.
7.3 Viewing the stage¶
The output is an OpenUSD stage. Any USD-capable viewer opens it, including
usdview and NVIDIA Omniverse. Cnuas does not require a viewer to be installed
and does not ship one.
8. Verifying the installation¶
Run the checks that match the path installed. Each is fast and each fails loudly rather than skipping.
# Host only
make -C src/cnuasgpu/lib/test check
source .venv-cnuas/bin/activate && pytest tests
# ORV3 firmware
make -C bmc/firmware/orv3 test
# Documentation and datasheets, if editing them
python scripts/check-doc-format.py
python scripts/datasheet/build.py
The Validation Matrix records which of these suites covers which claim, and how many cases each contributes.
9. Common problems¶
| Symptom | Cause | Fix |
|---|---|---|
cnuassmi prints a zeroed row |
The device failed to open | Lower CNUAS_HOST_ARENA_MB or check stderr |
| Arena silently smaller than requested | Value rejected | Read the stderr notice, use 1 to 1048576 MiB |
-device help shows no Cnuas devices |
Stale or wrong QEMU | Rerun ninja in qemu/build |
| Guest cannot map BAR1 | Old build without the 64-bit BAR | Rebuild QEMU |
| Device realise fails on size | Not a power of two, or out of range | Use a power of two from 16 MiB to 1 TiB |
cnuas command not found |
Virtual environment not active | source .venv-cnuas/bin/activate |
| Submodule directories empty | Cloned without submodules | git submodule update --init --recursive |
| Guest tests skip | No golden image | cnuas-tools image build |
10. Related reading¶
| Document | Covers |
|---|---|
| Minimum System Requirements | Sizing a host |
| User Guide | Running and configuring Cnuas |
| Developer Guide | Building, APIs, CI |
| Extending Cnuas | Adding emulated hardware |
| Build System | Bazel targets and packaging |
| Deployment | Installing from released artefacts |
| RoCE and InfiniBand How-To | Fabric bring-up in detail |