Cnuas User Guide¶

This guide is for running Cnuas. It assumes the software is installed as described in the Installation Guide.

Cnuas presents one control plane over everything it emulates, whether that is a single Soft-GPU on a laptop or a two-sled ORV3 rack with a power shelf. The same commands work in both cases and report what is actually present rather than what a configuration file claims.

1. Concepts¶

Term	Meaning
Soft-GPU	The accelerator running as a host userspace library
Emulated device	The same accelerator as a QEMU PCI device inside a guest
Sled	One emulated compute node, a guest VM with accelerators and a NIC
Rack	Sleds, the fabric, the power shelf and the management controller
Fabric	CnuasLink, the accelerator to accelerator interconnect
Switch	CnuasSwitch, the RoCE and InfiniBand network switch
RMC	The rack management controller, OpenBMC on an emulated AST2600
Facility twin	The OpenUSD campus model driven by rack power

A component that can run without a guest can be used on its own. The Soft-GPU, the switch, the fabric and the facility twin are all usable with no VM at all.

2. Configuration¶

The control plane is configured entirely from the environment. There is no configuration file to edit and no state hidden in one. The same binary behaves the same on a workstation, a CI agent and inside a lab VM.

2.1 Control-plane settings¶

Variable	Default	Controls
`CNUAS_RUN_DIR`	`/var/run/cnuas`	Base directory for the switch socket
`CNUAS_SWITCH_SOCKET`	`$CNUAS_RUN_DIR/mgmt.sock`	Switch management socket
`CNUASLINK_RUN_DIR`	`/var/run/cnuaslink`	Base directory for the fabric socket
`CNUAS_CNUASLINK_SOCKET`	`$CNUASLINK_RUN_DIR/mgmt.sock`	Fabric management socket
`CNUAS_GPU_SYSFS`	`/sys/class/cnuasgpu`	Accelerator sysfs class root
`CNUAS_GPU_DEV_DIR`	`/dev`	Where accelerator device nodes are looked for
`CNUAS_NIC_SYSFS`	`/sys/class/infiniband`	RDMA device sysfs root
`CNUAS_VM_TOOL`	`cnuas-tools vm`	Command used to drive VM lifecycle
`CNUAS_DATA_DIR`	`$XDG_DATA_HOME/cnuas`	Per-user state directory
`CNUAS_VMS_DIR`	`$CNUAS_DATA_DIR/vms`	Per-VM state
`CNUAS_API_HOST`	`0.0.0.0`	REST API bind address
`CNUAS_API_PORT`	`8080`	REST API port
`CNUAS_TIMEOUT`	`5.0`	Socket and subprocess timeout in seconds

2.2 Sled BMC settings¶

A blade listed here is powered through its BMC using Redfish and is never started behind the BMC's back, so the emulation stays accurate about who owns power state.

Variable	Default	Controls
`CNUAS_SLEDS`	empty	Blades as `name=url,name=url`
`CNUAS_SLED_USER`	`root`	Redfish user
`CNUAS_SLED_PASSWORD`	`0penBmc`	Redfish password
`CNUAS_SLED_VERIFY_TLS`	`false`	Verify the BMC certificate

TLS verification is off by default because a BMC generates a self-signed certificate on first boot. Turn it on where a real chain is installed.

export CNUAS_SLEDS="sled0=https://127.0.0.1:2444,sled1=https://127.0.0.1:2445"

A malformed entry in CNUAS_SLEDS is dropped rather than raising, so one typo cannot stop the control plane from reporting on every other machine.

2.3 Accelerator settings¶

Variable	Default	Controls
`CNUAS_HOST_ARENA_MB`	`512`	Soft-GPU device memory in MiB, 1 to 1048576
`CNUAS_DEVICE_BACKEND`	automatic	`host` or `device`, to pin the backend

Left unset, the library uses the emulated character device when one is present and the host arena when one is not, announcing the fallback once on stderr. Set CNUAS_DEVICE_BACKEND=device in a test that must not pass on the wrong backend.

3. The command line¶

cnuas is the single entry point. Every subcommand accepts --json for machine-readable output, which is the form to use in scripts.

Command	Covers
`cnuas system`	Inventory, health and versions
`cnuas gpu`	Accelerator device control
`cnuas nic`	NIC and RDMA device control
`cnuas switch`	RoCE and InfiniBand fabric control
`cnuas fabric`	CnuasLink accelerator fabric control
`cnuas vm`	Virtual machine lifecycle
`cnuas sled`	Sled BMC host power control
`cnuas api`	Run the REST API service
`cnuas version`	Control-plane version

3.1 Finding out what is present¶

cnuas system health
cnuas system inventory
cnuas system versions
cnuas --json system inventory | jq .

health probes every component and reports reachability. inventory enumerates what the host actually has. On a host-only installation it reports the Soft-GPU and no VMs, which is correct rather than an error.

3.2 Accelerators¶

cnuas gpu list
cnuassmi

cnuassmi is the accelerator status tool and is the analogue of a vendor SMI utility. It prints the backend in use, the device memory mapped and the memory in use. It works on the host against the Soft-GPU and inside a guest against the emulated device, with the same output format.

A row of zeroes means the device did not open. The usual cause is an arena that could not be mapped, and the reason is on stderr.

3.3 Switch and fabric¶

cnuas switch ports
cnuas switch telemetry
cnuas fabric ports
cnuas fabric telemetry

Port mode, link state, priority flow control, explicit congestion notification and transmission selection are all settable. The detailed bring-up sequence for RoCE and InfiniBand traffic is in the RoCE and InfiniBand How-To.

3.4 Virtual machines¶

cnuas vm list
cnuas vm status sled0
cnuas vm up sled0 --gpus 1
cnuas vm down sled0
cnuas vm lab

cnuas vm lab brings up the default two-VM lab, which is the two-sled rack the rest of the documentation refers to. To reach a guest shell, use the tooling package.

cnuas-tools vm ssh sled0

Where a VM is a blade behind a sled BMC, power it through the BMC instead, so that the BMC's view of power state stays true.

cnuas sled list
cnuas sled power sled0
cnuas vm reset sled0 --reset-type GracefulShutdown

4. REST API¶

The same operations are available over HTTP. The service is the control plane, not a separate implementation, so the CLI and the API cannot disagree.

cnuas api --host 127.0.0.1 --port 8080

Path	Purpose
`/health`	Liveness
`/api/v1/system/inventory`	What the host has
`/api/v1/system/versions`	Component versions
`/api/v1/switch/ports`	Switch ports
`/api/v1/switch/ports/{port}/mode`	Set port mode
`/api/v1/switch/fdb`	Ethernet forwarding database
`/api/v1/switch/lft`	InfiniBand linear forwarding table
`/api/v1/switch/sm`	Subnet manager state
`/api/v1/fabric/ports`	CnuasLink ports
`/api/v1/fabric/gpu-fdb`	Accelerator fabric forwarding
`/api/v1/fabric/telemetry`	Fabric counters

Interactive documentation is served from the running service and is also built into the documentation site as Swagger UI and ReDoc.

curl -s http://127.0.0.1:8080/api/v1/system/inventory | jq .

Errors use one model across the CLI and the API, so a failure reported by one is recognisable from the other. The model is described in Programmability.

5. Redfish¶

Cnuas is a Redfish client, not a Redfish server. The Redfish service is provided by OpenBMC running on the emulated management hardware, which is what makes the interaction representative of a real rack.

Resource	Used for
`/redfish/v1/Systems/system`	Reading blade power state
`/redfish/v1/Systems/system/Actions/ComputerSystem.Reset`	Power actions

Supported ResetType values are On, ForceOff, GracefulShutdown, ForceRestart, GracefulRestart and PowerCycle. The default is ForceRestart. An unsupported value is rejected with the list of accepted ones rather than being passed through to the BMC.

cnuas vm reset sled0 --reset-type On
cnuas vm reset sled0 --reset-type GracefulShutdown

The BMC can also be driven directly, which is useful when checking whether a problem is in Cnuas or in the BMC.

curl -sk -u root:0penBmc https://127.0.0.1:2444/redfish/v1/Systems/system | jq .PowerState

6. The ORV3 power shelf¶

The power shelf is the part of the rack that is not a computer. Six power supply units and six battery backup units sit on one RS-485 multi-drop segment and answer Modbus requests from the management controller.

renode bmc/renode/scripts/cnuas-orv3-power-shelf.resc

Node	Unit addresses
PSU, slots 0 to 5	0xC0 to 0xC5
BBU, slots 0 to 5	0x40 to 0x45

Addresses follow the ORV3 scheme, two type bits then three rack bits then three device bits. Every node hears every frame and only the node whose address matches answers, which is what makes multi-drop work.

6.1 Changing the load while it runs¶

The rack load is not a strap. It is re-read by the firmware on every plant tick, so it can be changed on a running shelf and the reported watts move.

mach set "psu0"; sysbus WriteDoubleWord 0x30000010 30000

The value is milliamps. The shelf starts at 21000, which is roughly a 6 kW rack on a 48 V busbar shared across six supplies in a five plus one shelf, so it comes up carrying a believable idle load rather than reporting zero watts.

To drive all twelve nodes from a profile rather than by hand, use the load driver.

renode-test bmc/renode/tests/orv3_load_driver.py

6.2 Reading it from OpenBMC¶

Inside OpenBMC the shelf is read by rackmon over Modbus, exactly as on real ORV3 hardware. The management controller reaches the segment through the TCP bridge on port 3485.

7. Datacentre simulation¶

The facility twin models a hyperscale campus and can be driven from the power the emulated rack is actually drawing.

7.1 Inspecting the design¶

cnuas-facility rooms
cnuas-facility power

rooms prints the room programme building by building, including data halls, horizontal distribution areas, coolant distribution rooms and support space. power prints the roll-up from rack count and per-rack draw, which is how a campus figure in the hundreds of megawatts is arrived at from a per-rack number.

7.2 Generating the stage¶

cnuas-facility build --out out/campus.usda

Use a .usda suffix for text, which is diffable and reviewable. Use .usdc for binary, which is faster to load. The choice is made by the suffix.

7.3 Driving it from live measurements¶

With the ORV3 shelf running, the twin consumes real measurements instead of a static estimate.

cnuas-facility live --stage out/campus.usda --duration 60 --interval 1 --fps 24

Option	Meaning
`--stage`	Stage to read
`--out`	Write here instead of over the input stage
`--duration`	Seconds to sample for
`--interval`	Seconds between samples
`--fps`	USD time codes per second

This is the path that ties an emulated rack drawing emulated watts to a facility power figure. Change the load on a PSU and the campus number moves.

The design behind the model is in Facility Twin Design and the walkthrough is in Datacentre Simulation.

8. Running a model on the Soft-GPU¶

The accelerator is usable without any guest. Size the arena for the model first, since the default 512 MiB suits the test suite rather than a model.

export CNUAS_HOST_ARENA_MB=16384
cnuassmi

Confirm the reported device memory matches what was asked for. If it does not, the value was rejected and the reason is on stderr.

Cnuas makes no claim about how fast this is. Speed is not a goal of the current phase. The goal is that the interfaces behave the way accelerator interfaces behave, so that software written against them is exercised faithfully.

9. Troubleshooting¶

Symptom	Likely cause	Action
`cnuassmi` shows zeroes	Device failed to open	Read stderr, lower the arena
Arena smaller than requested	Value rejected	Use 1 to 1048576 MiB
`cnuas switch` cannot connect	Daemon not running or wrong socket	Check `CNUAS_SWITCH_SOCKET`
`cnuas fabric` cannot connect	Fabric daemon not running	Check `CNUAS_CNUASLINK_SOCKET`
`cnuas gpu list` empty in a guest	Driver not loaded	Load the guest modules
Sled power action refused	Unsupported `ResetType`	Use one of the six documented values
Redfish TLS failure	Self-signed BMC certificate	Leave `CNUAS_SLED_VERIFY_TLS` off
Shelf reports zero watts	Load not seeded	Write the load word, see section 6.1
Campus power does not move	Twin not driven live	Use `cnuas-facility live`

Document	Covers
Minimum System Requirements	Sizing a host
Installation Guide	Getting Cnuas onto a host
Developer Guide	Building, APIs, CI
Programmability	CLI and REST reference
RoCE and InfiniBand How-To	Fabric bring-up
OCP Rack Management	BMC and Redfish design
Datacentre Simulation	The campus model