Cnuas Developer Guide¶

This guide is for people changing Cnuas rather than running it. It covers the build system, the programming interfaces, the coding standards the tree actually enforces, continuous integration, and the practices that keep the emulation accurate.

For adding a new emulated hardware component, see Extending Cnuas, which walks through the case the architecture is built around.

1. Repository layout¶

Cnuas is a superproject with submodules. A clone without submodules will not build.

Path	Contains
`src/cnuasgpu`	Accelerator, its own repository, libraries, compiler and tools
`src/cnuasnic`	NIC device, guest kernel module and userspace
`src/cnuasswitch`	RoCE and InfiniBand switch daemon and CLI
`src/cnuaslink`	Accelerator to accelerator fabric
`src/driver`	Guest kernel drivers
`src/webui`	Web interface
`qemu`	QEMU fork carrying the Cnuas PCI devices
`linux`	Guest kernel tree
`bmc`	ORV3 satellite firmware, Renode platforms, rack management controller
`cnuas`	Control plane, Python
`tools`	Packaging, image and VM lifecycle tooling, Python
`facility`	OpenUSD facility twin, Python
`docs`	Documentation and datasheet sources
`tests`	Superproject Python tests
`packaging`	Debian packaging
`upstream`	Material for upstream submissions

2. Build system¶

Cnuas is mid-migration to Bazel. Some components build with Bazel today and some still use Make. Both are supported and neither is deprecated yet.

2.1 Bazel¶

bazel build //...
bazel build //:all-binaries
bazel build //:all-debs
bazel test //tests:switch_tests

2.2 Make¶

The accelerator, the ORV3 firmware and the guest kernel modules use Make.

make -C src/cnuasgpu            # libraries, tools and driver
make -C src/cnuasgpu check      # accelerator test suites
make -C bmc/firmware/orv3 test  # ORV3 host unit tests
make -C bmc/firmware/orv3 firmware

2.3 QEMU¶

The Cnuas PCI devices are in this tree's QEMU fork, not upstream QEMU.

cd qemu/build && ninja
./qemu-system-x86_64 -device help | grep -i cnuas

A binary in qemu/build older than the device sources silently lacks the device. Check the timestamps before concluding that a device change had no effect.

2.4 Python¶

source .venv-cnuas/bin/activate
pip install -e ./cnuas -e ./facility -e ./tools
pytest tests

Run pytest tests rather than bare pytest. The latter picks up vendored trees.

Full detail is in Build System.

3. Programming interfaces¶

3.1 Device API, `libcnuasdev`¶

The lowest layer. It opens a device, allocates device memory, moves data and drives the fabric. It is the layer that hides whether the accelerator is the emulated PCI device or the host Soft-GPU, and it is where a new backend is added.

Function	Purpose
`cnuasdev_open`	Open device by index
`cnuasdev_query_info`	Fill a `struct cnuasdev_device_info`
`cnuasdev_alloc`	Allocate device memory, returns a device offset
`cnuasdev_free`	Release device memory
`cnuasdev_memcpy`	Copy between host and device
`cnuasdev_mmap`	Map device memory into the process
`cnuasdev_munmap`	Unmap
`cnuasdev_link_id`	Read the fabric identifier
`cnuasdev_link_status`	Read fabric link state
`cnuasdev_link_send_raw`	Send a raw fabric frame

Device memory is addressed by uint64_t offset, not by host pointer, so buffers past 4 GiB address correctly on both backends.

Errors are negative values of cnuasdev_status_t, covering invalid arguments, no device, input and output failure, out of memory, permission, unsupported, try again, timed out, message size and not connected.

3.2 Runtime API, `libcnuasrt`¶

The runtime mirrors the shape of a vendor accelerator runtime, so that code written against one is recognisable here.

Function	Purpose
`cnuasInit`, `cnuasShutdown`	Runtime lifecycle
`cnuasGetDeviceCount`, `cnuasSetDevice`, `cnuasGetDevice`	Device selection
`cnuasGetDeviceProperties`	Capabilities of a device
`cnuasMalloc`, `cnuasFree`	Device allocation
`cnuasMemcpy`	Transfers
`cnuasDeviceSynchronize`	Completion
`cnuasSgemm`, `cnuasSgemmNT`	Matrix multiply, normal and transposed
`cnuasModuleLoad`, `cnuasModuleUnload`	CnuasIR modules
`cnuasLaunchKernel`	Kernel launch
`cnuasGetErrorString`	Error text

Capabilities are reported rather than assumed. Query cnuasGetDeviceProperties and branch on what is present. Do not infer a capability from the presence of a device.

3.3 BLAS API, `libcnuasblas`¶

Naming and calling convention mirror cuBLAS with a cnuasblas prefix. cnuasblasCreate does not call cnuasInit; the caller owns runtime lifecycle.

Function	Purpose
`cnuasblasCreate`, `cnuasblasDestroy`	Handle lifecycle
`cnuasblasSscal`, `cnuasblasSaxpy`	Vector scaling and update
`cnuasblasSdot`, `cnuasblasSnrm2`	Reductions
`cnuasblasSgemv`	Matrix vector product
`cnuasblasSgemm`	Matrix matrix product
`cnuasblasGetStatusString`	Error text

3.4 Collectives API, `libcnuasccl`¶

Function	Purpose
`cnuasccl_comm_init_rank`, `cnuasccl_comm_destroy`	Communicator lifecycle
`cnuasccl_comm_rank`, `cnuasccl_comm_size`	Membership
`cnuasccl_comm_gpu_of`, `cnuasccl_comm_rank_of`	Rank and device mapping
`cnuasccl_comm_ring_next`, `cnuasccl_comm_ring_prev`	Ring topology
`cnuasccl_broadcast`, `cnuasccl_all_reduce`	Collectives
`cnuasccl_type_size`, `cnuasccl_status_str`	Helpers

Rank bootstrap is described in CnuasCCL Rank Bootstrap.

3.5 Python API¶

The control plane is importable, not only a command line tool. The CLI and the REST service are two front ends over the same service layer, which is why they cannot disagree.

Module	Role
`cnuas.config`	`Settings`, resolved from the environment
`cnuas.service`	Operations, the single source of behaviour
`cnuas.adapters`	One adapter per component
`cnuas.models`	Request and response models
`cnuas.errors`	`CnuasError` and its subclasses
`cnuas.api.app`	The REST application
`cnuas.cli`	The command line front end

from cnuas.config import Settings
from cnuas.service import Service

service = Service(Settings.from_env())
print(service.system_inventory())

Adapters exist for the switch, the fabric, the accelerator, the NIC, VMs, sleds and the system as a whole. A new component gets a new adapter and is then reachable from both front ends without further work.

3.6 REST API¶

Routes are declared in cnuas/src/cnuas/api/app.py under an /api/v1 prefix. CnuasError is mapped to an HTTP response by a single exception handler, so error shape is uniform.

Adding an endpoint means adding a service method, then a route that calls it. Do not put behaviour in the route.

3.7 Redfish¶

Cnuas consumes Redfish, it does not serve it. The service is OpenBMC on the emulated management hardware. The client lives in cnuas/src/cnuas/adapters/ sled.py and uses /redfish/v1/Systems/system and the ComputerSystem.Reset action.

Keeping Cnuas on the client side is deliberate. A Redfish implementation written specially for the emulator would be tested against itself and would prove nothing about real rack management software.

4. Coding standards¶

These are the standards the tree enforces mechanically. A change that violates one fails a build or a check rather than a review.

4.1 C¶

Rule	Enforced by
C99	`-std=c99`
No warnings	`-Wall -Wextra -Werror`
Freestanding for firmware	`-ffreestanding -fno-builtin`
Portable core, no MCU or host assumptions	Host unit tests build the same sources

The ORV3 firmware sources build twice, once for Cortex-M and once natively for the host tests. Anything that cannot build both ways belongs in the target directory, not in the shared core.

4.2 Python¶

Rule	Value
Formatter	`black`, line length 100
Target version	Python 3.11
Type hints	Used throughout, `from __future__ import annotations`
Test runner	`pytest`, `testpaths = ["tests"]`

4.3 Documentation¶

Documentation is checked, not merely reviewed.

python scripts/check-doc-format.py
python scripts/datasheet/build.py
mkdocs build --strict

Rule	Reason
No em dash or en dash, no double hyphen standing in for one	House style
No heading of the form word, colon, sentence	House style
No bold label followed by a colon	Use a table instead
A discouraged word list is rejected	House style

Code fences, inline code, link targets and HTML comments are exempt, so command lines and identifiers do not trip the checker.

Datasheet PDFs embed the git hash. Build the documentation site after committing, otherwise the published artefact is labelled dirty.

4.4 Commits¶

A commit hook is available in the tree and is opt in, because attribution must be truthful.

git config core.hooksPath scripts/git-hooks

Convention	Value
Subject	Short, imperative, prefixed by area
Body	Wrapped at 72 columns
Sign off	`git commit -s`

Verify trailers landed with git log -1 --format='%(trailers)'. A trailer written without a blank line above it is silently not a trailer.

5. Testing¶

5.1 What to run¶

Suite	Command	Needs a guest
Accelerator libraries	`make -C src/cnuasgpu/lib/test check`	No
Accelerator overall	`make -C src/cnuasgpu check`	No
ORV3 firmware	`make -C bmc/firmware/orv3 test`	No
Control plane and tooling	`pytest tests`	Partly
Switch	`bazel test //tests:switch_tests`	No
Facility twin	`pytest facility/tests`	No
Documentation	`scripts/check-doc-format.py`	No

The majority of Cnuas is testable with no VM at all. This is deliberate. A suite that needs a guest cannot run on a CI agent without nested virtualisation, so behaviour that can be covered without one is.

Guest tests skip rather than fail when the golden image is absent. A skip is not a pass. Check the skip count when a change should have been exercised.

5.2 What a test should assert¶

Cnuas emulates hardware, so the failure mode that matters is an emulation that is wrong in a way nothing notices. Tests should assert against an independent authority wherever one exists.

Area	Authority used
CnuasIR encoding	RISC-V binutils, as an oracle
ORV3 registers	The vendored rackmon register maps
Redfish	A real OpenBMC service
Verbs	The rdma-core provider and its tools

A test that only checks the emulator against itself proves that the code does what it does.

5.3 Validation matrix¶

Validation Matrix records which suite covers which claim and how many cases each contributes. A change to the number of tests changes that document, and the paper cites those numbers, so the two move together.

6. Continuous integration¶

6.1 Azure DevOps¶

Azure DevOps is what Cnuas runs today. The pipeline is azure-pipelines.yml at the repository root, triggered on main, on an ubuntu-24.04 image.

Job	Covers
Switch and packages	Bazel binaries and Debian packages
ORV3 firmware	Register map sync, unit tests, Cortex-M build
Facility twin	Campus planner, power model, USD emission
Document formatting	The formatting policy checker
Datasheets and site	Branded PDFs and a strict MkDocs build
CnuasIR toolchain	Packer, validator and disassembler against binutils
Host libraries	Compute, CnuasIR, CnuasBLAS, CnuasLink, CnuasCCL

No job needs nested virtualisation, which is what allows the pipeline to run on hosted agents. The agent and self-hosted strategy, including the FPGA agent with PCIe passthrough planned for a later phase, is in CI Strategy.

6.2 GitHub Actions¶

Cnuas does not currently ship a GitHub Actions workflow, so nothing below is running today. It is reference material for adding one. The jobs in section 6.1 map onto Actions without change, since none of them needs a special agent. Mirror the Azure DevOps job names so results are comparable.

Every step here uses a command from section 5.1 that runs on an ordinary hosted agent with no nested virtualisation.

name: cnuas
on:
  push:
    branches: [main]
  pull_request:

jobs:
  host-libraries:
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/checkout@v4
        with:
          submodules: recursive
      - run: sudo apt-get update -q
      - run: sudo apt-get install -y -q libjansson-dev libcurl4-openssl-dev
      - run: make -C src/cnuasgpu/lib/test check

  orv3-firmware:
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/checkout@v4
        with:
          submodules: recursive
      - run: make -C bmc/firmware/orv3 check-regmap
      - run: make -C bmc/firmware/orv3 test
      - run: sudo apt-get update -q && sudo apt-get install -y -q gcc-arm-none-eabi
      - run: make -C bmc/firmware/orv3 firmware

  control-plane:
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/checkout@v4
        with:
          submodules: recursive
      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'
          cache: pip
      - run: pip install -e ./cnuas -e ./facility -e ./tools
      - run: pytest tests

  docs:
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/checkout@v4
        with:
          submodules: recursive
      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'
          cache: pip
      - run: |
          sudo apt-get update -q
          sudo apt-get install -y -q pandoc texlive-xetex texlive-fonts-recommended
          pip install mkdocs mkdocs-material pymdown-extensions
      - run: python scripts/check-doc-format.py
      # Datasheet PDFs must exist before mkdocs runs. docs/datasheets/index.md
      # links each sheet as ../pdf/<Name>_Datasheet.pdf and the hook registers
      # those from out/datasheets/, so a strict build with no PDFs aborts with
      # one broken link per sheet.
      - run: python scripts/datasheet/build.py
      - run: mkdocs build --strict

submodules: recursive is not optional. A checkout without it leaves the accelerator directory empty and the accelerator job fails in a way that looks like a build error rather than a missing checkout.

Cache the pip directory and, if Bazel jobs are added, the Bazel repository directory. Neither depends on the change under test.

The datasheet build must run before the strict site build. This is not a preference. Datasheet links are validated against the MkDocs file set, so with no PDFs present a strict run aborts with one broken link per sheet.

6.3 Jenkins¶

Cnuas does not currently ship a Jenkinsfile, so this too is reference material. Jenkins is the right choice where the FPGA agent and hardware-attached runners of a later phase are involved, because those cannot be hosted agents. Stage names should match the Azure DevOps job names.

pipeline {
    agent { label 'ubuntu-24.04' }

    stages {
        stage('Checkout') {
            steps {
                checkout scm
                sh 'git submodule update --init --recursive'
            }
        }
        stage('Host Libraries') {
            steps { sh 'make -C src/cnuasgpu/lib/test check' }
        }
        stage('ORV3 Firmware') {
            steps {
                sh 'make -C bmc/firmware/orv3 check-regmap'
                sh 'make -C bmc/firmware/orv3 test'
            }
        }
        stage('Control Plane') {
            steps {
                sh '''
                    python3 -m venv .venv-ci
                    . .venv-ci/bin/activate
                    pip install -e ./cnuas -e ./facility -e ./tools
                    pytest tests
                '''
            }
        }
        stage('Documentation') {
            steps {
                sh '''
                    . .venv-ci/bin/activate
                    python scripts/check-doc-format.py
                    python scripts/datasheet/build.py
                    mkdocs build --strict
                '''
            }
        }
    }
}

Each sh step is a separate shell, so a virtual environment activated in one step is not active in the next. Activate it inside every step that needs it, as above. This is the most common cause of a Jenkins stage failing with a command not found while the same commands work by hand.

6.4 Rules that apply to any of the three¶

Rule	Reason
Submodules must be checked out for accelerator jobs	A shallow clone silently skips them
Documentation jobs run the format checker and a strict site build	Both fail on real problems
A guest-dependent job must fail on skip, not pass	A skip is not evidence
Validation counts must be regenerated, not hand-edited	The paper cites them

7. Practices¶

The project relies on the following habits.

Practice	Why
Execute a claim before writing it down	Plausible commands are often wrong
Report what was found, not what was expected	An emulator that flatters itself is useless
Make a fallback announce itself once	A silent fallback hides its cause
Fail loudly on an unusable configuration	Better than a device that realises and cannot be mapped
State a specification separately from an implementation	A specified opcode is not a working one
Prefer an independent oracle in tests	Self-agreement proves nothing
Rebuild before testing a device change	A stale binary shows the old behaviour
Check timestamps when a change appears to have no effect	Usually a stale build

Cnuas makes no speed or performance claims. Performance is not a goal of the current phase, and a change should not be justified by it.

Document	Covers
Extending Cnuas	Adding an emulated component
Installation Guide	Toolchain setup
User Guide	Runtime behaviour
Build System	Bazel targets and packaging
CI Strategy	Agents, caching, branch policy
Programmability	CLI and REST reference
Driver and Userspace	Guest driver design
Validation Matrix	Evidence for each claim
Upstreaming	Identifier registration and submissions