Cnuas
Cnuas (pronounced Knoo-us, the Irish word for a cluster) is a fully software-emulated AI and HPC infrastructure. Virtual GPUs, virtual GPU-peer fabric, virtual InfiniBand and RoCE switches and virtual RDMA NICs, all built from scratch and developed the way the Linux kernel is, so unmodified Linux guests run real distributed workloads on simulated silicon.
cnuas is the unified control plane. A single command surface drives the switch
fabric, the GPU fabric, GPUs, RDMA NICs and VM lifecycle, and the same service layer is
exposed as a REST API for automation. Every CLI capability is also an API endpoint.
$ cnuas system health ┏━━━━━━━━━━━━━┳━━━━━━━━━━━┓ ┃ Component ┃ Available ┃ ┡━━━━━━━━━━━━━╇━━━━━━━━━━━┩ │ cnuasswitch │ yes │ │ cnuaslink │ yes │ │ cnuasgpu │ yes │ │ cnuasnic │ yes │ │ vm │ yes │ └─────────────┴───────────┘ $ cnuas switch set-mode 2 STRICT_IB { "status": "ok", "port": 2, "mode": "STRICT_IB" } $ cnuas vm up vm-a --gpus 1 vm-a running, CnuasNIC + CnuasGPU attached
Cnuas does not use host-based shortcuts like Linux bridges or Soft-RoCE. Instead it builds custom QEMU PCIe devices and their corresponding Linux kernel drivers, so the guest sees a real RDMA HCA and a real GPU on its PCIe bus. From the application point of view it is talking to hardware, even though every byte is moving through software.
cnuas_net.ko, cnuas_ib.ko) probe the device and register with ib_corelibibverbs works unmodified, so ibv_rc_pingpong, perftest and NCCL all runNot all developers, researchers and students can get access to a rack-scale GPU cluster, or to the cutting-edge AI/HPC GPU and accelerator platforms inside it. Existing Linux virt approaches (bridges, Soft-RoCE) hide the hardware semantics that matter, such as DCB, ECN marking, IB LIDs and GPU peer DMA. Cnuas keeps those semantics in software so what you learn here transfers directly to production-grade AI/HPC GPU and accelerator platforms.
Cnuas is built from four hardware components, each its own open-source project under PacketFive, plus a software stack that mirrors the NVIDIA developer experience. Cnuas ties them together as the integration and orchestration layer.
PCIe device model exposing standard BARs, doorbells and DMA queues. The companion
kernel drivers register a netdev and an RDMA HCA. Guest applications use libibverbs
as if it were Mellanox or NVIDIA hardware.
Standalone C daemon acting as a top-of-rack switch, running on the host with no guest. An
epoll event loop listens on
per-port UNIX sockets, classifies frames (Ethernet, RoCEv2 or InfiniBand LRH), and
forwards using a per-pipeline forwarding table over a JSON-over-UDS control plane.
SIMT accelerator with Streaming Multiprocessors, warp scheduling, shared memory and HBM-equivalent device memory, with the tensor MAC applied inside the matrix multiply rather than exposed as a separately addressable tile unit. It runs in two forms that execute the same kernels: a Soft-GPU inside an ordinary host process, with no hypervisor, guest or kernel module, and a QEMU PCIe device for guest-side and rack deployments. It is building toward a complete NVIDIA-equivalent userland. Today that means Cnuas Compute, the CUDA-equivalent runtime, cnuassmi, and a cnuascc compiler that lowers device source to CnuasIR for the scalar subset; vector code generation and CnuasCCL collectives are in progress.
Soft-GPU → Datasheet → PDF ↓Independent GPU-to-GPU interconnect, an NVSwitch equivalent, served by a host daemon that
needs no guest. A VM with a CnuasGPU but no
CnuasNIC can still do GPU peer DMA. It uses ivshmem as the fast path for
cross-VM shared memory.
CnuasGPU is building toward a complete NVIDIA-equivalent userland, so the same programming model and tooling students use here transfer directly to production GPUs. Each tool uses clean Cnuas naming and mirrors a familiar NVIDIA counterpart. Components marked Planned are specified in the design documents but carry no code in the tree yet, Partial means some parts ship and others are specified only, and every Shipping claim is backed by a named test in the Validation Matrix. See the Software Stack Datasheet for the full breakdown.
The CUDA-equivalent runtime stack. libcnuasrt.so mirrors the CUDA
Runtime and libcnuasdev.so the device API.
Device compiler that lowers GPU kernels to CnuasIR, the CnuasGPU intermediate representation. The front end and scalar code generation are implemented, so a kernel now compiles to an object; vector code generation and the Clang and LLVM pipeline are still to come.
nvcc + PTX PartialMulti-GPU collective communications, including AllReduce and Broadcast, running over the CnuasLink peer fabric with a RoCE fallback. v0.1 runs over the bootstrap channel.
NCCL PartialOne-sided GPU-to-GPU shared memory for partitioned workloads, using CnuasLink put and get as the transport.
NVSHMEM PlannedDevice management and monitoring CLI. It lists GPUs, reports utilisation and memory, and exposes realistic telemetry.
nvidia-smi ShippingFleet health checks and diagnostics, reading per-SM and chip-wide counters for cluster management.
DCGM PlannedKernel profiler that surfaces per-SM occupancy and chip-wide performance counters for tuning workloads.
Nsight Compute PlannedCnuasBLAS v0.1 ships as libcnuasblas.so, providing dense linear algebra over the
runtime kernels. CnuasDNN, CnuasFFT, CnuasSPARSE and CnuasSOLVER are specified only.
Exposes CnuasGPU devices to Docker and other container runtimes for reproducible GPU workloads.
nvidia-container-toolkit PlannedConfigure, set up and interact with the entire system from a scriptable CLI or a REST API. Both front ends call the same service layer, so nothing is CLI-only or API-only.
Interactive Rich tables by default, and --json on any
command for scripting. Command groups cover every domain of the rack.
cnuas switch, RoCE and InfiniBand fabric, 18 commandscnuas fabric, CnuasLink GPU peer fabriccnuas gpu, CnuasGPU device controlcnuas nic, CnuasNIC and RDMA device controlcnuas vm, virtual-machine lifecyclecnuas system, inventory, health and versionsA FastAPI service (cnuas api) with OpenAPI docs, so any
language or orchestrator can drive the platform over HTTP.
$ cnuas api --host 0.0.0.0 --port 8080 # OpenAPI docs at http://host:8080/docs $ curl host:8080/api/v1/switch/ports [{ "port": 0, "mode": "AUTO", "link": "up" }, { "port": 2, "mode": "STRICT_IB", "link": "up" }]
The reference deployment follows the Open Compute Project Open Rack v3, the rack design the modern AI buildout was created around. Two 1OU TOR switches (CnuasSwitch and CnuasLink) sit at the top of the rack, above a management host and eight 2OU VM blades acting as GPU compute nodes. Below them the rack carries what a real ORv3 carries: a pair of 1OU power shelves feeding the 48 V busbar, a 3OU battery backup shelf at the foot, and blanking panels over the bays this deployment does not use. The U-heights are visual in the emulator, but the layout maps 1:1 to real ORv3 deployments, making the same diagrams useful for teaching, design and real lab build-outs.
The CnuasSwitch fabric carries every Ethernet frame, including RoCEv2 and RDMA
on UDP 4791, the same path NCCL's IB transport, perftest and
ibv_rc_pingpong flow over. The CnuasLink fabric is independent
and carries only GPU-to-GPU peer traffic. A guest can use one, the other, or both,
exactly as in a real GPU rack.
Cnuas models a hyperscale AI campus as one artefact, from the silicon up to the site. The rack that runs the workload is emulated in software, and the campus that would house it is generated as an OpenUSD stage that opens in NVIDIA Isaac Sim or any USD viewer. Site electrical load is computed from the same racks the emulator runs, so a change in compute load moves the number on the grid connection.
The walkthrough film, the rack level data hall plan, the full power roll-up and the commands that regenerate all of it are on the simulation page.
The design documents and datasheets describe what Cnuas is. The guides describe how to install it, how to run it and how to extend it, and the roadmap records what is built, what is planned and what is blocked. Day-to-day work is tracked in public on the GitHub project board, so the roadmap tables, the project plan graphic and the open issues are the same set of items.
Size a host before installing. Cnuas has a host-only form that needs no virtualisation, no root and no guest kernel, and knowing which form you want saves most of the setup.
Requirements →Install the host-only or the rack-scale software, bring up QEMU, the ORV3 shelf and the datacentre simulation.
Install →Configure and operate Cnuas day to day: environment, CLI, REST, Redfish, the power shelf and the facility twin.
User guide →Build Cnuas with Bazel or Make, use the C and Python APIs, and meet the coding, testing and CI standards the tree is held to.
Developer guide →Add a new emulated component, either on the ORV3 RS-485 segment with its Modbus register map, or reached over Ethernet and IP.
Extend →Every workstream, its status and the open design decisions behind it. Epics and Tasks on the board link straight to the issues, pull requests and commits in the component repos. The project plan draws the same board as a roadmap graphic you can pan, zoom and download as a PDF.
Roadmap → Project plan → Project board ↗Every Cnuas component, hardware, software and firmware, has a product datasheet. Each one is generated from a single Markdown source into three editions, so the web page, the printable PDF and the plain-text file cannot drift apart. Document ids, revisions and statuses below are read from the datasheet build itself. Published means implemented and covered by tests, Partial that some parts ship and others are specified only, and Preview that the component is designed but carries no implementation yet.
| Document | Datasheet | Part | Status | Download |
|---|---|---|---|---|
| Hardware | ||||
DS-CNU-001 rev D |
Cnuas Platform | Platform |
Published | PDF · TXT |
DS-CNU-002 rev C |
CnuasSwitch | cnuas-vswitchd |
Published | PDF · TXT |
DS-CNU-003 rev C |
CnuasNIC | cnuas-vnic |
Published | PDF · TXT |
DS-CNU-004 rev G |
CnuasGPU | cnuasgpu |
Published | PDF · TXT |
DS-CNU-005 rev C |
CnuasLink | cnuasgpu-link-switchd |
Published | PDF · TXT |
| Software | ||||
DS-CNU-010 rev D |
Cnuas Software Stack | Software map |
Published | PDF · TXT |
DS-CNU-011 rev B |
CnuasRT | libcnuasrt.so |
Published | PDF · TXT |
DS-CNU-012 rev G |
CnuasDev | libcnuasdev.so |
Published | PDF · TXT |
DS-CNU-013 rev A |
Kernel Modules | cnuas_net.ko, cnuas_ib.ko, cnuasgpu.ko |
Published | PDF · TXT |
DS-CNU-014 rev A |
Verbs Provider | libcnuas-rdmav34.so |
Published | PDF · TXT |
DS-CNU-019 rev B |
Math Libraries | libcnuasblas.so and four specified |
Partial | PDF · TXT |
DS-CNU-020 rev A |
Control Plane | cnuas, cnuas-api |
Published | PDF · TXT |
DS-CNU-021 rev A |
Cnuas Tools | cnuas-tools |
Published | PDF · TXT |
DS-CNU-022 rev A |
Management Tools | cnuassmi, cnuas-cli, cnuaslink-cli |
Published | PDF · TXT |
| Firmware and facility | ||||
DS-CNU-031 rev C |
CnuasBMC | cnuas-rackmond |
Published | PDF · TXT |
DS-CNU-030 rev A |
Facility Twin | cnuas-facility |
Published | PDF · TXT |
| Preview stack | ||||
DS-CNU-015 rev C |
CnuasCC | models nvcc |
Partial | PDF · TXT |
DS-CNU-016 rev D |
CnuasIR | models PTX |
Partial | PDF · TXT |
DS-CNU-017 rev C |
CnuasCCL | models NCCL |
Partial | PDF · TXT |
DS-CNU-018 rev A |
CnuasSHMEM | models NVSHMEM |
Preview | PDF · TXT |
Cnuas is an open platform for infrastructure research and hands-on learning, and it is the environment HiCAIN courses run against. Students and researchers run real distributed-training workloads, RDMA microbenchmarks and MPI collectives against the emulated fabric, building the skills modern AI cluster operations demand, on a laptop.
No physical GPUs, NICs or switches required. A multi-node GPU cluster fits on a single workstation.
IB LIDs, GRH and LRH headers, DCB pause frames and ECN marking, all the things that matter for real production debugging.
Open-source under the Apache License 2.0, modifiable and reproducible. Ideal for academic networking and HPC research where commercial silicon is a black box.
Cnuas stands on the open-source stack the industry runs on. Compute and devices are
emulated with QEMU, the rack follows the Open Compute Project Open Rack v3, and the
out-of-band management plane integrates OpenBMC and Renode for realistic Baseboard
Management Controller behaviour, serving the DMTF
Redfish
1.17.0 API so real management clients drive the rack unmodified. The campus around
that rack is written as an
OpenUSD
stage, so it opens in NVIDIA Isaac Sim, usdview, Blender or any other
USD-capable tool. Beyond emulation, CnuasGPU targets the Microchip PolarFire SoC FPGA on
the Icicle Kit, a PCIe-attached board with hard RISC-V cores, so the same accelerator
stack can be validated on real silicon. The RTL for that fabric is in the tree and passes
simulation against host-computed references. Nothing is proven on the board yet, because
the PCIe endpoint is vendor IP and silicon, so the FPGA attachment is still in progress
rather than released.
Device and machine emulation
Open Rack v3 reference design
Firmware-accurate virtual BMC
Redfish 1.17.0, the DMTF standard the
BMC serves for out-of-band management
Campus written as an OpenUSD stage,
rendered and simulated in Isaac Sim
Deterministic BMC simulation
PolarFire SoC FPGA, the RISC-V and PCIe
target for CnuasGPU on real silicon
QEMU, Open Compute Project, OpenBMC, Renode, OpenUSD, NVIDIA, Isaac Sim, Microchip and PolarFire, and their logos, are trademarks of their respective owners. They are shown to indicate the technologies Cnuas builds on and interoperates with. Cnuas is not affiliated with, sponsored by or endorsed by these projects or companies. The "Built on OpenBMC" logo is by the OpenBMC Project, used under CC BY 4.0. The NVIDIA logo is reproduced unaltered from NVIDIA's logo and brand usage page. OpenUSD was created by Pixar and is governed by the Alliance for OpenUSD, of which NVIDIA is a founding member; it is grouped above because Cnuas uses it together with Isaac Sim, not because NVIDIA owns it.
Cnuas is a product of PacketFive, the trading name of Packet Five Networks Limited, a research and development company based in Dublin, Ireland. We build open infrastructure for the AI/HPC datacenter, spanning simulation, training and facility security.
Cnuas is developed in the open and is the environment HiCAIN training courses run against. For commercial enquiries, research collaboration or support, get in touch with the team.
Packet Five Networks Limited, 51 Bracken Road, Sandyford, Dublin D18 CV48, Ireland. CRO 764119.