Cnuas OCP Rack Management Design¶

1. Overview¶

Cnuas today emulates the data plane of an Open Compute Project Open Rack v3 (ORv3): the CnuasSwitch RoCE/IB TOR, the CnuasLink GPU peer fabric, and eight VM blades carrying CnuasNIC and CnuasGPU, all wired over UNIX-domain sockets. What it does not yet have is the out-of-band management plane that turns a pile of servers into a manageable rack, the Baseboard Management Controllers (BMCs), the Rack Management Controller (RMC), and the isolated management network that ties them together.

This document proposes that management plane. The goal is a complete OCP Virtual Rack that a student or engineer can run on a laptop and manage exactly the way a datacenter operator manages real hardware, over Redfish and IPMI, on a dedicated management network, with per-sled power control, sensors, serial console, and firmware update flows.

The design keeps the property that makes Cnuas useful in the first place. A full rack, compute plus management, still fits on one workstation.

Real world	Cnuas
DC-SCM BMC (ASPEED AST2600) per sled	OpenBMC on QEMU `ast2600`, mock agent as fallback
Switch management controller	Per-switch BMC (OpenBMC or mock agent)
Rack Management Controller on power shelf	RMC (OpenBMC or mock agent)
Battery system and power-shelf microcontrollers	Real firmware in Renode over RS-485 or PMBus
1GbE out-of-band management switch	Isolated management bridge
Redfish aggregator / datacenter orchestrator	`cnuas` CLI and REST API

2. Why the management plane is separate¶

In a real ORv3 rack the BMC does not run on the top-of-rack switch, and there is no single central controller that owns everything. Management is distributed across three levels, and the out-of-band network is physically isolated from the high-speed data fabric. Cnuas should mirror those boundaries so the emulator teaches the real topology rather than a simplification.

flowchart TB subgraph OOB["Out-of-band management network (isolated)"] direction TB RMC["RMC agent power shelf, PSUs, BBU, rack thermal"] SB1["CnuasSwitch BMC"] SB2["CnuasLink BMC"] B0["sled BMC 0"] B1["sled BMC 1"] B7["sled BMC 7"] end ORCH["cnuas CLI and REST API Redfish aggregator, orchestrator"] ORCH --> RMC ORCH --> SB1 ORCH --> SB2 ORCH --> B0 ORCH --> B1 ORCH --> B7 classDef mgmt fill:#e0e7ff,stroke:#3730a3,color:#1e1b4b,font-size:16px classDef orch fill:#dcfce7,stroke:#166534,color:#052e16,font-size:16px class RMC,SB1,SB2,B0,B1,B7 mgmt class ORCH orch

2.1 Sled level¶

Every compute blade carries its own BMC. In modern OCP designs the BMC is decoupled from the motherboard onto a DC-SCM (Datacenter Secure Control Module), which holds the BMC silicon (typically an AST2500 or AST2600), the hardware root of trust, and the out-of-band NIC. Its job is to manage power, thermal, firmware, and telemetry for that one sled only.

2.2 Switch level¶

The CnuasSwitch and CnuasLink switches each carry their own management controller. They manage their own ASIC, power, and optics. They do not manage the compute blades. On the management network they appear as ordinary nodes with their own Redfish endpoint.

2.3 Rack level¶

The RMC sits on the power shelf. It talks to the centralized PSUs, the battery backup unit, and rack thermal sensors over PMBus and I2C, then aggregates that physical-infrastructure data upward. It does not manage the operating systems on the blades.

3. BMC fidelity, the anchoring decision¶

A virtual BMC can be built at three levels of fidelity. The decision drives the topology, the repository layout, and the per-sled cost, so it is settled first.

Tier	What a BMC is	Protocols	Cost per BMC	Fidelity
A. Mock agent	A small daemon that speaks real Redfish and IPMI and models power, thermal, and sensors as data	Real protocols, modeled internals	Tens of MB	Protocol accurate
B. Real OpenBMC on QEMU	An unmodified OpenBMC image on a QEMU `ast2600` machine	Real firmware, real `bmcweb`	128 to 256 MB plus boot time	Firmware accurate
C. Renode	Real MCU firmware in a deterministic simulator with buses wired to the BMC	Depends on firmware	Medium	Bus and timing accurate

3.1 Decision¶

Tier B, real OpenBMC on QEMU ast2600, is the primary path. Tier A mock agents are the lightweight fallback. Tier C Renode has a distinct role, described in section 3.3.

Teaching OCP management is a primary objective of the platform, so the default must be the real thing. A student needs to run actual bmcweb, perform a real firmware update, and read real IPMI sensors, none of which a mock agent can do faithfully. QEMU ast2600 runs a full, unmodified OpenBMC userland on the exact machine type OpenBMC upstream tests against, so what the student learns transfers directly to real hardware.

Tier A stays available because a full rack of real BMCs has a real footprint, roughly eleven OpenBMC images (eight sleds, two switch BMCs, one RMC) at 128 to 256 MB and a boot each. The rack is therefore composable rather than all-or- nothing. A Redfish power-control lesson can boot two real BMCs while the blades not under inspection run mock agents. A full eight-blade demo can drop blades to mock agents to stay within a laptop. Both tiers sit behind the same cnuas orchestration, so the operator experience does not change with the mix.

3.2 Sourcing the OpenBMC image¶

Two upstream sources feed Tier B.

The openbmc/openbmc QEMU ARM path (bitbake obmc-phosphor-image, booted on qemu-system-arm -M ast2600-evb) provides a working stock BMC. This is the default image for "give me a BMC that boots".
The meta-aspeed Yocto layer carries the AST2500 and AST2600 SoC support. It is the base for a Cnuas-branded image with our Redfish Managers and Chassis names, our sensor topology, and a firmware-version string that makes update demonstrations meaningful.

3.3 Renode, the satellite microcontroller simulator¶

Renode is not a rack and not the BMC. QEMU ast2600 runs the BMC, which is a Linux-class SoC. Renode's strength is the deeply embedded microcontrollers the BMC talks to over sideband buses, small MCUs running real firmware on a deterministic clock. An OCP rack carries several of these, and they are the natural place for Renode in the ecosystem.

Satellite MCU	Bus to BMC or RMC	Role Renode plays
PSU on the power shelf	RS-485 / Modbus RTU	PSU firmware, AC and DC telemetry, fan and thermal behaviour, power-cap and fault injection
BBU with its Battery Management System	RS-485 / Modbus RTU	Real BMS firmware, cell balancing, state-of-charge and state-of-health telemetry, fault injection
Fan or thermal controller	PWM plus I2C	Closed-loop fan-curve firmware the BMC reads and overrides
Optics module, QSFP-DD (CMIS)	I2C two-wire	Module diagnostic monitoring the switch BMC polls
DC-SCM root of trust and SPDM	SPI plus MCTP over I2C	Attestation and measured-boot deep-dive bench
Power sequencer or hot-swap controller	I2C plus GPIO	Sequencing and electronic-fuse behavior

The ORV3 power shelf is the first satellite to be built, and it is described in full in section 3.4. PMBus and I2C satellites, when they arrive, bridge over an I2C-bus socket in the same spirit.

3.4 The ORV3 power shelf RS-485 segment¶

Open Rack v3 does not manage its power shelf over PMBus. Rack management is Modbus RTU over a multi-drop RS-485 segment at 19200 baud, which is the bus Meta's rackmon speaks in OpenBMC. PMBus and I2C exist inside some vendors' PSUs as a private sideband, but they are not the rack-management path, and an implementation that used them would not run unmodified rackmon.

Unit addresses encode position on the shelf. The address byte is [T1 T0][R2 R1 R0][D2 D1 D0]: a device-type field, a rack field and a slot field. PSUs use type 11, so they land on 0xC0 to 0xF7; BBUs use type 01, so they land on 0x40 to 0x7F. For rack 0 the Cnuas shelf is therefore six PSUs at 0xC0 to 0xC5 and six BBUs at 0x40 to 0x45.

RS-485 is UART plus Modbus at the protocol level, so the framing itself is unremarkable. What matters, and what QEMU cannot express, is that the segment is multi-drop: one master and many slaves share one pair of wires, every node hears every frame, and only the addressed node answers. A QEMU serial chardev is strictly point to point, so it can be one endpoint of the segment but it cannot be the segment.

The split follows from that. The segment and its twelve satellites live in Renode; the RMC keeps running real OpenBMC on QEMU ast2600, preserving the Tier B decision in section 3.1. A custom Renode external, Rs485Segment, is the wire: it distributes every byte to every other node on the segment and terminates one leg in a TCP socket, which QEMU attaches to with -serial tcp:127.0.0.1:3485. Renode's stock UARTHub cannot serve here because nothing in Renode can put a socket on it, so the segment and the socket endpoint are the same object.

  OpenBMC RMC (QEMU ast2600, running rackmon)
           |
           |  TCP 127.0.0.1:3485
           |
    ======= Rs485Segment (Renode) ======================
       |     |                      |     |
     psu0 .. psu5              bbu0 .. bbu5
      0xC0 - 0xC5                0x40 - 0x45

The satellites run real firmware, not a scripted responder: a portable C Modbus RTU slave and plant model that also builds for the host, so the ORV3 register and multi-drop behaviour stays covered by unit tests in CI, where no simulator is available. One firmware image serves all twelve nodes; personality comes from a strap block the platform writes before reset, mirroring the RS485_Addr pins on real hardware.

The register maps are generated from Meta's own rackmon map files, vendored under bmc/firmware/orv3/regmaps/ with their Apache-2.0 attribution, so the emulated registers are byte-compatible with what real rackmon expects. See bmc/firmware/orv3/README.md for how to build and run the shelf.

The master on the other end of that segment is cnuas-rackmond, a portable C Modbus RTU master under bmc/rmc/ that runs on the RMC's OpenBMC and sweeps the twelve satellites, with cnuas-rackmon as the operator CLI. It follows rackmon2 in role and register usage but is a fresh implementation, because it has to cross build and link statically into a stock image with no package manager.

Two properties of the emulated wire had to be designed around, and both are worth recording because neither is obvious from a schematic. Every node on a multi-drop segment hears every reply, not just every request, so a payload byte that happens to equal a listener's own unit address can start a bogus frame in that listener and swallow the front of the next real request. The satellite firmware therefore validates the CRC before dispatching and realigns on the next plausible frame start when validation fails, which makes framing self synchronising rather than dependent on inter-byte timing that a socket transport does not preserve. Separately, a stock OpenBMC image starts obmc-console-server on every tty it probes, including UART6; a console server holding the port consumes the satellites' replies, so the RMC's systemd unit displaces it.

The readings do not stop at the daemon. cnuas-rackmond --dbus publishes them on the BMC's own system bus as 56 sensor objects under /xyz/openbmc_project/sensors, which is what the stock OpenBMC tooling reads, so busctl sees them and bmcweb serves them under /redfish/v1/Chassis/chassis/Sensors with no change to the image. Two consequences of how the object mapper works shaped that design. The mapper takes property values only from InterfacesAdded or from GetManagedObjects on a manager it already knows, so objects are announced at startup rather than merely served on request. The mapper also parks any association whose endpoint is not on the bus, and on a stock image nothing publishes a chassis, so the RMC also publishes the shelf's inventory item at /xyz/openbmc_project/inventory/system/chassis and announces it first. That is the correct owner in any case: the RMC is the shelf manager.

3.5 The sled BMC to blade seam¶

The rack level sideband and the sled level sideband are different problems. The RMC's is a bus with twelve devices on it, which is why it needed a simulator. The sled BMC's is four wires to one host: a power button, a reset line, and two status inputs. That does not need a third emulator, but it does need to stay wires, because a sled BMC's power state machine, its Redfish ComputerSystem and its IPMI chassis commands all sit on top of those pins. Replacing them with a remote procedure call into the hypervisor would make the management path a Cnuas invention rather than the one the firmware actually implements.

QEMU's AST2600 GPIO model registers each pin as a QOM boolean property, so both directions are reachable from outside the machine over QMP with no changes to QEMU and no changes to the firmware. cnuas-tools sledbmc bridge sits in that gap: it polls the two lines the BMC drives, applies ATX button semantics to them, and reports the two the BMC reads.

ATX semantics matter here rather than a simple on/off level, because that is what the firmware assumes. A momentary assertion of POWER_OUT toggles the host, a hold of four seconds is the hard-off override and the eventual release of that hold is not a second press, and a pulse on RESET_OUT restarts the host without a graceful shutdown. PS_PWROK follows the blade, and POST_COMPLETE asserts a configurable delay after power-on, which is what lets the BMC's own state machine settle instead of oscillating.

The names POWER_OUT, RESET_OUT, PS_PWROK and POST_COMPLETE are declared as gpio-line-names in the cnuas-sled device tree. That makes them the contract between the two sides: libgpiod consumers on the BMC, x86-power-control among them, find the lines by name, and the bridge drives the same pins from the host. Neither side may rename them alone.

4. Cnuas management topology¶

flowchart TB subgraph RACK["Cnuas rack (ORv3, 24 OU)"] direction TB subgraph PWR["Power shelf"] RMC["RMC agent rackmon, rack thermal"] PSUBBU["6x PSU + 6x BBU Renode satellites"] end subgraph TOR["Top of rack"] HS["cnuas-vswitchd"] HSB["CnuasSwitch BMC"] HL["cnuasgpu-link-switchd"] HLB["CnuasLink BMC"] end subgraph BLADE["Compute blade (x8)"] VM["QEMU guest CnuasNIC + CnuasGPU"] BMC["sled BMC agent"] end end BMC -- "virtual power, reset" --> VM BMC -- "serial over LAN" --> VM HSB --- HS HLB --- HL RMC -- "RS-485, Modbus RTU" --> PSUBBU MGMT{{"Management bridge out-of-band, isolated"}} BMC --- MGMT HSB --- MGMT HLB --- MGMT RMC --- MGMT classDef data fill:#fef3c7,stroke:#b45309,color:#78350f,font-size:16px classDef mgmt fill:#e0e7ff,stroke:#3730a3,color:#1e1b4b,font-size:16px classDef net fill:#f1f5f9,stroke:#334155,color:#0f172a,font-size:16px class VM,HS,HL data class BMC,HSB,HLB,RMC,PSUBBU mgmt class MGMT net

Two properties carry over from real racks. The management bridge is a separate, isolated network from the UNIX-socket data fabric, so a student can see the same plane separation they would find in production. And each blade's BMC controls only its own blade, never a neighbor.

5. How it reuses what already exists¶

The hard integration points are already built. The management plane mostly wires existing Cnuas capabilities to standard protocols.

Virtual power maps to the existing VM lifecycle. A blade BMC's On, ForceOff, and Reset actions call the same VM adapter that backs cnuas vm up and cnuas vm down. A Redfish ComputerSystem.Reset becomes a QEMU lifecycle transition.
Serial over LAN maps to the QEMU serial chardev. The console socket the VM layer already exposes becomes the BMC's SOL stream.
The cnuas REST API becomes the Redfish aggregator. It is the datacenter orchestrator role, fanning out to each endpoint on the management bridge. This is a new command group next to the existing ones, for example cnuas bmc, cnuas power, and cnuas rack.
Sensors derive from real emulator state where possible. GPU device-memory use, link state, and port counters already exist in the adapters and can feed synthetic thermal and power readings, so the numbers move when the workload moves.

6. Proposed component and surface¶

A new component, working name CnuasBMC (package cnuas-bmc), is the harness that integrates BMCs into the rack rather than a BMC itself. In the primary Tier B path the Redfish and IPMI surface is served by real OpenBMC (bmcweb and the phosphor IPMI stack) running on QEMU ast2600. CnuasBMC boots those images, wires each one's power control and serial console to the matching QEMU blade, places it on the management bridge, and hosts the optional Tier A mock agent for blades a given scenario chooses to run lightweight.

The mock agent, when used, runs one binary in three roles selected at start, sled, switch, or rmc, serving the same Redfish and IPMI surface a real OpenBMC would, so an operator cannot tell a mock blade from a real one over the network.

Redfish surface, first cut (served by OpenBMC in Tier B, by the mock agent in Tier A):

Resource	Roles	Purpose
`/redfish/v1/Systems/{id}`	sled	Power state, boot, reset
`/redfish/v1/Chassis/{id}/Thermal`	all	Temperatures, fans
`/redfish/v1/Chassis/{id}/Power`	rmc, sled	Power draw, PSU, BBU
`/redfish/v1/Managers/{id}`	all	The BMC itself, firmware version
`/redfish/v1/AggregationService`	orchestrator	Rack-wide view via `cnuas`

IPMI over LAN covers the classic verbs for teaching, chassis power control, sensor reads (SDR), and Serial over LAN, so ipmitool works against a blade the same way it would against real hardware.

6.1 Repository layout¶

The management plane lives under bmc/ in the Cnuas tree, extractable to a PacketFive/CnuasBMC repository later.

Path	Contents
`bmc/meta-cnuas/`	OpenBMC Yocto layer adding the Cnuas machines (`cnuas-sled`, `cnuas-tor`, `cnuas-rmc`) on AST2600, with a sensor topology example and branding hooks. Structured for contribution to OpenBMC.
`bmc/renode/`	Renode platforms and scripts for the sideband microcontrollers (BMS, PSU, fan, optics, sequencer, front panel), each bridged to the BMC over a socket.
`bmc/firmware/orv3/`	Portable C firmware for the ORV3 satellites: Modbus RTU slave, PSU and BBU plant models, host tests.
`bmc/rmc/`	`cnuas-rackmond` and `cnuas-rackmon`, the Modbus RTU master that runs on the RMC and the CLI that queries it.
`bmc/scripts/`	`run-rmc.sh`, which starts the Renode shelf and the RMC's OpenBMC wired together.
`tools/src/cnuas_tools/sledbmc/`	The sled BMC to blade power bridge: a QMP client, the GPIO backend, and the ATX power state machine.

meta-cnuas is the Cnuas-branded meta-aspeed consumer. It is added to an OpenBMC bblayers.conf next to meta-aspeed and the phosphor layers, then a machine is built with bitbake obmc-phosphor-image and booted on qemu-system-arm -M ast2600-evb. cnuas-rmc and cnuas-sled carry their own device trees in the layer, for the RS-485 segment and the host power-control lines respectively; cnuas-tor still reuses the evb tree.

6.2 PCIe link fidelity on the blades¶

Independent of the BMC, the blade endpoints now advertise a realistic PCI Express link. The CnuasNIC (cnuas-vnic) and CnuasGPU (cnuasgpu) QEMU devices program a PCIe endpoint capability and a Link Capabilities register, defaulting to Gen5 (32 GT/s) x16 and overridable with the x-speed and x-width device properties. No serdes link is simulated; the values exist so lspci -vv inside a blade reports the generation and width a modern accelerator and RDMA NIC would, which matters when a lesson inspects the fabric from the host side. Gen6 is a roadmap value; the PolarFire FPGA path stays at its real Gen2 x4.

The emulated blade endpoints identify as PCI vendor 0x1af4 (Red Hat / Qumranet) with experimental device IDs 0x10f0 (CnuasNIC), 0x10f1 (CnuasGPU), and 0x10f2 (the switch IB NodeInfo). These come from the QEMU 1af4:10f0-10ff range reserved for experimental use without registration (see qemu/docs/specs/pci-ids.rst). They carry no PCI-SIG assignment because the devices are emulation-only. Before any of this code is submitted upstream to QEMU or the Linux kernel, or shipped in a product, the IDs must be replaced with an officially assigned 1b36 device ID reserved through the QEMU PCI ID maintainer. The earlier 0x1ed5 vendor ID was dropped because it belongs to a real vendor (Moore Threads) and cannot be squatted on.

7. Suggested delivery phases¶

Management bridge and one real BMC. Bring up the isolated bridge and boot one OpenBMC image on QEMU ast2600 on it. Reach bmcweb over Redfish and the BMC over IPMI. Wire its power control to the matching blade's VM lifecycle, so a Redfish Reset and cnuas power on both boot the blade.
Serial over LAN and sensors. Bridge the blade's QEMU serial console to the BMC as SOL, and connect the OpenBMC sensor tree to emulator state so Thermal and Power readings move with the workload.
Mock agent fallback. Add the Tier A cnuas-bmc agent presenting the same surface, so a scenario can mix real and mock blades to stay within a laptop.
Switch BMCs and the RMC. Add the two switch BMCs and the power-shelf RMC, with a modeled PSU and BBU behind the RMC. The RMC half of this is done: an OpenBMC image boots on QEMU with UART6 on the Renode RS-485 segment, and cnuas-rackmond on it sweeps all six PSUs and all six BBUs and publishes them on D-Bus, where Redfish serves them under /redfish/v1/Chassis/chassis/Sensors. The cnuas-rmc machine in meta-cnuas now describes that properly, with its own device tree declaring UART6 as the RS-485 segment and a recipe that builds and enables the daemon; what has been run end to end is still the stock image with the binaries copied in.
Aggregation in cnuas. Add the cnuas bmc, cnuas power, and cnuas rack command groups and the Redfish aggregation endpoint that fans out across the bridge.
Renode satellites. The power shelf is done, twelve satellites on the segment running real firmware. Bring up the battery system as real BMS firmware in Renode, reached from the RMC or power-shelf BMC over an emulated RS-485 or PMBus segment, then extend to the other satellite microcontrollers in section 3.3.

8. Open decisions¶

These need a call before Phase 1 starts.

Management bridge implementation. A plain Linux bridge that the host owns, or a small Cnuas-native management switch so the whole rack still runs without root and without host network changes.
OpenBMC image source. Ship the stock obmc-phosphor-image first, or invest in a Cnuas-branded meta-aspeed build from the start so the Redfish names and firmware-version strings match the rest of the platform.
Repository placement. cnuas-bmc as its own PacketFive/CnuasBMC repo and submodule, matching CnuasNIC, CnuasSwitch, CnuasGPU, and CnuasLink, or as a package inside the existing cnuas tree.
Identity and security depth. How far to model the DC-SCM root of trust and the SPDM attestation flow, which is valuable for teaching but is a project on its own, and a natural fit for the Renode satellite track.