Cnuas demonstration runbook¶
A rehearsed sequence for showing Cnuas to an audience, with the exact commands, what each one is meant to reveal, and what to do when something does not come up.
The sequence assumes one host running Cnuas, and for the fabric comparison a second host cabled to it. Where a step needs the second host it says so.
docs/guides/RDMA_InfiniBand_and_RoCE.md is the companion note for an
audience that does not work with interconnects, and can be circulated
beforehand.
1. Before the audience arrives¶
1.1 Bring the stack up¶
This starts the ORv3 power shelf, the switch and fabric daemons, the endpoints that populate the switch's fabric ports, and the Rack UI. It waits for each to answer before reporting, so if it prints the summary block everything is up.
Export what it tells you to export, in the shell you will demonstrate from.
export CNUAS_SWITCH_SOCKET=$PWD/.demo-run/cnuas/mgmt.sock
export CNUAS_CNUASLINK_SOCKET=$PWD/.demo-run/cnuaslink/mgmt.sock
export PATH=$PWD/src/cnuasgpu/tools/cnuassmi:$PATH
1.2 Confirm it is healthy¶
Every component must read yes. If any reads no, section 7 has the
recovery for it.
1.3 Start the rack management controller¶
It takes two to three minutes to reach a login prompt, so start it now and come back to it in section 2.4.
1.4 If the InfiniBand comparison is part of the demonstration¶
A subnet manager must be running somewhere on the fabric, and only one.
Both ends should read State: Active with a nonzero Base lid. Leaving
this until the audience is present is an option, because starting it in
front of them is the clearest available demonstration of what a subnet
manager does, and section 3.2 uses it that way.
For the RoCE comparison, soft-RoCE must exist on both hosts.
Neither of these survives a reboot, so confirm both on the day.
1.5 Have these ready¶
- a browser on the Rack UI,
http://<host>:8500 - two terminals on the first host, one for the CLI and one for the server side of the RDMA example
- one terminal on the second host
- one terminal with root, for
tcpdump
2. The rack¶
2.1 Open the Rack UI¶
Everything on the page is read from a running system. The rack elevation is drawn from the shelf's actual population, the watts come from the power shelf over its RS-485 segment, the fabric table is the RDMA hardware the kernel reports, and the control plane panel is the same health probe the CLI runs.
Each rack on the elevation is polled separately. The two power sections
carry their own ORv3 unit addresses, 0xc0 to 0xc5 for the first rack
and 0xc8 to 0xcd for the second, and their watts differ because they
are two independent reads rather than one rack drawn twice.
The sled rows are the switch's fabric ports. A row is green when an endpoint is attached to that port, and it carries the MAC address the switch learned. Its blue port stub brightens and the row reports a frame rate while frames are moving, so a transfer started later in the demonstration is visible on the elevation as it runs.
The header carries two further controls. About opens a panel with the
author, the project home, the components that make up the platform and the
open source projects it is built on, and is also reachable directly at
http://<host>:8500/#about. Power and sizing calculator opens the two
calculators covered in section 6.1.
2.2 Show that the power is real¶
The load tracks what the machine is doing, so the figures move when the machine is worked.
Rack input climbs over the next few seconds and falls back when the load expires. The point is that this is not a playback. The shelf is running the same plant model the Cortex-M firmware runs, and the sled load is the work the audience just watched being started.
The conversion efficiency figure moves too, in the opposite direction, because a switching supply is less efficient at low load. That is the efficiency curve in the model, not a constant.
2.3 Read the same shelf the way a BMC does¶
python3 - <<'EOF'
from cnuas_facility.orv3 import ShelfClient
shelf = ShelfClient(host="127.0.0.1", port=3485)
reading = shelf.read_shelf()
for psu in reading.psus:
print(f"PSU {psu.unit:#04x} {psu.model} "
f"{psu.output_watts:8.1f} W out {psu.input_watts:8.1f} W in")
print(f"shelf {reading.output_watts/1000:.2f} kW out, "
f"{reading.input_watts/1000:.2f} kW in")
EOF
This is Modbus RTU function code 3 over a multi-drop RS-485 segment, addressed by ORv3 unit identifier, which is what the specification calls for and what rack management software expects to find.
2.4 Log in to the rack management controller¶
The Rack UI reads the shelf over a socket. The rack management controller reads it as a piece of rack hardware does, from OpenBMC running on an emulated AST2600 with the segment on UART6.
--external-shelf attaches the controller to the segment the demonstration
stack is already serving. Each connection to that segment gets its own
request stream, so the controller and the Rack UI both poll without
disturbing each other. The controller reaches a login prompt in two to
three minutes, so start it before the audience arrives.
Log in as root with 0penBmc. This is a full OpenBMC userspace, not a
shell script pretending to be one.
cnuas-rackmond is the Cnuas addition to the image. It owns the RS-485
segment from boot and polls the shelf on its own, so cnuas-rackmon status
prints the supplies and battery packs the controller has found by address,
with their input and output power and their state of charge.
The same readings come back over Redfish, which is how a fleet manager would collect them.
Leave this console open. The next section changes what it reports.
2.5 Cut the power¶
The shelf carries an operator control channel alongside the RS-485 segment. Faults are injected there rather than by writing registers the ORv3 map does not define.
Open a second terminal on the rack telemetry view and leave it running where the audience can see it next to the Rack UI.
Now take the utility feed away from the first rack only.
Three things happen at once. The six supplies in that rack drop to zero
output and zero AC input, because there is nothing left to rectify. The
DC bus stays up at about 1.9 kW, held by the battery packs. The rack
header on the elevation turns amber and reads on batteries with the
falling state of charge. The second rack is untouched throughout, which is
the point of injecting the fault per rack.
The packs discharge, and the state of charge falls. At 70 percent the
shelf sheds the rack: the supplies are held off, the DC bus goes to zero,
and the rack header turns red and reads shed. The threshold is a
setting, not a constant.
Restore the feed and the rack repowers and begins recharging.
The demonstration stack populates the shelf with 0.35 Ah packs so the
descent from full charge to the shed threshold takes about a minute. Full
size packs are the default everywhere else and give a ride through of
roughly an hour, which is the right number for the hardware and the wrong
number for a live audience. BBU_CAPACITY_AH in scripts/demo/start.sh
sets it.
Individual racks can also be shed directly, without touching the feed, to show the same supply cutoff without waiting for a discharge.
./scripts/demo/shelfctl.py reset returns every rack to the utility feed
and clears any shed state, and is the safe thing to run if the sequence is
interrupted.
2.6 Rack level telemetry¶
shelfctl.py watch is the whole rack in one view: feed state, supply
state, state of charge, DC bus, supply output and AC input, per rack,
refreshed continuously. It reads the same shelf as the Rack UI and the
rack management controller, so the three agree.
The REST API carries it too, for anything that wants to consume it.
power_state gives the feed and shed state per rack with the threshold in
force. shelf gives the measured input and output power, the conversion
efficiency and the per unit readings.
3. The fabric¶
3.1 Inventory¶
The first is the RDMA hardware present on the host. The second is the emulated top of rack switch, whose fabric ports are up because endpoints are attached to them.
3.2 The subnet manager, if InfiniBand hardware is present¶
Stop the subnet manager first, and show the consequence.
The ports fall back to Initializing with Base lid: 0. The cable has
not moved and the adapter is healthy. An InfiniBand port with no subnet
manager cannot pass traffic, because nothing has given it an address or
programmed the path to it.
Both ports return to Active with a LID. Then show the fabric as the
subnet manager sees it.
The emulated switch has the same facility, so the concept can be shown with no hardware at all.
3.3 Congestion control, the RoCE side of the same problem¶
cnuas switch set-pfc 0 --on --priority 3
cnuas switch set-ecn 0 --on --threshold 150 --probability 10
cnuas switch telemetry | head -20
InfiniBand gets losslessness from credit flow control in hardware. RoCE has to be configured into it, on every switch in the path. These are the knobs that configuration turns.
4. One application, both fabrics¶
This is the part of the demonstration that carries the argument, so it is worth not rushing.
4.1 The program¶
examples/rdma/msgstream.c is a client and server in one binary. It sets
up a reliable connected queue pair, streams messages with two-sided sends,
and finishes with a one-sided RDMA write straight into the peer's memory.
It is around seven hundred lines and the fabric appears in exactly one
branch of it.
4.2 Over InfiniBand¶
On the first host:
On the second:
The fabric report names InfiniBand, gives both LIDs, and states that the addresses were assigned by a subnet manager. The messages arrive, then the one-sided write lands in the server's buffer without the server's code doing anything at all.
4.3 Over RoCE¶
The same binary, a different device.
./msgstream --server --device roce0 --ib-port 1
./msgstream --client hicain1 --device roce0 --ib-port 1 --count 4
Nothing was rebuilt and nothing was reconfigured. The report now names Ethernet, prints a GID instead of a LID, and says the LID is not used on this fabric.
If the RoCE side is soft-RoCE, which implements RDMA in the kernel over an ordinary Ethernet interface, do not let the two sets of timings be read as a comparison of the fabrics. One path is an offload engine on a dedicated adapter and the other is software on a general purpose NIC. The comparison being made here is of the programming model and the addressing, which are what the single unchanged binary demonstrates.
4.4 The clincher¶
With the RoCE run going, capture it.
192.168.1.49.49552 > 192.168.1.52.4791: UDP, length 128
192.168.1.52.49553 > 192.168.1.49.4791: UDP, length 20
Remote memory access, inside ordinary UDP, on the same wire as everything else. The 128 byte datagrams are the messages and the 20 byte ones are acknowledgements.
Then run the same capture during the InfiniBand test. Nothing appears, and not because the traffic is hidden. There is no IP layer to capture.
5. The accelerator¶
The Soft-GPU runs entirely in userspace, with no kernel module and no guest. The SM and lane figures are the geometry of the compute backend selected for this CPU, so on a machine with AVX-512 the lane count is sixteen and on one with AVX2 it is eight.
Device memory is configurable, which matters because model weights have to fit in it.
6. The facility¶
The rack figures are the input to a campus model, which closes the loop between the work the host is doing and the size of a grid connection.
The roll-up is computed from the rack inventory, so 1,920 racks give an IT load of 253.2 MW and, at the design PUE, a 291.2 MW grid connection. Change the rack count and every figure moves, including the generator count.
Then drive the model from the shelf that is already running. The live
bridge writes into an existing stage, so generate one first. live reads
build/campus.usda unless told otherwise.
Each sample reads the emulated power supplies over RS-485, converts to a rack wattage, and scales the fleet by measured utilisation. With the host idle the projection sits near a third of the envelope. Put the host under load, as in section 2.2, and it walks up towards 291 MW.
That same stage is ordinary OpenUSD, so it opens in NVIDIA Isaac Sim, usdview, Blender or any other USD capable tool. None of those are installed on the demonstration hosts, so present the campus from the rendered film and the generated plans rather than from a live session.
docs/assets/media/cnuas-datacentre-simulation.mp4docs/assets/facility/cnuas-campus-plan.svgdocs/assets/facility/cnuas-datahall-plan.svg
6.1 The calculators¶
The same models are on the Rack UI at http://<host>:8500/calculator,
which is the form to use when the audience wants to change an assumption
and see the answer move.
Facility power. Set the number of data halls, the racks in each hall,
the rack type and how hard the racks are worked, then set the cooling
approach and the ancillary load. The page reports the IT load, the
mechanical and electrical overhead, the load at the utility feed, the heat
to reject, and the efficiency two ways, as PUE and as DCiE. It also sizes
the plant that follows, the generators and the transformers. The rack types
and their draw come from cnuas_facility.catalog, so the figures agree
with cnuas-facility power on the command line.
Emulation sizing. The other question is what it takes to run the
emulation of a facility rather than to power one. Give the page the
machines that are available and the shape of the rack to emulate, and it
returns the racks one host can carry, the limiting resource, and the fleet
needed to reach a target rack count. The costs are the marginal cost of one
more rack measured on a running deployment, and they live in
cnuas_facility.sizing.
The two together answer the question a supervisor tends to ask next. The model says a 253 MW site, and the sizing page says how many machines are needed to emulate it.
6.2 Joining two instances¶
One instance is bounded by the machine it runs on. Beyond that, instances are joined at their top of rack switches and behave as one fabric. Port 8 on each virtual switch is the uplink, and the bridge carries it over TCP.
# on the first host
scripts/demo/uplink_bridge.py --run-dir .demo-run/cnuas --listen 3487
# on the second host
scripts/demo/uplink_bridge.py --run-dir .demo-run/cnuas --connect hicain1:3487
A switch port carries one Ethernet frame per datagram, so each frame is given a two byte length on the wire and the boundary is restored on the far side. Flooding, address learning and subnet management then cross the join the way they do inside one instance, which is what makes a rack on the second host reachable from an application on the first.
Show it by running the client from section 4 on one host against the server on the other. Nothing in the program changes.
7. When something does not come up¶
A CLI command reports a component unavailable. The daemon is not running or the socket path is not exported. Check both.
cnuas system health
ls -l $CNUAS_SWITCH_SOCKET $CNUAS_CNUASLINK_SOCKET
tail -20 .demo-run/vswitchd.log
Switch fabric ports read DOWN. Nothing is plugged into them. The start script attaches endpoints; confirm that process survived.
The Rack UI shows the power shelf unreachable. The shelf daemon exited.
InfiniBand ports read Initializing with lid 0. No subnet manager, or two of them arguing. Exactly one instance must run on the fabric.
msgstream hangs on connect. A server from an earlier run is still holding the bootstrap port and has taken the connection. Clear it before every run.
msgstream reports no device. The RDMA device name is wrong, or soft-RoCE was not recreated after a reboot.
The rack management controller reports every unit absent, with a rising
timeout count. The controller is talking to a UART the emulator does not
model. The Cnuas RMC puts the RS-485 segment on UART6, and only a QEMU that
models all thirteen AST2600 UARTs wires the sixth serial index to it. A
distribution QEMU that models five accepts the option, connects the socket
and then silently discards everything the firmware writes. Build the
emulator from the qemu submodule and point QEMU at it.
The rack management controller never reaches a login prompt. It takes two to three minutes from a cold flash. If it is longer than that, the flash image is wrong; it has to be exactly 64 MiB.
The shelf reads zero output with the batteries discharging and no fault was injected. A fault is still in force from an earlier run.
The tcpdump capture is empty on RoCE. The capture started after the run finished. Start the capture first, then the client, in that order, and give the capture a timeout longer than the run.
8. Cutting the demonstration short¶
If time runs out, the three things that carry the argument are the live rack power responding to real work in section 2.2, the same binary running on both fabrics in sections 4.2 and 4.3, and the packet capture in section 4.4. Everything else can be described from the Rack UI in one pass.