Skip to content

Cnuas VM Lifecycle, design

Status: design, Phase 6.2. Author: Cnuas project. Companion docs: - Cnuas_CI_Strategy.md (Phase 6.0e, ADO self-hosted agents). - Cnuas_Build_System.md (Bazel / Make for component builds). Audience: developers running Cnuas locally, CI engineers wiring ADO pipelines to consume the golden images, and packagers producing Debian artefacts.

1. Why this exists

The Phase 1, 5 scripts/vm-manager.sh was a developer hack: it downloaded a stock Ubuntu cloud image, dropped a cloud-init seed, and booted QEMU in the user's terminal. Drivers were not installed; the user had to build cnuasgpu.ko inside the running VM against linux-headers-$(uname -r) of whatever kernel the cloud image happened to ship. Userspace (libcnuasrt, cnuaslink-cli, cnuas-vswitchd) had to be copied in by hand or via an ad-hoc genisoimage payload.

That works for the original author and nobody else. Phase 6.2 replaces it with a proper VM lifecycle pipeline:

  1. A golden image (cnuas-vm-vX.Y.Z.qcow2) with all Cnuas userspace .debs installed plus a pinned 6.19.0-cnuas+ kernel and its pre-built .ko modules (cnuasgpu, cnuas_net, cnuas_ib), cloud-init configured for SSH-from-host.
  2. A Python tooling package (cnuas-tools) with a Typer CLI that builds the image, builds the .debs (kernel + drivers + userspace), and brings up VMs through libvirt.
  3. A second, parallel boot path for kernel developers who want to iterate on cnuasgpu.ko against the in-tree linux/ submodule, that stays a thin script (scripts/boot-vm-kernel-dev.sh) for now.

The golden-image path is Approach A; the kernel-dev path is Approach C in the design-discussion shorthand. Approach B (DKMS in the guest) is rejected for v1 because we pin a single kernel and ship its .kos pre-built; DKMS can be re-introduced later for multi-kernel support (see §13.2).

2. Goals and non-goals

Goals

# Goal
G1 A user can run cnuas-tools vm up vm-a on a fresh machine and have /dev/cnuasgpu0 plus libcnuasrt.so working in the guest in < 30 seconds of wall time.
G2 The golden image is reproducible: same inputs (Ubuntu base image, the pinned linux/ submodule commit, our .debs, recipe) produce a byte-identical *.qcow2 plus a manifest with sha256s.
G3 Tooling is pure-Python (Poetry-managed). External CLI tools (virsh, qemu-img, virt-customize, genisoimage) are wrapped through a single run() helper so a future libvirt-native engine is a drop-in replacement.
G4 The same code path is used by humans and by CI (ADO pipelines). The pipelines just call the same Typer subcommands.
G5 Versioning: every artefact (.deb, .qcow2) is stamped with a semantic version + git SHA + manifest.
G6 The guest runs one known kernel, 6.19.0-cnuas+ from the in-tree linux/ submodule. Users don't build kernels; users don't run DKMS; users don't see "module verification failed" surprises. See §13.

Non-goals (now)

# Non-goal Phase that covers it
N1 A native libvirt-python engine. (Subprocess + virsh is fine for v1.) Phase 6.3
N2 Publishing images to GitHub Releases automatically. Phase 6.4 (CI)
N3 An OS image other than Ubuntu 24.04 Noble. Future
N4 An architecture other than x86_64. Phase 7+
N5 Live-migration, snapshots, pools, anything operational beyond up/down. Future
N6 Replacing the Bazel/Make builds for the C/C++ components. We call them; we don't replace them. Never
N7 Supporting non-Cnuas kernels in the guest (DKMS path). Multi-kernel support can be added in a follow-up release if needed; see §13.2. Post-USENIX
N8 Upstreaming the Cnuas kernel patch / drivers to mainline. We will, but only after the USENIX Cnuas paper publishes. Post-USENIX

3. Architecture overview

flowchart TB subgraph DEV["developer / CI agent"] subgraph TOOLS["cnuas-tools (Typer CLI, Python 3.11+)"] T1["cnuas-tools pkg build"] T2["cnuas-tools image build"] T3["cnuas-tools vm up vm-a"] end T1 -->|"&rarr;"| B1["existing make / bazel builds"] T2 -->|"&rarr;"| B2["virt-customize + qemu-img"] T3 -->|"&rarr;"| B3["virsh (subprocess)"] end ART["<b>out/</b><br/>cnuas-libcnuasrt_0.2.0_amd64.deb<br/>cnuas-gpu-modules_0.2.0_amd64.deb<br/>cnuas-nic-modules_0.1.0_amd64.deb<br/>cnuas-vswitchd_0.1.0_amd64.deb<br/>cnuas-cli_0.1.0_all.deb &bull; cnuas-link-cli_0.1.0_all.deb<br/><b>out/kernel/</b> linux-image / headers / libc-dev + config<br/><b>out/</b> cnuas-vm-v0.2.0.qcow2 (+ .sha256, .manifest.json)<br/>~2.6 GB sparsified, 6.19.0-cnuas+ pre-installed"] subgraph DOM["libvirt domain - qemu+kvm, q35, host-passthrough, 4 vCPU / 4 GB"] D1["disk: backing-store overlay<br/>cnuas-vm-v0.2.0.qcow2 (RO) &larr; vm-a/disk.qcow2 (RW overlay)"] D2["seed.iso (cloud-init NoCloud)<br/>hostname, ssh keys, network"] D3["-netdev user (mgmt, hostfwd 2222)<br/>-device cnuas-vnic,socket=...<br/>-device cnuasgpu,gpu_id=N"] end TOOLS -->|"produces"| ART ART -->|"at boot time"| DOM classDef tool fill:#dbeafe,stroke:#1e40af,stroke-width:2px,color:#1e3a8a; classDef build fill:#fef3c7,stroke:#b45309,stroke-width:2px,color:#78350f; classDef art fill:#e0e7ff,stroke:#3730a3,stroke-width:2px,color:#1e1b4b; classDef dom fill:#dcfce7,stroke:#166534,stroke-width:2px,color:#14532d; class T1,T2,T3 tool; class B1,B2,B3 build; class ART art; class D1,D2,D3 dom;

4. The Python tooling package (cnuas-tools)

4.1 Project layout

cnuas/
  tools/
    pyproject.toml                 # Poetry; details in §4.2
    poetry.lock                    # checked in
    README.md
    .pre-commit-config.yaml

    src/
      cnuas_tools/
        __init__.py                # exposes __version__
        cli.py                     # Typer root: `cnuas-tools <subcmd>`
        config.py                  # global config, search paths, env vars
        log.py                     # rich-based logging
        run.py                     # subprocess.run wrapper with proper errors
        version.py                 # reads version from pyproject

        pkg/                       # `cnuas-tools pkg ...`
          __init__.py
          cli.py
          binary.py                # plain (non-DKMS) .deb assembler
          kmod.py                  # kernel-module .deb helper
          python_cli.py            # Python wheel + wheelhouse + venv .deb
          targets/
            _core.py               # target registry / ABC
            cnuas_linux_image.py  # builds the linux .deb set
            cnuasgpu_modules.py       # cnuasgpu.ko binary .deb
            cnuasnic_modules.py       # cnuas_net.ko + cnuas_ib.ko
            libcnuasrt.py
            cnuas_vswitchd.py
            cnuas_cli.py
            cnuaslink_cli.py

        image/                     # `cnuas-tools image ...`
          __init__.py
          cli.py
          builder.py
          manifest.py
          recipe.py                # loads YAML recipes
          recipes/
            cnuas-vm.yaml
          configs/
            cnuas.config          # kernel-config overlay applied at
                                   # build time on top of x86_64_defconfig

        vm/                        # `cnuas-tools vm ...`
          __init__.py
          cli.py
          engine.py                # abstract VmEngine base class
          engine_subprocess.py     # uses virsh, qemu-img
          engine_libvirt.py        # stub for Phase 6.3
          cloud_init.py
          network.py               # mgmt libvirt net definition
          state.py
          templates/
            domain.xml.j2
            user-data.yaml.j2
            meta-data.yaml.j2
            network-mgmt.xml

        release/                   # `cnuas-tools release ...`
          __init__.py
          cli.py
          github.py                # wraps `gh` CLI (deferred, Phase 6.4)

    tests/
      test_cli.py
      pkg/
        test_binary.py
        test_targets.py
        test_cli.py
      image/
        test_builder.py
        test_manifest.py
      vm/
        test_engine_subprocess.py
        test_cloud_init.py
        test_network.py
      fixtures/
        fake_virsh.py
        fake_qemu_img.py

4.2 pyproject.toml

[build-system]
requires = ["poetry-core>=1.9"]
build-backend = "poetry_core.masonry.api"

[tool.poetry]
name = "cnuas-tools"
version = "0.2.0"
description = "Cnuas tooling: package builds, golden-image generation, VM lifecycle"
authors = ["PacketFive <cnuas@packetfive.io>"]
license = "GPL-2.0-or-later OR MIT"
readme = "README.md"
packages = [{ include = "cnuas_tools", from = "src" }]

[tool.poetry.dependencies]
python = "^3.11"
typer = "^0.12"
rich = "^13.7"
pyyaml = "^6.0"
jinja2 = "^3.1"

[tool.poetry.group.dev.dependencies]
pytest = "^8.3"
pytest-cov = "^5.0"
black = "^24.8"
flake8 = "^7.1"
flake8-bugbear = "^24.8"
flake8-pyproject = "^1.2"
isort = "^5.13"
mypy = "^1.11"
pre-commit = "^3.8"
poethepoet = "^0.30"

[tool.poetry.scripts]
cnuas-tools = "cnuas_tools.cli:app"

[tool.black]
line-length = 100
target-version = ["py311"]

[tool.isort]
profile = "black"
line_length = 100

[tool.flake8]
max-line-length = 100
extend-ignore = ["E203", "W503"]
select = ["E", "W", "F", "B", "C"]

[tool.mypy]
strict = true
python_version = "3.11"

[tool.poe.tasks]
fmt = ["black src tests", "isort src tests"]
lint = ["flake8 src tests", "mypy src"]
test = "pytest -q"
check = ["fmt", "lint", "test"]

4.3 CLI surface

Top-level:

cnuas-tools --help
cnuas-tools --version
cnuas-tools pkg ...
cnuas-tools image ...
cnuas-tools vm ...
cnuas-tools release ...

pkg

cnuas-tools pkg list                          # list targets
cnuas-tools pkg build [TARGET ...]            # build one or all
cnuas-tools pkg clean

Targets: cnuas-linux-image, cnuas-gpu-modules, cnuas-nic-modules, cnuas-libcnuasrt, cnuas-vswitchd, cnuas-cli, cnuas-link-cli. Default: all.

Each target maps to a small Python class subclassing Target with a single build(ctx) method that returns a BuildResult. Targets internally invoke existing make/bazel/poetry build commands; the Python layer adds version stamping, output location, and packaging into .debs (see pkg/binary.py, pkg/kmod.py, pkg/python_cli.py). See §5 for full details.

image

cnuas-tools image build [--recipe RECIPE] [--out DIR]
cnuas-tools image inspect IMAGE.qcow2        # print manifest
cnuas-tools image verify IMAGE.qcow2         # check sha256

The default recipe is recipes/cnuas-vm.yaml. The build outputs cnuas-vm-vX.Y.Z.qcow2, its .sha256, and a .manifest.json.

vm

cnuas-tools vm up NAME [--gpus N] [--port P]
cnuas-tools vm down NAME
cnuas-tools vm console NAME
cnuas-tools vm ssh NAME [-- CMD ...]
cnuas-tools vm scp [LOCAL] NAME:[REMOTE]
cnuas-tools vm list
cnuas-tools vm lab                            # vm-a + vm-b + switch
cnuas-tools vm lab-down
cnuas-tools vm destroy NAME [--purge]

Per-VM state lives in ~/.local/share/cnuas/vms/<name>/state.json, ssh port, libvirt domain name, switch socket, GPU count, image revision. state.json is the single source of truth; tools read from it and refresh from virsh dominfo.

release

cnuas-tools release publish VERSION           # Phase 6.4
cnuas-tools release verify VERSION

Stubs for now; implementation in Phase 6.4.

4.4 The run() wrapper

All external commands go through one helper. This is the seam where we later swap virsh calls for libvirt-python.

# src/cnuas_tools/run.py
import subprocess
from typing import Sequence

class RunError(RuntimeError):
    def __init__(self, cmd, returncode, stdout, stderr):
        super().__init__(
            f"Command failed ({returncode}): {' '.join(cmd)}\n"
            f"---stdout---\n{stdout}\n---stderr---\n{stderr}"
        )
        self.cmd, self.returncode = cmd, returncode
        self.stdout, self.stderr = stdout, stderr

def run(cmd: Sequence[str], *, capture: bool = True,
        check: bool = True, env: dict | None = None,
        cwd: str | None = None, timeout: float | None = None) -> subprocess.CompletedProcess:
    proc = subprocess.run(
        list(cmd),
        capture_output=capture,
        text=True,
        env=env,
        cwd=cwd,
        timeout=timeout,
    )
    if check and proc.returncode != 0:
        raise RunError(cmd, proc.returncode, proc.stdout, proc.stderr)
    return proc

No magic, no shell. Tests inject a fake by monkey-patching run at module level.

5. Package builds

5.1 Target overview

Target What it ships Build engine Layer
cnuas-linux-image linux-image-6.19.0-cnuas+, linux-headers-6.19.0-cnuas+, linux-libc-dev make x86_64_defconfig + merge_config.sh configs/cnuas.config + make bindeb-pkg kernel
cnuas-gpu-modules pre-built cnuasgpu.ko under /lib/modules/6.19.0-cnuas+/extra/cnuas/ + modules-load.d out-of-tree make M=... against linux/ kernel module
cnuas-nic-modules pre-built cnuas_net.ko + cnuas_ib.ko + modules-load.d same kernel module
cnuas-libcnuasrt libcnuasdev.so.0, libcnuasrt.so.0, headers (compute backend archive is statically linked) existing Makefile under src/cnuasgpu/lib/ userspace lib
cnuas-vswitchd switch daemon binary + systemd unit existing Makefile under src/cnuasswitch/switch/ service
cnuas-cli Python wheel + transitive-dep wheelhouse + /usr/bin/cnuas-cli launcher poetry build + pip wheel tool
cnuas-link-cli same pattern as cnuas-cli from src/cnuaslink/cli/ same tool

Kernel .debs land under out/kernel/; everything else lands directly in out/. This separation keeps the smaller component .debs clearly distinct from the larger kernel artefacts.

5.2 Kernel package set (cnuas-linux-image)

Cnuas pins guest VMs to a single Linux kernel built from the linux/ submodule of the Cnuas repo. The submodule is at PacketFive/linux@cnuas-v6.19.0, upstream Linux v6.19 plus exactly one Cnuas-specific patch:

--- a/include/uapi/rdma/ib_user_ioctl_verbs.h
+++ b/include/uapi/rdma/ib_user_ioctl_verbs.h
@@ -253,6 +253,7 @@ enum rdma_driver_id {
    RDMA_DRIVER_ERDMA,
    RDMA_DRIVER_MANA,
    RDMA_DRIVER_IONIC,
+   RDMA_DRIVER_CNUAS,
 };

This single uapi addition lets cnuas_ib.ko claim its driver identity. Everything else in the Cnuas kernel stack (cnuasgpu.c, cnuas_net.c, cnuas_ib.c) builds out-of-tree against an unmodified kernel.

The kernel .config is reproducible:

  1. make x86_64_defconfig in the submodule.
  2. scripts/kconfig/merge_config.sh -m .config tools/src/cnuas_tools/image/configs/cnuas.config.
  3. make olddefconfig.

configs/cnuas.config is ~70 lines of explicit CONFIG_*=y/m covering KVM-guest virtio, networking, RDMA, ext4, cgroups+namespaces, BTF, and CONFIG_LOCALVERSION="-cnuas". The resolved final .config is shipped at out/kernel/config-6.19.0-cnuas so users on other distributions can use it as a baseline.

Build invocation:

make -C linux \
    -j$(nproc) \
    KDEB_PKGVERSION=6.19.0-cnuas.1 \
    bindeb-pkg

ccache is auto-used when present (CC="ccache gcc"). Cold build on a 12-core box is ~8, 12 minutes; warm ccache rebuild is ~30 seconds. Output: linux-image-6.19.0-cnuas+_*.deb (~16 MB), linux-headers-6.19.0-cnuas+_*.deb (~9.5 MB), linux-libc-dev_*.deb (~1.5 MB).

The + suffix on the kernel release is appended by mainline scripts/setlocalversion whenever HEAD is not at an upstream vX.Y tag. Our submodule tip is one commit past v6.19, so the script appends +, this is the canonical "modified upstream tree" marker and we embrace it. uname -r inside the guest reads 6.19.0-cnuas+.

5.3 Kernel-module packages (binary, not DKMS)

For Cnuas's pre-USENIX release we ship pre-built .ko files in plain binary .debs rather than DKMS source .debs. The rationale:

  • One supported kernel = one supported set of .kos. DKMS exists precisely to handle the "many kernels per install" case, which we explicitly do not support yet.
  • Pre-built .kos install in O(1), no compiler in the guest, no install-time race against linux-headers-* apt hooks.
  • The image stays small (no kernel sources, no build-essential).

cnuas_tools.pkg.kmod.build_kmod_deb() produces these. For each target the flow is:

  1. Locate the built kernel source at <repo>/linux/ (must already have Module.symvers, i.e. pkg build cnuas-linux-image was run first).
  2. make -C linux M=<driver_src_dir> modules -j$(nproc).
  3. Stage the resulting .ko(s) under /lib/modules/6.19.0-cnuas+/extra/cnuas/<name>.ko.
  4. Drop a /etc/modules-load.d/<deb_name>.conf listing the modules to autoload at boot.
  5. Generate DEBIAN/control with Depends: linux-image-6.19.0-cnuas+ | linux-image-extra.
  6. postinst runs depmod -a 6.19.0-cnuas+; prerm does the same.

A future DKMS path is documented as a follow-on: when we want to support users on non-Cnuas kernels, we add a parallel set of targets cnuas-gpu-dkms / cnuas-nic-dkms. The current binary-module path stays as the default.

5.4 Userspace .debs

libcnuasrt and cnuas-vswitchd build via their existing component Makefiles. The tooling wrappers:

  • cnuas-libcnuasrt, runs make -C src/cnuasgpu/lib, stages libcnuasdev.so.*, libcnuasrt.so.*, plus cnuasdev.h, cnuasrt.h, cnuas_compute_ops.h. Symlinks for SONAME + unversioned dev names.
  • cnuas-vswitchd, runs make -C src/cnuasswitch/switch, stages the cnuas-vswitchd binary at /usr/bin/ and the bundled systemd unit at /lib/systemd/system/cnuas-vswitchd.service.

Built via cnuas_tools.pkg.python_cli.build_py_cli_deb():

  1. poetry build -f wheel in the source repo.
  2. pip wheel -w <wheelhouse> to collect all transitive deps as wheels.
  3. Stage the wheels under /opt/cnuas/wheels/<pkg>/.
  4. postinst creates /opt/cnuas/venvs/<pkg> via python3 -m venv, pip install --no-index --find-links from the wheelhouse, and writes a one-line /usr/bin/<cli> launcher.

This avoids Debian's PEP 668 "managed environment" restrictions on system Python while keeping the install offline and reproducible.

6. The golden image

6.1 Recipe (recipes/cnuas-vm.yaml)

name: cnuas-vm
description: Cnuas VM golden image
version: 0.2.0

base:
  source: cloud-image
  url: https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img
  sha256: <pinned>

disk:
  size_gb: 32

packages:
  apt:
    # Userspace runtime deps only. No DKMS / build-essential /
    # linux-headers-generic - we ship pre-built kernel modules.
    # No linux-image-generic - we install cnuas-linux-image and
    # purge the stock kernel in post_install.
    - libgomp1
    - libjansson4
    - libcurl4
    - python3
    - python3-venv
    - openssh-server
    - rdma-core
    - libibverbs1
    - ibverbs-utils
    - perftest
    - ethtool
    - pciutils
    - cloud-init
    - kmod
  cnuas:
    # Kernel first so module .debs find /lib/modules/6.19.0-cnuas+/.
    - cnuas-linux-image          # 3 .debs (image, headers, libc-dev)
    - cnuas-gpu-modules        # cnuasgpu.ko (loads at boot via modules-load.d)
    - cnuas-nic-modules        # cnuas_net.ko + cnuas_ib.ko
    - cnuas-libcnuasrt
    - cnuas-vswitchd
    - cnuas-cli
    - cnuas-link-cli

user:
  name: cnuas
  sudo: "ALL=(ALL) NOPASSWD:ALL"
  password: cnuas
  ssh_pwauth: yes

# Modules autoload via the modules-load.d files baked into the
# cnuas-*-modules .debs, so modules_load here stays empty (avoids
# double-registration).
modules_load: []

services:
  enable:
    - ssh
    - cnuas-vswitchd
  disable:
    - unattended-upgrades
    - apt-daily.timer
    - apt-daily-upgrade.timer

udev:
  - file: 90-cnuas.rules
    rule: 'KERNEL=="cnuasgpu[0-9]*", MODE="0660", GROUP="cnuas"'

post_install:
  - echo "Cnuas golden image (kernel 6.19.0-cnuas+)" > /etc/motd
  - mkdir -p /run/cnuas
  # Remove the stock Ubuntu 6.8 kernel; we boot 6.19.0-cnuas+ only.
  - apt-get -y purge 'linux-image-6.8*' 'linux-headers-6.8*' 'linux-modules-6.8*' 'linux-image-generic' || true
  - update-grub2 || update-grub || true

6.2 Builder steps (cnuas_tools.image.builder.ImageBuilder.build)

  1. Pre-flight check that qemu-img, virt-customize, virt-resize, virt-sparsify, virt-ls, virt-filesystems are all on $PATH.
  2. Fetch base image if not cached; verify sha256.
  3. virt-resize --expand copies the base into out/cnuas-vm-<ver>.qcow2 at the target disk size, growing the ext4 root in the same pass. (Plain qemu-img resize only grows the qcow2 file; cloud-init's growpart fires only at first boot, but virt-customize never boots, so we must expand the FS up front.)
  4. Stage .debs under out/work/debs/. Resolves cnuas-linux-image to all matching linux-*.deb under out/kernel/, every other entry to a single <name>_*.deb from out/.
  5. virt-customize as a single invocation. The first --run-command is a multi-step shell script that (a) brings up eth0 in the appliance VM (libguestfs on Ubuntu 24.04 does not do this automatically), (b) writes a minimal /etc/resolv.conf, and (c) runs apt-get update, apt-get install <recipe.apt>, then dpkg -i of the staged Cnuas .debs with apt-get -fy install as a fallback. Subsequent --run-commands configure modules-load.d, enable/disable services, install udev rules, edit /etc/ssh/sshd_config, and run post-install commands.
  6. virt-sparsify in place.
  7. virt-ls /lib/modules picks up the installed kernel release for the manifest.
  8. Manifest + sha256 sidecars written next to the qcow2.

Key gotchas that drove the v1 design:

  • virt-customize --network is not the default on Ubuntu 24.04 libguestfs 1.52 despite the man page; pass it explicitly.
  • The libguestfs appliance uses SLIRP at 169.254.0.0/16 with gateway 169.254.2.2 and DNS proxy 169.254.2.3, not the qemu default 10.0.2.0/24.
  • virt-customize wipes /etc/resolv.conf between --run-command invocations, so any apt-network setup must happen inside the same command that does the apt operations.
  • The Ubuntu cloud image ships with no /etc/resolv.conf at rest (cloud-init writes it at boot).

The whole pipeline takes 4, 8 minutes on a modern workstation (virt-resize ~2 min, apt update + apt install ~2 min, dpkg + depmod ~1 min, sparsify ~1 min).

Manifest format (cnuas_tools.image.manifest):

{
  "manifest_version": 1,
  "name": "cnuas-vm",
  "version": "0.2.0",
  "built_at": "2026-05-18T15:44:13Z",
  "host": "cnuas1",
  "git_sha": "2953c878f3781285b4541cc49c543db707893a5c",
  "base_image": {
    "url": "https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img",
    "sha256": null
  },
  "kernel_version": "6.19.0-cnuas+",
  "components": {
    "cnuas-cli":                   {"version": "0.1.0",         "deb_sha256": "..."},
    "cnuas-gpu-modules":         {"version": "0.2.0",         "deb_sha256": "..."},
    "cnuas-link-cli":            {"version": "0.1.0",         "deb_sha256": "..."},
    "cnuas-nic-modules":         {"version": "0.1.0",         "deb_sha256": "..."},
    "cnuas-libcnuasrt":             {"version": "0.2.0",         "deb_sha256": "..."},
    "cnuas-vswitchd":              {"version": "0.1.0",         "deb_sha256": "..."},
    "linux-headers-6.19.0-cnuas+": {"version": "6.19.0-cnuas.1","deb_sha256": "..."},
    "linux-image-6.19.0-cnuas+":   {"version": "6.19.0-cnuas.1","deb_sha256": "..."},
    "linux-libc-dev":               {"version": "6.19.0-cnuas.1","deb_sha256": "..."}
  },
  "image_sha256": "ee3a779796ef6006a096c0ec60d05975c412fa6eecb1915bbeb2742bef..."
}

6.3 Idempotency

image build is purely functional: same inputs → same outputs. We achieve this by:

  • Pinning the base image sha256 in the recipe (mandatory for releases; optional during development).
  • Pinning the linux/ submodule to a specific commit (currently PacketFive/linux@cnuas-v6.19.0).
  • Disabling unattended-upgrades, apt-daily.timer, etc., before package install so dpkg runs deterministically.
  • Calling virt-sparsify to normalize qcow2 metadata.

7. VM lifecycle

7.1 Networking

Two networks per running lab:

Network Type Purpose
cnuas-mgmt libvirt NAT, 10.42.0.0/24 SSH from host, apt, NTP
cnuas-fabric UNIX sockets (/run/cnuas/port_N.sock) Cnuas vNIC ↔ switch

cnuas-mgmt is defined once on the host:

<network>
  <name>cnuas-mgmt</name>
  <forward mode='nat'/>
  <bridge name='cnuas-mgmt' stp='on'/>
  <ip address='10.42.0.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='10.42.0.10' end='10.42.0.99'/>
      <host mac='52:54:00:42:00:01' name='vm-a' ip='10.42.0.11'/>
      <host mac='52:54:00:42:00:02' name='vm-b' ip='10.42.0.12'/>
    </dhcp>
  </ip>
</network>

cnuas-fabric is not a libvirt network, it is a directory of UNIX sockets owned by cnuas-vswitchd, attached to each VM via the custom -device cnuas-vnic,socket_path=.... The switch is started separately (see §7.3).

7.2 Per-VM domain XML (template)

Rendered from templates/domain.xml.j2:

<domain type='kvm'>
  <name>cnuas-{{ name }}</name>
  <uuid>{{ uuid }}</uuid>
  <memory unit='GiB'>{{ ram_gb }}</memory>
  <vcpu placement='static'>{{ vcpus }}</vcpu>
  <cpu mode='host-passthrough' check='none'/>
  <os>
    <type arch='x86_64' machine='q35'>hvm</type>
    <boot dev='hd'/>
  </os>
  <devices>
    <emulator>{{ qemu_path }}</emulator>

    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='{{ disk_path }}'/>
      <target dev='vda' bus='virtio'/>
    </disk>

    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='{{ seed_iso }}'/>
      <target dev='sda' bus='sata'/>
      <readonly/>
    </disk>

    <interface type='network'>
      <mac address='{{ mac }}'/>
      <source network='cnuas-mgmt'/>
      <model type='virtio'/>
    </interface>

    <!-- Cnuas vNIC and CnuasGPU devices live in <qemu:commandline>
         until upstream libvirt adds direct support for them. -->
    <qemu:commandline xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
      <qemu:arg value='-device'/>
      <qemu:arg value='cnuas-vnic,socket_path={{ fabric_socket }}'/>
      {% for gpu in gpus %}
      <qemu:arg value='-device'/>
      <qemu:arg value='cnuasgpu,gpu_id={{ gpu.id }}'/>
      {% endfor %}
    </qemu:commandline>

    <serial type='pty'><target type='isa-serial' port='0'/></serial>
    <console type='pty'><target type='serial' port='0'/></console>
    <graphics type='none'/>
  </devices>
</domain>

7.3 vm up flow

cnuas-tools vm up vm-a --gpus 1
  1. Read ~/.local/share/cnuas/vms/vm-a/state.json if it exists. If the domain is already running, return.
  2. Ensure cnuas-mgmt libvirt network is defined and active. If not, define + start.
  3. Ensure switch is running on /run/cnuas/. If not, start cnuas-vswitchd --run-dir /run/cnuas --daemon.
  4. Resolve image path: ~/.local/share/cnuas/images/cnuas-vm-<ver>.qcow2. If not present, error (cnuas-tools image build first).
  5. Allocate per-VM directory ~/.local/share/cnuas/vms/vm-a/.
  6. qemu-img create -F qcow2 -b <golden> -f qcow2 disk.qcow2.
  7. Generate cloud-init seed:
  8. user-data.yaml: hostname, SSH public key (from ~/.ssh/id_*.pub), ssh_pwauth: yes, growpart, no upgrades.
  9. meta-data.yaml: instance-id, hostname.
  10. network-config.yaml: DHCP via ens3.
  11. genisoimage -o seed.iso -V cidata -r -J user-data meta-data network-config.
  12. Render domain.xml from template, fill in UUID, MAC, paths, GPU list.
  13. virsh define vm-a/domain.xml.
  14. virsh start cnuas-vm-a.
  15. Wait up to 60 s for SSH (poll ssh -o ConnectTimeout=2).
  16. Update state.json with pid, console_pts, ip, ssh_port.

7.4 vm down

virsh shutdown cnuas-vm-a    # ACPI; falls back to destroy after 30s

The overlay disk is preserved so the VM's own state survives across restarts. vm destroy --purge deletes the overlay.

7.5 vm ssh and vm scp

Discover the VM's IP from state.json (refreshed from virsh domifaddr when stale), then exec ssh cnuas@<ip> with sensible default options (-o StrictHostKeyChecking=accept-new).

7.6 vm lab

Brings up the full demo: 1. Switch. 2. vm up vm-a --gpus 1. 3. vm up vm-b --gpus 1.

Equivalent to running vm up for both with the right defaults.

7.7 vm console

virsh console cnuas-vm-a, same as today. Useful when SSH is broken; otherwise prefer vm ssh.

8. CI / ADO integration

(Phase 6.4, sketched here so this doc reflects the contract.)

Pipeline stage Pool Command
lint cnuas-linux cd tools && poetry install && poetry run poe lint
test cnuas-linux cd tools && poetry install && poetry run pytest
pkg-build cnuas-linux-big cnuas-tools pkg build
image-build cnuas-linux-big cnuas-tools image build (artefact: qcow2 + manifest)
lab-up cnuas-linux-big cnuas-tools vm lab && pytest -q tests/e2e/
release cnuas-linux cnuas-tools release publish $(VERSION)

Image and .deb artefacts are uploaded to ADO pipeline artefacts and mirrored to GitHub Releases by release publish.

9. Versioning

  • Each component repo has its own semver (cnuas-libcnuasrt → src/cnuasgpu/lib's SO_VERSION; cnuas-vswitchd → src/cnuasswitch).
  • cnuas-tools carries its own semver.
  • The golden image version is set by the user at build time (--version 0.2.0) and recorded in the manifest with the resolved component versions. There is no automatic image-version derivation in v1, that's a Phase 6.4 enhancement.

10. Testing strategy

10.1 Unit tests

tests/ mocks subprocess.run via a fixture that captures the intended command. Cover:

  • Recipe parsing.
  • Binary .deb layout (BinaryDebSpec, file/symlink staging, control, postinst/prerm).
  • Kernel-module .deb layout (KModSpec, modules-load.d snippet, depmod hooks).
  • Image builder pre-flight (refuses cleanly when libguestfs tools are missing).
  • Domain XML rendering for various GPU counts. (Phase 6.2e)
  • Cloud-init seed contents. (Phase 6.2e)
  • State-file lifecycle. (Phase 6.2e)
  • Error paths (missing image, libvirt-not-installed, fabric socket missing).

10.2 Integration tests (host)

A small tests/integration/ that actually invokes virt-customize against a tiny synthetic disk (a 64 MiB blank qcow2) to verify the subprocess calls succeed end-to-end. Skipped if virt-customize is not installed. Runs in the cnuas-linux-big pool.

10.3 End-to-end tests (nightly)

tests/e2e/ builds the real golden image, brings up vm-a, SSHes in, runs sgemm_smoke and compute_backend_smoke, asserts both pass with GFLOPS within a tolerance band. Runs in the cnuas-linux-big pool nightly.

11. Migration plan from current state

Phase Action
6.2a Design doc (this file).
6.2b tools/ skeleton (Poetry, Typer root). Stubs only. ✅
6.2c Implement pkg build for libcnuasrt, cnuas-vswitchd, the two Python CLIs (initially with DKMS targets for the kernel modules). ✅ (DKMS targets later replaced, see 6.2f.)
6.2d Implement image build. Produce the first cnuas-vm-v0.2.0.qcow2. ✅
6.2e Implement vm up/down/ssh/console/lab against libvirt. (in progress)
6.2f Replace DKMS targets with pinned-kernel binary targets: cnuas-linux-image builds the kernel .deb set from the linux/ submodule; cnuas-gpu-modules and cnuas-nic-modules ship pre-built .kos. Recipe rewrite. ✅
6.3 Migrate engine_subprocess → engine_libvirt (native libvirt-python bindings) behind the same VmEngine ABC.
6.4 ADO pipelines (pkg-build, image-build, vm-e2e, release publish).
6.5 Drop scripts/vm-manager.sh entirely; scripts/boot-vm-kernel-dev.sh (Approach C) stays for kernel-dev iteration.

12. Deferred decisions

Decisions that remain open for this component are carried in the Cnuas Roadmap.

13. Kernel pinning (Phase 6.2f rationale)

Cnuas pins guest VMs to a single Linux kernel, 6.19.0-cnuas+, built from the in-tree linux/ submodule. This is a deliberate trade-off:

Pro Con
Users do not build kernels or wait for DKMS on first boot. Image release cadence is tied to kernel release cadence.
Driver .kos are matched to the exact kernel they ship with, no unknown symbol surprises. Users cannot trivially upgrade their guest kernel without rebuilding the image.
Reproducibility: same Cnuas commit → same kernel binary → same .ko binaries → same image sha256. The Cnuas-specific patch (RDMA_DRIVER_CNUAS) cannot be merged upstream until our USENIX paper publishes, so the linux/ fork is meaningful infrastructure.
Simpler .debs: no dkms.conf, no install-time compile, no linux-headers-generic in the image. Removes the (theoretical) ability for security-conscious users to apt-upgrade to a more recent kernel.
Matches how mainline CUDA / NVIDIA HCA stacks ship for production. More work for us to bump kernel versions later, we'll be doing periodic rebases of the Cnuas patch.

13.1 The Cnuas kernel patch

PacketFive/linux@cnuas-v6.19.0 (commit 84d11a07) is upstream v6.19 plus this single patch:

--- a/include/uapi/rdma/ib_user_ioctl_verbs.h
+++ b/include/uapi/rdma/ib_user_ioctl_verbs.h
@@ -253,6 +253,7 @@ enum rdma_driver_id {
    RDMA_DRIVER_ERDMA,
    RDMA_DRIVER_MANA,
    RDMA_DRIVER_IONIC,
+   RDMA_DRIVER_CNUAS,
 };

The new uapi enum value is what cnuas_ib.ko registers as. Once the USENIX paper publishes we send the same patch (plus the driver itself) to linux-rdma@vger.kernel.org for upstreaming and deprecate this fork.

13.2 Multi-kernel future (optional)

If/when we want to support non-Cnuas kernels, the migration is:

  1. Add a new target cnuas-gpu-dkms (mirror of the deprecated one) producing source-only .debs. Same for cnuas-nic-dkms.
  2. The recipe gains an optional package_set: field selecting either pinned (current default) or dkms (multi-kernel).
  3. The image-build pipeline either installs `linux-image-6.19.0-cnuas+
  4. cnuas--modules(pinned) ordkms + build-essential + linux-headers-generic + cnuas--dkms` (dkms).
  5. The two paths can coexist in a single image with appropriate conflicts in the .deb metadata.

Not in v1 scope.

14. Out-of-scope reference, kernel-dev (Approach C)

For developers iterating on cnuasgpu.c, the workflow is unchanged from today: scripts/boot-vm-kernel-dev.sh does a direct-kernel boot using the in-tree linux/ submodule kernel + src/cnuasgpu/driver/cnuasgpu.ko built against the same source. No image, no DKMS. Documented in Cnuas_Driver_Design.md.