Cnuas CI Strategy, Azure DevOps with Self-Hosted Agents on `cnuas2`¶

Status: design, Phase 6.0e. Author: Cnuas project. Companion docs: - Cnuas_Build_System.md (Bazel/Make conventions) - src/cnuasgpu/docs/CnuasGPU_FPGA_PolarFire.md (FPGA acceptance, §8 references this doc)

Audience: developers who maintain the pipelines; bring-up engineers who install or upgrade agents; reviewers triaging red builds.

1. Goals¶

The Cnuas CI must:

Run KVM-accelerated guests, every functional/e2e test in this project boots one or more QEMU/KVM VMs. Microsoft-hosted ADO agents give no nested KVM (and even if they did, they are slow and non-deterministic for our case). We need real hardware.
Expose PCIe devices, the PolarFire SoC Icicle Kit (Phase 6.5+) must be passed through to a dedicated agent VM. Microsoft-hosted can't do this.
Survive PR review at sub-15-minute latency for the common "edit a source file, push a PR" workflow.
Keep cnuas1 interactive, it is the developer's primary box. CI lives on cnuas2. A stuck pipeline must not lock out cnuas1.
Be reproducible: any agent VM image is fully derivable from versioned scripts under cnuas/ci/, no clicked-through, hand-built golden images.

2. Topology¶

flowchart TB ADO["Azure DevOps SaaS org: packetfive  •  project: cnuas pools: cnuas-linux, cnuas-linux-big, cnuas-fpga"] subgraph H1["cnuas1 - DEV ONLY"] H1D["Xeon W-2133, 12c/64G  •  Ubuntu 24.04 vm-a, vm-b (manual) for interactive development NO CI AGENTS"] end subgraph H2["cnuas2 - CI HOST (Ubuntu 24.04, libvirt+KVM, nested KVM)"] A1["ado-agent-01 (Linux-small) 4 vCPU, 16 GB, 80 GB • pool: cnuas-linux caps: CNUAS_KVM=1"] A2["ado-agent-02 (Linux-small) identical to -01 (parallelism for PRs)"] ABIG["ado-agent-big (Linux-big) 8 vCPU, 32 GB, 200 GB • pool: cnuas-linux-big caps: CNUAS_KVM=1, CNUAS_LARGE=1 e2e (vm-a+vm-b inside)"] AFPGA["ado-agent-fpga (Phase 6.5) 4 vCPU, 16 GB • pool: cnuas-fpga PCIe passthrough: Icicle Kit • caps: CNUAS_FPGA=1"] end ADO -->|"HTTPS (agent → ADO)"| H2 classDef saas fill:#dbeafe,stroke:#1e40af,stroke-width:2px,color:#1e3a8a; classDef dev fill:#fef3c7,stroke:#b45309,stroke-width:2px,color:#78350f; classDef agent fill:#dcfce7,stroke:#166534,stroke-width:2px,color:#14532d; class ADO saas; class H1D dev; class A1,A2,ABIG,AFPGA agent;

cnuas2 baremetal does not itself run an agent; only its VMs do. That keeps the host kernel boring and lets us reimage agent VMs without losing the libvirt control plane.

3. ADO organisation layout¶

Object	Name
Organisation	`packetfive` (`dev.azure.com/packetfive`)
Project	`cnuas`
Agent pools	`cnuas-linux`, `cnuas-linux-big`, `cnuas-fpga`
Variable groups	`cnuas-secrets`, `cnuas-build-versions`
Service connections	`github-packetfive` (read-only mirror), `docker-ghcr` (artefacts)

3.1 Pool purposes¶

Pool	Agents	Used by
`cnuas-linux`	2 small VMs	PR validation, unit tests, lint, doc builds
`cnuas-linux-big`	1 big VM	Nightly e2e (vm-a + vm-b), large Bazel builds, kernel module compilation
`cnuas-fpga`	1 VM with Icicle Kit	FPGA bitstream sim, hardware acceptance tests

3.2 Routing pipelines to pools¶

In each azure-pipelines.yml:

jobs:
  - job: pr_lint
    pool:
      name: cnuas-linux
    steps: [...]

  - job: e2e
    pool:
      name: cnuas-linux-big
      demands:
        - CNUAS_KVM -equals 1
    steps: [...]

  - job: fpga_acceptance
    pool:
      name: cnuas-fpga
      demands:
        - CNUAS_FPGA -equals 1
    condition: eq(variables['Build.Reason'], 'Schedule')
    steps: [...]

4. Agent VM provisioning¶

4.1 Host (`cnuas2`) prep, one-time¶

# Enable nested KVM (Intel; AMD is kvm-amd nested=1)
echo 'options kvm-intel nested=Y' | sudo tee /etc/modprobe.d/kvm-nested.conf
sudo modprobe -r kvm-intel || true
sudo modprobe kvm-intel
cat /sys/module/kvm_intel/parameters/nested   # → Y

# libvirt + tools
sudo apt-get install -y qemu-kvm libvirt-daemon-system virtinst \
    cloud-image-utils ovmf bridge-utils
sudo usermod -aG libvirt,kvm "$USER"

# A simple NAT'd virtual network is enough; default 'default' libvirt net is fine.

4.2 Golden agent image¶

We do not install ADO agent into a "pet" VM. Instead, we provision each VM from a cloud-init image plus a versioned bootstrap script:

cnuas/ci/
  agent-cloud-init.yaml         # cloud-init: users, sudo, ssh keys
  agent-bootstrap.sh            # idempotent setup: ADO agent + tools
  README.md
  Makefile                      # `make agent NAME=ado-agent-01 SIZE=small`
  pool.env.example              # AZP_URL, AZP_POOL, AZP_TOKEN placeholders

agent-bootstrap.sh (sketch):

#!/usr/bin/env bash
# Run as root inside a fresh Ubuntu 24.04 VM.
set -euxo pipefail

apt-get update -q
apt-get install -y -q \
    build-essential bazel-bootstrap clang lld gdb make cmake ccache \
    python3 python3-venv python3-pip pytest \
    qemu-system-x86 qemu-utils \
    libjansson-dev libcurl4-openssl-dev \
    libibverbs-dev rdma-core ibverbs-providers \
    pkg-config git curl jq verilator

# Build tools needed for CnuasGPU kernel module
apt-get install -y -q linux-headers-generic

# ADO agent user
useradd -m -s /bin/bash -G kvm,libvirt ado || true

# Install agent
sudo -u ado bash -c '
  cd ~ado
  mkdir -p agent && cd agent
  curl -sL https://vstsagentpackage.azureedge.net/agent/3.241.0/vsts-agent-linux-x64-3.241.0.tar.gz | tar xz
  ./bin/installdependencies.sh
'

# Configure as service (PAT supplied via env)
sudo -u ado bash -c "
  cd ~ado/agent
  ./config.sh --unattended \
    --url '${AZP_URL}' \
    --auth pat --token '${AZP_TOKEN}' \
    --pool '${AZP_POOL}' \
    --agent '${AZP_AGENT}' \
    --acceptTeeEula \
    --replace
"
cd ~ado/agent
./svc.sh install ado
./svc.sh start

Note: this is GUI-free, fully scripted. The image is reproducible from cnuas/ci/; a stuck VM is destroyed and re-created in minutes.

4.3 Per-VM resources (recommended)¶

Agent	vCPU	RAM	Disk	Nested KVM	Notes
`ado-agent-01`	4	16 GB	80 GB	Yes	PR validation
`ado-agent-02`	4	16 GB	80 GB	Yes	PR validation, parallelism
`ado-agent-big`	8	32 GB	200 GB	Yes	E2E, big builds
`ado-agent-fpga`	4	16 GB	80 GB	n/a	+ PCIe passthrough of Icicle Kit

Set CPU mode host-passthrough so AVX2/AVX-512/RVV detection inside the VM matches the bare host, required for the compute-backend selection logic.

4.4 Capabilities¶

ADO auto-detects many caps (Python, Bash, etc.). Custom ones we add via ~ado/.agent_capabilities:

CNUAS_KVM=1
CNUAS_ARCH=x86_64
CNUAS_AVX2=1
CNUAS_AVX512=0      # set 1 if the bare host CPU has it
CNUAS_LARGE=1       # only on -big agent
CNUAS_FPGA=1        # only on -fpga agent

Restart the agent (svc.sh stop && svc.sh start) for caps to publish.

5. PAT and secrets¶

5.1 PAT issuance¶

One PAT per agent, single scope Agent Pools (read, manage), 90-day expiry, rotated by calendar reminder.

Store the PAT only on the agent VM, file mode 0600, owned by root, read by the bootstrap script during initial config.sh. Once ./config.sh finishes, the PAT is no longer needed; the agent holds its own session credential.

5.2 Pipeline secrets¶

ADO variable groups, scoped to the project:

Group	Contents
`cnuas-secrets`	container registry token, signing key (passphrase-protected)
`cnuas-build-versions`	`BAZEL_VERSION`, `CLANG_VERSION`, etc. (non-secret defaults)

Variables marked secret are masked in logs and never exported to the job environment unless the step explicitly maps them in env:.

6. Caching strategy¶

Two layers:

ADO Pipeline Caching (Cache@2 task), stored in ADO cloud storage, keyed by lockfile hash. Use for:
~/.cache/bazel/_bazel_ado/install (Bazel install base)
~/.ccache
~/.cache/pip
Host-side bind mounts, for things too big or too volatile for cloud caching:
/var/cache/ccache on cnuas2 → mounted into each agent VM read-write. Shared between agents; ccache handles concurrency.
/var/cache/bazel-disk on cnuas2 → similar, shared Bazel disk cache (build --disk_cache=/cache/bazel).

Bind mounts go via 9p / virtiofs. Use virtiofs for performance.

<filesystem type='mount' accessmode='passthrough'>
  <driver type='virtiofs'/>
  <source dir='/var/cache/ccache'/>
  <target dir='ccache_host'/>
</filesystem>

Inside the VM:

mkdir -p /var/cache/ccache
mount -t virtiofs ccache_host /var/cache/ccache

7. Per-repo pipelines¶

Each component repo has its own azure-pipelines.yml.

7.1 `cnuas/azure-pipelines.yml` (existing, update)¶

Already exists; targets vmImage: ubuntu-24.04 (Microsoft-hosted). Swap it to pool: { name: cnuas-linux } for PR validation, add a nightly e2e stage targeting cnuas-linux-big with full submodule checkout (submodules: recursive), KVM smoke test, and pytest.

Two ordering constraints apply in the Test stage, neither of which is obvious from reading the steps:

The Docs job runs scripts/datasheet/build.py before mkdocs build --strict. docs/datasheets/index.md links every sheet as ../pdf/<Name>_Datasheet.pdf, and docs/hooks/datasheet_pdfs.py registers those paths from out/datasheets/. MkDocs validates internal links against its file set, so with no PDFs rendered the strict build aborts with one broken link per sheet. Any new pipeline that runs mkdocs has to keep this order.
The CnuasIRTools job installs binutils-riscv64-unknown-elf before running make -C src/cnuasgpu/tools check. That suite holds its instruction decoder against objdump as an external oracle, and skips the comparison when no toolchain is present. Skipping is the right behaviour on a developer machine and the wrong one in CI, where it would silently retire the only independent check on the decoder.

7.2 `src/cnuasgpu/azure-pipelines.yml` (new)¶

trigger: { branches: { include: [main, 'release/*'] } }
pr:      { branches: { include: [main] } }

stages:
  - stage: lint
    jobs:
      - job: lint
        pool: { name: cnuas-linux }
        steps:
          - checkout: self
          - script: make -C lib/libcnuasrt lint
            displayName: 'libcnuasrt lint'

  - stage: build_test
    jobs:
      - job: userspace
        pool: { name: cnuas-linux }
        steps:
          - script: make -C lib/libcnuasrt && make -C lib/test test
            displayName: 'libcnuasrt + tests'

      - job: kmod
        pool: { name: cnuas-linux-big }
        steps:
          - script: make -C driver KDIR=/lib/modules/$(uname -r)/build
            displayName: 'cnuasgpu.ko (out-of-tree)'

  - stage: nightly
    condition: eq(variables['Build.Reason'], 'Schedule')
    jobs:
      - job: e2e
        pool: { name: cnuas-linux-big, demands: ['CNUAS_KVM -equals 1'] }
        steps:
          - script: ./tests/run-vm-smoke.sh
            displayName: 'QEMU VM smoke'

      - job: fpga
        pool: { name: cnuas-fpga, demands: ['CNUAS_FPGA -equals 1'] }
        condition: succeeded()
        steps:
          - script: ./fpga/host/pcie-bringup.sh
            displayName: 'FPGA PCIe bring-up'
          - script: ./tests/run-fpga-smoke.sh
            displayName: 'FPGA SGEMM smoke'

7.3 `src/cnuaslink/azure-pipelines.yml` and `src/cnuasswitch/azure-pipelines.yml`¶

Same skeleton: lint on small pool, build+pytest on small pool, e2e (switch + cli + vm peers) on big pool nightly.

7.4 `src/cnuasnic/azure-pipelines.yml`¶

Driver build + rdma-core integration. Big pool for the kernel module build; small pool for userspace tests.

8. Branch / PR policies¶

Configured under Project Settings → Repos → Branch policies → main:

Policy	Setting
Require minimum reviewers	1
Check for linked work items	optional
Check for comment resolution	required
Build validation	`cnuas-pr`, `cnuasgpu-pr`, `cnuaslink-pr`, `cnuasswitch-pr`, `cnuasnic-pr` (all auto-queued on every PR push, all on `cnuas-linux`)
Status checks	`lint`, `unit-tests` must be green
Limit merge types	"Squash" preferred

Nightly pipelines run on a schedule:

schedules:
  - cron: '0 3 * * *'      # 03:00 UTC nightly
    displayName: nightly
    branches: { include: [main] }
    always: true

9. FPGA agent (Phase 6.5+), PCIe passthrough¶

When the Icicle Kit is physically attached to cnuas2:

lspci -nn | grep -i microsemi, find the BDF.
Bind to vfio-pci on cnuas2 boot (modprobe vfio-pci ids=11aa:1556 adjust to the actual PCIe vendor/device of the Icicle Kit endpoint).
In the ado-agent-fpga libvirt XML:

<hostdev mode='subsystem' type='pci' managed='yes'>
  <source>
    <address domain='0x0000' bus='0x42' slot='0x00' function='0x0'/>
  </source>
</hostdev>

Inside the VM: lspci -nn should now show 1af4:10f1 (our CnuasGPU IDs as presented by the FPGA bitstream).
modprobe cnuasgpu → /dev/cnuasgpu0 appears → fully software-identical to QEMU.

The agent advertises CNUAS_FPGA=1; fpga-tagged jobs land here.

10. Observability¶

Each agent's _diag/ rotates daily; uploaded to ADO automatically on job failure.
A small Prometheus exporter on cnuas2 host scrapes virsh and publishes agent_running{name=...} so we can detect dead agents.
Pipeline failures fire an email + (later) a webhook to the team chat.

11. Disaster recovery¶

Agent VM dies → destroy + re-provision from cnuas/ci/ scripts. Ten minutes start-to-finish; capabilities and PATs are re-issued by the bootstrap.
cnuas2 baremetal fails → pipelines queue. A spare host (any Linux workstation with KVM) can run the same cnuas/ci/ scripts and bring up agents under the same pool names; ADO doesn't care where they live.
ADO outage → pipelines pause; nothing else affected. Self-hosted agents resume on reconnect.

12. Migration plan (from current Microsoft-hosted `ubuntu-24.04`)¶

Step	Action
1	Stand up `ado-agent-01` on cnuas2; register to `cnuas-linux`.
2	Add a second pipeline targeting `cnuas-linux` in parallel with the existing Microsoft-hosted one; compare results for a week.
3	Flip the canonical pipeline to `cnuas-linux`; keep Microsoft-hosted as a fallback in a separate `azure-pipelines.microsoft.yml`.
4	Add `ado-agent-02` for parallelism.
5	Stand up `ado-agent-big`; add nightly stage.
6	(Phase 6.5) Stand up `ado-agent-fpga`; enable fpga schedule.

13. Deferred decisions¶

Decisions that remain open for this component are carried in the Cnuas Roadmap.

Cnuas CI Strategy, Azure DevOps with Self-Hosted Agents on cnuas2¶