Skip to content

Build sandbox

Draft Defined in RFC 0001 §9 Default backend unshare(2)

A reproducible description is only half of a reproducible build. The build itself must not be able to reach anything the description does not mention, such as a library that happens to be installed on the host, a file in a home directory, or a server on the network. Lattice prevents this by running every build in a fresh set of Linux namespaces that it creates itself through the unshare(2) system call. This default needs no Docker, Podman or Bubblewrap. A machine can instead use an OCI container runtime or a lightweight virtual machine, described under Backends.

Isolation

Namespace What the build sees
User It runs as root inside the sandbox, but only as the unprivileged Lattice user on the host.
Mount Its own filesystem tree, which the host never sees and which disappears when the build ends.
Network A loopback interface and nothing else. There is no route to any other machine.
PID Its own process tree, in which the builder is PID 1. Host processes are invisible.
UTS A fixed hostname, localhost, so the host's name cannot end up in a build.
IPC No shared memory, semaphores or message queues with host processes.

Filesystem

The root of the sandbox is an empty tmpfs. Lattice gives it a writable /build for the build to work in and a writable /lattice/store, in which the build creates each output at its final store path. The $out variable holds that path, so an installation prefix the build records is already correct. The store paths of the build's declared inputs are mounted read-only at their usual locations, a fresh /proc is mounted for the sandbox's own processes, and /dev holds only null, zero, full, random, urandom and a /dev/shm. An input that is itself a symbolic link is recreated as a link, never mounted, so it cannot expose the host path it names. Nothing else from the host is present, so the builder itself must be a program inside one of the inputs.

One build, start to finish

---
config:
  themeVariables:
    fontSize: 16px
  flowchart:
    useMaxWidth: false
  sequence:
    useMaxWidth: false
---
sequenceDiagram
    participant L as Lattice
    participant S as Sandbox
    L->>S: start a fresh helper process
    S->>S: create namespaces
    S->>S: build root, mount inputs read-only
    S->>S: drop capabilities, no new privileges,<br/>seccomp filter
    S->>S: run builder in /build
    S->>S: copy outputs out of the tmpfs
    S-->>L: result

When the builder exits, the kernel ends every process it started, because they all belong to the sandbox's PID namespace. Lattice then copies each output the derivation declared out of the sandbox. The copy never follows a symbolic link the build made, so a link cannot point it at a host file. If an output is missing, holds a FIFO, socket or device file, or the builder failed, nothing is published.

Hardening

The builder holds no capabilities, so although it is root inside the sandbox it cannot unmount or change its read-only inputs. It cannot gain privileges either, because Lattice sets the kernel's "no new privileges" flag before starting it. A seccomp filter then refuses the few system calls that would leave something dangerous in the store or reach kernel state the namespaces do not isolate: setting the setuid or setgid bit, setting extended attributes, using kernel keyrings, and io_uring, which would bypass the filter. A refused call returns an error rather than ending the build. Setuid and setgid bits that a file was created with are dropped when the outputs are copied out. Core dumps are off, open files are capped, the umask is 022, and a build that runs too long or stays silent too long is stopped.

When Lattice runs under a service manager that delegates a cgroup version 2 subtree to it, as lattice daemon does on PyrixOS with Delegate=yes, each build also gets its own cgroup. The settings max_memory_mib, max_cpu and max_processes become its memory.max, cpu.max and pids.max, swap is off, and removing the cgroup after the build ends any process the build left behind, even one that escaped into a new session. Without a delegated subtree these three limits are not applied, and the cgroups line of lattice backend-status says why.

Build environment

A build sees only the environment variables its derivation declares, one variable per output, such as $out, holding the output's store path, and a fixed set that it cannot change.

Variable Value Why
SOURCE_DATE_EPOCH 315532800 Tools that embed a date use 1980-01-01 instead of the clock.
TZ, LANG UTC, C.UTF-8 Output does not depend on the host's time zone or language.
HOME /homeless It does not exist, so no user configuration is read.
TMPDIR /build Temporary files stay in the sandbox.
PATH /path-not-set Programs come only from store paths the derivation names.

Every setting of the sandbox maps to a field of the standard OCI runtime configuration, and lattice sandbox-config prints that configuration for any derivation, so the sandbox can be reviewed with existing container security tools.

$ lattice sandbox-config hello.pyb hello
$ lattice backend-status
namespaces: available
oci: available
microvm: no guest kernel and init are configured

Backends

Backend Runs the build in Isolation boundary
Namespaces (default) Linux namespaces created by Lattice, with nothing else installed The host kernel
OCI container A rootless runtime such as crun, runc or youki, with no daemon and no image The host kernel
MicroVM A lightweight virtual machine under Firecracker, Cloud Hypervisor or OpenVMM The hypervisor, with the build on its own guest kernel

With the microVM backend the build no longer depends on the host kernel. The guest kernel and a small guest init are named in the settings, usually as files in store paths, and both are recorded in the provenance of every path built on that backend. The virtual machine has no network device. The build runs as root of the guest without capabilities, with the kernel settings under /proc mounted read-only. Its inputs arrive on a read-only disk image and its outputs leave as an archive on a second disk, which Lattice unpacks with the same rules as the other backends. Firecracker and Cloud Hypervisor are tested; OpenVMM follows its documented command line and is untested, and the Hyper-V utility VM on Windows is not implemented yet.

Every backend gives the build the same filesystem, hardening and environment, and none gives it a network. If the chosen backend is unavailable, Lattice refuses to build rather than falling back to a weaker one. Because a reproducible build gives the same bytes on any backend, lattice check --backend microvm hello.pyb hello rebuilds a path in a virtual machine and reports any file that differs from the copy in the store.

sandbox_backend = "microvm"

[sandbox]
runtime = "crun"      # for the oci backend

[sandbox.microvm]
vmm = "firecracker"
kernel = "/lattice/store/<hash>-linux-guest/vmlinux"
init = "/lattice/store/<hash>-cloche-guest-init/init"

Host requirements

The sandbox needs unprivileged user namespaces, which most current Linux kernels provide. Some distributions restrict them by default, for example Ubuntu through the kernel.apparmor_restrict_unprivileged_userns setting. In that case Lattice reports the problem and refuses to build, rather than falling back to a build without isolation. lattice backend-status names the setting that blocks the sandbox. The OCI backend needs its runtime program installed. The microVM backend needs KVM, the virtual machine monitor and mkfs.ext4, and every file of the inputs must be readable by the user who builds.

Implementation

The sandbox is Cloche, a separate Python package that runs any program in this kind of sandbox and knows nothing of derivations or the store. Other projects can use it on its own, from Python or with the cloche command. It calls the few system calls the standard library lacks, such as mount and pivot_root, through ctypes.

Cloche module Contents
models.py Command, Bind and Limits, the Espalier models of one run
spec.py The sandbox description every backend consumes, and its OCI form
environment.py The fixed environment and the rules the program's variables must follow
seccomp.py The denied system calls, as a BPF program and as OCI rules
libc.py mount, pivot_root, prctl and capability calls through ctypes
namespaces.py The namespace backend and the helper process that sets up each run
oci.py The OCI backend, which drives a rootless runtime
microvm.py, guest_init.c, newc.py The microVM backend, its guest init and the archive format of its outputs
runner.py Runs one program, records its log and enforces the time limits
cgroups.py The cgroup of each run under a delegated subtree, with its memory, CPU and process limits
probe.py Tests which backends can run on the machine

Lattice's lattice.sandbox module turns a derivation into a Cloche Command. The builder becomes the program, env and the output paths become the environment, each input is bound read-only at its store path from the host store, and the store directory is an empty writable directory in which the builder creates its outputs. Lattice adds the rules that involve derivations: a builtin: builder never reaches the sandbox, the builder must be inside an input, and env may not set an output's name. Cloche's errors become BuildFailed and BackendUnavailable.

The full specification, including the order of system calls, why the builder needs a second fork, and the provenance and compliance targets, is in RFC 0001 §9.

Using Cloche outside PyrixOS

Cloche runs any program, not only builders. A program that must run code it does not trust, such as a build tool, a test runner or a grader for submitted code, can give it only the host paths it needs and collect the files it declares.

import tempfile

from cloche import Bind, Command, Limits, run

command = Command(
    name="hello",
    program="/usr/bin/sh",
    args=("-c", "echo hello > /build/greeting"),
    binds=tuple(Bind(source=p, destination=p) for p in ("/usr", "/lib", "/lib64")),
    outputs={"greeting": "/build/greeting"},
)
outcome = run(command, tempfile.mkdtemp(), Limits(timeout=60))

The same run from the shell is cloche run --bind /usr --bind /lib --bind /lib64 --output greeting=/build/greeting /usr/bin/sh -c 'echo hello > /build/greeting', and cloche status reports whether the machine can run the sandbox.