Calibration and Validation¶

Terms¶

Calibration fits model parameters using a training set. Validation scores the fitted model against a separate held-out set. Reporting fit error as prediction error is not valid.

The first implemented correction is:

[ \hat{t}=\max(0,\alpha t_{\mathrm{analytic}}+\beta). ]

Two or more distinct analytic predictions fit (\alpha) and (\beta) by ordinary least squares. One observation fits a scale through the origin and is only a preliminary calibration. A non-positive fitted scale is rejected.

1. Define the context of use¶

Before measuring, write down the decision the model will support. Examples:

comparing queue counts for one NIC revision and firmware;
estimating one accelerator kernel family over a stated tensor range;
modelling ORV3 polling latency for one controller and bus topology.

Do not begin with a claim such as "predicts all Cnuas hardware performance."

2. Record target identity¶

At minimum preserve:

Area	Required details
Hardware	Product, revision, serial-independent configuration
Firmware	Version and settings
Clocks and power	Frequency, governor, thermal and power policy
Links	Generation, width, speed, MTU, encoding
Software	OS, kernel, driver, runtime, compiler, emulator versions
Workload	Packet, message, or tensor shape; batch; operation count
Contention	Queue, flow, lane, rank, and concurrency counts
Measurement	Instrument, repetitions, warm-up, statistic

Use this complete description as the calibration target string or associate it with a target manifest stored beside the profile.

3. Split observations before fitting¶

Use separate files:

training.json:

{
  "observations": [
    {"predicted_ns": 1000, "measured_ns": 1170, "label": "4 KiB, one queue"},
    {"predicted_ns": 2000, "measured_ns": 2270, "label": "8 KiB, one queue"},
    {"predicted_ns": 4000, "measured_ns": 4470, "label": "16 KiB, one queue"}
  ]
}

held-out.json:

{
  "observations": [
    {"predicted_ns": 1500, "measured_ns": 1725, "label": "6 KiB, one queue"},
    {"predicted_ns": 3000, "measured_ns": 3360, "label": "12 KiB, one queue"}
  ]
}

The labels are required experimental context even though the current parser accepts an empty label.

4. Fit¶

cnuas-timing calibrate cnuasnic \
  "target product/revision; firmware; 100G; MTU; one queue; software versions" \
  training.json > calibration.json

The output records:

{
  "component": "cnuasnic",
  "target": "target product/revision; firmware; 100G; MTU; one queue; software versions",
  "scale": 1.1,
  "offset_ns": 70.0,
  "observations": 3,
  "fit_rmse_ns": 0.0
}

Fit RMSE describes the training data only.

5. Validate held-out data¶

cnuas-timing validate calibration.json held-out.json

The report contains:

Metric	Meaning
MAE	Mean absolute duration error
RMSE	Root mean squared error, more sensitive to large errors
MAPE	Mean absolute percentage error
Maximum absolute error	Worst held-out miss

Also report error by workload regime. One aggregate can hide a packet-rate failure at small messages or a bandwidth failure at large messages.

6. Decide whether the affine model is adequate¶

Inspect residuals against:

work size;
queue and flow count;
concurrency;
compute versus memory intensity;
temperature and clock state.

A pattern means the profile is missing a resource or the affine correction is too simple. Do not add more coefficients only to reduce training error. Add a physically interpretable stage, split the operating range, or use a specialist simulator.

7. Apply a calibration in Python¶

from cnuas_timing import Observation, fit, load_profile

profile = load_profile("profiles/cnuas-analytic-v0.json")
calibration = fit(
    "cnuasnic",
    "named target and workload conditions",
    [Observation(1000, 1170), Observation(2000, 2270)],
)
calibrated_nic = calibration.apply(profile.component("cnuasnic"))

Create a new TimingProfile with the calibrated component and use dump_profile to preserve it. Never overwrite the analytic reference profile.

Publication checklist¶

State the evidence class.
Publish profile and pipeline versions.
Publish raw observations when permitted.
Identify training and held-out rows.
Report measurement variability, not only model error.
Report the workload range.
Avoid extrapolation outside that range.
Keep the target name with every calibrated result.