NeuralBenchError
run.py.NeuralBenchError
Base error for the neural benchmark harness.
ConfigError
run.py.ConfigError · inherits NeuralBenchError
Raised when config.yaml is missing required keys or has bad values.
validate_config
run.py.validate_config(cfg)
Raise ConfigError if cfg is missing required structure.
| Parameter | Type | Default | Description |
|---|---|---|---|
cfg |
dict |
- | (undocumented) |
model_params
run.py.model_params(cfg, name)
Resolve (chronax_params, nf_params) for name.
Models are auto-discovered, so config carries no per-model block by default;
cfg['overrides'][name] is an optional override. MAE loss is forced on both
sides. A standard model needs no override — both libraries fall back to their
(matching) defaults.
| Parameter | Type | Default | Description |
|---|---|---|---|
cfg |
dict |
- | (undocumented) |
name |
str |
- | (undocumented) |
Returns: tuple[dict, dict]
load_config
run.py.load_config(path)
Load and validate config.yaml.
A missing or unreadable file surfaces as ConfigError (not a bare OSError) so every config failure stays inside the NeuralBenchError taxonomy.
| Parameter | Type | Default | Description |
|---|---|---|---|
path |
str |
- | (undocumented) |
Returns: dict
load_results_tolerant
run.py.load_results_tolerant(path)
Read the results CSV, dropping any torn line left by a crash.
The worker streams one full row per seed via csv.DictWriter, so a clean line has exactly len(FIELDS) fields (error messages with commas/newlines are quoted). We parse with the stdlib csv reader — quote-aware, so a multi-line quoted error field is reassembled into one record — and keep only rows whose field count is exactly len(FIELDS). A crash-truncated trailing line has too few fields and is dropped. This does NOT rely on NaN-key/ParserError: pandas NaN-pads a short final line instead of raising, so a torn line truncated after the key columns must be caught by field count, not by dropna-on-keys.
| Parameter | Type | Default | Description |
|---|---|---|---|
path |
str | Path |
- | (undocumented) |
Returns: pd.DataFrame
done_keys
run.py.done_keys(df)
Set of (model, dataset, library, seed) rows already present.
| Parameter | Type | Default | Description |
|---|---|---|---|
df |
pd.DataFrame |
- | (undocumented) |
Returns: set
remaining_seeds
run.py.remaining_seeds(model, dataset, library, seeds, done)
Seeds not yet present for this {model,dataset,library}.
| Parameter | Type | Default | Description |
|---|---|---|---|
model |
str |
- | (undocumented) |
dataset |
str |
- | (undocumented) |
library |
str |
- | (undocumented) |
seeds |
Sequence[int] |
- | (undocumented) |
done |
set[tuple] |
- | (undocumented) |
Returns: list[int]
summarize
run.py.summarize(df, warmup_seeds, group_cols)
Mean/std over all seeds for MAE/sMAPE/wall-clock, plus an after-warmup wall-clock mean/std over the non-warmup seeds (spec §8).
| Parameter | Type | Default | Description |
|---|---|---|---|
df |
pd.DataFrame |
- | (undocumented) |
warmup_seeds |
int |
- | (undocumented) |
group_cols |
list |
- | (undocumented) |
Returns: pd.DataFrame
paired_accuracy
run.py.paired_accuracy(df, model, dataset)
| Parameter | Type | Default | Description |
|---|---|---|---|
df |
pd.DataFrame |
- | (undocumented) |
model |
str |
- | (undocumented) |
dataset |
str |
- | (undocumented) |
Returns: dict
paired_speed
run.py.paired_speed(df, model, dataset, warmup_seeds)
| Parameter | Type | Default | Description |
|---|---|---|---|
df |
pd.DataFrame |
- | (undocumented) |
model |
str |
- | (undocumented) |
dataset |
str |
- | (undocumented) |
warmup_seeds |
int |
- | (undocumented) |
Returns: dict
is_canonical
run.py.is_canonical(df, models, datasets, libs, seeds)
True iff df is exactly one clean, full, single-run set — the only file the gate will certify (spec §9). Requires: 1. every (model,dataset,library,seed) present exactly once (no missing/dup), 2. no error rows, and 3. per (model,dataset,library) the iter_idx values are exactly range(len(seeds)).
Check 3 is what makes a RESUMED file non-canonical: a resume runs only the remaining seeds, so the worker restarts iter_idx at 0 and the group's iter_idx multiset (e.g. [0,0,1,1,2]) no longer equals range(len(seeds)) even though every seed is present once. Resumed seeds re-pay JIT warmup, so their wall-clock is contaminated and the gate must refuse.
| Parameter | Type | Default | Description |
|---|---|---|---|
df |
pd.DataFrame |
- | (undocumented) |
models |
Sequence[str] |
- | (undocumented) |
datasets |
Sequence[str] |
- | (undocumented) |
libs |
Sequence[str] |
- | (undocumented) |
seeds |
Sequence[int] |
- | (undocumented) |
Returns: bool
accept_gate_report
run.py.accept_gate_report(df, models, datasets, libs, seeds, warmup_seeds)
Per-cell paired accept-gate table + overall verdict, or a non-canonical
notice (spec §8/§9). libs is the REQUIRED library set (both chronax and
nixtla) — a run missing a library, resumed, incomplete, or carrying error
rows is non-canonical. Returns text; never raises, never exits non-zero.
| Parameter | Type | Default | Description |
|---|---|---|---|
df |
pd.DataFrame |
- | (undocumented) |
models |
Sequence[str] |
- | (undocumented) |
datasets |
Sequence[str] |
- | (undocumented) |
libs |
Sequence[str] |
- | (undocumented) |
seeds |
Sequence[int] |
- | (undocumented) |
warmup_seeds |
int |
- | (undocumented) |
Returns: str
check_nf_venv
run.py.check_nf_venv(libs, venv_py=NF_VENV_PY)
Fail fast up front if NF is requested but .venv-nf is absent (spec §10).
| Parameter | Type | Default | Description |
|---|---|---|---|
libs |
(undocumented) | - | (undocumented) |
venv_py |
(undocumented) | NF_VENV_PY |
(undocumented) |
stream_worker_to_csv
run.py.stream_worker_to_csv(model, dataset, library, config_path, csv_path, done, seeds)
Spawn one worker for {model,dataset,library}; append each streamed RESULT_JSON row to csv_path (skipping already-done keys). Resumable by construction.
On resume the worker is handed only the remaining seeds, so it restarts iter_idx at 0 — which is exactly what is_canonical (Task 9) uses to flag a resumed file as non-canonical and refuse a verdict.
| Parameter | Type | Default | Description |
|---|---|---|---|
model |
(undocumented) | - | (undocumented) |
dataset |
(undocumented) | - | (undocumented) |
library |
(undocumented) | - | (undocumented) |
config_path |
(undocumented) | - | (undocumented) |
csv_path |
(undocumented) | - | (undocumented) |
done |
(undocumented) | - | (undocumented) |
seeds |
(undocumented) | - | (undocumented) |
nf_baseline_metadata
run.py.nf_baseline_metadata(nf_name, h, input_size, nf_params, venv_py=NF_VENV_PY)
Query .venv-nf for env versions + the fully-resolved model hyperparameters so protocol.json stays reconstructible even if NF's architecture defaults drift later (spec §13). One short subprocess in the isolated venv.
| Parameter | Type | Default | Description |
|---|---|---|---|
nf_name |
(undocumented) | - | (undocumented) |
h |
(undocumented) | - | (undocumented) |
input_size |
(undocumented) | - | (undocumented) |
nf_params |
(undocumented) | - | (undocumented) |
venv_py |
(undocumented) | NF_VENV_PY |
(undocumented) |
Returns: dict
write_baseline
run.py.write_baseline(model, df, cfg)
Write baselines/
| Parameter | Type | Default | Description |
|---|---|---|---|
model |
(undocumented) | - | (undocumented) |
df |
(undocumented) | - | (undocumented) |
cfg |
(undocumented) | - | (undocumented) |
check_committed_report
run.py.check_committed_report(chronax_df, committed_summary, warmup_seeds)
Coarse (unpaired) chronax-vs-committed-NF-summary check (spec §8 --check-committed). Not the paired canonical verdict; cheap iteration.
Requires a post-recapture summary (with the after-warmup wall-clock column); the migrated legacy GRU summary predates it, so guard explicitly rather than KeyError mid-report.
| Parameter | Type | Default | Description |
|---|---|---|---|
chronax_df |
pd.DataFrame |
- | (undocumented) |
committed_summary |
pd.DataFrame |
- | (undocumented) |
warmup_seeds |
int |
- | (undocumented) |
Returns: str