run_benchmark

run_benchmark

Execute benchmark orchestration over configured datasets and models.

run_benchmark(config_path, model_filter=None, dataset_filter=None, forecast_mode=False) -> None

Loads benchmark configuration, applies optional dataset/model filters, dispatches per-model jobs to benchmark_suite.py subprocesses, and aggregates returned JSON metrics into benchmark result artifacts.

Parameter	Type	Default	Description
config_path	str	-	Absolute or relative path to configuration YAML.
model_filter	Optional[str]	None	Restrict run to a single model.
dataset_filter	Optional[str]	None	Restrict run to one dataset name or external dataset path.
forecast_mode	bool	False	Enable forecast artifact mode instead of benchmark aggregation mode.

Returns: None (Writes artifacts and prints progress to stdout.) Raises: subprocess.CalledProcessError (Captured and logged per worker run.)

process_forecast_result

run_benchmark

Persist a single forecast result to CSV and PNG for inspection and reporting.

process_forecast_result(record, output_dir, dataset_name, library, model_name) -> None

Consumes a forecast-mode result record from the benchmark worker (containing y_train, y_test, predictions, and optionally MAPE). Writes a CSV file under output_dir/data with columns y_true, split (train/test), y_pred (forecast aligned to test period), library, and model. Then generates a matplotlib figure plotting training series, test actuals, and forecast series with a title including dataset, library, model, and MAPE; saves the figure as a PNG under output_dir/plots. Used by run_benchmark when forecast_mode is True to produce per-model artifacts without changing the worker's JSON contract.

Parameter	Type	Default	Description
record	Dict[str, Any]	-	Must contain "y_train", "y_test", "predictions" (lists or array-like), and optionally "MAPE". Keys are as returned by run_single_model in forecast mode.
output_dir	str	-	Base directory for results; data and plots are written to output_dir/data and output_dir/plots.
dataset_name	str	-	Human-readable dataset name for filenames and plot title.
library	str	-	"chronax" or "statsforecast" for filenames and title.
model_name	str	-	Model name for filenames and title.

Returns: None (Writes one CSV and one PNG; prints the saved plot basename to stdout.) Raises: KeyError (If record is missing "y_train", "y_test", or "predictions".) Raises: IOError (If writing the CSV or PNG fails.)

plot_results

run_benchmark

Generate benchmark summary visualizations from an aggregated results CSV.

plot_results(csv_path) -> None

Loads the benchmark CSV (with columns Dataset, Model, Length, Time_Warm_Sec, MAPE, etc.) and produces two types of plots in the same directory as the CSV. (1) Scalability plot: for each dataset, a subplot of Time_Warm_Sec vs Length (log-log) per model, to compare runtime scaling. (2) Accuracy plot: for each dataset, a subplot of MAPE vs Length (log x) per model, to compare forecast accuracy. Uses a fixed color map across models for consistency. Saves scalability_plot.png and accuracy_plot.png and prints their paths. If the CSV is missing, prints an error and returns without writing. Designed for post-run analysis and reporting; uses only matplotlib for portability.

Parameter	Type	Default	Description
csv_path	str	-	Absolute or relative path to the benchmark results CSV (e.g. benchmark_results.csv or benchmark_YYYYMMDD_HHMMSS.csv).

Returns: None (Writes up to two PNG files and prints their paths to stdout.)