Configure and Run a Performance Benchmark

Use this guide to set up and execute the Chronax benchmarking suite, allowing you to compare the performance (speed) and accuracy (error metrics) of Chronax models against other forecasting libraries.

Prerequisites

• Chronax must be installed in your environment.
• You must have access to the Chronax source repository, specifically the benchmarks/run_benchmark.py script.
• A YAML configuration file (e.g., config.yaml) must be created to define the experiment parameters.

Steps

1. Define Experiment Parameters in config.yaml

Start by defining the overall parameters for the benchmark in the experiment block of your configuration file. This controls the forecast horizon, seasonality, and the number of warm runs used for averaging execution time.

# config.yaml
experiment:
  scales: [1000]       # Lengths for synthetic data (e.g., 1000 points)
  horizon: 24          # The number of steps to forecast (h)
  seasonality: 12      # The primary seasonality period
  n_iterations: 5      # Number of warm runs for calculating average speed

2. Specify Datasets to Test

In the datasets block, list the time series you want to use. You can use built-in synthetic datasets (like "Trend") or specify external CSV files.

# config.yaml (continued)
datasets:
  - name: "Trend"           # Use a built-in synthetic dataset
  - name: "my-series"
    type: "external"
    path: "data/sales.csv"
    target_column: "value"  # Specify the column containing the time series data

3. Configure Chronax Models to Benchmark

In the models block, define which models should be run. To ensure the model is sourced from Chronax, set the library key to "chronax". Include any necessary model parameters in the params dictionary.

# config.yaml (continued)
models:
  - name: "AutoARIMA"
    library: "chronax"
    params: {}
  - name: "AutoETS"
    library: "chronax"
    params: { season_length: 12 }
  - name: "AutoETS_StatsForecast" # Example comparison model
    library: "statsforecast"
    params: { season_length: 12 }

4. Execute the Benchmark

Run the main script using the command line, pointing to your configuration file. The script will execute all combinations of models and datasets defined in the configuration.

python benchmarks/run_benchmark.py --config benchmarks/config.yaml

5. Filter the Run (Optional)

If you only want to run a subset of the defined configuration, use the command line arguments to filter by model name or dataset name/path.

To run only the AutoETS model defined in the config:

python benchmarks/run_benchmark.py --config benchmarks/config.yaml --model AutoETS

To run only against a specific external dataset:

python benchmarks/run_benchmark.py --config benchmarks/config.yaml --dataset "data/sales.csv"

Full example

This example defines an experiment comparing Chronax's AutoARIMA against a synthetic trend dataset, then executes the benchmark.

# benchmarks/config.yaml
experiment:
  scales: [500]
  horizon: 12
  seasonality: 12
  n_iterations: 3

datasets:
  - name: "Trend"

models:
  - name: "AutoARIMA"
    library: "chronax"
    params: {}
  - name: "Naive"
    library: "statsforecast"
    params: {}

# Execute the benchmark
python benchmarks/run_benchmark.py --config benchmarks/config.yaml

Next steps

• Review the output metrics, paying attention to Time_Warm_Sec for pure inference speed and MASE (Mean Absolute Scaled Error) for accuracy.
• Use the --forecast flag when running the script to generate CSV files of the forecasts instead of aggregate statistics.
• Consult the guides on specific Chronax models, such as AutoARIMA or AutoETS, to understand available params.