Esc
Ask AIAnswers may be inaccurate; check the linked pages.Esc
Ask anything about these docs, like how to get started or what a function does.

Running Chronax Neural Benchmarks

This guide shows you how to execute the performance comparison benchmarks between Chronax neural models and the neuralforecast library, allowing you to verify performance and speed metrics against committed baselines.

Prerequisites

  • You have cloned the Chronax repository locally.
  • You have an active Python environment where Chronax is installed (the project environment).
  • All commands are run from the repository root directory.

Steps

1. Set up the neuralforecast comparison environment (Optional)

If you intend to run benchmarks comparing Chronax against neuralforecast, you must first set up an isolated virtual environment for neuralforecast. This is necessary because its PyTorch/Lightning dependencies conflict with the JAX environment used by Chronax.

For Unix / Git Bash / WSL:

bash benchmarks/setup_nf_venv.sh

For Windows PowerShell:

python -m venv benchmarks\.venv-nf
.\benchmarks\.venv-nf\Scripts\pip.exe install -r benchmarks\requirements-nf.txt

2. Execute a full benchmark run

To run the full suite, comparing all auto-discovered neural models in Chronax against neuralforecast across all configured datasets, execute the main script.

python benchmarks/neural/run.py

3. Filter the benchmark run

You can restrict the run to specific models, datasets, or libraries to save time or focus testing.

To run only a specific model (e.g., GRU):

python benchmarks/neural/run.py --models GRU

To run only a specific dataset (e.g., AirlinePassengers):

python benchmarks/neural/run.py --datasets AirlinePassengers

To run Chronax models only (skipping the neuralforecast comparison):

python benchmarks/neural/run.py --libs chronax

4. Manage reference baselines

Benchmark results are saved to benchmarks/benchmark_results/neural/. You can manage the committed reference baselines stored in benchmarks/baselines/.

To refresh the committed baseline for a specific model after a successful run:

python benchmarks/neural/run.py --refresh-baseline GRU

To compare your current Chronax implementation against the committed baseline without re-running the potentially slow neuralforecast comparison:

python benchmarks/neural/run.py --check-committed --models GRU

Full example

This sequence sets up the comparison environment (if needed) and runs the benchmark for the GRU model, refreshing its baseline.

# 1. Set up the neuralforecast environment (Unix example)
bash benchmarks/setup_nf_venv.sh

# 2. Run the GRU benchmark
python benchmarks/neural/run.py --models GRU

# 3. Refresh the committed baseline for GRU
python benchmarks/neural/run.py --refresh-baseline GRU

Next steps

  • Review the dataset configurations and run parameters in benchmarks/neural/config.yaml.
  • Run the internal tests for the neural components using python -m pytest tests/neural/ -q.
  • Explore the available neural models in chronax.models (e.g., GRU, TFT).