Benchmark Chronax Models Against Other Libraries

This guide shows you how to use the built-in Chronax benchmarking suite to compare the performance (speed) and accuracy (error metrics) of Chronax models against models from other libraries, such as StatsForecast. You use this when evaluating which forecasting engine is best suited for your production environment.

Prerequisites

• You have cloned the Chronax repository and are running commands from the root directory.
• The required dependencies for both Chronax (JAX) and the comparison library (e.g., StatsForecast) are installed in your Python environment.
• The input data for external datasets is available (e.g., data.csv).

Steps

1. Configure the Experiment Parameters

Define the overall scope of the benchmark in the experiment section of your config.yaml. This controls the forecast horizon, seasonality period, and the number of warm runs used for averaging execution time.

experiment:
  scales: [500, 5000]       # Test series lengths of 500 and 5000
  horizon: 12               # Forecast 12 steps ahead
  seasonality: 12           # Assume monthly seasonality
  n_iterations: 10          # Run 10 warm iterations for stable timing

2. Define Datasets to Test

Specify the datasets the models will run against in the datasets section. You can use built-in synthetic datasets (like "Trend") or reference external CSV files.

If using external data, specify the type as "external", provide the path, and optionally define the target_column.

datasets:
  - name: "Trend"           # Use a built-in synthetic dataset
  - name: "my-sales-data"
    type: "external"
    path: "data/sales.csv"
    target_column: "revenue"

3. Select Models and Libraries

List the models you want to compare in the models section. For each model, specify its name, the library it belongs to (chronax or statsforecast), and any required params.

models:
  - name: "AutoARIMA"
    library: "chronax"      # Chronax model
    params: {}
  - name: "AutoETS"
    library: "statsforecast" # Comparison model
    params: { season_length: 12 }

4. Execute the Benchmark

Run the main benchmark script, pointing it to your configuration file. The script will iterate through all defined models and datasets, calculating performance and accuracy metrics.

python benchmarks/run_benchmark.py --config benchmarks/config.yaml

5. Filter the Run (Optional)

If you only want to test a specific model or dataset without modifying the configuration file, use the command line filters.

To run only the AutoARIMA model:

python benchmarks/run_benchmark.py --config benchmarks/config.yaml --model AutoARIMA

To run only against the external sales data:

python benchmarks/run_benchmark.py --config benchmarks/config.yaml --dataset "my-sales-data"

6. Generate Forecast Data (Optional)

If you need the raw forecast outputs (e.g., to generate plots or analyze specific predictions) instead of the aggregated metrics, run the script in forecast mode.

python benchmarks/run_benchmark.py --config benchmarks/config.yaml --forecast

Full example

This example shows a complete config.yaml and the command to run the benchmark comparing Chronax's AutoARIMA against StatsForecast's AutoETS on a synthetic dataset.

# benchmarks/config.yaml
experiment:
  scales: [1000]
  horizon: 24
  seasonality: 24
  n_iterations: 5

datasets:
  - name: "Seasonality"

models:
  - name: "AutoARIMA"
    library: "chronax"
    params: {}
  - name: "AutoETS"
    library: "statsforecast"
    params: { season_length: 24 }

python benchmarks/run_benchmark.py --config benchmarks/config.yaml

Next steps

• Review the output metrics, focusing on Time_Warm_Sec for pure inference speed and MASE for accuracy.
• See the guide on "Forecast a univariate series" using AutoARIMA.
• Explore how to add prediction intervals by reviewing the Interval_Overhead_Pct metric.
• Learn about configuring specific Chronax models like AutoETS or AutoARIMA.