run_benchmark

Entry point for running the benchmarking pipeline.

This module provides the BenchmarkRunner class which orchestrates the end-to-end benchmarking process including configuration loading, hyperparameter tuning, and model evaluation across multiple task-model combinations.

BenchmarkRunner

run_benchmark.BenchmarkRunner

Orchestrates the end-to-end benchmarking pipeline execution.

The BenchmarkRunner coordinates the execution of multiple benchmarking jobs, where each job represents a combination of a task (dataset) and model. It handles configuration loading, hyperparameter tuning, and result aggregation.

__init__(self, config_path, *, results_callback=None, task_status_callback=None)

Initialize benchmark runner with configuration.

Parameter	Type	Default	Description
config_path	str	-	Path to the configuration YAML file used for this benchmark run.
results_callback	Callable[..., Any] \| None	None	Optional per-window hook (job_id, model, task, window_idx, metrics, forecast_data?). Injected by downstream runners that persist per-window metrics. Omit for plain local runs.
task_status_callback	Callable[..., Any] \| None	None	Optional (task_name, model_name, status, error=None) hook called when each task starts / completes / fails. Downstream runners use this for per-task status tracking and buffer management.

__enter__(self)

Sets up temporary directories and loads datasets.

Returns: Self

__exit__(self, exc_type, exc_value, traceback)

Cleans up temporary resources and closes the logger.

Parameter	Type	Default	Description
exc_type	-	-	(undocumented)
exc_value	-	-	(undocumented)
traceback	-	-	(undocumented)

run(self)

Execute the end-to-end benchmarking pipeline.

Returns: list[str] (List of names of tasks that failed during execution.)