LSFEvaluator

lsf_evaluator.LSFEvaluator

Class to evaluate model on Long Sequence Forecasting benchmark datasets, maintained at https://github.com/thuml/Time-Series-Library.

Uses GluonTS's evaluation tools.

__init__(self, metrics=POINT_FORECAST_METRICS, datasets=[LSFDatasetName.ETTh1, LSFDatasetName.ETTh2, LSFDatasetName.ETTm1, LSFDatasetName.ETTm2, LSFDatasetName.weather], prediction_lengths=[96, 192, 336, 720], context_lengths=[1280], mode=Multivariate(1), num_samples=30, lsf_path='./data/', data_split='test', eval_stride=256, samples_per_batch=30, use_kv_cache=False)

Initializes the LSF Evaluator with configuration parameters for datasets, lengths, and evaluation settings.

Parameter	Type	Default	Description
metrics	list[gluon_metrics.Metric]	POINT_FORECAST_METRICS	(undocumented)
datasets	list[LSFDatasetName]	[LSFDatasetName.ETTh1, LSFDatasetName.ETTh2, LSFDatasetName.ETTm1, LSFDatasetName.ETTm2, LSFDatasetName.weather]	(undocumented)
prediction_lengths	list[int]	[96, 192, 336, 720]	(undocumented)
context_lengths	list[int]	[1280]	(undocumented)
mode	Multivariate	Multivariate(1)	(undocumented)
num_samples	int | None	30	(undocumented)
lsf_path	str	"./data/"	(undocumented)
data_split	str	"test"	(undocumented)
eval_stride	int	256	(undocumented)
samples_per_batch	int	30	(undocumented)
use_kv_cache	bool	False	(undocumented)

eval(self, model, checkpoint_name=None) -> tuple[pd.DataFrame, list[list[TotoSampleForecast]], list[TestData]]

Runs the evaluation loop over all configured datasets, prediction lengths, and context lengths using the provided model.

Parameters:

Parameter	Type	Default	Description
model	Toto	-	(undocumented)
checkpoint_name	str | None	None	(undocumented)

Returns: A tuple containing the evaluation results, the generated forecasts, and the test data used. - pd.DataFrame: Evaluation metrics aggregated across all configurations. - list[list[TotoSampleForecast]]: All generated forecasts. - list[TestData]: The test data used for evaluation.