leaderboard
chronax.leaderboard
Utilities for loading, processing, and aggregating model evaluation results into a structured leaderboard format, including handling scaling relative to a naive baseline and separating zero-inflated datasets.
NON_ZERO_METRICS
Metrics used for datasets that are not zero-inflated or low-variance.
Type: list
ZERO_METRICS
Metrics used for zero-inflated or low-variance datasets.
Type: list
LOW_VARIANCE_DATASETS
A predefined list of dataset names identified as having low variance.
Type: list
shifted_gmean(x, epsilon=1e-05, dim=-1)
Calculates the shifted geometric mean of an array.
| Parameter | Type | Default | Description |
|---|---|---|---|
x |
(undocumented) | - | (undocumented) |
epsilon |
float |
1e-05 |
(undocumented) |
dim |
int |
-1 |
(undocumented) |
load_and_process_csv(path, boomlet_benchmark)
Loads a CSV file containing model results, processes dataset names, and optionally filters for Boomlet benchmark datasets.
| Parameter | Type | Default | Description |
|---|---|---|---|
path |
(undocumented) | - | (undocumented) |
boomlet_benchmark |
(undocumented) | - | (undocumented) |
load_model_results(models_dir, boomlet_benchmark)
Loads results from multiple model directories, ensuring the seasonalnaive model is present and placed last in the list.
| Parameter | Type | Default | Description |
|---|---|---|---|
models_dir |
(undocumented) | - | (undocumented) |
boomlet_benchmark |
(undocumented) | - | (undocumented) |
Returns: (dfs, model_names)
separate_zero_inflated_data(dfs)
Separates the input list of DataFrames into two lists: one containing non-zero-inflated/non-low-variance datasets, and one containing zero-inflated/low-variance datasets.
| Parameter | Type | Default | Description |
|---|---|---|---|
dfs |
(undocumented) | - | (undocumented) |
Returns: (non_zero_dfs, zero_dfs)
scale_by_naive(df, naive_df, metrics)
Scales the metric values in a DataFrame relative to the corresponding values in the naive baseline DataFrame.
| Parameter | Type | Default | Description |
|---|---|---|---|
df |
(undocumented) | - | (undocumented) |
naive_df |
(undocumented) | - | (undocumented) |
metrics |
(undocumented) | - | (undocumented) |
replace_invalid_values(dfs, metrics)
Replaces infinite values in specified metric columns with NaN, and then fills NaNs with the column mean across all datasets for that model.
| Parameter | Type | Default | Description |
|---|---|---|---|
dfs |
(undocumented) | - | (undocumented) |
metrics |
(undocumented) | - | (undocumented) |
process_benchmark_model_results(is_scale_by_naive, dfs, metrics)
Cleans invalid values and optionally scales model results by the naive baseline.
| Parameter | Type | Default | Description |
|---|---|---|---|
is_scale_by_naive |
(undocumented) | - | (undocumented) |
dfs |
(undocumented) | - | (undocumented) |
metrics |
(undocumented) | - | (undocumented) |
format_number(num)
Formats a number for display in the leaderboard. Uses scientific notation for absolute values >= 100, otherwise formats to three decimal places.
| Parameter | Type | Default | Description |
|---|---|---|---|
num |
(undocumented) | - | (undocumented) |
rename_metrics(df)
Renames metric columns in the DataFrame for display purposes.
| Parameter | Type | Default | Description |
|---|---|---|---|
df |
(undocumented) | - | (undocumented) |
get_leaderboard(dfs, names, agg_func, metrics, ranking_metric='eval_metrics/mean_weighted_sum_quantile_loss')
Generates a leaderboard by combining results, calculating ranks based on a specified metric, and aggregating results using the provided aggregation function.
| Parameter | Type | Default | Description |
|---|---|---|---|
dfs |
(undocumented) | - | (undocumented) |
names |
(undocumented) | - | (undocumented) |
agg_func |
(undocumented) | - | (undocumented) |
metrics |
(undocumented) | - | (undocumented) |
ranking_metric |
str |
'eval_metrics/mean_weighted_sum_quantile_loss' |
(undocumented) |
get_separate_zero_inflated_leaderboard(non_zero_dfs, zero_dfs, dfs_names, agg_func, non_zero_metrics, zero_metrics)
Generates two separate leaderboards (for non-zero and zero-inflated datasets) and merges them.
| Parameter | Type | Default | Description |
|---|---|---|---|
non_zero_dfs |
(undocumented) | - | (undocumented) |
zero_dfs |
(undocumented) | - | (undocumented) |
dfs_names |
(undocumented) | - | (undocumented) |
agg_func |
(undocumented) | - | (undocumented) |
non_zero_metrics |
(undocumented) | - | (undocumented) |
zero_metrics |
(undocumented) | - | (undocumented) |