Configuration manager for the benchmarking pipeline.

This module provides comprehensive validation and management of configuration files using Pydantic to ensure they comply with the expected schema before execution. The ConfigManager class handles validation of benchmark configurations, model settings, task configurations, and system settings.

ValidationError

config_manager.ValidationError

Custom exception for configuration validation errors.

__init__(self, message)

Parameter	Type	Default	Description
message	str	-	(undocumented)

ConfigManager

config_manager.ConfigManager

Configuration manager with comprehensive validation rules.

This class handles the validation and management of all configuration files in the benchmarking pipeline, including benchmark configurations, model settings, task configurations, and system settings.

__init__(self, config_path)

Initialize the configuration manager.

This method performs initialization in the following order: 1. Initializes the Logger 2. Loads the main benchmark configuration 3. Loads evaluation settings from tasks/settings.yaml 4. Extracts models to be evaluated 5. Initializes evaluation configuration and settings 6. Initializes Logger with TensorBoard support based on evaluation settings 7. Initializes model hyperparameters and settings 8. Initializes task configurations

Parameter	Type	Default	Description
config_path	str	-	Path to the main benchmark configuration YAML file.

init_tasks(self) -> Dict[str, TaskConfig]

Initialize the tasks.

Discovers and loads all TaskConfig objects from task.yaml files in directories found via find_task_directories using self.task_path.

Returns: Dict[str, TaskConfig] (Mapping of task directory names to their validated TaskConfig objects.) Raises: FileNotFoundError or ValidationError if a task.yaml file is missing or invalid.

init_models_config(self, models_config) -> Dict[str, ModelConfig]

Initialize the model hyperparameters.

This method initializes the model hyperparameters.

Parameter	Type	Default	Description
models_config	Dict[str, Any]	-	(undocumented)

Returns: Dict[str, ModelConfig] (Dictionary mapping model names to their validated ModelConfig objects.)

init_model_setting(self) -> Dict[str, Any]

Validate model execution settings for models specified in the configuration.

This method finds and validates model-specific settings.yaml files recursively in the models directory. Only processes models that are defined in self.models_evaluated.

Returns: Dict[str, Any] (Dictionary mapping model names to their validated execution settings (Python version, device, conda environment)) Raises: ValidationError If validation fails, models directory doesn't exist, or a model settings file is invalid

generate_run_configs(self)

Generate unified configurations for each task-model combination.

This method yields JobConfig instances that combine: - A benchmark configuration with the specific task path and single model hyperparameters - System settings - Model execution settings for the specific model - Task configurations for the specific task

For each task and each model, a separate JobConfig is generated.

Yields: Tuple[JobConfig, int] (A tuple of (JobConfig, task_idx) where JobConfig is the aggregated configuration and task_idx is always 0.)