edsnlp.tune

Check if Plotly is installed. If not warn the user. Plotly is needed by optuna.visualization to produce tuning visual results.

Returns: bool: True if Plotly is installed, False otherwise.

Update a configuration dictionary with tuned hyperparameter values.

This function modifies a given configuration dictionary by updating the specified hyperparameters with values from either a dictionary or an Optuna trial object. The updated configuration and training keyword arguments are returned.

Parameters:

config : dict The configuration dictionary to be updated. tuned_parameters : dict A dictionary specifying the hyperparameters to tune. values : dict, optional A dictionary of parameter names and their corresponding values to update the configuration. Used when trial is not provided. trial : optuna.trial.Trial, optional An Optuna trial object to sample parameter values. Used when values is not provided.

Returns:

tuple - kwargs : dict The resolved training keyword arguments from the updated configuration. - updated_config : dict The modified configuration dictionary.

Perform two-phase hyperparameter tuning using Optuna.

This method executes a two-phase tuning strategy. In the first phase, all specified hyperparameters are tuned. Based on their computed importance, only the most important hyperparameters (top 50% by importance) are selected for fine-tuning in the second phase, while the less important hyperparameters are frozen to their best values from phase one.

Parameters:

config : dict The configuration dictionary for the model and training process. hyperparameters : dict A dictionary specifying the hyperparameters to tune. output_dir : str Directory where tuning results, visualizations, and best parameters will be saved. checkpoint_dir : str, Path to save the checkpoint file. n_trials : int, optional The total number of trials to execute across both tuning phases. This number will be split between the two phases, with approximately half of the trials assigned to each phase. If not provided, trials run until timeout for each phase. viz : bool Whether or not to include visual features (False if Plotly is unavailable). metric_paths : list[tuple[str, ...]] Metric paths used to evaluate trials (mean if multiple). study : optuna.study.Study, optional Optuna study containing previous trials when resuming from checkpoint. skip_phase_1 : bool, optional Whether or not to skip phase 1 (in case of resuming from checkpoint). Default is False. timeout : float, optional Timeout in seconds for each phase. If provided, it is applied to each phase. pruner : dict, optional Optuna pruner configuration. Pass None to disable pruning. The default uses Optuna's MedianPruner. Visit Optuna docs for more information: https://optuna.readthedocs.io/en/stable/reference/pruners.html. Allowed pruner types (in {"type": TYPE, "some_arg": ...}) are:

- "median": MedianPruner
- "nop": NopPruner
- "percentile": PercentilePruner
- "threshold": ThresholdPruner
- "successive_halving": SuccessiveHalvingPruner
- "hyperband": HyperbandPruner

execution : {"inprocess", "dvc"}, optional Execution backend for trials. Default is "inprocess". seed : int, optional Base seed used for deterministic trial seeds. Default is 42. seed_path : str, optional Config path to the seed parameter for DVC overrides. Default is "train.seed". training_seeds : list[int], optional Fixed seed list for multi-seed aggregation. Default is None. use_seeds_for_phase_1 : bool, optional Use training_seeds during phase 1. Default is True for single-phase tuning, otherwise False. gpu_ids : list[int], optional List of GPU IDs to use for parallel trial execution. Default is None, which uses a single GPU. parallel_trials : bool, optional Whether to run trials in parallel across multiple GPUs when using DVC execution. Default is False. raw_config : object, optional Preloaded raw config to avoid re-reading config_path. raw_kind : {"yaml","cfg"}, optional Kind of raw_config when provided.

Perform hyperparameter tuning for a model using Optuna.

Parameters:

config_meta : dict Metadata for the configuration file, containing at least the key "config_path" which specifies the path to the configuration file. hyperparameters : dict A dictionary specifying the hyperparameters to tune. The keys are the parameter names, and the values are dictionaries containing the following fields: - "path": List[str] representing the path to the parameter in config. - "type": The type of parameter ("float", "int", "categorical", "ordered_categorical"). - "low": (optional) Lower bound for numerical parameters. - "high": (optional) Upper bound for numerical parameters. - "step": (optional) Step size for numerical parameters. - "log": (optional) Whether to sample numerical parameters on a log scale. - "choices": (optional) List of values for categorical parameters. Required for "ordered_categorical". output_dir : str Directory where tuning results, visualizations, and best parameters will be saved. checkpoint_dir : str, Path to save the checkpoint file. gpu_hours : float, optional Deprecated. Total GPU time available for tuning, in hours. Used to derive timeout for backward compatibility. timeout : float or str, optional Total time budget for tuning, in seconds. You can also pass a duration string like "30s", "10m", "1.5h", or "2d". If two_phase_tuning is True, the timeout is split equally between the two phases. n_trials : int, optional Number of trials for tuning. If not provided, tuning runs until timeout. two_phase_tuning : bool, optional If True, performs two-phase tuning. In the first phase, all hyperparameters are tuned, and in the second phase, the top half (based on importance) are fine-tuned while freezing others. Default is False. seed : int, optional Random seed for reproducibility. Default is 42. metric : str or list[str], optional Metric(s) used to evaluate trials. If multiple, their mean is optimized. Default is "ner.micro.f". keep_checkpoint : bool, optional If True, keeps the checkpoint file after tuning. Default is False. pruner : dict, optional Optuna pruner configuration. Pass None to disable pruning. Supported types are "median", "nop", "percentile", "threshold", "successive_halving" and "hyperband". The default uses Optuna's MedianPruner. execution : {"inprocess", "dvc"}, optional Execution backend for trials. Default is "inprocess". seed_path : str, optional Config path to the seed parameter for DVC overrides. Default is "train.seed". training_seeds : list[int], optional Fixed seed list for multi-seed aggregation. Default is None. use_seeds_for_phase_1 : bool, optional Use training_seeds during phase 1. Default is True for single-phase tuning, otherwise False. gpu_ids : list[int], optional GPU ids to use for DVC subprocesses. Default is None. parallel_trials : bool, optional Whether to run trials in parallel across multiple GPUs when using DVC, or run them sequentially (you may parametrize your training job to parallelize your batches across these GPUs). Default is False.

!!! warning "

    When using `parallel_trials`, the 2nd, 3rd, etc. trials will be started
    at the same time, and will therefore not benefit from the previous trials'
    results. This may lead to suboptimal tuning results than
    running sequentially. However, if you have many GPUs and a time budget, and
    can only train on one GPU at a time, this is a good way to fully utilize
    your resources to improve your tuning performance (as you will get N times
    more trials in the same time).