Tuning Utilities

General Utilities

deepobs.tuner.tuner_utils.rerun_setting(runner, optimizer_class, hyperparam_names, optimizer_path, seeds=array([43, 44, 45, 46, 47, 48, 49, 50, 51]), rank=1, mode='final', metric='valid_accuracies')

Reruns a hyperparameter setting with several seeds after the tuning is finished. Defaults to rerun the best setting. :param runner: The runner which was used for the tuning. :type runner: framework runner.runner :param optimizer_class: The optimizer class that was tuned. :type optimizer_class: framework optimizer class :param hyperparam_names: A nested dictionary that holds the names, the types and the default values of the hyperparams. :type hyperparam_names: dict :param optimizer_path: The path to the optimizer to analyse the best setting on. :type optimizer_path: str :param seeds: The seeds that are used to rerun the setting. :type seeds: iterable :param rank: The ranking of the setting that is to rerun. :type rank: int :param mode: The mode by which to decide the best setting. :type mode: str :param metric: The metric by which to decide the best setting. :type metric: str

deepobs.tuner.tuner_utils.write_tuning_summary(optimizer_path, mode='final', metric='valid_accuracies')

Writes the tuning summary to a json file in the optimizer_path. :param optimizer_path: Path to the optimizer folder. :type optimizer_path: str :param mode: The mode on which the performance measure for the summary is based. :type mode: str :param metric: The metric which is printed to the tuning summary as 'target' :type metric: str

deepobs.tuner.tuner_utils.generate_tuning_summary(optimizer_path, mode='final', metric='valid_accuracies')

Generates a list of dictionaries that holds an overview of the current tuning process. Should not be used for Bayesian tuning methods, since the order of evaluation is ignored in this summary. For Bayesian tuning methods use the tuning summary logging of the respective class.

Parameters:

• optimizer_path (str) -- Path to the optimizer folder.
• mode (str) -- The mode on which the performance measure for the summary is based.
• metric (str) -- The metric which is printed to the tuning summary as 'target'

Returns:

A list of dictionaries. Each dictionary corresponds to one hyperparameter evaluation of the tuning process and holds the hyperparameters and their performance. setting_analyzer_ranking (list): A ranked list of SettingAnalyzers that were used to generate the summary

Return type:

tuning_summary (list)

class deepobs.tuner.tuner_utils.log_uniform(a, b, base=10)

A log uniform distribution that takes an arbitrary base.

__init__(a, b, base=10)

Parameters:

• a (float) -- Lower bound.
• b (float) -- Range from lower bound.
• base (float) -- Base of the log.

Bayesian Specific Utilities

deepobs.tuner.bayesian_utils.plot_1d_bo_posterior(optimizer_path, step, resolution, xscale='linear', show=True)

Plots the one dimensional GP posterior of the Bayesian tuning process. The tuning process must have been done for only one hyperparameter (i.e. one dimensional). :param optimizer_path: Path to the optimizer which was tuned. :type optimizer_path: str :param step: The step of the tuning process for which the posterior is plotted. :type step: int :param resolution: Resolution of the plot, i.e. number of x-values. :type resolution: int :param xscale: The scaling for the x-axis. :type xscale: str :param show: Whether to show the plot or not. :type show: bool

Returns:The figure and axes of the plot. Return type:tuple

deepobs.tuner.bayesian_utils.plot_2d_bo_posterior(optimizer_path, step, resolution, show=True)

Plots the two dimensional GP posterior of the Bayesian tuning process. The tuning process must have been done for exactly two hyperparameters (i.e. two dimensional). :param optimizer_path: Path to the optimizer which was tuned. :type optimizer_path: str :param step: The step of the tuning process for which the posterior is plotted. :type step: int :param resolution: Resolution of the plot, i.e. number of x-values. :type resolution: int :param show: Whether to show the plot or not. :type show: bool

Returns:Figure and axes of the plot. Return type:tuple