Validate Output

deepobs.analyzer.check_output(results_path)[source]

Iterates through the results folder an checks all outputs for format and completeness. It checks for some basic format in every json file and looks for setting folders which are empty. It further gives an overview over the amount of different settings and seed runs for each test problem and each optimizer. It does not return anything, but it prints an overview to the console.

Parameters:results_path (str) -- Path to the results folder.

Plot Optimizer Performances

deepobs.analyzer.plot_optimizer_performance(path, fig=None, ax=None, mode='most', metric='valid_accuracies', reference_path=None, show=True, which='mean_and_std')[source]

Plots the training curve of optimizers and addionally plots reference results from the reference_path

Parameters:
  • path (str) -- Path to the optimizer or to a whole testproblem (in this case all optimizers in the testproblem folder are plotted).
  • fig (matplotlib.Figure) -- Figure to plot the training curves in.
  • ax (matplotlib.axes.Axes) -- The axes to plot the trainig curves for all metrices. Must have 4 subaxes (one for each metric).
  • mode (str) -- The mode by which to decide the best setting.
  • metric (str) -- The metric by which to decide the best setting.
  • reference_path (str) -- Path to the reference optimizer or to a whole testproblem (in this case all optimizers in the testproblem folder are taken as reference).
  • show (bool) -- Whether to show the plot or not.
  • which (str) -- ['mean_and_std', 'median_and_quartiles'] Solid plot mean or median, shaded plots standard deviation or lower/upper quartiles.
Returns:

The figure and axes with the plots.

Return type:

tuple

deepobs.analyzer.plot_testset_performances(results_path, mode='most', metric='valid_accuracies', reference_path=None, show=True, which='mean_and_std')[source]

Plots all optimizer performances for all testproblems.

Parameters:
  • results_path (str) -- The path to the results folder.
  • mode (str) -- The mode by which to decide the best setting.
  • metric (str) -- The metric by which to decide the best setting.
  • reference_path (str) -- Path to the reference results folder. For each available reference testproblem, all optimizers are plotted as reference.
  • show (bool) -- Whether to show the plot or not.
  • which (str) -- ['mean_and_std', 'median_and_quartiles'] Solid plot mean or median, shaded plots standard deviation or lower/upper quartiles.
Returns:

The figure and axes.

Return type:

tuple

Get the Best Runs

deepobs.analyzer.plot_results_table(results_path, mode='most', metric='valid_accuracies', conv_perf_file=None)[source]

Summarizes the performance of the optimizer and prints it to a pandas data frame.

Parameters:
  • results_path (str) -- The path to the results directory.
  • mode (str) -- The mode by which to decide the best setting.
  • metric (str) -- The metric by which to decide the best setting.
  • conv_perf_file (str) -- Path to the convergence performance file. It is used to calculate the speed of the optimizer. Defaults to None in which case the speed measure is N.A.
Returns:

A data frame that summarizes the results on the test set.

Return type:

pandas.DataFrame

deepobs.analyzer.get_performance_dictionary(optimizer_path, mode='most', metric='valid_accuracies', conv_perf_file=None)[source]

Summarizes the performance of the optimizer.

Parameters:
  • optimizer_path (str) -- The path to the optimizer to analyse.
  • mode (str) -- The mode by which to decide the best setting.
  • metric (str) -- The metric by which to decide the best setting.
  • conv_perf_file (str) -- Path to the convergence performance file. It is used to calculate the speed of the optimizer. Defaults to None in which case the speed measure is N.A.
Returns:

A dictionary that holds the best setting and it's performance on the test set.

Return type:

dict

Plot Hyperparameter Sensitivity

deepobs.analyzer.plot_hyperparameter_sensitivity(path, hyperparam, mode='final', metric='valid_accuracies', xscale='linear', plot_std=True, reference_path=None, show=True)[source]

Plots the hyperparameter sensitivtiy of the optimizer.

Parameters:
  • path (str) -- The path to the optimizer to analyse. Or to a whole testproblem. In that case, all optimizer sensitivities are plotted.
  • hyperparam (str) -- The name of the hyperparameter that should be analyzed.
  • mode (str) -- The mode by which to decide the best setting.
  • metric (str) -- The metric by which to decide the best setting.
  • xscale (str) -- The scale for the parameter axes. Is passed to plt.xscale().
  • plot_std (bool) -- Whether to plot markers for individual seed runs or not. If False, only the mean is plotted.
  • reference_path (str) -- Path to the reference optimizer or to a whole testproblem (in this case all optimizers in the testproblem folder are taken as reference).
  • show (bool) -- Whether to show the plot or not.
Returns:

The figure and axes of the plot.

Return type:

tuple

Estimate Runtime

deepobs.analyzer.estimate_runtime(framework, runner_cls, optimizer_cls, optimizer_hp, optimizer_hyperparams, n_runs=5, sgd_lr=0.01, testproblem='mnist_mlp', num_epochs=5, batch_size=128, **kwargs)[source]

Can be used to estimates the runtime overhead of a new optimizer compared to SGD. Runs the new optimizer and SGD seperately and calculates the fraction of wall clock overhead.

Parameters:
  • framework (str) -- Framework that you use. Must be 'pytorch' or 'tensorlfow'.
  • runner_cls -- The runner class that your optimizer uses.
  • optimizer_cls -- Your optimizer class.
  • optimizer_hp (dict) -- Its hyperparameter specification as it is used in the runner initialization.
  • optimizer_hyperparams (dict) -- Optimizer hyperparameter values to run.
  • n_runs (int) -- The number of run calls for which the overhead is averaged over.
  • sgd_lr (float) -- The vanilla SGD learning rate to use.
  • testproblem (str) -- The deepobs testproblem to run SGD and the new optimizer on.
  • num_epochs (int) -- The number of epochs to run for the testproblem.
  • batch_size (int) -- Batch size of the testproblem.
Returns:

The output that is printed to the console.

Return type:

str