Optimizer Analyzer

class deepobs.analyzer.analyze_utils.OptimizerAnalyzer(path, opt, metric, testproblem, conv_perf)[source]

DeepOBS analyzer class for an optimizer (and a specific test problem).

This class will give access to all relevant information regarding this optimizer such as the best performing hyperparameter setting or the number of settings.

Parameters:
  • path (str) -- Path to the parent folder of the optimizer folder (i.e. the test problem folder).
  • opt (str) -- Name of the optimizer (folder).
  • metric (str) -- Metric to use for this test problem. If available this will be test_accuracies, otherwise test_losses.
  • testproblem (str) -- Name of the test problem this optimizer (folder) belongs to.
  • conv_perf (float) -- Convergence performance of the test problem this optimizer (folder) belongs to.
name

Name of the optimizer (folder).

metric

Metric to use for this test problem. If available this will be test_accuracies, otherwise test_losses.

testproblem

Name of the test problem this optimizer (folder) belongs to.

conv_perf

Convergence performance for this test problem.

settings

Dictionary of hyperparameter settings for this optimizer (on this test problem) where the key is the name of the setting (folder) and the value is an instance of the SettingAnalyzer class (see below).

num_settings

Total number of settings for this optimizer (and test problem)

get_best_setting_best()[source]

Returns the setting for this optimizer that has the best overall performance using the metric (test_losses or test_accuracies) defined for this test problem. In contrast to get_best_setting_final in not only looks at the final performance per setting, but the best performance per setting.

Returns:Instance of the SettingAnalyzer class with the best overall performance
Return type:SettingAnalyzer
get_best_setting_final()[source]

Returns the setting for this optimizer that has the best final performance using the metric (test_losses or test_accuracies) defined for this test problem.

Returns:Instance of the SettingAnalyzer class with the best final performance
Return type:SettingAnalyzer
get_bm_table(perf_table, mode='most')[source]

Generates the overall performance table for this optimizer.

This includes metrics for the performance, speed and tuneability of this optimizer (on this test problem).

Parameters:
  • perf_table (dict) -- A dictionary with three keys: Performance, Speed and Tuneability.
  • mode (str) -- Whether to use the setting with the best final (final) performance, the best overall (best) performance or the setting with the most runs (most). Defaults to most.
Returns:

Dictionary with holding the performance, speed and tuneability measure for this optimizer.

Return type:

dict

get_setting_most_runs()[source]

Returns the setting with the most repeated runs (with the same setting, but probably different seeds).

Returns:Instance of the SettingAnalyzer class with the most repeated runs.
Return type:SettingAnalyzer
plot_lr_sensitivity(ax, mode='final')[source]

Generates the learning rate sensitivity plot for this optimizer. This plots the relative performance (relative to the best setting for this optimizer) against the learning rate used in this setting.

This assumes that all settings or otherwise equal and only different in the learning rate.

Parameters:
  • ax (matplotlib.axes) -- Handle to a matplotlib axis to plot the learning rate sensitivity onto.
  • mode (str) -- Whether to use the final (final) performance or the best (best) when evaluating each setting. Defaults to final.
plot_performance(ax, mode='most')[source]

Generates a performance plot for this optimzer using one hyperparameter setting.

Can either use the setting with the best final performance, the best overall performance or the setting with the most runs.

This function will plot all four possible performance metrics (test_losses, train_losses, test_accuracies and train_accuracies).

Parameters:
  • ax (list) -- List of four matplotlib axis to plot the performancs metrics onto.
  • mode (str) -- Whether to use the setting with the best final (final) performance, the best overall (best) performance or the setting with the most runs (most) when plotting. Defaults to most.