Simple Example

This tutorial will show you an example of how DeepOBS can be used to benchmark the performance of a new optimization method for deep learning.

This simple example aims to show you some basic functions of DeepOBS, by creating a run script for a new optimizer (we will use the Momentum optimizer as an example here) and running it on a very simple test problem.

We show this example for TensorFlow and PyTorch respectively.

Create new Run Script

The easiest way to use DeepOBS with a new optimizer is to write a run script for it. This run script will import the optimizer and list its hyperparameters. For the Momentum optimizer in TensorFlow this is

"""Example run script using StandardRunner."""

import tensorflow as tf
from deepobs import tensorflow as tfobs

optimizer_class = tf.train.MomentumOptimizer
hyperparams = {"learning_rate": {"type": float},
               "momentum": {"type": float, "default": 0.99},
               "use_nesterov": {"type": bool, "default": False}}

runner = tfobs.runners.StandardRunner(optimizer_class, hyperparams)
runner.run(testproblem='quadratic_deep', hyperparams={'learning_rate': 1e-2}, num_epochs=10)

You can download this example run script tensorflow and use it as a template.

For the Momentum optimizer in PyTorch it is

"""Example run script using StandardRunner."""

from torch.optim import SGD
from deepobs import pytorch as pt

optimizer_class = SGD
hyperparams = {"lr": {"type": float},
               "momentum": {"type": float, "default": 0.99},
               "nesterov": {"type": bool, "default": False}}

runner = pt.runners.StandardRunner(optimizer_class, hyperparams)
runner.run(testproblem='quadratic_deep', hyperparams={'lr': 1e-2}, num_epochs=10)

You can download this example run script pytorch and use it as a template.

The DeepOBS runner needs access to an optimizer class with the same API as the TensorFlow/PyTorch optimizers and a list of additional hyperparameters for this new optimizers.

Run new Optimizer

You can now just execute the above mentioned script to run Momentum on the quadratic_deep test problem. You can change the arguments in the run() method to run other test problems, other hyperparameter settings, different number of epochs, etc.. If you want to make the script command line based, you can simply remove all arguments in the run() method an parse them from the command line. For TensorFlow this would look like this:

python runner_momentum_tensorflow.py quadratic_deep --bs 128 --learning_rate 1e-2 --momentum 0.99 --num_epochs 10

We will run it a couple times more this time with different learning_rates

python runner_momentum_tensorflow.py quadratic_deep --bs 128 --learning_rate 1e-3 --momentum 0.99 --num_epochs 10
python runner_momentum_tensorflow.py quadratic_deep --bs 128 --learning_rate 1e-4 --momentum 0.99 --num_epochs 10
python runner_momentum_tensorflow.py quadratic_deep --bs 128 --learning_rate 1e-5 --momentum 0.99 --num_epochs 10

For PyTorch this would look like this:

python runner_momentum_pytorch.py quadratic_deep --bs 128 --lr 1e-2 --momentum 0.99 --num_epochs 10

We will run it a couple times more this time with different lr

python runner_momentum_pytorch.py quadratic_deep --bs 128 --lr 1e-3 --momentum 0.99 --num_epochs 10
python runner_momentum_pytorch.py quadratic_deep --bs 128 --lr 1e-4 --momentum 0.99 --num_epochs 10
python runner_momentum_pytorch.py quadratic_deep --bs 128 --lr 1e-5 --momentum 0.99 --num_epochs 10

Analyzing the Runs

We can use DeepOBS's analyzer module to automatically find the best hyperparameter setting. First note, that the runner writes the output in a directory tree like:

<results_name>/<testproblem>/<optimizer>/<hyperparameter_setting>/

In the above example, the directory of the run outputs for TensorFlow would be:

./results/quadratic_deep/MomentumOptimizer/...

And for PyTorch:

./results/quadratic_deep/SGD/...

We pass the path to the optimizer directory to the analyzer functions. This way, we can get the best performance setting, a plot for the corresponding training curve and a plot that visualizes the hyperparameter sensitivity.

For TensorFlow and PyTorch:

from deepobs import analyzer

# get the overall best performance of the MomentumOptimizer on the quadratic_deep testproblem
performance_dic = analyzer.get_performance_dictionary('./results/quadratic_deep/SGD')
print(performance_dic)

# plot the training curve for the best performance
analyzer.plot_optimizer_performance('./results/quadratic_deep/SGD')

# plot again, but this time compare to the Adam baseline
analyzer.plot_optimizer_performance('./results/quadratic_deep/SGD',
                                    reference_path='../DeepOBS_Baselines/baselines_tensorflow/quadratic_deep/MomentumOptimizer')

You need to change the results directory accordingly, i.e. in our example it would be

'./results/quadratic_deep/SGD'

for TensorFlow (as in the example above) and

'./results/quadratic_deep/MomentumOptimizer'

for PyTorch.

You can download the script and use it as a template for further analysis: example analyze script tensorflow.

Note that you can also select a reference path (here the deepobs baselines for TensorFlow) to plot reference results as well. You can download the latest baselines from GitHub