Estimate Runtime¶
A convenience script to estimate the run time overhead of a new optimization method compared to SGD.
By default this script runs SGD as well as the new optimizer 5
times for
3
epochs on the multi-layer perceptron on MNIST while measuring the time.
It will output the mean run time overhead of the new optimizer for these runs.
Optionally the setup can be changed, by varying the test problem, the number of epochs, the number of runs, etc. if this allows for a fairer evaluation.
Usage:
Run a new run script and compare its runtime to SGD.
usage: deepobs_estimate_runtime.py [-h] [--test_problem TEST_PROBLEM]
[--data_dir DATA_DIR] [--bs BS] [--lr LR]
[-N NUM_EPOCHS] [--num_runs NUM_RUNS]
[--saveto SAVETO]
[--optimizer_args OPTIMIZER_ARGS]
run_script
Positional Arguments¶
run_script | Path to the new run_script. |
Named Arguments¶
--test_problem | Name of the test problem to run both scripts. Default: "mnist_mlp" |
--data_dir | Path to the base data dir. If not set, deepobs uses its default. Default: "data_deepobs" |
--bs, --batch_size | |
The batch size (positive integer). Default: 128 | |
--lr, --learning_rate | |
The learning rate of both SGD and the new optimizer, defaults to 1e-5. Default: 1e-05 | |
-N, --num_epochs | |
Total number of training epochs per run. Default: 3 | |
--num_runs | Total number of runs for each optimizer. Default: 5 |
--saveto | Folder for saving a txt files with a summary. |
--optimizer_args | |
Additional arguments for the new optimizer |