Test Problems

Currently DeepOBS includes twenty-six different test problems. A test problem is given by a combination of a data set and a model and is characterized by its loss function.

Each test problem inherits from the same base class with the following signature.

class deepobs.tensorflow.testproblems.testproblem.TestProblem(batch_size, weight_decay=None)[source]

Base class for DeepOBS test problems.

  • batch_size (int) -- Batch size to use.
  • weight_decay (float) -- Weight decay (L2-regularization) factor to use. If not specified, the test problems revert to their respective defaults. Note: Some test problems do not use regularization and this value will be ignored in such a case.

The dataset used by the test problem (datasets.DataSet instance).


A tensorflow operation initializing the test problem for the training phase.


A tensorflow operation initializing the test problem for evaluating on training data.


A tensorflow operation initializing the test problem for evaluating on test data.


A tf.Tensor of shape (batch_size, ) containing the per-example loss values.


A scalar tf.Tensor containing a regularization term (might be a constant 0.0 for test problems that do not use regularization).


A scalar tf.Tensor containing the mini-batch mean accuracy.


Sets up the test problem.

This includes setting up the data loading pipeline for the data set and creating the tensorflow computation graph for this test problem (e.g. creating the neural network).


Some of the test problems described here are based on more general implementations. For example the Wide ResNet 40-4 network on Cifar-100 is based on the general Wide ResNet architecture which is also implemented. Therefore, it is very easy to include new Wide ResNets if necessary.