The Bayesian optimization algorithm

The aim of a Bayesian hyperparameter optimizer is to construct a probability model of the objective function and use it to select the hyperparameters that work best for use in the real objective function. Bayesian statistics allow us to foresee not only a value, but a distribution, and this is the success of this methodology.

The Bayesian method, when compared with the two methods already dealt with (grid search and random search), stores the results of the past evaluation, which it uses to form a probabilistic model that associates the hyperparameters with a probability of a score on the objective function.

This model is called a surrogate of the objective function and is much easier to optimize than the objective function itself. This result is obtained by following this procedure:

1. A surrogate probability model of the objective function is constructed.
2. The hyperparameters that give the best results on the surrogate are searched.
3. These hyperparameters are applied to the real objective function.
4. The surrogate model is updated by incorporating the new results.
5. Repeat steps 2–4 until you reach the pre-established iterations or the maximum time.

In this way, the surrogate probability model is updated after each evaluation of the objective function. To use a Bayesian hyperparameter optimizer, several libraries are available: scikit-optimize, spearmint, and SMAC3.