Restart strategies for hyperparameter optimization

An important aspect of generating models is finding the best model parameters. This is an optimization problem and classic algorithms can be used to tackle it (if no model specific algorithm is available). The tricky bit is that this optimization problem depends on the data distribution. If the amount and distribution of data is different the optimal set of model parameters will most likely differ as wel. In essence this is a dynamic optimization problem which can be tackled using a suitable algorithm, e.g., PSO.

However, one may also wish to use classical algorithms in this context. The question then is, what do you do when the algorithm has converged. Once new data has arrived, what do you do? Continue from the last solution, restart randomly somewhere else, etc..?

The SUMO-Toolbox provides 4 restart strategies, of which intelligent is the default. The plot below shows the results when each is applied to Kriging models where the theta parameters are optimized using pattern search. The underlying problem is to approximate the 3D Ackley function. Included in the comparison is also krigingego. Here the Efficient Global Optimization (EGO) algorithm is used to predict the kriging parameters.



Conclusion? Simply continuing from the last solution is not always a good idea apparently. It will depend on how dynamic your problem is, and how large your search population is.

--Dirk