Model selection shootout

A crucial part of generating surrogate models automatically is choosing a good model selection metric to estimate the model accuracy and drive the hyperparameter optimization. Cross validation is a popular compromise here but it has its problems and requires the model to be re-trained.

We are working on a new smoothness based measure, and I did some initial tests comparing it to some other measures. The example is the 2D LNA using Kriging and pattern search. The plots are shown below. The curves are the average of 30 runs, the standard deviations have been omitted for clarity. As you can see, the smoothness metric seems to be a step in the right direction. Interesting is the very poor performance of the dynamic validation set...a bug or the real deal?

--Dirk

Ps: notice the bump between 100-200 samples, due to the heterogeneous sample distribution (as before)?