Model selection shootout in 3D

Hello again,

This is a followup to my last entry on model selection. There we saw that the smoothness measure we have defined seems to make some sense, at least for Kriging on that particular 2D problem. I now repeated the same tests but then in 3D:



First of all, ignore the obvious issue that the error increases as we get more data. This is a known problem of standard Kriging if you use it together with adaptive sampling. We have discussed this elsewhere so its not the focus of this post (it makes you wonder though how Kriging can ever be used for global modeling since it needs way to much data to keep it stable).

Rather we were interested in the smoothness measure and how it compared to the other measures. Quite surprisingly it does very well (again, ignoring for the moment that the model fit is rubbish). It does well on its own, but even better when combined with another measure, see the dramatic effect it has on SampleError. Note also, surprisingly, how even the dense validation set (= the 'true' generalization estimator) goes haywire.

Anyways, the next step is now to switch to an example that Kriging can actually model properly (or switch to a different method all together) and further scale up the dimensions.

--Dirk