SUMO - Surrogate Modeling Lab blog

SUMO-Toolbox 7.0.2 Released + Publication

2010-08-10T01:51:00.000-07:00

Hello,

We have just released version 7.0.2 of the SUMO-Toolbox. This is the second incremental update since the toolbox was released under an open source license.

This coincides with a new summary publication that should be used whenever referring to the toolbox:

A Surrogate Modeling and Adaptive Sampling Toolbox for Computer Based Design
D. Gorissen, K. Crombecq, I. Couckuyt, T. Dhaene, P. Demeester,
Journal of Machine Learning Research,
Vol. 11, pp. 2051-2055, July 2010.

More information can be found on the SUMO Toolbox wiki.

--Dirk

SUMO-Toolbox 7.0 Released - Open Source

2010-02-18T06:38:00.000-08:00

We are very proud to announce the 7.0 release of the SUrrogate MOdeling (SUMO) Toolbox. The main novelty of this release is that from now on the SUMO Toolbox will now be available under a open source license (AGPLv3) for non-commercial use.

Details are available on the SUMO website, see: http://www.sumowiki.intec.ugent.be/index.php/License_terms

Besides the adoption of an open source license for non-commercial use, this release has seen many bug fixes and improvements in the Kriging and SampleEvaluator components.

[*] Download instructions can be found here:

http://www.sumo.intec.ugent.be/?q=SUMO_toolbox#download

[*] The full changelog and release history is available here:

http://www.sumowiki.intec.ugent.be/index.php/Whats_new
http://www.sumowiki.intec.ugent.be/index.php/Changelog

All users are strongly advised to upgrade (remember to delete old versions first). Upgrade instructions can be found here:

http://www.sumowiki.intec.ugent.be/index.php/FAQ#Upgrading

If you encounter any problems when downloading or using the toolbox please let us know here:

http://www.sumowiki.intec.ugent.be/index.php/Contact

Enjoy! :)

--Dirk

3D Surface Modeling

2009-11-11T11:00:00.000-08:00

In an earlier post I wrote about how the SUMO framework could be extended and applied to classification problems. While doing that it struck me that the SUMO Toolbox could be similarly used for 3D surface modeling.

With 3D surface modeling, I mean the fitting of 3D geometric data in order to reproduce the shapes of various 3D objects like cubes, spheres, chairs, tables, dragons, etc. Ideally this leads to a closed analytic expression that fully describes the object.

Work to this end has already been done of course, using RBF models (FastRBF) and neural gas models (I cant seen to find the link right now). However, since it was quite straightforward to implement (only a new example was needed, the modeling code did not have to change) I thought I would quickly add an example and associated demo file. This will be available in 6.3.

Wat is needed is a triangular surface mesh (in standard Matlab format) and the surface normals for each triangle (can be easily calculated). Given such a mesh I added a Matlab function that can decide whether any given point is inside or outside the mesh (using the excellent InPolyhedron function by Luigi Giaccari). Putting these two together in a new 3DModel example then allows any of the SUMO toolbox model types or sample selection algorithms to be used for fitting the object.

But how does it work? Well its very simple, the idea is to regard the problem as a classification (or regression) problem: (1) fit a model on the 3D points, use -1 and 1 as output values to indicate if a point is inside or outside the object, (2) of the final model, plot the isosurface at isovalue = 0

The object should magically appear. Of course, the more complicated the model the more data will be needed to get the details right and keep things smooth. As a proof of concept example I used SVM models and a simple sphere:

Not the fanciest and shiniest example, nor the nicest visualization, but it proves the point :) Adding more complicated models (teapots, dragons, sculptures, ...) is now trivial, limited only by available computer memory and processing power...

Of course many improvements can be made to the straightforward approach described here. But that is just a matter of some spare time and motivation :)

What would be interesting (and easy) though is to couple this with the LOLA sample selection algorithm. Since it should seek out the boundary automatically. Potentially saving a lot of time...

--Dirk

PS: naturally the models need to be closed for this to work

SUMO Lab YouTube Channel

2009-10-24T02:38:00.000-07:00

We always had a collection of videos available. Putting these online as a collection of avi links is a bit too Web 1.0 :)

Therefore we started a YouTube channel where we hope to add some stuff from time to time:

http://www.youtube.com/sumolab

Feel free to make suggestions or leave comments.

--Dirk

Surrogate Models for Classification

2009-10-24T02:08:00.000-07:00

From our FAQ:

Question: Does the SUMO Toolbox support classification problems?
Short Answer: Yes, now it does
Long Answer: see below

At the SUMO Lab we spend most of our time on the problem of generating an accurate surrogate model (metamodel) for a given data set or simulation code with a minimum number of data points (= adaptive sampling, sequential design, active learning). The goal is to make this this process as efficient and pain-free as possible.

To aid this work we developed the Matlab SUMO Toolbox which implements a number of frameworks and abstractions to facilitate:

model selection
model complexity selection (= hyperparameter optimization)
adaptive sampling (= active learning)
Design of Experiments (DoE)
data visualization
data interfacing
distributed execution of simulations

The work always revolved around regression/function approximation type problems. However, many of the algorithms and sub problems we encountered are equally applicable to classification. So wouldn't it be possible to leverage the SUMO framework for classification problems as well? Encouraged by some comments of one of the toolbox users I looked into this.

It turned out that with only 30mins work I had a first demo ready. Since some of the model types inside SUMO already support classification internally (e.g., the SVM models) I just needed to add some extra options and tweak the model plotting code somewhat.
The result is that now you can use the SUMO plugins for hyperparameter optimization, model selection, adaptive sampling, etc. and apply them to classification problems. The code will become available in version 6.3. If you want to play around with it earlier, just let me know.

As a proof of principle example I took the classical two spiral problem and configured SUMO to use SVM models (parameters optimized with DIRECT) and the density based sample selection algorithm. The resulting movie generated by SUMO is given below:

Remark that while the basic support for classification is there, our focus remains on the classic surrogate modeling problem (regression). So dont expect major developments in this area anytime soon. Rather, the purpose of this was just to show that it can be done quite easily. The basic support is there and now its up to an interested somebody to pick it up and improve or extend it as needed :)

Remark also that exactly the same could be done for Time Series prediction. If I turn out to have a spare hour here or there I might do a similar post on that as well :)

--Dirk

SUMO-Toolbox v6.2.1 released!

2009-10-20T05:28:00.000-07:00

After a lot of hard work and testing I am happy to announce the availability of version 6.2.1 of the SUMO-Toolbox. This is a bugfix release, hot on the heels of our 6.2 release.

This version has seen a lot of internal cleanups and feature improvements. You can find information about new features here.

The full list of changes is available here

For download information see here. Screenshots can be found here.

Note that post-release bugs may have been found so remember to check the known bugs page. For how to upgrade see the Upgrading FAQ entry here.

Any feedback, be it positive or negative, is much appreaciated. :)

--Dirk

Scalability of Delaunayn

2009-08-25T02:08:00.001-07:00

As part of our work on a new model selection metric (Linear Reference Model (LRM) selection) we needed an idea of how Matlabs' triangulation routine (based on qhull) scales with the number of points and dimensionality.

The two plots below show the results of a simple test we did on a high end desktop machine. It turns out that the main limitation is memory usage, which in turn is very closely linked with the dimensionality. Beyond 6 dimensions with a couple thousand points we would just get a malloc memory error. For those cases we are looking at an iterative or approximate implementation. The Hull code by Clarkson also seems promising.

--Dirk

Working towards 6.2

2009-07-28T05:08:00.001-07:00

Its been a while since the last post but progress is continuing with Karel doing some fancy dynamic sample selector work and Ivo vastly improving and updating everything related to (Blind) Kriging models and Kriging based optimization.

The fruit of this work will see the light in 6.2 which will be released sometime this summer. To get an idea of what is coming you can take a look at a tentative, incomplete, changelog here.

--Dirk

SUMO-Toolbox version 6.1.1 released

2009-04-17T09:21:00.000-07:00

I am happy to announce version 6.1.1 of the SUMO-Toolbox. This is a bugfix release that fixes a few small bugs of 6.1 and polishes a few features.

You can find information about new features here.

The full changelog is available here

For download information see here.

For how to upgrade see the Upgrading FAQ entry here.

Any feedback, be it positive or negative, is much appreaciated. :)

--Dirk

2009-03-05T07:48:00.000-08:00

While I was performing tests for an upcoming paper on sequential design strategies, I came across the following interesting snippet. I was comparing different sequential design strategies with a standard Latin hypercube using the same number of samples, in order to assess the advantage of using sequential design over the most popular one-shot experimental design: the Latin hypercube. I used the Peaks function, which is a built-in matlab benchmark function, as a test case. The peaks function looks like this:

The Peaks function is the sum of several translated and scaled gaussian distributions. As can be seen on the figure, it is very nonlinear near the origin, but almost flat near the boundaries. It is therefore an excellent function to demonstrate the power of sequential design strategies.

The LOLA-Voronoi hybrid sequential design strategy, which is the default choice when using the SUMO Toolbox, was compared against other methods that are available in the toolbox, using Kriging as the model of choice because it internally uses gaussian basis functions and is therefore expected to model the Peaks function very efficiently. The tests showed that LOLA-Voronoi performed the best of all the methods, needing only 122 samples for an accuracy of 1%. However, how does it compare to a Latin hypercube?

the Latin hypercube used in the SUMO Toolbox is optimized using the algorithm described in:
V. R. Joseph and Y. Hung. Orthogonal-maximin latin hypercube designs. Statistica Sinica, 18:171-186, 2008.

It turns out that a Latin hypercube performs much worse than all of the sequential design strategies, save for random sampling. But what is even more surprising, is the effect that LOLA-Voronoi can have on a Latin hypercube. On average, a 150 sample Latin hypercube was needed to achieve the same accuracy as LOLA-Voronoi with 122 samples. However, if one generated a 145 sample Latin hypercube, augmented with one sample from LOLA-Voronoi, the average error drops below the one obtained from a 150 sample Latin hypercube! One run where the Latin hypercube achieved an error of 1,17% but one extra sample dropped the error to 0,4% is depicted below:

The Latin hypercube samples are drawn as circles, the additional LOLA-Voronoi sample as a square. It can be seen from the figure that the Latin hypercube does a nice job of covering up the design space quite evenly. However, there are some visible gaps, such as the one on the bottom left, the one on the top left and one where the LOLA-Voronoi sample is now located. The first two areas are relatively unimportant, as the peaks function behaves very linearly in these regions. However, the last area is of paramount importance, as it lies on a steep slope near the global optimum of the function. LOLA-Voronoi immediately identifies this region as both highly non-linear and undersampled.

This clearly illustrates that the surrogate modelling community might be giving way too much credit to the Latin hypercube design, especially since a lot of people don't even optimize their Latin hypercube, and just use a randomly generated one! Further tests will be performed to see if this problem also appears in other situations.

SUMO-Toolbox version 6.1 released

2009-02-26T14:17:00.001-08:00

After a lot of hard work and testing I am happy to announce the availability of version 6.1.

You can find information about new features here.

The full changelog is available here

For download information see here.

Note that post-release bugs may have been found so remember to check the known bugs page.

For how to upgrade see the Upgrading FAQ entry here.

Many thanks to those who helped test pre-release versions! Special thanks in this regard to Joel Kuhn from the University of Toronto.

Any feedback, be it positive or negative, is much appreaciated. :)

--Dirk

Restart strategies for hyperparameter optimization

2009-02-16T11:23:00.001-08:00

An important aspect of generating models is finding the best model parameters. This is an optimization problem and classic algorithms can be used to tackle it (if no model specific algorithm is available). The tricky bit is that this optimization problem depends on the data distribution. If the amount and distribution of data is different the optimal set of model parameters will most likely differ as wel. In essence this is a dynamic optimization problem which can be tackled using a suitable algorithm, e.g., PSO.

However, one may also wish to use classical algorithms in this context. The question then is, what do you do when the algorithm has converged. Once new data has arrived, what do you do? Continue from the last solution, restart randomly somewhere else, etc..?

The SUMO-Toolbox provides 4 restart strategies, of which intelligent is the default. The plot below shows the results when each is applied to Kriging models where the theta parameters are optimized using pattern search. The underlying problem is to approximate the 3D Ackley function. Included in the comparison is also krigingego. Here the Efficient Global Optimization (EGO) algorithm is used to predict the kriging parameters.

Conclusion? Simply continuing from the last solution is not always a good idea apparently. It will depend on how dynamic your problem is, and how large your search population is.

--Dirk

Rational model parameter optimization comparison

2009-02-11T09:18:00.000-08:00

Problem:

Every model type (Kriging, Neural nets, ...) has a set of model parameters that control the complexity of the model. When the SUMO-Toolbox generates models automatically, it tries to find the best fitting model by optimizing over the model parameters. Different algorithms are available for doing so (PSO, Differential Evolution, GA, BFGS, etc., and even multi-objective ones like NSGA-II)

This all works quite well for all model types, with the exception of the rational models. Until recently there were only two options for rational model order selection: a custom stochastic hill climber (sequential) and a GA. Neither of which gave very good performance in some problems. With version 6.1 rational now supports most (if not all) of the other optimization algorithms. Thus it was time for a comparison...

Setup:

LNA modeling problem with 3 inputs and 1 real valued output (input noise)
samples are selected in batches of 50 up to a maximum of 1000 using the default sample selector
each optimization algorithm was allowed to generate 50 models (100 for PSO and the GA algorithms)
model generation is driven by the BEEQ error on a dense validation set and the MinMax error
the model parameters are the weights for each parameter (3), the flags (3, binary) and the percentage of degrees of freedom (1)
repeat 10 times

Results:

Strangely enough building random models works best!
With some algorithms the error even increases with the number of data points!
It also turned out that most (if not all) of the final model still contained poles in some places.

Caution:

These results are not final, paper quality results but should give a good indication of the problem
The standard deviations were not plotted since it makes the graph unreadable, but they were very large
This is not the easiest problem to model as other tests have shown, then again, one would intuitively expect better results
The model parameters are a mix of discrete and continous variables, that does not make the optimization any easier
A maximum of 50 models is not very high for some algorithms (CMA-ES).
The results may be better with complex data

Conclusion

Rational order selection (and pole avoidance) is a difficult problem that will require specialized algorithms to solve this properly. This is what one would intuitively expect but its nice to see this in numbers. The next step would be to actually implement such an order selection algorithm and see how it compares.

--Dirk

Pre-release snapshots of 6.1

2009-02-09T14:54:00.000-08:00

Hello!

With the official release of the SUMO-Toolbox version 6.1 drawing near I have been making development snapshots available and recommending them over the last stable release (6.0.1).

If anybody is interested in a snapshot they should just send an email to sumo@intec.ugent.be.

Any feedback is much appreciated

Expect an official announcement within the next two weeks.

Happy modeling!

Dirk

Dynamic Multiobjective Hyperparameter Optimization

2008-09-09T01:56:00.000-07:00

In order to produce better models it is useful to enforce multiple criteria. This effectively turns the hyperparameter optimization problem into a multiobjective one. However, it does not stop there, since data ponits are sampled incrementally (this is the case if they are expensive) it makes the hyperparameter optimization surface dynamic (see the previous posts for some movies). So one may ask, what happens to the Pareto front as more data becomes available. A little movie that illustrates this nicely is shown below.

The two criteria are the min/max 20% validation error and the custom smoothness measure we have developed.

--Dirk

Comparison of surrogate model types

2008-09-04T04:37:00.000-07:00

Recently we have obtained a dataset kindly donated by Alex Forrester (University of Southampton). In fact there are 20 datasets each representing the same 4-dimensional problem, namely the band-averaged vibration of a truss-like structure, but with a different number of samples (ranging from 10 to 200 with intervals of 10). Each dataset is arranged in a Latin Hypercube.
For more information about the problem itself we refer to "Global Optimization of Deceptive Functions With Sparse Sampling" by Forrester and Jones.

So we wanted to try out several surrogate model types from the SUMO toolbox and compare it with the Kriging implementation of Forrester, additionally we implemented Blind Kriging based on R code obtained from Joseph Roshan (Georgia Tech). See "Blind Kriging: A New Method for Developing Metamodels" by Roshan et al.

After constructing the surrogate model a Mean Square Error (MSE) was calculated based on a validation set of 100 samples. Afterwards we plotted these errors versus the sample size used to create the surrogate model.

It should be noted that the hyperparameters of all surrogate models, with the exception of Blind Kriging and Kriging (Forrester), are optimized using 5-fold cross validation. The last two use a log-likelihood to determine the hyper parameters.

As you can see the preliminary results are not really conclusive of what is the best surrogate model type. Surprisingly enough, the Kriging model of the DACE toolbox does well with a low number of samples but gets really worse towards the end. The RBF model follows the same trend.

More interesting is the fact that the other Kriging implementations are about the same. Also noteworthy is that the 2nd custom Kriging run had more time to find the right parameters (that's why it performs a bit better at the end).

Blind Kriging is quite interesting as the implementation is very rough at the moment. The hyper parameters can be better optimized as well as a better search in the variable selection phase is possible. We expect at least to slightly improve its results. Looking at the QQ-plot of blind kriging (plotting the prediction vs validationset) we can see that the prediction error over the whole range is quite uniform (which is nice).

The author would like to thank Forrester et al. for kindly providing their results and the dataset.

UPDATE:
To analyse the performance of Blind Kriging a plot was made of the Bayesian variable selection phase. The leave-one-out cross validation score (CV) is plotted against the number of terms in the regression part. The lowest CV score determines the number of terms chosen for the final Blind Kriging model, this minimum is denoted by a star.

Ivo
Dirk

Dynamic optimization 2

2008-08-27T07:39:00.000-07:00

Another, more dynamic example:

--Dirk

Dynamic optimization

2008-08-27T01:35:00.000-07:00

The SUMO-Toolbox generates surrogate models (= metamodels, response surface models) automatically while adaptively sampling the design space. This means that the hyperparameter optimization problem is a dynamic one, it changes as more data becomes available. This causes some problems later on in the modeling process but also allows for some smart optimizations.

To see what really changes, the SUMO-Toolbox allows you to generate a movie of the process. An example (though admittedly, not the best one) is the following:

It shows how the surface changes as more data arrives. When I have a more extreme example I will post it as well.

--Dirk

Model selection shootout in 3D

2008-08-27T01:17:00.000-07:00

Hello again,

This is a followup to my last entry on model selection. There we saw that the smoothness measure we have defined seems to make some sense, at least for Kriging on that particular 2D problem. I now repeated the same tests but then in 3D:

First of all, ignore the obvious issue that the error increases as we get more data. This is a known problem of standard Kriging if you use it together with adaptive sampling. We have discussed this elsewhere so its not the focus of this post (it makes you wonder though how Kriging can ever be used for global modeling since it needs way to much data to keep it stable).

Rather we were interested in the smoothness measure and how it compared to the other measures. Quite surprisingly it does very well (again, ignoring for the moment that the model fit is rubbish). It does well on its own, but even better when combined with another measure, see the dramatic effect it has on SampleError. Note also, surprisingly, how even the dense validation set (= the 'true' generalization estimator) goes haywire.

Anyways, the next step is now to switch to an example that Kriging can actually model properly (or switch to a different method all together) and further scale up the dimensions.

--Dirk

Optimization of a wheel fairing airfoil

2008-08-18T02:52:00.000-07:00

Recently, we have applied the SUMO toolbox on an application from the aerospace industry. It is based on ``Airfoil Geometry Design for Minimum Drag'' by Z. Wang [#!WangZheng2005!#] and is a nice example of airfoil design. Whang is concerned with finding an optimal design of a wheel fairing (or wheel pant) on a solar powered vehicle. When designing airfoils, the typical goal is to optimize the lift-to-drag ratio, i.e., design an airfoil that has a high lift while having not a too high drag coefficient. However for a wheel fairing for a vehicle or airplane the main aim is to have a low drag coefficient. There are several published standard airfoils, for example the NACA series of the, now dissolved, National Advisory Committee for Aeronautics [#!uiic2008!#] who created series of airfoils for different purposes using analytical equations (see figure below for an example NACA profile). Nonetheless, in many cases it is useful and more efficient to create a custom-made design. See wikipedia for more information about wheel fairings.

Additionally it is always interesting to have a complete idea of the behaviour of the different parameters. As such we tried to model the objective function (for optimizing wheel fairing airfoils) using the SUMO toolbox. We utilized the open source xfoil program for simulation with the following objective:

The resulting surrogate model (a 4-14-2-1 neural network) had an accuracy of 1% (RRSE) on a validation set.

Due to the ANN model being cheap to evaluate we can now apply as many optimization algorithms as we want on this surrogate model. In this case we just applied the DIviding RECTangles (DIRECT) algorithm, resulting in the following wheel fairing airfoil:

Ivo--

Model selection shootout

2008-08-15T06:05:00.000-07:00

A crucial part of generating surrogate models automatically is choosing a good model selection metric to estimate the model accuracy and drive the hyperparameter optimization. Cross validation is a popular compromise here but it has its problems and requires the model to be re-trained.

We are working on a new smoothness based measure, and I did some initial tests comparing it to some other measures. The example is the 2D LNA using Kriging and pattern search. The plots are shown below. The curves are the average of 30 runs, the standard deviations have been omitted for clarity. As you can see, the smoothness metric seems to be a step in the right direction. Interesting is the very poor performance of the dynamic validation set...a bug or the real deal?

--Dirk

Ps: notice the bump between 100-200 samples, due to the heterogeneous sample distribution (as before)?

Heterogeneous Pareto Front

2008-08-15T05:54:00.001-07:00

What happens if you take the automatic model type selection algorithm (based on heterogeneous evolution) implemented in the SUMO Toolbox and combine it with our new multi-objective model builder?

Well, you get (ideally) a pareto front which shows you which model type is most suitable for which output (or model selection metric). In addition it also shows you the correlation between different output (or metrics). This seems smarter than simply doing multiple runs in parallel.

The obligatory screenshot:
Admittedly its nothing to look at, and doesn't really give you any information, but its the proof of principle that counts :)

The context of this work is figuring out how to solve, what I like to call the 5 percent problem. I.e., an engineer wants a model to be accurate within 5%. But what does this mean, especially if you don't know what the response looks like? There are 101 ways to get 5% and it is very difficult to agree on what to use upfront while understanding the implications.

--Dirk

First post!

2008-08-15T05:33:00.000-07:00

During our research into surrogate modeling and its applications there often are times that you think "Mmm this is interesting, cool, or weird..., maybe it could be of use to somebody?"

Well, since blogging is all the hype these days I finally decided to syndicate some of these things into a common blog. No guarantees are given that posts will be regular, lets just take it one step at a time and see how it goes :)

Cheers!
Dirk