July 14, 2015

Mixture of Approximations


It is a quite common situation when a single global surrogate model [2,3] is not able to provide an accurate modeling of the physical phenomenon (represented by some input-output dependency) at hand due to the following reasons:

  • dependency to be modelled is spatially inhomogeneous,
  • input vectors are distributed inhomogeneously in the design space and form clusters,
  • training sample is huge, up to million points, so usual methods can not process it.

The situation when the dependency to be approximated is inhomogeneous is quite often, for example, in mechanics when computing critical buckling modes (see [1,4]). In this case the dependency can have discontinuities and derivative-discontinuities which prevents constructing accurate surrogate model.

When training data is inhomogeneous (for example, the set of input vectors forms some clusters in the design space) single global surrogate model can be not enough accurate even if the dependency is continuous and rather smooth. For example, in case of clustered input data there is no enough data in regions between domains corresponding to clusters, that is why global nonlinear surrogate model can have artifacts in these regions.

Quite natural solution is to perform preliminary space partitioning and use a mixture of approximators (see [1,3,4]). The idea is to decompose input space into subdomains such that in each subdomain the variability of a given dependency is lower than in all the design space. If approximations are constructed for each subdomains and then are "glued", then more accurate surrogate model can be obtained compared to the global surrogate model, constructed at once for the whole design space.

Another advantage of splitting training data into subsets is that it allows handling huge training sets. Building surrogate model using all the points in huge training set can be too time- and memory-consuming. Splitting training set into smaller subsets decreases time and memory consumption.

Therefore, surrogate model in general is not represented as a single approximator, but is represented as a mixture of approximators (MoA). This technique is implemented in pSeven by DATADVANCE.

Usage example

In this section as an illustration we present application of Mixture of Approximators approach, realized in pSeven platform, to some artificial dependency. To measure quality of constructed surrogate model a mean average error MAE is used.

The dependency we want to approximate is a two-dimensional discontinuous function:

where  is a Heaviside function:

Here is the figure of this function:

Figure 1: 2D discontinuous function

Figure 1: 2D discontinuous function

To construct surrogate model for this function we will use Mixture of Approximators with default parameters. The number of clusters is selected automatically from the range from 2 to 10. In order to apply pSeven to solve this problem we performed the following steps:

1) Develop a workflow for constructing surrogate models. The workflow is given in Figure 2 and consists of:

  • Two composite blocks for generation of train and test datasets,
  • Surrogate model builder and player

2) Analyze the obtained results and draw plots of constructed surrogate models, see Figure 3.


Figure 2. Workflow for surrogate model construction

Figure 3. Analysis of results of surrogate models construction

The results are depicted in Figures 4 and 5. We can see that usage of a smooth global surrogate model in this case is not efficient at all: MAE is equal to 0.41 versus 0.09 for MoA surrogate modeling technique.

Figure 4: Approximation of 2D discontinuous function using smooth global surrogate modeling technique based on Gaussian Process regression

Figure 5: Approximation of 2D discontinuous function using MoA technique


By Evgeny Burnaev, Scientific consultant, DATADVANCE



[1] D. Bettebghor, N. Bartoli, S. Grihon, J. Morlier, and M. Samuelides. Surrogate modeling approximation using a mixture of experts based on em joint estimation. Structural and Multidisciplinary Optimization, 43:243–259, 2011.

[2] A. Forrester, A. Sobester and A. Keane. Engineering design via surrogate modelling: a practical guide. Progress in astronautics and aeronautics. J. Wiley, 2008.

[3] T. Hastie, R. Tibshirani and J. Friedman. The elements of statistical learning: data mining, inference, and prediction. Springer, 2008.

[4] Grihon S., Burnaev E.V., Belyaev M.G. and Prikhodko P.V. Surrogate Modeling of Stability Constraints for Optimization of Composite Structures// Surrogate-Based Modeling and Optimization. Engineering applications. Eds. by S. Koziel, L. Leifsson. Springer, 2013. P. 359-391.