July 14, 2015

Mixture of Approximations

Updated 11.12.2018

Introduction

It is a quite common situation when a single global approximation model [2,3] is not able to provide an accurate modeling of the physical phenomenon (represented by some input-output dependency) at hand due to the following reasons:

• dependency to be modeled is spatially inhomogeneous,
• input vectors are distributed inhomogeneously in the design space and form clusters,
• training sample is huge, up to million points, so usual methods can not process it.

The situation where the dependency that needs to be approximated is inhomogeneous occurs quite often, for example, in computational mechanics where critical buckling modes (see [1,4]) are calculated. In this case, the dependency can have discontinuities and derivative-discontinuities which prevent building an accurate approximation model.

When training data is inhomogeneous (for example, the set of input vectors forms some clusters in the design space) single global approximation model can be not accurate enough, even if the dependency is continuous and rather smooth. For example, in case of clustered input data, there is not enough data in regions between domains corresponding to clusters, that is why the global nonlinear approximation model can have artifacts in these regions.

A quite natural solution is to perform preliminary space partitioning and use a mixture of approximators (see [1,3,4]). The idea is to decompose input space into subdomains such that in each subdomain the variability of a given dependency is lower than in all the design space. If approximations are constructed for each subdomain and "glued" after, then the more accurate approximation model can be obtained compared to the global approximation model, constructed at once for the whole design space.

Another advantage of splitting training data into subsets is that it allows handling huge training sets. Building approximation model using all the points in huge training set can be too time- and memory-consuming. Splitting the training set into smaller subsets decreases time and memory consumption.

Therefore, the approximation model, in general, is not represented as a single approximatior but is represented as a mixture of approximators (MoA). This technique is implemented in pSeven.

Example

In this section, as an illustration, we present application of Mixture of Approximators approach, realized in pSeven, to some artificial dependency. To measure the quality of built approximation model a mean average error MAE is used.

The dependency we want to approximate is a two-dimensional discontinuous function:

where is a Heaviside function:

Here is the figure of this function:

Figure 1: 2D discontinuous function

To build an approximation model for this function we will use Mixture of Approximators with default parameters. The number of clusters is selected automatically from the range from 2 to 10. In order to apply pSeven to solve this problem we performed the following steps:

1) Develop a workflow for building approximation models. The workflow is given in Figure 2 and consists of:

• Two composite blocks for generation of train and test datasets,
• Approximation model builder and evaluation block

2) Analyze the obtained results and draw plots of built approximation models, see Figure 3.

Figure 2. Workflow for approximation model building

Figure 3. Analysis of approximation models building of results

The results are depicted in Figures 4 and 5. We can see that usage of a smooth global approximation model, in this case, is not efficient at all: MAE is equal to 0.41 versus 0.09 for MoA approximation modeling technique.

Figure 4: Approximation of 2D discontinuous function using smooth global approximation technique based on Gaussian Process regression

Figure 5: Approximation of 2D discontinuous function using MoA technique

References

[1] D. Bettebghor, N. Bartoli, S. Grihon, J. Morlier, and M. Samuelides. Surrogate modeling approximation using a mixture of experts based on em joint estimation.Structural and Multidisciplinary Optimization, 43:243–259, 2011.

[2] A. Forrester, A. Sobester and A. Keane. Engineering design via surrogate modelling: a practical guide. Progress in astronautics and aeronautics. J. Wiley, 2008.

[3] T. Hastie, R. Tibshirani and J. Friedman. The elements of statistical learning: data mining, inference, and prediction. Springer, 2008.

[4] Grihon S., Burnaev E.V., Belyaev M.G. and Prikhodko P.V. Surrogate Modeling of Stability Constraints for Optimization of Composite Structures // Surrogate-Based Modeling and Optimization. Engineering applications. Eds. by S. Koziel, L. Leifsson. Springer, 2013. P. 359-391.