March 12, 2019

Adaptive Design of Experiments in pSeven

Overview of Adaptive Design of Experiments

Design of Experiments (DoE) is a selection of inputs at which outputs of the model are measured to explore design space or to get as much information as possible about the model behavior using a small number of observations as possible. Design space exploration block in pSeven offers variety DoE techniques which provide the most uniform and optimal filling of the design space such as Latin hypercube sampling (LHS), Full factorial design, Sobol sequence, etc.

However, it also provides an advanced technique called Adaptive design which considers model behavior before adding new points and takes into account linear and non-linear constraints of the model. Self-explanatory, this technique is used for implementing the Adaptive Design of Experiments (ADoE) approach.

Generally, Adaptive design technique supports three usage scenarios:

• Uniform (feasible domain sampling)
• Explore (response surface improvement/ objective function behavior tracking)
• Contour (search for designs with given objective function value)

In this tech tip, we will consider the implementation of the first two. As an addition, the ability of Adaptive design to work with the initial sample will also be presented. Therefore, three main problems will be considered:

1. Linear and non-linear constraints. How much design points are feasible if sample sizes for regular space-filling DoE technique LHS and Adaptive design techniques are equal? Various sample sizes are considered.
2. Adaptive response. How do LHS and Adaptive design techniques generate new design points for further building of a predictive model? How is the predictive model improved with the addition of new design points?
3. Sample-based ADoE. How does Adaptive design technique perform in problems, where the addition of incremental points is required? A particular case of using Adaptive design technique without the initial model is demonstrated.

1. Linear and Non-Linear Constraints

To illustrate the difference between LHS and Adaptive design techniques for a problem with linear and non-linear constraints let’s consider a design space of two variables:

\(x_{1}, x_{2}∈[-5,5]\).

There is one linear constraint

\(x_{2}-x_{1}<4\),

and one non-linear constraint (shown in Fig. 1)

\([1+0.1(x_{1}+2)^2]\cdot[1-(0.2x_{2})^2]<3\),

 

Fig. 1. Non-linear constraint function

With taking into account the constraints, a feasible domain of variables in 2D space looks as follows in Fig. 2.

Fig. 2. Feasible domain plot in 2D space

Various sample sizes were generated with LHS technique: \(N = 10\), \(N = 20\) and \(N = 100\) (number of designs or sample size). The general result of DoE is divided into two groups: All designs and Feasible designs.

All designs group includes all generated points and response values evaluated by the model. Feasible designs group includes points that satisfy all constraints, values of continuous variables are within bounds, and all discrete or categorical variables have values matching the levels defined for these variables. For different sample sizes feasible and unfeasible points are shown in Fig. 3.

Fig. 3. Design points generated by LHS technique

Samples with the same sizes as above were generated with Adaptive design technique and the result is represented in Fig. 4.

Fig. 4. Design points generated by the Adaptive design technique

As can be seen from the figure above for space-filling LHS technique the ratio of the feasible design to the total number of designs is a constant approximately equal for the different sample sizes. For Adaptive design technique, the percent of feasible points grows with the increasing of sample size. The graph below clearly demonstrates this affirmation (Fig. 5).

Fig. 5. The fraction of feasible points generated by LHS and Adaptive design techniques

2. Adaptive Response

A special response type 'Adaptive' is available in the Design space exploration block. The Adaptive design technique analyzes the behavior of 'Adaptive' responses and generates more design points in “areas of interest” — for example, the areas with high gradients of the response function. Note that only Adaptive design technique supports 'Adaptive' responses. Other techniques treat adaptive responses in the same way as 'Evaluation' responses.

Unlike space-filling DoE techniques, like LHS, Adaptive design builds an approximation during the run time. It starts with generating initial design by one of the space-filling techniques for the original function. Adaptive design technique then builds an approximation based on this initial sample and enriches sample iteratively adding points to the most interesting regions.

To illustrate the difference between LHS and Adaptive design techniques for 'Adaptive' response type let’s consider a function of two variables shown in Fig. 6 and try to build a predictive model based on designs generated by those techniques respectively. For building predictive models, the Gaussian Processes (GP) technique is selected.

Fig. 6. Original function for adaptive response case

Designs points \((N = 8, N = 10\) and \(N = 13)\) generated by Latin Hypercube sampling and the corresponding trained predictive models are shown in Fig. 7.

Fig. 7. Designs generated by LHS technique and corresponding predictive models

It can be seen that the quality of the second predictive model is lower than the first one. LHS generates a fixed number of points at once; therefore, increasing the sample size does not always guarantee an improvement of the predictive model.

Adaptive design technique iteratively adds points to the training set, minimizing the uncertainty of the approximation. Designs generated by Adaptive design and corresponding predictive models are represented in Fig. 8.

Fig. 8. Designs generated by Adaptive design and corresponding predictive models

In the case with the number of designs \(N = 8 \) the Adaptive design used 3 points as a sample for training an initial approximation and added remaining points one by one to explore the response function. This detailed information is available in the Run log when you process your workflow in pSeven. The same number of points for initial approximation was used in the second case and in the last case it was 6 points for the initial approximation.

As a resume for obtained results values of the prediction root-mean-squared error (RMS) metric is shown in Fig. 9. A source of reference data used for models validation is a test sample with 1000 points of sample size. Lower values of RMS are better. It means that the quality of the predictive model is higher.

Fig. 9. RMS prediction error metric for the predictive models built by LHS and Adaptive design

3. Sample-Based ADoE

Initial sample is an optional input data sent to the Design space exploration block - an existing designs sample which can be used in several ways depending on the block configuration. The initial sample can contain either values of variables, or values of both variables and responses.

Adaptive design technique can use an initial sample to build initial approximations of the response functions, thus improving generation quality.

Let’s see how the Adaptive design technique operates with the initial sample. A function which will be approximated is the same as shown in Fig. 6. Initial sample input in Design space exploration block accepts initial data in the form of a single matrix or separately for each variable and response. Fig. 10 demonstrates design points generated by the Adaptive design technique and a corresponding predictive model. It is seen from plots above that the Adaptive design technique generates new points adaptively.

Fig. 10. Design points generated with the initial sample and corresponding predictive model

By default, it is assumed that evaluations of response function are available from some other block in a workflow (for example, simulation model). However, with Adaptive design, you can add a response to your task definition but disable its evaluation, for example, when the initial model is missing.

A special feature of the Adaptive design is when your configuration includes an adaptive type response which is not evaluated by the model. In this case, the technique requires an initial sample for this response and will use it to train an internal model which is evaluated instead of the model (Fig. 11).

Fig. 11. Design points generated by Adaptive design without the model evaluation

Adaptive design technique will generate new points and return response values as empty for all new designs. None value means that a response was not evaluated for the corresponding design point.

As an addition to the above, a technical implementation of sample-based ADoE using Adaptive design technique is shown in Fig. 12-13. This process does not require special skills and can be easily realized in pSeven.

Summary

In this tech tip, we showed three possible cases of using the Adaptive design technique for ADoE in pSeven. Key features of this technique are represented in visual comparison with the space-filling Latin hypercube sampling (LHS). The significant difference of the Adaptive design technique is in its’ special approach to the generation of new design points.

Adaptive design is especially useful for the generation of a sample on which further a predictive model will be built. Note also that Adaptive design technique does not always require the response function (model). If your configuration includes only constraint and evaluation type responses, this technique generates a uniformly distributed sample with regard to constraints.