# Integration Basics¶

This tutorial shows the basics of integrating a third-party tool into a workflow. It explains a generic solution applicable to any program that can read parameters from the command line or from a text file (script).

## Before You Begin¶

This tutorial requires an existing Tutorials project. If you have not created this project yet, see Tutorial Project first.

• Open the Tutorials project.

• Create a new empty workflow. If you have never created a workflow before, see the Simple Workflow tutorial.

• Save the workflow as IntegrationBasics.

• Switch to Workspace and verify that the workflow file (IntegrationBasics.p7wf) is added to the project. You can see it in the Project pane.

• While in Workspace, verify that the following files exist in the project:

• The example executable to be integrated in the workflow:
• tools\toy_solver.exe (in Windows).
• Three files tools/toy_solver, tools/toy_solver_x86, tools/toy_solver_x64 (in Linux). The tools/toy_solver file is the initial script, that chooses the tools/toy_solver_x86 or the tools/toy_solver_x64 executable.
• templates\example_solver_input.dat — a sample of solver’s input.
• templates\example_solver_output.dat — a sample of solver’s output.

• Switch to Edit and select the workflow you have just created to continue with the tutorial.

The example executable (Toy Solver) is a simple command line tool which imitates a solver program. It requires an input file containing several commands, some of which set values of input variables. The tool evaluates some function of these variables and prints a “report” containing function values and some additional information.

The task, in general, is to create a workflow that allows to specify values of variables and to get function values in pSeven without running Toy Solver manually. This workflow has to generate a valid input file with specified values, launch Toy Solver, then capture and parse its output. In other words, the workflow creates a wrapper around Toy Solver and allows to automate its usage. This wrapper is the first step to integrating an external executable into approximation and optimization workflows: pSeven supports using a workflow as a block, so once you have created a wrapper, you can use it in various other tasks as a ready custom component.

## Solution¶

The task of integrating a command line tool into a workflow can be divided into the following general steps:

• Investigate the tool. You need to know how to start it from command line and how to pass parameters.
• Configure a Program block to launch the tool. This block executes a command line which can accept parameters from other blocks in the workflow.
• Generate input for the tool so it can be used when Program launches the executable. In this tutorial, the input is a temporary file prepared by a Text block.
• Capture the output and parse it to obtain data that other blocks can use. In this tutorial, the output is also a temporary file, and it is parsed by another Text block.

### Tool Investigation¶

To make the generic integration solution explained in this tutorial possible, there are certain requirements for the tool you want to integrate:

1. It must support non-interactive command line interface — that is, it must allow starting it from a command prompt and must be able to read input data from a file or directly from the command line.
2. It must output to a file. It can also just print the output, since screen output is easily redirected to a file.

Our example tool, the Toy Solver, accepts an input file containing values of variables, evaluates some function, and prints the results (but does not save them to a file). Its basic command line syntax is:

> toy_solver.exe input_file


for Windows, or

$./toy_solver input_file  for Linux. For more details, you can run Toy Solver without parameters to print help. For example, in Windows: > toy_solver.exe Example Toy Solver. Calculates values and gradients of a vector function in 2D. Both input and output are 2-dimensional. Function domain is limited to [0.0, 1.0] in 2D - if any input is not in range [0.0, 1.0], the solver exits with an error. Variable values are set in the input script (see usage). Results are output to terminal or a file. Usage: toy_solver.exe [-n] infile [> outfile] toy_solver.exe this message toy_solver.exe infile execute the script from infile and print result on screen (see script template) toy_solver.exe infile > outfile execute the script from infile and print result to outfile Options: -n notch: adds a region of undefined behavior: X1 - X2 > 0.95 (unit square corner); if set, the solver also exits with an error if input point is in the notch region Script template: #TOYSOLVER SETUP INPUT = X1 X2 OUTPUT = F1 F2 BEGIN SET X1 = 0.5 SET X2 = 0.5 SOLVE F1 F2 RETURN When making a new input script, the template should be copied as is, only changing the X1 and X2 values.  To begin, try running Toy Solver without parameters. • Open a command prompt. • Change to the tools subdirectory in the Tutorials project. • Run the Toy Solver executable in this directory. In Windows: > toy_solver.exe  In Linux: $ ./toy_solver


When Toy Solver is run with an input file, it prints calculation results. There is a template input script (example_solver_input.dat) located in the templates subdirectory of your project.

• Try running Toy Solver with an input file.

In Windows:

> toy_solver.exe ..\templates\example_solver_input.dat


In Linux:

$./toy_solver ../templates/example_solver_input.dat  Screen output should be: Solving in [5.000000e-01, 5.000000e-01]. Result: Function: 5.000000e-01 3.841688e+00 Gradient: 1.000000e+00 0.000000e+00 -1.658312e+00 7.643199e+00  The workflow, however, needs an output file it can read, not the screen output. This is not a problem since output redirection is a standard OS function. • Try redirecting Toy Solver output to a file. In Windows: > toy_solver.exe ..\templates\example_solver_input.dat > ..\templates\test_output.dat  In Linux: $ ./toy_solver ../templates/example_solver_input.dat > ../templates/test_output.dat


This time nothing is printed, but you can find test_output.dat in the templates subdirectory. Compare this file to the screen output from the previous step.

Note

The example_solver_output.dat file in templates is a pre-generated output which is the same as test_output.dat from above.

As you can see, mass calculations with Toy Solver are far from being convenient. If you were to run it manually, each evaluation would require you to:

1. Edit the input file to change values of variables.
2. Execute the command.
3. Copy function values from the output to save them somewhere.

This tutorial automates the steps above, so you can just specify values of variables as workflow inputs and make pSeven do the rest of the job. Further, this solution can be used in workflows that require multiple evaluations — for example, to integrate Toy Solver in an optimization cycle, as shown in the Integrated Component Optimization tutorial.

### Tool Launch¶

Now that you know how to execute Toy Solver manually, the next step is to configure its launch from the workflow.

• Add a Program block. Name it Launcher.

The Launcher block has to be configured to run Toy Solver with an input file as a command line parameter and then redirect the output to another file.

Begin with adding the input and output files to Launcher. Open its configuration; to add a file, click in the Files pane.

• Specify the port name: infile.
• Direction: input.
• Specify the file name: input.dat.

• Port name: outfile.
• Direction: output.
• File name: output.dat.

Verify file settings in the Files pane.

Note

File names are optional. You can leave file name empty and use port names in commands instead. The syntax is @{port_name} — for example, @{infile}; when Launcher executes the command, it substitutes actual file names where port names are specified.

On the main tab, specify the program name, relative path, command line parameters (the input file) and redirect standard output to the output file.

• Program: input toy_solver.exe in Windows, ./toy_solver in Linux.
• After specifying the program, verify that the relative path is set to Relative to the sandbox directory (the default setting).
• Arguments: input input.dat (or @{infile}, see the note above).
• Input/Output redirection: select “outfile” for stdout. Toy Solver’s screen output will be sent to this file.

Verify the command preview.

In Windows:

toy_solver.exe input.dat >output.dat


In Linux:

./toy_solver input.dat >output.dat


Compare the above with the command lines you have used to start Toy Solver when examining it. When the workflow runs, Launcher will start when it receives a file to the infile port. This file is temporary and does not exist on disk. To make it available to Toy Solver, Launcher will create a temporary directory and copy the received file there under the name you have specified when adding the file (“input.dat”). If the file name was not specified (@{port_name} syntax is used), then the name will be generated at random, and Launcher will substitute this generated name to the command line.

As you can note, the command line does not include the path to the Toy Solver executable, so Launcher does not yet know where to look for it. You could specify an absolute path to the program (like C:\Users\UserName\Documents\pSeven\Tutorials\tools\toy_solver.exe), but this has an unwanted effect of making the project not portable: if you move the Tutorials directory, you then need to edit Launcher configuration again to change the path. In this tutorial, a better solution is to place the Toy Solver executable into same temporary directory that is created by Launcher.

In pSeven this directory is called “sandbox” (see section Sandbox for details). The Program block uses the sandbox as its current working directory, so all relative paths are resolved inside the sandbox. To configure the sandbox, switch to the Sandbox tab in Launcher configuration.

The question is how to make the Toy Solver executable appear in the Launcher sandbox, so that the block can find this executable by a relative path. There are two possible ways:

1. The simple way is to specify that the tools subdirectory itself is the sandbox (input tools in the Sandbox field). The tools subdirectory then becomes the working directory for Launcher.
2. The recommended way is to specify that the tools subdirectory is the sandbox prototype (input tools in the Prototype field) and let the block create temporary sandboxes when it runs (leave the Sandbox field empty). Contents of the sandbox prototype are copied to the sandbox directory when the block starts. Since you do not specify a path in the Sandbox field, the block will create a sandbox directory with a randomized name in the system directory for temporary files, and then copy files from the prototype to that temporary sandbox.

The first configuration is acceptable, but may be unsafe in certain cases. In particular, it causes file conflicts if your Program block runs in a Composite block with enabled parallelization (you can see this further in the Parallelization and Batch Optimization tutorials). The second configuration is preferred because it guarantees that each block will work in its own sandbox, the block will never change any files in the prototype directory, and the sandbox will contain all required files.

• Input the path in the Prototype field: tools. Note that the block automatically transforms your input to a relative path: .\tools in Windows, ./tools in Linux, where . means the current project directory.
• Leave other settings on the Sandbox tab default. The Sandbox field should be empty.

Since the sandbox prototype path is relative, the block will look for this directory inside the current project and copy its contents to the temporary sandbox when it runs. No matter where you move the Tutorials directory, Launcher will get the Toy Solver executable from Tutorials/tools and run it inside the sandbox to avoid conflicts.

Launcher configuration is now complete. Next steps are to generate the input file and to process the Toy Solver’s output.

### Generating Input¶

Program input files can be generated using a Text block.

• Add a Text block. Name it InputGenerator.

To generate an input file for the program, Text requires a sample of such file as its input. This file is also automatically used as template in block configuration. As mentioned before, there is a sample input file for our tool found at templates\example_solver_input.dat — this file shall be the input template.

Open the InputGenerator configuration and click in the Input file pane to bring up the Configure file dialog.

Note

Do not change the output file settings for InputGenerator — the output is configured by default.

• In the File origin pane choose the Project origin.
• Click and navigate to templates\example_solver_input.dat.
• Leave other settings default and click to close the dialog.

The input file is set, and InputGenerator configuration window now displays its contents in the Template file pane.

The next task is to replace the values of variables in the template (“X1” and “X2”) with values from the InputGenerator input ports.

• Click and drag your mouse over the two values in the template, selecting a rectangular area.

When you release the mouse button, a pop-up appears. Now you can specify the name of a variable and select the operation to perform (write to file or read from file).

• Input the name for a new variable: x. Note that no variables were added yet, so this variable will be created automatically.
• Click to write the value of a variable to the generated file.

Note that the variable named “x” now appears on the Variables pane. Also, search and replace operations are added, shown on the Operations pane.

Adding a variable also creates the InputGenerator.x port of the RealVector type. The block will substitute the value received to this port into the template.

You can now preview the generated output file to verify that value replacements work as expected.

• In the Variables pane, add a test value for the “x” variable. For example, a vector (0.1, 0.2).
• Click in the Operations pane toolbar to test the replacements. If they are correct, you should see that the values in the template are replaced with specified test values. Note that the template shows the location of replacements. This test does not change the template file itself, you just get a preview of the generated output file.
• Click in the Operations pane toolbar to cleanup after the test.

Testing the replacements completes the InputGenerator configuration. The output file in Text blocks is configured by default, so you do not have to change any settings in the Output file pane.

### Parsing Output¶

Parsing the program’s output is very similar to generating input. The output file is sent to a Text block which extracts values from specified locations in the file and uses a template file to configure operations.

• Add another Text block. Name it OutputParser.

Add a template file to the OutputParser configuration.

• Click in the Template file pane to open the file dialog.
• In the file dialog, select example_solver_output.dat from the templates subdirectory in your project.

Note

Do not change the input file settings for OutputParser — the input will be a file received from Launcher.

The sample file is set, and OutputParser configuration window now displays its contents.

The next task is to extract two values from the “Function:” line and send them to an output port.

• Like before, click and drag the selection over function values. Select only numbers, not the whole line.
• Input the name for a new variable: f. The variable in block configuration will be created automatically.
• Click to read the values from file and assign them to the variable.

Adding this variable creates the OutputParser.f port of the RealVector type. The block will read values from the file it receives and output them to f as a single vector.

You can now test the block to verify that the file is parsed correctly.

• Click in the Operations pane toolbar. The block parses the template file and shows the found values on the Variables pane.
• Click in the Operations pane toolbar to cleanup after the test.

The test completes the OutputParser configuration. The input file configuration should be left default, so you do not have to change any settings in the Input file pane. The output file configuration also does not require any changes, but you can disable the output file if you wish (this file is not needed in the current workflow).

## Workflow¶

Finished workflow is a typical integration wrapper.

This workflow can be used in two ways:

1. Run Toy Solver with various inputs and collect the results. Note that you can set a fixed run name to store all outputs to the same record in the project database (see the Results and Reports tutorial for more details).
2. Import it as a block into a new workflow and use it for sampling, approximation, optimization and other tasks.

## Results¶

To test the workflow, input the values of variables on the Input pane in Run.

Note that Toy Solver crashes if any of the input values is out of the $$[0.0, 1.0]$$ range. This is intended behavior: the Conditions and Error Handling tutorial explains how to handle such errors in the workflow and uses Toy Solver as an example.

After you run the workflow, the results from Toy Solver appear in the Outputs pane.

All inputs and outputs are also automatically stored to the project database for analysis.

## Conclusion¶

The workflow from this tutorial has one critical flaw: it stops with an error when input values are out of valid range. In real tasks there is always a possibility that the integrated program produces some unexpected results, and the behavior of Toy Solver is a simplified example. However, such errors can be handled in pSeven, as it is further explained in Conditions and Error Handling.

• Save the workflow you have created in this tutorial: it will be required further.

Note that the Conditions and Error Handling tutorial continues with the same workflow and explains an important step that has to be done before our wrapper can be used for other tasks.