Integration Basics

This tutorial shows the basics of integrating a third-party tool into a workflow. It explains a generic solution applicable to any program that can read parameters from the command line or from a text file (script).

Before You Begin

This tutorial requires an existing prjTutorials project. If you have not created this project yet, see Tutorial Project first.

  • Open the prjTutorials project.

  • Create a new empty workflow. If you have never created a workflow before, see the Simple Workflow tutorial.

  • Save the workflow as wfIntegrationBasics.

  • Switch to Workspace and verify that the workflow file (IntegrationBasics.p7wf) is added to the project. You can see it in the Project pane.

  • While in Workspace, verify that the following files exist in the project:

    • The example executable to be integrated in the workflow:
      • tools\toy_solver.exe (in Windows).
      • Three files tools/toy_solver, tools/toy_solver_x86, tools/toy_solver_x64 (in Linux). The tools/toy_solver file is the initial script, that chooses the tools/toy_solver_x86 or the tools/toy_solver_x64 executable.
    • templates\example_solver_input.dat — a sample of solver’s input.
    • templates\example_solver_output.dat — a sample of solver’s output.

    If these files are not found, see Tutorial Project, section Adding Files.

  • Switch to Edit and select the workflow you have just created to continue with the tutorial.

Task

The example executable (Toy Solver) is a simple command line tool which imitates a solver program. It requires an input file containing several commands, some of which set values of input variables. The tool evaluates some function of these variables and prints a “report” containing function values and some additional information.

The task, in general, is to create a workflow that allows to specify values of variables and to get function values in pSeven without running Toy Solver manually. This workflow has to generate a valid input file with specified values, launch Toy Solver, then capture and parse its output. In other words, the workflow creates a wrapper around Toy Solver and allows to automate its usage. This wrapper is the first step to integrating an external executable into approximation and optimization workflows: pSeven supports using a workflow as a block, so once you have created a wrapper, you can use it in various other tasks as a ready custom component.

Solution

The task of integrating a command line tool into a workflow can be divided into the following general steps:

  • Investigate the tool. You need to know how to start it from command line and how to pass parameters.
  • Configure a Program block to launch the tool. This block executes a command line which can accept parameters from other blocks in the workflow.
  • Generate input for the tool so it can be used when Program launches the executable. In this tutorial, the input is a temporary file prepared by a Text block.
  • Capture the output and parse it to obtain data that other blocks can use. In this tutorial, the output is also a temporary file, and it is parsed by another Text block.

Tool Investigation

To make the generic integration solution explained in this tutorial possible, there are certain requirements for the tool you want to integrate:

  1. It must support non-interactive command line interface — that is, it must allow starting it from a command prompt and must be able to read input data from a file or directly from the command line.
  2. It must output to a file. It can also just print the output, since screen output is easily redirected to a file.

Our example tool, the Toy Solver, accepts an input file containing values of variables, evaluates some function, and prints the results (but does not save them to a file). Its basic command line syntax is:

> toy_solver.exe input_file

for Windows, or

$ ./toy_solver input_file

for Linux.

For more details, you can run Toy Solver without parameters to print help. For example, in Windows:

> toy_solver.exe

Example Toy Solver.

Calculates values and gradients of a vector function in 2D. Both input and
output are 2-dimensional. Function domain is limited to [0.0, 1.0] in 2D -
if any input is not in range [0.0, 1.0], the solver exits with an error.
Variable values are set in the input script (see usage). Results are output
to terminal or a file.

Usage: toy_solver.exe [-n] infile [> outfile]

  toy_solver.exe                   this message
  toy_solver.exe infile            execute the script from infile and print
                                   result on screen (see script template)
  toy_solver.exe infile > outfile  execute the script from infile and print
                                   result to outfile

Options:

  -n      notch: adds a region of undefined behavior: X1 - X2 > 0.95 (unit
          square corner); if set, the solver also exits with an error if input
          point is in the notch region

Script template:

#TOYSOLVER

SETUP
  INPUT = X1 X2
  OUTPUT = F1 F2
BEGIN
  SET X1 = 0.5
  SET X2 = 0.5
  SOLVE F1 F2
RETURN

When making a new input script, the template should be copied as is, only
changing the X1 and X2 values.

To begin, try running Toy Solver without parameters.

  • Open a command prompt.

  • Change to the tools subdirectory in the prjTutorials project.

  • Run the Toy Solver executable in this directory.

    In Windows:

    > toy_solver.exe
    

    In Linux:

    $ ./toy_solver
    
../_images/page_tutorials_integration_01_cmd.png

When Toy Solver is run with an input file, it prints calculation results. There is a template input script (example_solver_input.dat) located in the templates subdirectory of your project.

  • Try running Toy Solver with an input file.

    In Windows:

    > toy_solver.exe ..\templates\example_solver_input.dat
    

    In Linux:

    $ ./toy_solver ../templates/example_solver_input.dat
    

Screen output should be:

Solving in [5.000000e-01, 5.000000e-01].

Result:

Function:  5.000000e-01  3.841688e+00
Gradient:  1.000000e+00  0.000000e+00
          -1.658312e+00  7.643199e+00

The workflow, however, needs an output file it can read, not the screen output. This is not a problem since output redirection is a standard OS function.

  • Try redirecting Toy Solver output to a file.

    In Windows:

    > toy_solver.exe ..\templates\example_solver_input.dat > ..\templates\test_output.dat
    

    In Linux:

    $ ./toy_solver ../templates/example_solver_input.dat > ../templates/test_output.dat
    

This time nothing is printed, but you can find test_output.dat in the templates subdirectory. Compare this file to the screen output from the previous step.

Note

The example_solver_output.dat file in templates is a pre-generated output which is the same as test_output.dat from above.

As you can see, mass calculations with Toy Solver are far from being convenient. If you were to run it manually, each evaluation would require you to:

  1. Edit the input file to change values of variables.
  2. Execute the command.
  3. Copy function values from the output to save them somewhere.

This tutorial automates the steps above, so you can just specify values of variables as workflow inputs and make pSeven do the rest of the job. Further, this solution can be used in workflows that require multiple evaluations — for example, to integrate Toy Solver in an optimization cycle, as shown in the Integrated Component Optimization tutorial.

Tool Launch

Now that you know how to execute Toy Solver manually, the next step is to configure its launch from the workflow.

  • Add a Program block. Name it Launcher.

The Launcher block has to be configured to run Toy Solver with an input file as a command line parameter and then redirect the output to another file.

Begin with adding the input and output files to Launcher. Open its configuration; to add a file, click b_blconf_add in the Files pane.

Add the input file.

../_images/page_tutorials_integration_02_infile.png
  • Specify the port name: infile.
  • Select direction, input.
  • Specify the file name: input.dat.

Similarly, add the output file.

  • Port name: outfile.
  • Direction: output.
  • File name: output.dat.

Verify file settings in the Files pane.

../_images/page_tutorials_integration_03_files.png

Note

File names are optional. You can leave file name empty and use port names in commands instead. The syntax is @{port_name} — for example, @{infile}; when Launcher executes the command, it substitutes actual file names where port names are specified.

On the main tab, specify the program name, relative path, command line parameters (the input file) and redirect standard output to the output file.

../_images/page_tutorials_integration_04_ssconf.png
  • Program: input toy_solver.exe in Windows, ./toy_solver in Linux.
  • After specifying the program, verify that the relative path is set to Relative to the sandbox directory (the default setting).
  • Arguments: input input.dat (or @{infile}, see the note above).
  • Input/Output redirection: select “outfile” for stdout. Toy Solver’s screen output will be sent to this file.

Verify the command preview.

In Windows:

toy_solver.exe input.dat >output.dat

In Linux:

./toy_solver input.dat >output.dat

Compare the above with the command lines you have used to start Toy Solver when examining it. When the workflow runs, Launcher will start when it receives a file to the infile port. This file is temporary and does not exist on disk. To make it available to Toy Solver, Launcher will create a temporary directory and copy the received file there under the name you have specified when adding the file (“input.dat”). If the file name was not specified (@{port_name} syntax is used), then the name will be generated at random, and Launcher will substitute this generated name to the command line.

As you can note, the command line does not include the path to the Toy Solver executable, so Launcher does not yet know where to look for it. You could specify an absolute path to the program (like C:\Users\UserName\Documents\pSeven\Tutorials\tools\toy_solver.exe), but this has an unwanted effect of making the project not portable: if you move the Tutorials directory, you then need to edit Launcher configuration again to change the path. In this tutorial, a better solution is to place the Toy Solver executable into same temporary directory that is created by Launcher.

In pSeven this directory is called “sandbox” (see section Sandbox for details). The Program block uses sandbox as its current working directory, so all relative paths are resolved inside the sandbox. The question is how to make the Toy Solver executable appear in the Launcher sandbox (since we want to keep its relative path in the block configuration). This can be done in two ways:

  1. Set the sandbox path to the tools subdirectory in the project, which hence becomes the working directory for Launcher. This is acceptable but may be unsafe in certain cases.
  2. Specify that the tools subdirectory is the sandbox prototype. The contents of prototype are copied to the sandbox when the block starts. This configuration is preferred because the prototype is read-only.

To configure the sandbox, switch to the Sandbox tab in Launcher configuration.

../_images/page_tutorials_integration_05_sbox.png
  • Input the path in the Prototype field: tools. Note that the block automatically transforms your input to a relative path (.\tools in Windows, ./tools in Linux, where . means the current project directory).
  • Leave other settings on the Sandbox tab default.

The key feature of this configuration is that a relative sandbox prototype path means that the prototype is a subdirectory of the current project. No matter where you move the Tutorials directory, Launcher will search for the Toy Solver executable in Tutorials/tools and copy it to the sandbox (temporary working directory).

Launcher configuration is now complete. Next steps are to generate the input file and to process the Toy Solver’s output.

Generating Input

Program input files can be generated using a Text block.

  • Add a Text block. Name it InputGenerator.

To generate an input file for the program, Text requires a sample of such file as its input. This file is also automatically used as template in block configuration. As mentioned before, there is an sample input file for our tool found at templates\example_solver_input.dat — this file shall be the input template.

Open the InputGenerator configuration and click b_browse in the Input File pane to bring up the file selection dialog.

Note

Do not change the output file settings for InputGenerator — the output is configured by default.

../_images/page_tutorials_integration_06_conffile.png
  • In the File origin pane choose the Project origin.
  • Click b_browse and navigate to templates\example_solver_input.dat.
  • Leave other settings default and click b_ok to close the dialog.

The input file is set, and InputGenerator configuration window now displays its contents in the Template File pane.

../_images/page_tutorials_integration_07_templateset.png

The next task is to replace the values of variables in the template (“X1” and “X2”) with values from the InputGenerator input ports.

  • Click and drag your mouse over the two values in the template, selecting a rectangular area.
../_images/page_tutorials_integration_08_qsel1.png

When you release the mouse button, a pop-up appears. Now you can specify the name of a variable and select the operation to perform (write to file or read from file).

../_images/page_tutorials_integration_09_qsel2.png
  • Input the name for a new variable: x. Note that no variables were added yet, so this variable will be created automatically.
  • Click b_blconf_write to write the value of a variable to the generated file.

Note that the variable named “x” now appears on the Variables pane. Also, search and replace operations are added, shown on the Operations pane.

Adding a variable also creates the InputGenerator.x port of the RealVector type. The block will substitute the value received to this port into the template.

You can now preview the generated output file to verify that value replacements work as expected.

../_images/page_tutorials_integration_10_preview.png
  • In the Variables pane, add a test value for the “x” variable. For example, a vector (0.1, 0.2).
  • Click b_blconf_play in the Operations pane toolbar to test the replacements. If they are correct, you should see that the values in the template are replaced with specified test values. Note that the template shows the location of replacements. This test does not change the template file itself, you just get a preview of the generated output file.
  • Click b_blconf_sweep in the Operations pane toolbar to cleanup after the test.

Testing the replacements completes the InputGenerator configuration. The output file in Text blocks is configured by default, so you do not have to change any settings in the Output File pane.

Parsing Output

Parsing the program’s output is very similar to generating input. The output file is sent to a Text block which extracts values from specified locations in the file and uses a template file to configure operations.

  • Add another Text block. Name it OutputParser.

Add a template file to the OutputParser configuration.

  • Click b_browse in the Template File pane to open the file dialog.
  • In the file dialog, select example_solver_output.dat from the templates subdirectory in your project.

Note

Do not change the input file settings for OutputParser — the input will be a file received from Launcher.

The sample file is set, and OutputParser configuration window now displays its contents.

../_images/page_tutorials_integration_11_sampleset.png

The next task is to extract two values from the “Function:” line and send them to an output port.

  • Like before, click and drag the selection over function values. Select only numbers, not the whole line.
../_images/page_tutorials_integration_12_qsel.png
  • Input the name for a new variable: f. The variable in block configuration will be created automatically.
  • Click b_blconf_read to read the values from file and assign them to the variable.

Adding this variable creates the OutputParser.f port of the RealVector type. The block will read values from the file it receives and output them to f as a single vector.

You can now test the block to verify that the file is parsed correctly.

../_images/page_tutorials_integration_13_parsetest.png
  • Click b_blconf_play in the Operations pane toolbar. The block parses the template file and shows the found values on the Variables pane.
  • Click b_blconf_sweep in the Operations pane toolbar to cleanup after the test.

The test completes the OutputParser configuration. The input file configuration should be left default, so you do not have to change any settings in the Input File pane. The output file configuration also does not require any changes, but you can disable the output file if you wish (this file is not needed in the current workflow).

Workflow

Finished workflow is a typical integration wrapper.

../_images/page_tutorials_integration_14_wf.png

This workflow can be used in two ways:

  1. Run Toy Solver with various inputs and collect the results. Note that you can set a fixed run name to store all outputs to the same record in the project database (see the Results and Reports tutorial for more details).
  2. Import it as a block into a new workflow and use it for sampling, approximation, optimization and other tasks.

Results

To test the workflow, input the values of variables on the Input pane in Run.

Note that Toy Solver crashes if any of the input values is out of the \([0.0, 1.0]\) range. This is intended behavior: the Conditions and Error Handling tutorial explains how to handle such errors in the workflow and uses Toy Solver as an example.

../_images/page_tutorials_integration_15_wfin.png

After you run the workflow, the results from Toy Solver appear in the Outputs pane.

../_images/page_tutorials_integration_16_wfout.png

All inputs and outputs are also automatically stored to the project database for analysis.

Conclusion

The workflow from this tutorial has one critical flaw: it stops with an error when input values are out of valid range. In real tasks there is always a possibility that the integrated program produces some unexpected results, and the behavior of Toy Solver is a simplified example. However, such errors can be handled in pSeven, as it is further explained in Conditions and Error Handling.

  • Save the workflow you have created in this tutorial: it will be required further.

Note that the Conditions and Error Handling tutorial continues with the same workflow and explains an important step that has to be done before our wrapper can be used for other tasks.