September 17, 2015

Workflow Hierarchy, Import and Export

 

Task decomposition and structuring is very important in engineering. It allows to solve problems step by step and helps to understand dependencies and links between problem parts. pSeven inherently supports hierarchy in workflows thanks to the std.Composite block. It is a special block which functions as a wrapper for a group of blocks or another workflow. Grouping blocks into std.Composite gives the following advantages:

  • Visually simplifies and logically structures  a workflow   
  • Provides the ability to re-use certain workflows or their parts in new workflows
  • Enables collaborative work on complicated workflows
  • Simplifies the approach to parallel computation
  • Allows to re-use previously evaluated designs stored in the  std.Composite data cache

Decomposing a workflow into subtasks

Consider a dimension reduction (DR) workflow (similar one is found in the pSeven examples):

 

The workflow does the following:

  1. Train a DR model using the training data sample and pass it to the compressor and decompressor.
  2. Read data for computation and compress it using the trained model.
  3. Process data and make some calculations.
  4. Restore processed data into the original design space, using the trained model, and output this data as the result.

Let us see how can we improve this workflow. We can group the Train Sample and Train blocks into the single Prepare Model block, and also group Huge Data and Compressor into the Prepare Data block:

 

As a result we have the following improvements:

  • Better visual representation. Workflow is easier to understand, the Prepare Model block clearly starts the workflow
  • Better logical structure. The Prepare Data block now depends on Prepare Model. We do not need to read huge data if DR model fails to build for some reason

Note that pSeven tries to run blocks in parallel if possible, so in the initial workflow both std.CSVParser blocks launch simultaneously. In the changed workflow Prepare Data will be started only after Prepare Model finishes, since an std.Composite block behaves like a separate workflow, and in order to start it needs data to all connected inputs.

 
 

Re-using workflows

Optimization problem with an external solver is a typical task for pSeven, but working with external solvers usually requires some preparations. We can make a workflow to run the solver (prepare data in a text file, execute, read values from the resulting text file and send them to outputs) and test it without std.Optimizer. This will save us a lot of time, since we can test our solver on different input data without configuring and running other blocks.

After the Solver workflow is ready, we can import it as an std.Composite block into an optimization workflow and start working on an optimization problem. Similarly, the same Solver workflow can be used to train an approximation model.

 

Importing a workflow copies it inside a std.Composite block, and workflow inputs and outputs become ports of this block. New port names are copied from the aliases set for workflow inputs and outputs in workflow configuration. If the aliases were not set, then new names are copied from the original port names. This way the new names are always the same as the names seen in Run for the imported workflow. Note that if you import into an existing block, the workflow replaces all block contents. It is possible to reimport workflow into resulting std.Composite, if ports and types of std.Composite are compatible with imported workflow ports, links to std.Composite will be preserved. Also, path to workflow is stored inside std.Composite during import process and used during reimport (you can always change it in Import Worfklow dialog). Advanced settings of Import Workflow dialog allows you customize resulting std.Composite block:
  • Copy worklow input values as defaults of std.Composite input ports.
  • Merge imported workflow configuration parameters into resulting workflow configuration.
  • Keep monitoring settings from imported workflow.

 

Along with import it is possible to export any std.Composite block into a separate workflow and re-use it the same way as Solver above. After export, block input and output ports become workflow inputs and outputs seen in Run. Export keeps parameter and monitoring settings related to the exported block, but not parameter presets. This means that monitored ports of nested blocks will also be monitored in the exported workflow, and parameters will be added to workflow configuration, but their values are not kept (the workflow will contain only an empty default preset).

Workflow hierarchy is a fundamental pSeven feature. It allows to use the top-down approach to workflow construction and enables collaboration. std.Composite blocks, workflow import and export facilitate working with complex workflows developed together. Other features, such as parallel execution and caching make std.Composite essential for every pSeven user.

 

By Sergey Komissarov, Senior software engineer, DATADVANCE