Workflow

This page contains general information about pSeven workflow structure and introduces advanced features available in workflows.

General Information

In general, a workflow represents the process of solving a particular task. Each workflow is stored into a separate binary file with an .p7wf extension, for example, MyWorkflow.p7wf; the name of this file is the workflow’s name, too. However, a workflow is always a part of some project and cannot be opened or executed without opening the project first. One of the reasons is that the project provides a common database which stores results from all workflows. It allows to create reports consolidating data obtained from different workflows in the same project.

_images/page_workflow_01_struct.png

Main elements of a workflow are blocks 1 and links 2.

Blocks are functional elements that process input data and generate output. Blocks are isolated at OS level: they run in separate processes which do not share memory. Block isolation improves the application stability because even if some block encounters an error during execution, the workflow can continue to run.

Links are data routing elements. Blocks mainly communicate through links (in certain cases you can also use the block sandbox to transfer data; see section Sandbox for more details). It is important that links also define the order of block execution. In the example above, Model starts only when it receives data from Sample; you do not need to specify starting time or order because pSeven resolves such data dependencies automatically. This rule is known as the data flow concept.

See also

Simple Workflow tutorial
Provides a basic example of workflow editing and block configuration.
Block Configuration
Common block properties, settings, and functions.

Hierarchical Workflow Structure

In pSeven you can create custom blocks that contain a number of nested blocks. There is a special block type, Composite, which in this case is used as a container.

There are several ways to add a Composite block to the workflow:

  • Add an empty Composite block from the block library, then edit its contents (add nested blocks).
  • Add nested blocks first, select them and click b_group on the edit toolbar to put selected blocks into a new Composite block (or select Group... from the edit toolbar context menu b_context). Note that the reverse operation, b_ungroup Ungroup is also available from the context menu.
  • Import another workflow as a block (see section Workflow Import).

Composite blocks can be used to simply group blocks (for example, to make complex workflows easier to read), but they also provide several advanced functions. First of all, every workflow is in fact contained inside a Composite block; this persistent block is called the workflow root. Thus importing a workflow actually copies its root block. Next, Composite blocks allow to create parallel and cached regions in the workflow; these features are described in more detail below (see sections Parallelization and Caching, respectively).

Cycles

Workflows can contain cycles; a typical example is an optimization cycle controlled by an Optimizer block.

_images/page_workflow_02_optloop.png

In this example, Optimizer send values of variables to Model, which calculates function values and sends them back to Optimizer.

A more generic example is a processing loop controlled by a Map block.

_images/page_workflow_03_maploop.png

In this example, Loop receives a data sample and sends data points sequentially to the loop body (blocks A, B, and so on). Results are returned to Loop which collects them and finally outputs a results sample.

Note that such blocks as Optimizer and Loop have non-trivial behavior: they can control execution of other blocks that constitute the body of a loop. For example, Optimizer is free to generate new values of variables (and hence start the Model block) until it finds the optimum; that is, only Optimizer decides when to stop the cycle. Such blocks are called cycle drivers.

See also

Sampling Loop tutorial
Generic loop example.
Optimization Basics tutorial
Simple optimization loop example.

Branching

pSeven allows to create links that connect the same output port to different input ports. In this case, each link transmits a copy of the data output by the source port.

_images/page_workflow_04_branch.png

In this example, both Model1 and Model2 receive the same data sample. These blocks start two workflow branches.

Connecting multiple links to the same target port is also allowed, although not recommended because in a general case it creates a race condition on the target port and workflow may stop with an error.

Conditional Branching

Compared to the simple branching shown above, a workflow can also contain alternative branches that start based on some condition.

_images/page_workflow_05_condbranch.png

In this example, the input sample is sent to the Condition block first. This block can, for example, check the sample size and send it either to Model1 or to Model2. It ensures that both model blocks never start simultaneously, so the block that processes results will receive only one data sample, and race condition is avoided.

Conditional branching can also be used to handle block errors, filter out incorrect inputs, and other similar purposes.

See also

Conditions and Error Handling tutorial
Provides an example of using the Condition block.

Parallelization

Parallelization in pSeven can be understood in different ways. Naturally, blocks that are parts of different workflow branches can start in parallel, since there are no data dependencies between them. Taking the example from section Branching, Model1 and Model2 start immediately when they receive data from Sample, so both run simultaneously. If Model1 finishes processing first, Results1 will start while Model2 is still running; none of the blocks waits for another unless it needs data from that block.

Certain blocks can create parallel threads internally. For example, Optimizer, ApproxBuilder and other Modeling blocks have an option that sets the number of parallel threads for a specific block and can be used to speed up optimization or model training.

Lastly, a special parallel execution feature is provided by Composite blocks: with parallelization enabled, this block creates several parallel processes for its nested blocks and distributes input data between them. For more details, see section Parallel Execution on the Composite block page. This feature is also explained in detail in the Parallelization tutorial.

See also

Composite block — Parallel Execution
More details on configuring parallel execution in Composite blocks.
Parallelization tutorial
Provides an example of configuring a parallel Composite block.

Caching

pSeven allows to create cached regions in workflows by using a Composite block that is configured to save its input and output values. The part of the workflow that is contained inside this block will start only if the received inputs are not found in the Composite block’s cache. It means that if the cached Composite repeatedly receives the same input, actual calculation is performed only once, thus saving some processing time.

Cached data can be stored in memory or saved to a file on disk. The memory cache is automatically cleared after running the workflow (the cached data is not saved). File cache is persistent: the data cached to file will be available in subsequent workflow runs, and can be cleared only manually. Using a file cache also allows to set the same cache for two or more Composite blocks. For more details on these cache configurations, see section Cache on the Composite block page.

Note that the workflow root block also supports caching, since it is in fact a top-level Composite block. This can be useful if your workflow has inputs that are specified by user in Run.

See also

Composite block — Cache
More details on cache configuration in Composite blocks.
Caching tutorial
Provides an example of cache usage in the Composite block.

Workflow as a Block

In pSeven, you can include an existing workflow as a block into the workflow you edit (import a workflow), or pick a block from your workflow and export it to a separate workflow. Both functions rely on Composite blocks and are found in the toolbar menu in Edit.

_images/page_workflow_07_wfblock_menu.png

Import Workflow... 1 loads a workflow from file and includes it into the workflow you edit as a Composite block. If you select an existing Composite block, the workflow is imported into this block. If no block is selected, you are offered to create a new Composite block on import.

Export Workflow... 2 is the opposite. If you select a Composite block in the workflow you edit, this block is exported to a new workflow. Export can also be used without selecting any block, if you are currently editing some Composite block — in this case it exports the block you edit.

See also

Workflow as a Block tutorial
Provides an example of importing an existing workflow as a new Composite block.

Workflow Import

To import a workflow as a block, you can do one of the following:

  • Deselect all (click an empty area in the workflow you edit), then select Import Workflow... from the edit toolbar menu. A new Composite block will be created on import. This new block will contain the imported workflow.
  • Select an existing Composite block, then select Import Workflow... from the edit toolbar menu. The workflow will be imported into the selected block.

In general, importing a workflow places it inside a Composite block, and workflow inputs and outputs become ports of this block. New port names are copied from the aliases set for workflow inputs and outputs in workflow configuration. If the aliases were not set, then new names are copied from the original port names. This way the new names are always the same as the names seen in Run for the imported workflow.

Note that if you import into an existing block, the workflow replaces all block contents (removes all nested blocks). Existing ports are also removed, unless:

  • the names of existing ports match the new port names (generated from workflow input and output aliases, or port names if aliases are not set, see above), and
  • the types of existing and new ports are compatible.

This feature allows to preserve existing links to the host Composite block, if ports are properly named and typed. It is primarily intended to support reimporting of the same workflow — for example, when the workflow was changed, but its inputs and outputs were not edited. In other cases, when pSeven is unable to “merge” existing and new ports, some links may be lost.

Clicking Import Workflow... opens the import dialog.

_images/page_workflow_08_wfblock_import.png

Basic settings are:

  • Block: the target Composite block. If you are importing into a new block, this field is empty, and you have to input a name for the new block. If you are importing into an existing block, this field is read-only.
  • Workflow: the workflow file to open.

Advanced settings below are related to workflow settings seen in Run (the Inputs pane and the Parameter settings pane).

Inputs

  • Copy input values to port defaults

    If input values are specified in the imported workflow (see the Inputs pane in Run), they can be copied to Composite block input ports.

    Clearing this checkbox only discards input values, general rules for port creation remain the same. Port values can also be edited or removed after import.

Configuration

  • Import parameter presets

    If the imported workflow has some options or ports selected as parameters (see the Parameter settings pane in Run), they are always added as parameters to the workflow you edit. However, parameter values stored in presets are not copied by default. Select this option if you want to add presets on import.

    You can choose the presets to add:

    • All: imports all presets.
    • Selected: imports only the presets you select from the drop-down list. Use Ctrl+Click to select multiple presets.

    If no presets exist in the workflow you edit, imported presets are simply added under the same names. However, this is usually not the case — at least, the “Default” preset exists in all workflows unless you intentionally delete it. To resolve this, pSeven creates all possible combinations of imported and existing presets, joining their names with an underscore.

    For example, if there are existing presets “Default” and “Test”, and you select two presets “P1” and “P2” on import, the result is four presets named “Default_P1”, “Default_P2”, “Test_P1”, and “Test_P2”.

    Note

    This setting potentially can create a very large number of presets in the workflow you edit. To avoid this, there is a limit for the number of presets.

    Note that pSeven does not remember how the presets were created — manually or automatically when importing a workflow. This means that all presets created on import are considered to be new presets of the workflow you edit. If you then import another workflow with presets, they are in turn combined with all presets created before; this happens even if you re-import the same workflow. Obviously, the number of presets in such scenarios grows exponentially, which quickly renders the workflow unusable. pSeven will detect this and warn you if the number of preset combinations exceeds 128.

    If you actively edit and re-import workflows, it is recommended to keep the preset import off until you have a final version of the imported workflow, and then import the final version with all required presets.

Monitoring

  • Keep monitoring settings

    If the imported workflow has monitoring enabled for some ports (see the monitoring settings on the Monitoring settings pane in Run), they are always added to the list of monitored ports in the workflow you edit. However, monitoring is disabled by default — so they appear on the Monitoring settings pane in Run after import, but corresponding checkboxes are cleared. If you want to apply the same monitoring settings as in the imported workflow, select this option.

Workflow Export

Workflow export works with Composite blocks and allows to create a new workflow from a block. When exporting, you can do one of the following:

  • Select an existing Composite block, then select Export Workflow... from the edit toolbar menu. Selected block will be exported to a new workflow (new file).
  • If you are currently editing a Composite block, deselect all (click an empty edit area), then select Export Workflow... from the edit toolbar menu. The block you are editing will be exported in this case.

In general, exporting a Composite block creates a new workflow file, block input and output ports become workflow inputs and outputs seen in Run. Export keeps parameter and monitoring settings related to the exported block (but not parameter presets). This means that monitored ports of nested blocks will also be monitored in the exported workflow, and parameters will be added to workflow configuration, but their values are not kept (the workflow will contain only an empty default preset).

When exporting a Composite block to a workflow, you only have to specify the name and location of a new workflow file after you click Export Workflow...; there are no additional settings.