July 28, 2015
HPC Made Easy
Let's consider the problem of performing a lot of numerical experiments using some third-party solver to generate the data to build an approximation model. Each solver’s run may be time-consuming and performing multiple calculations on a single computer can take plenty of time.
The calculations can be speed up by using computer clusters. pSeven provides integration with the following types of cluster resource manager software:
- LSF / OpenLava
Integration with computer clusters is performed by std.ShellScript block which also supports Job Array mechanism simplifying management of collections of similar tasks.
To solve the problem we use Approximation workflow from pSeven’s General Examples. The workflow generates a data sample which is used both as the solver input sample and the input part of the model training sample.
The Solver is a composite block containing the workflow to process a single vector of input data.
The Solver composite block expanded:
The option “Parallel execution” of Solver block is enabled and this allows the block to automatically process the input sample matrix as a list of input vectors. The option also defines how many instances of the internal workflow will be run at the same moment.
The Launcher block command:
How to Speed Up the Workflow
If the solver requires many CPU/memory resources then running multiple solvers on one computer will be not very effective.
Let’s modify the settings of Launcher block:
- batch manager: “slurm”
- ssh connection information (“Host”, “Port”, “Username”, “Password”)
- synchronization: (“local to remote”)
Now Launcher block will run the solver on the cluster automatically uploading input files and downloading output results. Due to parallel execution of Launcher instances, the solver on a cluster will be run in parallel too.
Using parallel execution of composite blocks along with running the solver on computer cluster is an easy way to speed up the workflow by just a few clicks. You may increase performance even further, by making the workflow prepare all input files before submitting the job to the cluster and making std.ShellScript run a whole bunch of calculations via a mechanism called Job Array.
By Stanislav Markevich, Senior software engineer, DATADVANCE