Run on a cluster
PEPPRO by itself does not specify any cluster resources, so you could just roll your own and submit individual jobs to a cluster however you choose. But because
PEPPRO is already
looper-compatible, the easier way is to use
looper's built-in template system, which
looper uses to build flexible shell scripts for job submission. These templates can be used to run jobs in a container, to submit to a cluster resource manager, or both.
looper templates, we must create a
divvy computing configuration file (compute_config.yaml) and point an environment variable (
DIVCFG) to that file. You then have access to any configured computing packages by using
looper --compute <package>, where
package can be any computing system you configure. In short, you will need to:
- Set up a compute configuration file that includes a containerized or cluster compute template (or both).
- Point the environment variable
DIVCFGto the location of this file.
- Run the pipeline with
looper run --compute PACKAGE(where
PACKAGEis specified in your
This enables you to adjust your computing preferences on-the-fly when you run a project.
The complete description of setting up
looper to use
DIVCFG is generic to any pipeline. If you want to use looper with containers or clusters, you should consult the complete docs in the looper documentation on configuring looper to use a cluster.