Jobs with GNU Parallel¶

GNU Parallel is a powerful tool to run multiple, independent but similar tasks efficiently within one job. In essence, it is similar to Slurm's array jobs and can be used both as an alternative to array jobs, or together with array jobs. GNU Parallel is available as a module on the HPC, and can be loaded with module load parallel in your batch script. The examples below show how you can use GNU Parallel by itself, and with array jobs.

Basic GNU Parallel Job¶

The Slurm script for a basic GNU Parallel job will look like the following:

basic-parallel-job.slurm

#!/bin/bash
#SBATCH --ntasks=28
#SBATCH --nodes=1
#SBATCH --time=00:01:00
#SBATCH --partition=standard
#SBATCH --account=YOUR_GROUP

module load parallel
seq 1 100 | parallel 'DATE=$( date +"%T" ) && sleep 0.{} && echo "Host: $HOSTNAME ; Date: $DATE; {}"'

In this example, we're make use of a full node with GNU Parallel. The meat of the command lies here:

seq 1 100 | parallel 'DATE=$( date +"%T" ) && sleep 0.{} && echo "Host: $HOSTNAME ; Date: $DATE; {}"'

seq 1 100 generates a list between 1 and 100 (inclusive), and we pipe that into a parallel command which will generate one task per element (so 100 tasks). GNU Parallel will find the space on our node as it works through the relevant tasks.

Inside the command:

DATE=$( date + "%T" ) sets DATE so we can visualize tasks and when they're being executed.
sleep 0.{} forces each task to sleep for 0.n seconds, where n is the input integer from the seq command. This means, for example, the 2^nd task will wait longer than the 10^th task, as can be seen in the output file. This is used to demonstrate that these tasks are being executed in parallel.
echo "HOST: $HOSTNAME ; Date: $DATE; {}" prints out information about the task. {} is piped input which, in this case, is an integer generated by seq between 1 and 100.

You can submit the script with sbatch basic-parallel-job.slurm. Unlike an array job, this will produce only one output file:

[netid@junonia ~]$ head slurm-74027.out 
Host: i10n18 ; Date: 16:45:55; 1
Host: i10n18 ; Date: 16:45:55; 10
Host: i10n18 ; Date: 16:45:55; 11
Host: i10n18 ; Date: 16:45:55; 12
Host: i10n18 ; Date: 16:45:55; 13
Host: i10n18 ; Date: 16:45:55; 14
Host: i10n18 ; Date: 16:45:55; 15
Host: i10n18 ; Date: 16:45:55; 16
Host: i10n18 ; Date: 16:45:55; 17
Host: i10n18 ; Date: 16:45:55; 2

Array Jobs and GNU Parallel¶

Sometimes you need to run a lot of jobs. More than can be reasonably accomplished using arrays since submitting thousands of jobs can be a problem for the system, and GNU Parallel can be challenging to make work in a multi-node environment. In this case, we can combine the forces of GNU Parallel and array jobs to distribute a chunk of tasks across multiple nodes where GNU Parallel will execute them.

The Slurm script in this case will look like the following:

array-and-parallel.slurm

#!/bin/bash
#SBATCH --job-name=Sample_Array_With_GNU_Parallel
#SBATCH --ntasks=94
#SBATCH --nodes=1
#SBATCH --time=00:05:00
#SBATCH --partition=standard
#SBATCH --account=<YOUR_ACCOUNT>
#SBATCH --array=1-400:200

module load parallel
seq ${SLURM_ARRAY_TASK_ID} $((${SLURM_ARRAY_TASK_ID}+199)) | parallel echo "JOB ID: $SLURM_JOB_ID HOST NODE: $HOSTNAME EXAMPLE COMMAND: ./executable input_{}"

The main difference with the basic example is using an array job with a step size. The syntax is:

--array=index1-index2:step_size

So in this case, --array=1-400:200 submits 2 jobs, one with array index 1, the other with array index 200. This allows us to run 200 tasks per subjob. The array indices are used to differentiate tasks. seq n m generates a sequence of integers from n to m (inclusive), so

seq ${SLURM_ARRAY_TASK_ID} $((${SLURM_ARRAY_TASK_ID}+199))

becomes seq 1 200 in the first job and seq 201 400 in the second.

You can submit the script with sbatch array-and-parallel.slurm. It will produce two output files, one for each job:

Generated output files

[netid@junonia ~]$ ls *.out
slurm-23714261_1.out  slurm-23714261_201.out

Output file contents

[netid@junonia ~]$ head *.out
==> slurm-23714261_1.out <==
1
JOB ID: 23714298 HOST NODE: r7u04n2 EXAMPLE COMMAND: ./executable input_1
JOB ID: 23714298 HOST NODE: r7u04n2 EXAMPLE COMMAND: ./executable input_2
JOB ID: 23714298 HOST NODE: r7u04n2 EXAMPLE COMMAND: ./executable input_3
JOB ID: 23714298 HOST NODE: r7u04n2 EXAMPLE COMMAND: ./executable input_4
JOB ID: 23714298 HOST NODE: r7u04n2 EXAMPLE COMMAND: ./executable input_5
JOB ID: 23714298 HOST NODE: r7u04n2 EXAMPLE COMMAND: ./executable input_7
JOB ID: 23714298 HOST NODE: r7u04n2 EXAMPLE COMMAND: ./executable input_6
JOB ID: 23714298 HOST NODE: r7u04n2 EXAMPLE COMMAND: ./executable input_8
JOB ID: 23714298 HOST NODE: r7u04n2 EXAMPLE COMMAND: ./executable input_9

==> slurm-23714261_201.out <==
201
JOB ID: 23714261 HOST NODE: r7u05n2 EXAMPLE COMMAND: ./executable input_201
JOB ID: 23714261 HOST NODE: r7u05n2 EXAMPLE COMMAND: ./executable input_202
JOB ID: 23714261 HOST NODE: r7u05n2 EXAMPLE COMMAND: ./executable input_203
JOB ID: 23714261 HOST NODE: r7u05n2 EXAMPLE COMMAND: ./executable input_204
JOB ID: 23714261 HOST NODE: r7u05n2 EXAMPLE COMMAND: ./executable input_205
JOB ID: 23714261 HOST NODE: r7u05n2 EXAMPLE COMMAND: ./executable input_207
JOB ID: 23714261 HOST NODE: r7u05n2 EXAMPLE COMMAND: ./executable input_206
JOB ID: 23714261 HOST NODE: r7u05n2 EXAMPLE COMMAND: ./executable input_208
JOB ID: 23714261 HOST NODE: r7u05n2 EXAMPLE COMMAND: ./executable input_209