Job array

1. Introduction

2. Limiting the maximum number of running array tasks at once

3. Naming output and error files

4. Cancel job arrays and tasks

5. More advanced examples

Introduction

When dealing with many identical jobs with different inputs or multiple runs with slight change of the parameters, one can try SLURM’s Job Array Support feature. This can be done by adding

into job submission script. Another option is use as the argument when use sbatch command

From the above example, it will create 10 independent jobs with the array index ranging from 0 to 9. All the jobs will use the same resource as specified on the slurm directive lines ( e.g. --cpus-per-tasks, --time etc.).

The slurm environment variable $SLURM_ARRAY_TASK_ID will be assign to each task, and users can use this variable to handle input and output files for that task.

Example :

myapp.sh

submit.sh

When sbatch submit.sh and used myqueue command, it will show <JOB ID>_1 to <JOB ID>_5 on the list

Limiting the maximum number of running array tasks at once

To limit the maximum number of simultaneously running tasks from the job array, one can use "%" separator. Example :

Naming output and error files

SLURM uses the %A and %a replacement strings for the master job ID and task ID, respectively.

Example:

Cancel job arrays and tasks

To delete some tasks of an array job

To delete all of the tasks of an array job

More advanced examples

1. Changing INPUT files by mapping job index to multiple data files in a file
   Case scenario : Run post-processing of 1000 data files and the list of data file names are contained in a data mapping file called files.txt . We want to use an array of 20 jobs, each element job processes 50 data files.
   For example, inside files.txt :

Template script :