Batch Directives

Tip

For a full list of directives, see Slurm's official documentation.

The first section of a batch script (after the shebang) always contains the Slurm Directives, which specify the resource requests for your job. The scheduler parses these in order to allocate CPUs, memory, walltime, etc. to your job request.

Allocations and Partitions

The partitions, or queues, on the UArizona HPC which determine the priority of your jobs and resources available to them are shown in the table below. With the exception of Windfall, these consume your monthly allocation. See our allocations documentation for more detailed information on each. The syntax to request each of the following is shown below:

NEW! Updated Partitions

To improve GPU resource availability, we have three new partitions: gpu_standard, gpu_windfall, and gpu_high_priority. This will prevent CPU-only jobs from occupying GPU node resources. GPU jobs must use the new partitions to be submitted successfully.

Buy-in users must use the --qos directive

If you're a member of a buy-in group and are trying to use your high priority hours, ensure you are including a --qos directive. When this directive is missing, you will recieve the error:

sbatch: error: QOSGrpSubmitJobsLimit
sbatch: error: Batch job submission failed: Job violates accounting/QOS policy (job submit limit, user's size and/or time limits)

Partition	Request Syntax	Comments
Standard	`#SBATCH --account=<PI GROUP> #SBATCH --partition=standard`	Request a CPU-only node using standard hours.
NEW! Standard GPU	`#SBATCH --account=<PI GROUP> #SBATCH --partition=gpu_standard #SBATCH --gres=gpu:<options>`	Request GPU resources using standard hours. See the GPUs section below for details on the `gres` directive.
Windfall	`#SBATCH --partition=windfall`	Unlimited access. Preemptible. Do not include an `--account` flag when requesting this partition.
NEW! Windfall GPU	`#SBATCH --partition=gpu_windfall #SBATCH --gres=gpu:<options>`	Request GPU resources using windfall. See the GPUs section below for details on the `gres` directive. Do not include an `--account` flag when requesting this partition.
High Priority	`#SBATCH --account=<PI GROUP> #SBATCH --partition=high_priority #SBATCH --qos=user_qos_<PI GROUP>`	Request a CPU-only node with high priority resources. Only available to buy-in groups.
NEW! High Priority GPU	`#SBATCH --account=<PI GROUP> #SBATCH --partition=gpu_high_priority #SBATCH --qos=user_qos_<PI GROUP> #SBATCH --gres=gpu:<options>`	Request GPU resources with high priority hours. Only available to buy-in groups. See the GPUs section below for details on the `gres` directive.
Qualified	`#SBATCH --account=<PI GROUP> #SBATCH --partition=standard #SBATCH --qos=qual_qos_<PI GROUP>`	Available to groups with an activate special project.
NEW! Qualified GPU	`#SBATCH --account=<PI GROUP> #SBATCH --partition=gpu_standard #SBATCH --qos=qual_qos_<PI GROUP> #SBATCH --gres=gpu:<options>`	Request GPU resources with qualified hours. Available to groups with an activate special project. See the GPUs section below for details on the `gres` directive.

CPUs

Each job must specify the requested number of CPUs with the --ntasks directive. This can be done in one of two ways:

If your application is making use of MPI or is executing simultaneous distinct processes, you can request <N> CPUs with
```
#SBATCH --ntasks=<N>
```
If you are using a multithreaded application, then you can request <N> CPUs with:
```
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=<N>
```

Nodes

Single vs. Multi-Node Programs

In order for your job to make use of more than one node, it must be able to make use of something like MPI.

If your application is not MPI-enabled, always set --nodes=1

The term node refers to the number of physical computers allocated to your job. The syntax to allocate <N> nodes to a job is:

#SBATCH --nodes=<N>

Time

The syntax for requesting time for your job is HHH:MM:SS or DD-HHH:MM:SS. The maximum amount of time that can be requested is 10 days for a batch job. More details in Job Limits.

#SBATCH --time=HHH:MM:SS

Memory and High Memory Nodes

Memory and CPUs are connected

More detailed information on memory and CPU requests can be found on our CPUs and Memory page.

Include Units

If you exclude gb from your memory request, Slurm will default to mb.

Memory is an optional flag. By default, the scheduler will allocate you the standard CPU/memory ratio available on the cluster.

Memory can either be requested with the --mem or --mem-per-cpu flags. The --mem flag indicates the amount of Memory per node to allocate to your job. If you are running multi-node MPI jobs with this flag, the total amount of memory you will receive will be mem\(\times\)nodes

The general syntax for requesting <N> GB of memory per node is

#SBATCH --mem=<N>gb

or, to request <N> GB of memory per CPU:

#SBATCH --mem-per-cpu=<N>gb

High Memory Node Requests

To request a high memory node, you will need the additional flag --constraint=hi_mem. It is recommended to use the exact directives below to avoid unexpected behavior.

Cluster	Command
Ocelote	`#SBATCH --mem-per-cpu=41gb #SBATCH --constraint=hi_mem`
Puma	`#SBATCH --mem-per-cpu=32gb #SBATCH --constraint=hi_mem`

GPUs

GPU partitions must be used

GPU jobs will need to use GPU-specific partitions. See the partitions section at the top of this page for details.

GPU options are per node

When using --gres=gpu:N, keep in mind that the total number of GPUs the job is allocated is N per node.

GPUs are an optional resource that may be requested with the --gres directive. For an overview of the specific GPU resources available on each cluster, see our resources page.

Cluster	Directive	Target
Puma	`#SBATCH --gres=gpu:1`	Request a single GPU. This will either target one Volta GPU (v100) or one A100 MIG slice, depending on availability. Only one GPU should be selected with this method to avoid being allocated multiple MIG slices.
	`#SBATCH --gres=gpu:nvidia_a100_80gb_pcie_2g.20gb`	Target one A100 MIG slice.
	`#SBATCH --gres=gpu:volta:N`	Request `N` V100 GPUs where 1≤`N`≤4
Ocelote	`#SBATCH --gres=gpu:N #SBATCH --mem-per-cpu=8gb`	Request `N` GPUs, where 1≤`N`≤2. This will target either one or two Pascals (p100s)

Job Arrays

Array jobs in Slurm allow users to submit multiple similar tasks as a single job. Each task within the array can have its own unique input parameters, making it ideal for running batch jobs with varied inputs or executing repetitive tasks efficiently. The flag for submitting array jobs is:

#SBATCH --array=<N>-<M>

where <N> and <M> are integers.

For detailed information on job arrays, see our job array tutorial.

Job Dependencies

Slurm job dependencies allow users to submit to a series of jobs that depend on each other using the flag and options:

--dependency=<type:jobid[:jobid][,type:jobid[:jobid]]>

For example, say job B depends on the successful completion of job A. Job B can be submitted as a dependency of job A using the following method:

[netid@junonia ~]$  sbatch A.slurm
Submitted batch job 1939000
[netid@junonia ~]$ sbatch --afterok:1939000 B.slurm

This tells the scheduler to hold job B until job A completes. The afterok is the dependency <type>, in this case it ensures that job B runs only if job A completes successfully. The different options for <type> are show below:

Dependency Type	Meaning
`after`	Job can begin after the specified job(s) have started
`afterany`	Job can begin after the specified job(s) have terminated. Job(s) will start regardless of whether the specified jobs failed or ran successfully
`afterok`	Job can begin after the specified job(s) have completed successfully. If the specified job(s) fail, the dependency will never run.
`afternotok`	Job can begin after the specified job(s) have failed. If the specified job(s) complete successfully, the dependency will never run.

Output Filenames

The default output filename for a slurm job is slurm-<jobid>.out. If desired, this can be customized using the directives

#SBATCH -o output_filename.out
#SBATCH -e output_filename.err

Filenames take patterns that allow for job information substitution. A list of filename patterns is shown below.

Variable	Meaning	Example Slurm Directive(s)	Sample Output
`%A`	A job array's main job ID	`#SBATCH --array=1-2 #SBATCH -o %A.out #SBATCH --open-mode=append`	`12345.out`
`%a`	A job array's index number	`#SBATCH --array=1-2 #SBATCH -o %A_%a.out`	`12345_1.out` `12345_2.out`
`%J`	Job ID plus stepid	`#SBATCH -o %J.out`	`12345.out`
`%j`	Job ID	`#SBATCH -o %j.out`	`12345.out`
`%N`	Hostname of the first compute node allocated to the job	`#SBATCH -o %N.out`	`r1u11n1.out`
`%u`	Username	`#SBATCH -o %u.out`	`netid.out`
`%x`	Job name	`#SBATCH --job-name=JobName #SBATCH -o %x.out`	`JobName.out`

Additional Directives

Command	Purpose
`#SBATCH --job-name=JobName`	Optional: Specify a name for your job. This will not automatically affect the output filename.
`#SBATCH -e output_filename.err #SBATCH -o output_filename.out`	Optional: Specify output filename(s). If `-e` is missing, stdout and stderr will be combined.
`#SBATCH --open-mode=append`	Optional: Append your job's output to the specified output filename(s).
`#SBATCH --mail-type=BEGIN\|END\|FAIL\|ALL`	Optional: Request email notifications. Beware of mail bombing yourself.
`#SBATCH --mail-user=email@address.xyz`	Optional: Specify email address. If this is missing, notifications will go to your UArizona email address by default.
`#SBATCH --export=VAR`	Optional: Export a comma-delimited list of environment variables to a job.
`#SBATCH --export=all`	Optional: Export your working environment to your job. This is the default.
`#SBATCH --export=none`	Optional: Do not export working environment to your job.

Examples and Explanations

The below examples are complete sections of Slurm directives that will produce valid requests. Other directives can be added (like output files), but they are not strictly necessary to submit a valid request. For simplicity, the Puma cluster is assumed when discussing memory and GPU resources. Note that these examples do not include the shebang #!bin/bash statement, which should be at the top of every Slurm script. Also, note that the order of directives does not matter.

Single CPUSingle NodeSingle GPU NodeMulti-NodeHigh-Memory Node

#SBATCH --job-name=hello_world
#SBATCH --account=your_group
#SBATCH --partition=standard
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --time=01:00:00

This example requests one CPU on one node for one hour. Easy!

#SBATCH --job-name=hello_world
#SBATCH --account=your_group
#SBATCH --partition=standard
#SBATCH --nodes=1
#SBATCH --ntasks=10
#SBATCH --time=01:00:00

10 CPUs are now requested. The default value of mem-per-cpu is assumed, therefore giving this job 50 GB of total memory. Specifying this value by including #SBATCH --mem-per-cpu=5gb will not change the behavior of the above request.

The example below will produce an equivalent request as above:

#SBATCH --job-name=hello_world
#SBATCH --account=your_group
#SBATCH --partition=standard
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --mem=50gb
#SBATCH --time=01:00:00

On Puma, up to 94 CPUs or 470 GB of memory can be requested.

NEW! July 31, 2024 Partitions update

Beginning July 31, GPU jobs must use a GPU partition. See the partitions section at the top of this page for details.

#SBATCH --job-name=hello_world
#SBATCH --account=your_group
#SBATCH --partition=gpu_standard
#SBATCH --nodes=1
#SBATCH --ntasks=10
#SBATCH --time=01:00:00
#SBATCH --gres=gpu:1

Note the gres=gpu:1 option and gpu_standard partition.

When requesting a multi-node job, up to 94 --ntasks-per-node can be requested on Puma. The numbers below are chosen for illustrative purposes and can be replaced with your choice, up to system limitations. It should be noted that there is no advantage to requesting multiple nodes when the total number of CPUs needed is less than or equal to the number of CPUs on one node.

#SBATCH --job-name=Multi-Node-MPI-Job
#SBATCH --account=your_group
#SBATCH --partition=standard
#SBATCH --ntasks=30
#SBATCH --nodes=3
#SBATCH --ntasks-per-node=10
#SBATCH --time=01:00:00

When requesting a high memory node, include both the --mem-per-cpu and --constraint directives.

#SBATCH --job-name=High-Mem-Job
#SBATCH --account=your_group
#SBATCH --partition=standard
#SBATCH --nodes=1
#SBATCH --ntasks=94
#SBATCH --mem-per-cpu=32gb
#SBATCH --constraint=hi_mem
#SBATCH --time=01:00:00

Groups and users are subject to limitations on resource usage. For more information, see job limits. ↩