HPC User Manual

The Qlustar shell setup supports tcsh and bash. There are global initialization files, that affect the setup of both shell types. Hence, the administrator has to modify only a single file to set environment variables, aliases and path variables for both shells and all users. These global files are:

Use the sbatch command to submit a batch script.

Important sbatch flags:

For submitting parallel jobs, a few rules have to be understood and followed. In general they depend on the type of parallelization and the architecture.

Allocating GPUs is very much cluster-specific. Depending on your local Slurm setup you might have dedicated GPU partitions, possibly coupled to QOS definitions, or you might have to specify GPUs directly using an option like --gres=gpu:<gpu_name>:<# of GPUs>. Ask your local administrator how to optimally submit GPU jobs on your cluster. Examples:

Interactive tasks like editing, compiling etc. are normally limited to the FrontEnd (login) nodes. For longer interactive sessions, one can also allocate resources on compute nodes with the commands salloc or directly srun. To specify the required resources, they mostly take the same options as sbatch.

salloc is special in that it returns a new shell on the node, where you submitted the job. You then need to use the command srun in front of the following commands to have them executed on the allocated resources. Please be aware, that if you request more than one task, srun will run the command once for each allocated task.

An example of an interactive session using srun looks as follows:

Often, partitions have walltime limits. Use the command sinfo --summarize to see the most important properties of the partitions in your cluster.

If no walltime is requested on the command line, the walltime set for the job will be the default walltime of the partition. To request a specific walltime, use the --time option to sbatch. For example:

will submit a job to the default partition, and request a walltime of 36 hours. If the job runs over 36 hours, it will be killed by the batch system.

To check the walltime limits and current runtimes for jobs, you can use the squeue command.

For many more squeue options, see the squeue man page.

The scancel command is used to delete jobs. Examples:

Common job states:

Use the sacct command to check on the states of completed jobs.

Show all your jobs in any state since midnight:

Show all jobs that failed since midnight:

Show all jobs that failed since March 1st 2016:

The completion status of a job is essentially the exit status of the job script with all the complications that entails. For example consider the following job script:

This script tries to run my_buggy_program which fails with an exit code 1. However, bash by default keeps executing, even if a command fails, so the script will eventually print 'DONE'. Since the exit status of a bash script is the exit status of the last command in the script, and echo returns 0 (SUCCESS), the script as a whole will exit with an exit code of 0. This signals sucess and the job state will show COMPLETED, since Slurm uses the script exit code to judge if a job completed sucessfully.

Similarly, if a command in the middle of the job script were killed for exceeding memory, the rest of the job script would still be executed and could potentially return an exit code of 0 (SUCCESS), resulting again in a state of COMPLETED.

Some more careful bash programming techniques can help ensure, that a job script will show a final state of FAILED if anything goes wrong.

Use set -e

Starting a bash script with set -e will tell bash to stop executing a script if a command fails. This will then signal failure with a non-zero exit code and will be reflected as a FAILED state in Slurm.

One complication with this approach is, that some commands will return non-zero exit codes. For example grepping for a string that does not exist.

Check errors for individual commands

A more selective and superior approach involves carefully checking the exit codes of the important parts of a job script. This can be done with conventional if/else statements or with conditional short circuit evaluation often seen in scripts. For example:

Array jobs can be used to create a sequence of jobs which will be submitted, controlled, and monitored as a single unit. This job sequence shares the same executable and resource requirements, but individual executions have different input files. The arguments -a or --array take an additional parameter that specify the array indices. Within an array job, you can read the environment variables SLURM_ARRAY_JOB_ID, which will be set to the first job ID of the array, and SLURM_ARRAY_TASK_ID, which will be set individually for each step.

Within an array job, you can use %a and %A in addition to %j and %N (described above) to make the output file name specific to the job. %A will be replaced by the value of SLURM_ARRAY_JOB_ID and %a will be replaced by the value of SLURM_ARRAY_TASK_ID.

Here is an example how an array job may look like:

For further details, please read the corresponding part of the general Slurm documentation.

You can use chain jobs to create dependencies between jobs. This is often the case, if a job relies on the result of one or more preceding jobs. Chain jobs can also be used, if the runtime limit of the batch queues is not sufficient for your job. Slurm has an option -d or --dependency that allows to specify that a job is only allowed to start if another job finished.

Here is an example of a chain job. It submits 4 jobs (described in a job file) that will be executed one after each other with different CPU numbers:

squeue will report all jobs on the cluster. squeue -u $USER will report your running jobs. Slurm commands like squeue are very flexible, so that you can easily create your own aliases.

Example of squeue:

Using the --mail-type=<type> option to sbatch, users can request email notifications from Slurm as certain events occur. Multiple event types can be specified as a comma separated list. For example:

Available event types:

After a job is submitted, some of the submission parameters can be modified using the scontrol command. Examples:

Change the job dependency:

Request a matlab license:

Job was submitted to the default partition, resend to gpu partition:

Reduce the walltime to 12 hrs:

Slurm provides several binding strategies to place and bind the tasks and/or threads of your job to cores, sockets and nodes.

The default allocation of tasks/threads for OpenMP, MPI and Hybrid (MPI and OpenMP) applications are as follows:

OpenMP

The illustration shows the default binding of a pure OpenMP job on 1 node with 8 CPUs on which 8 threads are allocated.

MPI

The illustration shows the default binding of a pure MPI job. 16 global ranks are distributed onto 2 nodes with 8 cores each. Each rank has 1 core assigned to it.

Hybrid (MPI and OpenMP)

In the illustration the default binding of a hybrid job is shown. Four global MPI ranks are distributed onto 2 nodes with 8 cores each. Each rank has 4 cores assigned to it.

The Slurm command sacct provides job statistics like memory usage, CPU time, energy usage etc. Examples:

Read the manpage (man sacct) for information on the provided fields.

Running on specific hosts

If you want to place your job onto specific nodes, there are two options for doing this. Either use -p to specify a partition with a host group that fits your needs. Or, use the -w (alternatively --nodelist) parameter with a single or list of nodes, where you want your job to be executed.

Useful external Slurm documentation links

Spack is managed by the spack command. Help is available by executing

and for sub-commands e.g.

Cluster users running the default Qlustar shell setup have an automatic Spack shell initialization. Others should source the correct initialization file for their shell from the variants at /usr/share/spack/root/share/spack. E.g. bash users would add a line

to their .bashrc. Once Spack shell initialization has been done, full completion support for Spack commands and packages is available.

To show all Spack packages installed on the cluster execute

As a more specific example, the following command searches for all OpenMPI versions/variants on the cluster:

Let’s have a closer look at the output of the command: It shows that in this installation two compiler variants (gcc@12.2.0 and oneapi@2022.2.1) of OpenMPI exist, each with three processor specific versions (haswell, zen2 and zen3) optimized for the different platforms. All versions are built for the OS ubuntu22.04 (aka Qlustar 13/jammy).

The Qlustar shell setup defines the following environment variables for convenient usage of Spack:

SPACK_OS
SPACK_TARGET
export SPACK_COLOR=always

If you intend to compile architecture specific binaries of your own programs, you can use these variables to setup a directory structure in your home directory of the form

By setting an additional variable VERSION, you can then create your executable and systematically relocate it with a few lines in your makefile like e.g.

In your job script (assume bash), you can then do the following

With this setup, you make sure that your jobs will always use the correct optimized binary on the nodes assigned by slurm.

In case users find a need to use their own private Spack installation, they can use the chaining mechanism to extend it with the official Qlustar Spack instance of the cluster. This allows to use all packages installed there together with the ones they install in their own instance. The chaining is accomplished by adding the following lines to the upstreams.yaml file of the private instance: