Configurations

Partitions

The partition is a computational resource organizing the nodes together in the same logical entity defined by one or more given specifications (whether physical or linked to resources).

To get a quick overview of the different partitions, you may use the sinfo command:

% sinfo
PARTITION       AVAIL  TIMELIMIT  NODES  STATE NODELIST
htc*               up   infinite      1   drng ccwslurm0130
htc*               up   infinite    139    mix ccwslurm[...]
htc*               up   infinite     50  alloc ccwslurm[...]
htc_interactive    up   infinite      1    mix ccwislurm0001
htc_interactive    up   infinite      1   idle ccwislurm0002
htc_highmem        up   infinite      1    mix ccwmslurm0001
gpu                up   infinite      6    mix ccwgslurm[0002,0100-0104]
gpu_interactive    up   infinite      2    mix ccwgislurm[0001,0100]
hpc                up   infinite      2  alloc ccwpslurm[0001-0002]
flash              up   infinite      1    mix ccwslurm0001
htc_daemon         up   infinite      1    mix ccwslurm0001

There are three major distinct partitions: htc, hpc and gpu, as well as their equivalents for interactive jobs: htc_interactive, hpc_interactive and gpu_interactive. Each of these partitions corresponds to one of the three computing platforms described on the page concerning the computing platform.

The flash partition, dedicates a whole worker to tests and debug. This partition is limited to one hour by its qos.

The htc_highmem partition is dedicated to jobs that need huge memory and allows a higer memory/CPU ratio.

The htc_daemon partition allows for jobs without time limit, but limited to one core and 1 GB of memory. This partition is limited to 10 jobs per user by its qos.

Note

In a simple way, single-core and multi-core jobs will be executed in the htc partition, parallel jobs using the InfiniBand in the hpc partition, and access to GPUs will be given through the gpu partition.

The sinfo command also indicates the job execution time restriction, the computation servers belonging to each of these partitions and their states.

The main options of the sinfo command are:

-a
displays all computing servers
-d
displays all off line computing servers
-l
displays the output in a long format
-p <partition>
displays the information for a specific partition

To display and read a partition detailed configuration you may use scontrol:

% scontrol show partition
PartitionName=htc
   AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL
   AllocNodes=ALL Default=YES QoS=N/A
   DefaultTime=NONE DisableRootJobs=YES ExclusiveUser=NO GraceTime=0 Hidden=NO
   MaxNodes=1 MaxTime=UNLIMITED MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED
   Nodes=ccwslurm[0001-0002,0005-0023]
   PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO
   OverTimeLimit=NONE PreemptMode=OFF
   State=UP TotalCPUs=1344 TotalNodes=21 SelectTypeParameters=NONE
   JobDefaults=(null)
   DefMemPerCPU=3000 MaxMemPerCPU=3100

   [...]
The command gives the main characteristics of the partitions:
  • authorized groups and accounts,
  • the associated qualities of service (see below),
  • the available resources in the partition,
  • the limits in terms of partition resources.

There is a distinction between the notion of groups and accounts. The first is the group as defined on the page Definition of laboratory and group. It therefore corresponds to an experience or a collaboration in which the user participates, and is the Unix group associated with the user (main group). The account corresponds to the entity that will be charged for the resources the job will use. It will therefore be a group the user participates to, but it may be different from the Unix group to which the user is currently associated.

To display the account a user is attached to:

% sacctmgr show user my_username

To display all the accounts a user is attached to:

% sacctmgr show user my_username withassoc

my_username being the user id.

Generally, the sacctmgr command allows to display and modify all the information related to the accounts.

For more details on the command, please refer to the help sacctmgr -h.

Note

In practice, when submitting a job, we can specify the partition and the account to use with the options --partition and --account respectively. Without any specification, Slurm will opt for the default partition, i.e. htc, and the user’s main account (see above).

Quality of service

Quality of service, or QoS, are a set of limits. The command used below scontrol show partitions displays the QoS authorized on a partition.

In order to list the available QoS, you may use the command sacctmgr:

% sacctmgr show qos format=Name,Priority,MaxWall,MaxSubmitPU
    Name   Priority     MaxWall MaxSubmitPU
  ---------- ---------- ----------- -----------
    normal          0  7-00:00:00        3000
     flash          0    01:00:00          10
       gpu          0  7-00:00:00         100
    daemon          0                      10

Here, we have restricted the output to only the name, priority, execution time and maximum limit of submitted jobs per user fields using the format option.

The normal QoS is applied by default to all jobs on. It therefore limits the execution time to a maximum of 7 days. The gpu QoS has the same time limitation but is limited to 100 submitted job at a time. The flash QoS limits the execution time to 1 hour. It’s automatically applied when flash partition is set. It should be preferred to carry out short tests needing a rapid result. The number of simultaneous jobs using this QoS is however limited to 10 per user. The daemon QoS is used with the htc_daemon partition and is useful to execute light processes that need to run during a long time. It is also limited to 10 jobs per user. To summarize:

normal
is used with htc, htc_interactive and hpc partitions.
gpu
is used with gpu and gpu_interactive partitions.
flash
is used only with flash partition.
daemon
is used only with htc_daemon partition.

Note

As a result, one doesn’t need to set QoS in its jobs: simply set a partition.