Configurations

Attention

The outputs below are displayed as an example of format and not for their content: the latter may change according to the computing platform maintenance modifications.

The user is invited to run the commands himself on the interactive servers to obtain up-to-date information.

User information

There is a distinction between the notion of group and account. The first is the Unix group corresponding to the collaboration the user is member of. It therefore corresponds to an experience or a collaboration in which the user participates. The account corresponds to the entity that will be charged for the resources the job will use.

To display all the accounts a user is attached to and the QoS the accounts are allowed to:

% sacctmgr show user withassoc <username> format=Account,QOS%30

<username> being the user id.

Note

Generally, the sacctmgr command allows to display and modify all the information related to the accounts. For more details on the command, please refer to the help sacctmgr -h.

By default the active account will be set on the user’s main group. For confirmation, or to switch from one default account to another, please refer to the syntax suggested in Account management to change temporarly the main group. To submit on a different account without modyifing the main group, use the -A | --account= option.

Partitions

The partition is a computational resource organizing the nodes together in the same logical entity defined by one or more given specifications (whether physical or linked to resources).

To get a quick overview of the different partitions, you may use the sinfo command:

% sinfo
PARTITION       AVAIL  TIMELIMIT  NODES  STATE NODELIST
htc*               up   infinite      1   drng ccwslurm0130
htc*               up   infinite    139    mix ccwslurm[...]
htc*               up   infinite     50  alloc ccwslurm[...]
htc_interactive    up   infinite      1    mix ccwislurm0001
htc_interactive    up   infinite      1   idle ccwislurm0002
htc_highmem        up   infinite      1    mix ccwmslurm0001
gpu                up   infinite      6    mix ccwgslurm[0002,0100-0104]
gpu_interactive    up   infinite      2    mix ccwgislurm[0001,0100]
hpc                up   infinite      2  alloc ccwpslurm[0001-0002]
flash              up   infinite      1    mix ccwslurm0001
htc_daemon         up   infinite      1    mix ccwslurm0001
dask               up   infinite    139    mix ccwslurm[...]

There are three major distinct partitions: htc, hpc and gpu, as well as their equivalents for interactive jobs: htc_interactive, hpc_interactive and gpu_interactive. Each of these partitions corresponds to one of the three computing platforms described on the page concerning the computing platform.

The flash partition, dedicates a whole node to job testing and debug. This partition is limited to 1 hour by its qos.

The htc_highmem partition is dedicated to jobs that need huge memory and is allowed a higher memory limit per jobs.

The htc_daemon partition generally allows you to run monitoring or orchestrating jobs: very long, but limited in resources. This partition is limited by its qos to 10 jobs per user.

The dask partition is dedicated to the Dask functionality on the Jupyter Notebook Platform. This partition shares the same compute servers as htc.

Note

In a simple way, single-core and multi-core jobs will be executed in the htc partition, parallel jobs using InfiniBand in the hpc partition, and access to the GPUs will be done through the gpu partition. Access to this last partition is restricted and depends on the resources request made by your computing group. Please contact user support for any additional information.

Details on submission resource limitations are described in the Required parameter limits paragraph.

The sinfo command also indicates the job execution time restriction, the computation servers belonging to each of these partitions and their states.

The main options of the sinfo command are:

-a

displays all computing servers

-d

displays all off line computing servers

-l

displays the output in a long format

-p <partition>

displays the information for a specific partition

-O "<output fields>"

displays in the output the mentioned fields. For the list of fields, please run the command man sinfo

To display and read a partition detailed configuration you may use scontrol:

% scontrol show partition
PartitionName=htc
   AllowGroups=ALL AllowAccounts=ALL AllowQos=normal
   AllocNodes=ALL Default=YES QoS=N/A
   DefaultTime=NONE DisableRootJobs=YES ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO
   MaxNodes=1 MaxTime=UNLIMITED MinNodes=0 LLN=YES MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED
   NodeSets=htc
   Nodes=ccwslurm[0002-0143,0168-0215,0312-0367]
   PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO
   OverTimeLimit=NONE PreemptMode=OFF
   State=UP TotalCPUs=18432 TotalNodes=246 SelectTypeParameters=NONE
   JobDefaults=(null)
   DefMemPerCPU=1024 MaxMemPerNode=UNLIMITED
   TRES=cpu=18432,mem=80590000M,node=246,billing=18432

 [...]
The command gives the main characteristics of the partitions:
  • authorized groups and accounts,

  • the associated qualities of service (see below),

  • the available resources in the partition,

  • the limits in terms of partition resources.

Note

In practice, when submitting a job, we can specify the partition and the account to use with the options --partition and --account respectively. Without any specification, Slurm will opt for the default partition, i.e. htc, and the user’s main account.

Nodes

Nodes are the physical machines hosting the computing resources such as CPU and memory. To obtain detailed information of each node on the computing platform, use the command below (example with the node ccwslurm0003; without this specification, the command gives the following amount of information for each node on the platform):

% scontrol show node ccwslurm0002
NodeName=ccwslurm0002 Arch=x86_64 CoresPerSocket=1
   CPUAlloc=25 CPUEfctv=64 CPUTot=64 CPULoad=24.12
   AvailableFeatures=htc,x86_64,el9
   ActiveFeatures=htc,x86_64,el9
   Gres=(null)
   NodeAddr=ccwslurm0002 NodeHostName=ccwslurm0002 Version=24.05.1
   OS=Linux 5.14.0-427.31.1.el9_4.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Aug 9 14:06:03 EDT 2024
   RealMemory=265000 AllocMem=80896 FreeMem=159929 Sockets=64 Boards=1
   MemSpecLimit=6000
   State=MIXED ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A
   Partitions=htc,dask
   BootTime=2024-08-26T17:07:57 SlurmdStartTime=2024-08-26T17:08:11
   LastBusyTime=2024-09-02T12:09:59 ResumeAfterTime=None
   CfgTRES=cpu=64,mem=265000M,billing=64
   AllocTRES=cpu=25,mem=79G
   CurrentWatts=0 AveWatts=0

Attention

The fields CPUTot and RealMemory give the node’s total CPU and memory hardware limits respectively. Make sure you do not exceed these limits when submitting your jobs. As a rule of thumb, if a job demands more than 200G on memory, it should be submitted on the htc_highmem partition.

For an overview of all the resources available on the computing platform, please refer to the Information on computing platform resources page.

Quality of service

The quality of service (QoS), is a rule associated with a partition or a job that allows it to be altered. It can for example modify the priority of a job, or limit the allocated resources. The command scontrol showed in the partitions paragraph allows also to view the QoS implemented on a given partition.

In order to list the available QoS, you may use the command sacctmgr:

% sacctmgr show qos format=Name,Priority,MaxWall,MaxSubmitPU
      Name   Priority     MaxWall MaxSubmitPU
---------- ---------- ----------- -----------
    normal          0  7-00:00:00        5000
     flash          0    01:00:00          10
       gpu          0  7-00:00:00         100
    daemon          0 90-00:00:00          10
      dask       1000  2-00:00:00

Here, we have restricted the output to only the name, priority, execution time and maximum limit of submitted jobs per user fields using the format option.

The normal QoS is applied by default to all jobs on. It therefore limits the execution time to a maximum of 7 days. The gpu QoS has the same time limitation but is limited to 100 submitted job at a time. The flash QoS limits the execution time to 1 hour for a number of simultaneous jobs limited to 10 per user. The daemon QoS is used with the htc_daemon partition and is useful to execute light processes that need to run during a long time. It is also limited to 10 jobs per user. To summarize:

normal

is used with htc, htc_interactive and hpc partitions.

gpu

is used with gpu and gpu_interactive partitions.

flash

is used only with flash partition.

daemon

is used only with htc_daemon partition.

Note

As a result, upon submission simply set the partition and the QoS will be automatically set.