Parallel jobs

Jobs called “parallel” are jobs running concurrent tasks distributed by UGE on different computing servers. These tasks are synchronized using an MPI interface. To submit such a job, you must first compile your code with an MPI library. The OpenMPI and MPICH libraries are supported by UGE and provided by CC-IN2P3 (see the page Software licenses).

Before submitting such a job it is necessary to create a file .mpd.conf with rights 600 (-rw-------) in your HOME containing a password:

% touch $HOME/.mpd.conf
% echo "Secretword = XXXX"> $HOME/.mpd.conf
% chmod 600 $HOME/.mpd.conf

The job is then submitted by specifying the openmpi or mpich2 parallel environment and the number of cores used:

% qsub -pe <openmpi|mpich2> <number_of_cores> -q <QueueName> <script>

Note

A parallel job can request an arbitrary number of cores, without exceeding 256.

The queues available for parallel jobs are the following:

  • pa_medium
  • pa_long
  • pa_longlasting

To know the queues limits please refer to the page Information on scheduling queues.

Attention

Access to all queues dedicated to parallel computing is restricted. You must contact your computing czar to access this type of resource. Restricted queues FAQ.

The output of a parallel job is written directly to its final destination during the execution of the job (not locally on the calculation server and then transferred to its final destination at the end of the job).

OpenMPI job

The script must load the desired OpenMPI environment, for example version 2.1.1 provided by the Computing Center

% source /pbs/software/centos-7-x86_64/openmpi/ccenv.sh 2.1.1

Then use mpiexec to execute the tasks corresponding to the number of cores requested:

% mpiexec -n $NSLOTS myprogram

The script can then be executed, for example:

% qsub -pe openmpi 64 -q pa_long test.sh

Note

The MPI tasks can be executed on several computing servers (up to one per server). In order to handle this disribution, the parallel environment openmpi_4, openmpi_8 and openmpi_16 force the scheduler to execute respectively 4, 8 and 16 tasks per server.

MPICH job

The script must load the desired MPICH environment, for example version 3.2 provided by the Computing Center

% source /pbs/software/centos-7-x86_64/mpich2/ccenv.sh 3.2

Then use mpiexec to execute the tasks corresponding to the number of cores requested:

% mpiexec -iface ib0 -np $NSLOTS myprogram

The script can then be executed, for example:

% qsub -pe mpich2 64 -q pa_long test.sh