Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
en:ge_submit_a_job_qsub [2018/01/17 15:14]
Quentin LE BOULC'H
en:ge_submit_a_job_qsub [2018/01/17 15:15] (current)
Quentin LE BOULC'H
Line 1: Line 1:
- +**This page is deprecated, please see [[utiliser_le_systeme_batch_ge_depuis_le_centre_de_calcul|Using the GE batch system at the computing ​centre]]**
-FIXME **This page is deprecated, please see [[utiliser_le_systeme_batch_ge_depuis_le_centre_de_calcul|Utiliser le système batch GE depuis le centre de calcul]]**\\ +
- +
- +
-====== GE: submit a job (qsub) ====== +
- +
-\\ +
-\\ +
-\\ +
-\\ +
-**WARNING** : After the upgrade of GE (8.1.6 to 8.2.1) planned for the 29/04/2015 you will notice some minor differences in **qstat** and **qacct** commands described below . You will also notice the old accounting file is no longer compatible with this new version of GE . +
-**qstat** command output : job_number is now written on 10 digits instead of 7 digits and submission_time format has changed . For example 02/20/2015 15:37:36 instead of Fri Feb 20 15:37:25 2015 . +
-<​code>​ +
-> qstat -r -s r -u santa_claus +
-job-ID ​    ​prior ​  ​name ​      ​user ​        state submit/​start at     ​queue ​                         jclass ​                        slots ja-task-ID  +
------------------------------------------------------------------------------------------------------------------------------------------------- +
-1234567890 0.00050 myjob   ​santa_claus ​      ​r ​    ​04/​29/​2015 11:​16:​04 ​ long@ccwsge0155.in2p3.fr ​                                  1 1 +
-</​code>​ +
-**qacct** command output : 6 additional fields " //cwd// ", " //​submit_host//​ ", " //​submit_cmd//​ ", " //​deleted_by//​ ", " //maxrss// " and " //maxpss// " . +
-<​code>​ +
-cwd          NONE +
-submit_host ​ cca005.in2p3.fr +
-submit_cmd ​  ​qsub +
-deleted_by ​  ​NONE +
-maxrss ​      ​0.000 +
-maxpss ​      ​0.000 +
-</​code>​ +
-**Accounting files** : here are paths of the old and new accounting files . +
-<​code>​ +
-Old version (8.1.6) : /​afs/​in2p3.fr/​common/​uge/​prod/​common/​accounting* +
-New version (8.2.1) : /​afs/​in2p3.fr/​common/​uge/​v8.2.1/​common/​accounting +
-</​code>​ +
-=====  Introduction ​ ===== +
- +
-**What is a job?**\\ +
-\\ +
-In the general sense, a job stands for an action the user wishes to perform on a computing machine of the Computing Centre. It could be an executable file, a set of commands, a script,... This job, which may consist of several tasks, will be preliminary developed and tested on the interactive access machines of Computing Centre before being submitted massively on the ones of the computing farm.\\ +
-\\ +
-**Job scheduling**\\ +
-\\ +
-The submission of the jobs is made using a job scheduler. The scheduler is the only entry point common to all users to submit jobs on the farm. The role of this scheduler is to receive the jobs submitted by users, to schedule and to submit them for execution on an appropriate and available computing machine (worker). The mutualisation of all computing resources (memory, disk space, CPU) for all users allows an optimal use of computing machines and the whole farm.\\ +
-\\ +
-For job scheduling, the Computing Centre has adopted an open source ​batch-queuing ​system ​//Sun Grid Engine// (SGE), later on known as Oracle Grid Engine after its acquisition of Sun by Oracle in 2010. To have more information,​ please visit the [[http://​gridengine.sunsource.net/​|Grid Engine]] project home pages.\\ +
-\\ +
-The submission of jobs is made by using the command-line interface.\\ +
-\\ +
-**Types of jobs**\\ +
-\\ +
-The Grid Engine system recognizes the following four basic classes of jobs: +
- ​**Batch Jobs** – Single segments of work. Typically, a batch job is executed once. +
- ​**Array Jobs** – Groups of similar work segments that can all be run in parallel but are completely independent of one another. All of the workload segments of an array job, known as tasks, are identical except for the data sets on which they operate. +
- ​**Multicores jobs** – Jobs using several cores on a single machine +
- ​**Parallel Jobs** – Jobs composed of cooperating tasks that must all be executed ​at the same time, often with requirements about how the tasks are distributed across the resources. Note that not just any program can run in parallel, it must be programmed as such and compiled against a particular MPI library. +
- ​**Interactive Jobs** – Jobs that provide the submitting user with an interactive login to available resources in the compute cluster. You can an interactive job when you are building and testing your scripts. But note that this is not the place to run long, computationally intensive or other jobs better suited to run in a batch mode. You can launch an interactive job with the command **//​qlogin//​** , develop your script and then write the job file. When it comes to running the job itself, submit it as a batch job. +
- +
-=====  Getting started ​ ===== +
- +
- +
-====  Grid Engine Environment ​ ==== +
- +
-The SGE environnement is defined automatically when you log in to an interactive machine cca.in2p3.fr (if you use CentOS 7 instead of SL6 connect to cca7.in2p3.fr). But in case you don't use our default environnement,​ you have to type the command ge_env after login or source /​usr/​local/​shared/​bin/​ge_env.{c,​}sh in a script in order to set the SGE environment.\\ +
-\\ +
-If your jobs need your shell profiles, you'll have to add at the beginning of the script (for bash) : +
-<​code>​ +
-#! /​usr/​local/​bin/​bash -l  +
-</​code>​ +
-or use : +
-<​code>​ +
-qsub -S "/​usr/​local/​bin/​bash -l"  +
-</​code>​ +
-- Environmental variables :\\ +
-BATCH_SYSTEM=GE\\ +
-ENVIRONMENT = ACCESS for the cca host machines.\\ +
-ENVIRONMENT = SEQUENTIAL_BATCH | PARALLEL_BATCH | INTERACTIVE_BATCH\\ +
-for submitted the jobs  +
-====  Submit a basic job (sequential monocore) ​ ==== +
- +
-To submit a batch job to GE, the command to use on an interactive access machine of Computing Centre is: +
-<​code>​ +
-qsub -P P_myproject [ options ] [ script | -- [ script args ]]  +
-</​code>​ +
-myproject is typically the name of your group. It can be a subproject in your group if a subproject has been created. It is mandatory to declare your project. This is typically the type of option that can be put in the file .sge_request in your $HOME directory or in the submission directory (which supersedes $HOME/​.sge_request). Type 'man sge_request'​ for more details.\\ +
-\\ +
-If you belong to several groups, use if necessary the "​newgroup"​ command in order to switch to the group corresponding to the project. See : https://​doc.cc.in2p3.fr/​en:​afs_changer_de_groupe \\ +
-\\ +
-In most cases, jobs have to use computing ​resources shared with other groups. In order to regulate the use of these resources, there is a notion of resource in GE. The resources used by the jobs must be specified with the option " **-l** " of **qsub** command. They work as followed: +
-<​code>​ +
--l resource=value +
-</​code>​ +
-If no file name is given, **qsub** command put the cursor on the following line and waits manual commands from you. These commands, typed one by one, will constitute the job to be executed. Input commands are closed by Ctrl-D.\\ +
-\\ +
-A batch job consist typically of shell scripts with a set of informations concerning your job. You can specify **qsub** options inside your script. These options should be at the beginning of the script, before all other lines of your job. Each line containing **qsub** options should start with the prefix " **#$** ". You can change this prefix with option **-C prefix** .\\ +
-\\ +
-The **qsub** command has a lot of options. To get the full list of all options, use the command: +
-<​code>​ +
-qsub -help +
-</​code>​ +
-Below, you will find the most important options.  +
-===  Computing resources declaration (CPU, MEMORY, DISK)  === +
- +
-\\ +
-To know the CPU, MEMORY, DISK needed by your job, you can do short time tests on an interactive machine or short time submissions on Grid Engine. Then the information concerning the ressources used are in the end banner of the log file you get when the job is finished.\\ +
-\\ +
-The output of your job will contain this type of information :\\ +
-\\ +
-Header : +
-<​code>​ +
-*************************************************************** +
-*                  Grid Engine Batch System ​                 +
-*           IN2P3 Computing Centre, Villeurbanne FR          +
-*************************************************************** +
-* User:                    santa_klaus ​                             +
-* Group: ​                  ​christmas ​                          +
-* Jobname: ​                ​STDIN ​                            +
-* JobID: ​                  ​495379 ​                           +
-* Queue: ​                  ​long ​                       +
-* Worker: ​                 ccwsge0006.in2p3.fr ​              +
-* Operating system: ​       Linux 2.6.18-238.12cc.el5 ​        +
-* Project: ​                ​P_christmas ​                        +
-*************************************************************** +
-* Submitted on:            Thu May 26 17:45:15 2011          +
-* Started on:              Thu May 26 17:45:35 2011          +
-*************************************************************** +
-</​code>​ +
-Footer : +
-<​code>​ +
-*************************************************************** +
-* Ended on:                Wed Oct 05 15:48:05 2016          +
-* Exit status: ​            ​0 ​                                +
-* Consumed ​                                                  +
-*   cpu (HS06): ​           11:​34:​10 ​                         +
-*   cpu scaling factor: ​   11.350000 ​                        +
-*   cpu time:              3669 / 259200 ​                    +
-*   ​efficiency: ​           90 %                              +
-*   ​io: ​                   13.87236 ​                         +
-*   ​vmem: ​                 1.129G ​                           +
-*   ​maxvmem: ​              ​6.240G ​                           +
-*   ​maxrss: ​               3.473G ​                           +
-*************************************************************** +
- +
-</​code>​ +
-**- Declaration of required CPU**\\ +
-\\ +
-The execution time expressed will take into account the difference in term of performance between the different machines in the farm. For a better definition, you will probably need to run some tests before the submission of a lot jobs. If the time of job execution exceeds the duration specified, it will receive a signal XCPU (signal in the UNIX sense), then, one minute after, a KILL signal to end job execution. To specify the duration, use:\\ +
-\\ +
-To specify the //cputime// (= the time the job needs), for ex. to ask 1 hour and 40 minutes, use: +
-<​code>​ +
--l ct=6000 (it's cpu time in seconds) +
-</​code>​ +
-or +
-<​code>​ +
--l ct=01:​40:​00 +
-</​code>​ +
-**- Declaration of required MEMORY**\\ +
-\\ +
-To specify the memory: +
-<​code>​ +
--l s_rss=2G +
-</​code>​ +
-for a maximum need of 2 GBytes (M). +
-<​code>​ +
-Possible units for memory: G, M, K, B  +
-</​code>​ +
-**Important : You should specify the unit and s_rss should be at least 64M.**\\ +
-\\ +
-**- Declaration of required DISK**\\ +
-\\ +
-During job execution on a machine of the farm, a local disk space of the machine is allocated. You can write and read the data on that space. It is accessible to your task through the environment variable **${TMPDIR}** . This space is cleaned at the end of your task by GE. You will need to copy the data you want to keep in an appropriate storage space. All the machines of the farm have not the same available space. In order to optimize your job submission, it is important to specify your needs. For that, you must use: +
-<​code>​ +
--l fsize=4096M +
-</​code>​ +
-if you need a maximum of 4096 MBytes (ou 4 GBytes). This space (directory) corresponds also the working directory of the task during in its execution. +
-<​code>​ +
-Possible units for this scratch space: G, M, K, B  +
-</​code>​ +
-**Important : You should specify the unit and fsize should be at least 64M. No files you read or write, even in case of distant access, should exceed this limit. Otherwise, your job will be killed.** .\\ +
-\\ +
-\\ +
-**- QUEUE**\\ +
-\\ +
-When submitting a job you should not specify a queue if not needed.\\ +
-The batch system will choose the best suited queue associated with the\\ +
-required ressources (CPU time, Memory, Disk).\\ +
-\\ +
-It's needed only for demon jobs, multicore jobs and parallel jobs and can be specified with : +
-<​code>​ +
--q <​QueueName>​ +
-</​code>​ +
-For demon jobs you should specify in addition the ressource demon with : +
-<​code>​ +
--q demon -l demon=1 +
-</​code>​ +
-The "​long"​ queue can be used for common jobs.\\ +
-\\ +
-See [[http://​cctools.in2p3.fr/​mrtguser/​info_sge_generale.php|SGE configuration]] to know the current configuration.\\ +
-\\ +
-To know the characteristics of a queue : +
-<​code>​ +
-qconf -sq <​QueueName>​  +
-</​code>​ +
-Other queues like longlasting,​ huge and demon have restricted access and you'll have to contact us if you want to access them.\\ +
-\\ +
-To know the available queues, use: +
-<​code>​ +
-qconf -sql +
-</​code>​ +
- +
-===  Storage resources declaration (HPSS, SPS, XROOTD, IRODS, DCACHE,​ORACLE) ​ === +
- +
-You have to declare the ressources you use in your jobs with the syntax below.\\ +
-N.B.: There is no specific resource per group to declare. The limitations for each group\\ +
-are handled by Grid Engine configuration based on the project (-P option) you declare.\\ +
-\\ +
-\\ +
-**- Use of HPSS** +
-<​code>​ +
--l hpss=1 +
-</​code>​ +
-**- Use of SPS** +
-<​code>​ +
--l sps=1 +
-</​code>​ +
-**- Use of XROOTD** +
-<​code>​ +
--l xrootd=1 +
-</​code>​ +
-**- Use of IRODS** +
-<​code>​ +
--l irods=1 +
-</​code>​ +
-**- Use of DCACHE** +
-<​code>​ +
--l dcache=1 +
-</​code>​ +
-**- Use of ORACLE** +
-<​code>​ +
--l oracle=1 +
-</​code>​ +
- +
-===  Other useful options ​ === +
- +
-**Options for standard output handling //stderr// and //​stdout//​**\\ +
-\\ +
-When you submit the job from the command line, you will see an information line indicating the number of job: +
-<​code>​ +
-> qsub hello_world.sh +
-Your job 1260 ("​hello_world.sh"​) has been submitted  +
-</​code>​ +
-The given number (here 1260) is a unique number used by GE to identify your job.\\ +
-\\ +
-When you submit your batch job, two files are created par default: +
- < //​job_name//​ > **.o** < //job_id// > (default standard output), et +
- < //​job_name//​ > **.e** < //job_id// > (default standard error output) +
- +
-The name of your job (< //​job_name//​ >) is par default the name of the script file without the prefix, but you can change it with the option " **-N** ".\\ +
-Please don't use french accents or special characters (like quotes) in the job name.\\ +
-\\ +
-When you submit an array job, the files you get are: +
- < //​job_name//​ > **.o** < //job_id// >.< //task-id// >, et +
- < //​job_name//​ > **.e** < //job_id// >.< //task-id// > +
- +
-and par default there is the same number of files as you have the tasks.\\ +
-\\ +
-The standard outputs **//​stdout//​** and **//​stderr//​** are redirected to spool space during job execution and copy back, by default, in $HOME directory. You can change this behaviour with the options " **-o** " et " **-e** ": +
-<​code>​ +
-> qsub -e path -o path myscript.sh +
-</​code>​ +
-The -j y option allows to merge the stdout and stderr in a single file.\\ +
-\\ +
-The total size of the outputs is limited to 25 Mo.\\ +
-\\ +
-**Job execution at specidifed date**\\ +
-\\ +
-You can submit a job asking its execution (RUNNING) on a precise date. For that you need to use the following option: +
-<​code>​ +
--a time +
-</​code>​ +
-with the format %%[[CC]]YY]MMDDhhmm[.SS]%%. See "man sge_types"​.\\ +
-\\ +
-**Job Name**\\ +
-\\ +
--N //​job_name//​ : allow to specify the name //​job_name//​ of the job. By default, GE uses the script name.\\ +
-Please don't use french accents or special characters (like quotes) in the job name.\\ +
-\\ +
-**Prefix of parameters in script**\\ +
-\\ +
--C //prefix// : allow to modify prefix used in a script to define options to GE. By default, "#​$"​ is used.\\ +
-\\ +
-**Email alerts**\\ +
-\\ +
-You can ask to be notified on some events using the -m option : -m //​mail_options//​\\ +
-The mail will be sent to the user or to an email address specified by the flag -M //email address//​\\ +
-\\ +
-Mail options:​\\ +
-//b// : send an email to the user at job execution start-up\\ +
-//e// : send an email to the user at job execution end\\ +
-//n// : do not send email. n is the default\\ +
-NB : avoid using "-m a" option (email when job is aborted) as this can badly overload the system when jobs are deleted massively.\\ +
-\\ +
-**Job priority**\\ +
-\\ +
-The priority of a job can be reduced with the -p option of qsub.\\ +
-By default it's 0 and you can only lower the job priority. The value\\ +
-of the priority has to be negative. +
-<​code>​ +
-qsub -p -50 myjob.sh +
-</​code>​ +
-When the job is already in queue its priority can only be decreased and not increased\\ +
-with the qalter command. +
-<​code>​ +
-qalter -p -200 myjob.sh +
-</​code>​ +
-**Job priority**\\ +
-\\ +
-The "​-clear"​ option removes all other qsub options (for instance in the script ) except the options following it in the qsub command-line.  +
-===  Altering resources of pending jobs  === +
- +
-The qalter command lets you change/​add/​remove a given resource for a pending job.\\ +
-\\ +
-**Examples :**\\ +
-\\ +
-- To modify the requested memory of a pending job (setting s_rss=2G) +
-<​code>​ +
-> qalter -mods l_hard s_rss 2G <​jobid>​ +
-> qalter -h U jobnumber1 jobnumber2 ... +
-</​code>​ +
-- To add a given resource to a pending job (adding sps=1) +
-<​code>​ +
-> qalter -adds l_hard sps 1 <​jobid>​ +
-</​code>​ +
-- To remove a given resource from a pending job (removing hpss=1) +
-<​code>​ +
-> qalter -clears l_hard hpss <​jobid>​ +
-</​code>​ +
-- To remove all resources from a pending job : +
-<​code>​ +
-> qalter -clearp l_hard <​jobid>​ +
-</​code>​ +
-- To replace the list of all "Hard Resources"​ by a new list : +
-<​code>​ +
-> qalter -l s_rss=2G,​sps=1,​hpss=1 <​jobid>​ +
-</​code>​ +
-Type "man qalter"​ for other options like change of queue, of email, ...  +
-===  Job dependencies ​ === +
- +
-If you want to submit a job that should stay in the hold state until other jobs\\ +
-have finished you can use the option : +
-<​code>​ +
-qsub -hold_jid <​jobid>​ +
-</​code>​ +
-You can specify a list of jobs : qsub -hold_jid 102,​103,​104,​105,​106 myscript.sh +
-=====  Other types of jobs  ===== +
- +
-Please read the "​getting started"​ section first for the options common to all types of jobs.  +
-====  Array jobs  ==== +
- +
-An array job is a job which consist of several tasks. In this case, the same script is to be run several times, the only difference between each run is the index number associated with the tasks. The index numbers will be exported to the jobs via the environment variable $SGE_TASK_ID.\\ +
-\\ +
-To submit an array job to GE, the command to use on an interactive access machine of Computing Centre is **qsub** with the option " **-t** ": +
-<​code>​ +
-> qsub -t min[-max[:​interval]] +
-</​code>​ +
-for example : +
-<​code>​ +
-> qsub -t 1-10 +
-</​code>​ +
-The option arguments //min// , //max// and //​interval//​ will be available through the environment variables $SGE_TASK_FIRST,​ $SGE_TASK_LAST and $SGE_TASK_STEPSIZE,​ respectively. But note, that is nothing to do with the execution order.  +
-====  Interactive jobs  ==== +
- +
-To submit interactive jobs, first log on cca.in2p3.fr and then connect to an interactive machine by using the qlogin command. Or if you use CentOS 7 connect to cca7.in2p3.fr and use : qlogin -l os=cl7.\\ +
-qlogin command has the same options as qsub command for specifying needed resources.\\ +
-\\ +
-For example: +
-<​code>​ +
->qlogin -l fsize=1G,​ct=1:​00:​00,​s_rss=1G +
-</​code>​ +
-You will be connected directly with a worker and your interactive task will be managed like a batch job.\\ +
-\\ +
-The message +
-<​code>​ +
-Your job 148273 ("​QLOGIN"​) has been submitted +
-</​code>​ +
-provides the id of your job.\\ +
-\\ +
-Once connected you can go to the local directory corresponding to your job id. +
-<​code>​ +
-cd /​scratch/​148273.1.interactive +
-</​code>​ +
-You'll have at your disposal the disk space specified by the fsize option.\\ +
-\\ +
-To use several cores specify the number of cores you will use with : +
-<​code>​ +
-qlogin -pe multicores <​number_of_cores>​ -q mc_interactive  +
-</​code>​ +
- +
-====  Multicore jobs  ==== +
- +
-Multicore jobs have restricted usage : you should contact CC-IN2P3 Helpdesk (https:​%%//​%%cc-usersupport.in2p3.fr) if you want to run multicore jobs.\\ +
-\\ +
-To run a job using several cores on a single machine use multicores environment and define the needed queue with : +
-<​code>​ +
--pe multicores <​number_of_cores>​ -q <​QueueName>​  +
-</​code>​ +
-The available queues for multicore jobs are named "​mc_long",​ "​mc_huge"​ and "​mc_longlasting"​. Access to mc_huge and mc_longlasting has to be specifically asked for. You have to chose the queue matching the needs of your jobs in terms of cpu, memory and scratch space. The cpu and the memory are expressed per core and the scratch space is global for the job.\\ +
-\\ +
-To know the characteristics of a queue : +
-<​code>​ +
-qconf -sq <​QueueName>​  +
-</​code>​ +
-On the worker node, the number of cores is accassible through the variable NSLOTS\\ +
-\\ +
-See [[http://​cctools.in2p3.fr/​mrtguser/​info_sge_generale.php|SGE configuration]] to know the current configuration.  +
-====  Parallel jobs  ==== +
- +
-Parallel jobs have restricted usage : you should contact CC-IN2P3 Helpdesk (https:​%%//​%%cc-usersupport.in2p3.fr) if you want to run parallel jobs.\\ +
-\\ +
-After having defined environment variables and having compiled the your code : use parallel Grid Engine environment with the option -pe equal either to openmpi or mpich2\\ +
-and define the needed queue with option -q either equal to pa_medium or pa_long depending on the cpu and memory needed. Access to pa_longlasting is restricted and should be allowed by your czar. The cpu and the memory are expressed per core.\\ +
-\\ +
-To know the characteristics of a queue : +
-<​code>​ +
-qconf -sq <​QueueName>​  +
-</​code>​ +
-See [[http://​cctools.in2p3.fr/​mrtguser/​info_sge_generale.php|SGE configuration]] to know the current configuration.\\ +
-\\ +
-**The parallel jobs are run on CentOS 7**\\ +
-\\ +
-You can connect to the interactive machines cca7 to compile your code under CentOS 7\\ +
-\\ +
-You need to specify the OS (qsub -l os=cl7) and use /​usr/​lib64/​openmpi or\\ +
-/​usr/​lib64/​mpich respectively for Open MPI and MPICH2.\\ +
-\\ +
-Examples :\\ +
-\\ +
-Open MPI : +
-<​code>​ +
--l os=cl7 -pe openmpi <​number_of_cores>​ -q <​QueueName>​  +
-</​code>​ +
-With in the script :  +
-<​code>​ +
- ​export OPENMPI_HOME=/​usr/​lib64/​openmpi/​ +
- ​export PATH=${OPENMPI_HOME}/​bin:​${PATH} +
- ​export LD_LIBRARY_PATH=${OPENMPI_HOME}/​lib:​${LD_LIBRARY_PATH} +
- ​export MANPATH=${OPENMPI_HOME}/​man:​${MANPATH} +
- ​MPIEXEC="/​usr/​lib64/​openmpi/​bin/​mpiexec"​ +
- ​$MPIEXEC -n $NSLOTS phello +
-</​code>​ +
-\\ +
-compilation :<​code>​ mpicc -o phello phello.c</​code>​ +
-\\ +
-Example of job submission : <​code>​ qsub -l os=cl7 -cwd -pe openmpi 16 -q pa_long phello.script</​code>​ +
-\\ +
-\\ +
-MPICH2 : +
-<​code>​ +
--l os=cl7 -pe mpich2 <​number_of_cores> ​ -q <​QueueName>​  +
-</​code>​ +
-With in the script :  +
-<​code>​ +
-export MPICH_HOME=/​usr/​lib64/​mpich +
-export PATH=${MPICH_HOME}/​bin:​${PATH} +
-export MANPATH=${MPICH_HOME}/​man:​${MANPATH} +
-MPIEXEC="/​usr/​lib64/​mpich/​bin/​mpiexec"​ +
-${MPIEXEC} -iface ib0 -np $NSLOTS phello +
-</​code>​ +
-\\ +
-compilation : <​code>​ mpicc -o phello phello.c </​code>​ +
-\\ +
-Example of job submission : <​code>​ qsub -l os=cl7 -cwd -pe mpich2 16 -q pa_long phello.script </​code>​ +
-\\ +
-Note that the ouput of the job is written directly to its final destination during the job execution\\ +
-and not written locally on the worker and transferred to its final destination at the end of the job.\\ +
-\\ +
-In order to use OpenMPI et MPICH2 libraries you'll have to create a file .mpd.conf in you $HOME with the rights 600 (-rw-------) and which contains : secretword=XXXX where XXXX is a secret word. Otherwise you'll get an error message like: mpdboot_ccali16 (handle_mpd_output 406): from mpd on ccali16, invalid port info:\\ +
-no_port  +
-====  GPU jobs  ==== +
- +
-GPU jobs have restricted usage : you should contact CC-IN2P3 Helpdesk (https:​%%//​%%cc-usersupport.in2p3.fr) if you want to run GPU jobs.\\ +
-\\ +
-The GPU jobs are run on CentOS 7 and the current platform consists of 10 Dell C4130 with 4 GPUs and 16 CPU cores per machine :\\ +
-  * 2 Xeon E5-2640v3 (8c @2.6 Ghz)\\ +
-  * 128 GB RAM\\ +
-  * 2 Nvidia Tesla K80 -> 4 GPU Nvidia GK210 with 12 GB DDR5\\ +
-\\ +
-CUDA 8.0 and OpenCL 1.2 environments are available in /​opt/​cuda-8.0.\\ +
-\\ +
-If you need CUDA or OpenCL libraries, you should add in your script :\\ +
-  * bash :\\ +
-<​code>​ +
- if ! echo ${LD_LIBRARY_PATH} | /bin/grep -q /​opt/​cuda-8.0/​lib64 ; then +
-       ​LD_LIBRARY_PATH=/​opt/​cuda-8.0/​lib64:​${LD_LIBRARY_PATH} +
- fi +
-</​code>​ +
-\\ +
-  * csh :\\ +
-<​code>​ +
- if ($?​LD_LIBRARY_PATH) then +
-       ​setenv LD_LIBRARY_PATH /​opt/​cuda-8.0/​lib64:​${LD_LIBRARY_PATH} +
- ​else +
-       ​setenv LD_LIBRARY_PATH /​opt/​cuda-8.0/​lib64 +
- ​endif +
-</​code>​ +
-\\ +
- +
-===  Multicore GPU jobs  === +
- +
-To submit a multicore GPU job you need to specify the OS (“-l os=cl7”), the GPU queue (for instance “-q mc_gpu_long”) the number of GPUs needed (for instance “-l GPU=2”, up to 4 GPUs are available per machine) and the number of cores needed (for instance “-pe multicores 8”, up to 16 cores per machine).\\ +
-\\ +
-In summary the **qsub** options are : +
-<​code>​ +
- qsub -l os=cl7,​GPU=<​number_of_gpus>​ -q <​QueueName>​ -pe multicores <​number_of_cores>​ ...   +
-</​code>​ +
-The available batch queues are : mc_gpu_medium,​ mc_gpu_long and mc_gpu_longlasting (restricted access). The “CUDA_VISIBLE_DEVICES” variable is set automatically. +
-<​code>​ +
-example :  qsub -l os=cl7,​GPU=2 -q mc_gpu_long -pe multicores 8 ...  +
-</​code>​ +
-\\ +
-===  Parallel GPU jobs  === +
- +
-To submit a parallel GPU job you need to specify the OS (“-l os=cl7”), the GPU queue (for instance “-q pa_gpu_long”),​ the number of GPUs needed ** by node** (for instance “-l GPU=4” to get 4 GPUs on each node. 1 to 4 GPUs are available per machine) and the openmpi environment + number of cores needed:\\ +
-\\ +
-There are 3 openmpigpu environments,​ which will allow you to control how many nodes you will get, based on how many CPU cores you request:​\\ +
-  * openmpigpu_4 : if you request N CPU cores, your job will run on N/4 nodes\\ +
-  * openmpigpu_2 : if you request N CPU cores, your job will run on N/2 nodes\\ +
-  * openmpigpu : if you request N CPU cores, your job will run on N/16 nodes (you fill up each node; thus pay attention to request -l GPU=4, to avoid a waste of GPU resources)\\ +
-\\ +
-Your script must contain some OpenMPI specific settings (including MPIEXEC launch), mentioned in https://​doc.cc.in2p3.fr/​en:​ge_submit_a_job_qsub#​parallel_jobs OpenMPI section.\\ +
-\\ +
-In summary the qsub options are : +
-<​code>​ +
- qsub -l os=cl7,​GPU=<​number_of_gpus_per_node>​ -q <​QueueName>​ -pe <​openmpi_env>​ <​number_of_cores>​ ... +
-</​code>​ +
-The available batch queue is : pa_gpu_long (restricted access). +
-<​code>​ +
-example :  qsub -l os=cl7,​GPU=3 -q pa_gpu_long -pe openmpigpu_2 6 my_script_for_3_nodes_and_9_GPUs.sh  +
-</​code>​ +
-\\ +
-===  Interactive GPU jobs  === +
- +
-The interactive GPU jobs are run with the qlogin command (with the same options as multicore job qsub) and choosing the mc_gpu_interactive queue.\\ +
-\\ +
- +
-In summary the **qlogin** options are : +
-<​code>​ +
- ​qlogin -l os=cl7,​GPU=<​number_of_gpus>​ -q <​QueueName>​ -pe multicores <​number_of_cores>​ ...   +
-</​code>​ +
-<​code>​ +
-example : qlogin -l os=cl7,​GPU=1 -q mc_gpu_interactive -pe multicores 1 ... +
-</​code>​ +
-=====  Examples ​ ===== +
- +
- +
-====  Simple example: Hello World! ​ ==== +
- +
-To submit a job in a queue, write your first script. Copy the following two lines in a file and save it as //​hello_world.sh//​ : +
-<​code>​ +
-#​!/​bin/​csh +
- +
-### Merge stdout et stderr in a single file +
-#$ -j y +
- +
-echo "​\nHello World!\n"​ +
- +
-echo 'my working directory is: ' +
-pwd +
- +
-echo 'on the host: '  +
-hostname +
- +
- +
-</​code>​ +
-To submit a job, type: +
-<​code>​ +
-> qsub hello_world.sh +
-</​code>​ +
-This command **//​qsub//​** will return: +
-<​code>​ +
-Your job 1260 ("​hello_world.sh"​) has been submitted  +
-</​code>​ +
-Standard outputs //stdout// and //stderr// will redirected in the same file which will be copied at the end of execution in current working submission directory. An email will be send to you at the beginning and at the end of the job submission. For that, email address is specified in the file. These emails contains:​\\ +
-\\ +
-- at the beginning of execution:​ +
-<​code>​ +
-Job 1260 (hello_world.sh) Started +
- ​User ​      = sloikkan +
- ​Queue ​     = long +
- ​Host ​      = ccwge0018.in2p3.fr +
- Start Time = 08/13/2010 15:53:03 +
-</​code>​ +
-- at the end of execution:​ +
-<​code>​ +
-Job 1260 (hello_world.sh) Complete +
- ​User ​            = sloikkan +
-Queue            = long@ccwge0018.in2p3.fr +
- ​Host ​            = ccwge0018.in2p3.fr +
- Start Time       = 08/13/2010 15:53:03 +
- End Time         = 08/13/2010 15:54:03 +
- User Time        = 00:00:00 +
- ​System Time      = 00:00:00 +
- ​Wallclock Time   = 00:01:00 +
- ​CPU ​             = 00:00:00 +
- Max vmem         = 12.930M +
- Max rss          = 2.120M +
- Exit Status ​     = 0 +
-</​code>​ +
-If you display now the o.file, you should see the text "Hello World!"​. Change the number by the number of your own job. +
-<​code>​ +
-> cat hello_world.sh.o1260 +
-Hello World! +
-</​code>​ +
- +
-====  Example of job submission with parameters ​ ==== +
- +
-<​code>​ +
-#​!/​usr/​local/​bin/​tcsh +
- +
-qsub <<eof  +
-$HOME/​test.sh arg1 arg2 arg3 +
-eof +
-</​code>​ +
-This command allows to give arguments arg1, arg2 and arg3 to the shell script test.sh. arg1 is read with variable $1, arg2 with $2 and arg3 with $3.  +
-====  Example of a job using a database ​ ==== +
- +
-Submission of a job using Oracle database, with 20 Go sized scratch, 1.5 Go of memory +
-<​code>​ +
-qsub -l oracle,​ct=00:​30:​00,​s_rss=1500M,​fsize=20GB test.sh  +
-</​code>​ +
- +
-====  Example of an array job  ==== +
- +
-When a jobs run the same code on several different input files or produce\\ +
-files using the same code with different input parameters using array jobs\\ +
-can be very useful. The basic principal is that a single job will launch\\ +
-several tasks at once.\\ +
-\\ +
-For example a job //​array_job.sh//​ like : +
-<​code>​ +
-#!/bin/sh +
- +
-### Merge the stdout et stderr in a single file +
-#$ -j y +
- +
-### set array job indices '​min-max:​interval'​ +
-#$ -t 1-4:2 +
- +
-### set the name +
-#$ -N hello_items +
- +
-echo "Task id = $SGE_TASK_ID"​ +
- +
-if [ -e input$SGE_TASK_ID.txt ]; then +
-while read file +
-do +
-echo "hello $file"​ +
-done < input$SGE_TASK_ID.txt +
-fi  +
-</​code>​ +
-reads the data in the files //​input[1-4].txt//​ and say hello to the items. +
-<​code>​ +
-#​input1.txt +
-world +
-moon  +
- +
-#​input2.txt +
-forest +
-sun +
- +
-#​input3.txt +
-monkey +
-people +
- +
-#​input4.txt +
-banan +
-apple +
-</​code>​ +
-To submit a job, tapez (options are in the script) : +
-<​code>​ +
-> qsub array_job.sh +
-Your job-array 1331.1-4:2 ("​hello_items"​) has been submitted +
- +
-</​code>​ +
-And there will be 2 ouputs (instead of 4 because the interval is equal to 2) : +
-<​code>​ +
-$ cat hello_items.o1331.1  +
-Task id = 1 +
-hello world +
-hello moon +
- +
-$ cat hello_items.o1331.3 +
-Task id = 3 +
-hello monkey +
-hello people +
-</​code>​ +
-=====  Check the status of a job  ===== +
- +
-Please read the following documentation [[:​en:​ge_list_of_commands]] .  +
-====  Information about current jobs  ==== +
- +
-The qstat command provides information on current tasks : +
-<​code>​ +
-qstat  +
-</​code>​ +
-Normal status : +
-<​code>​ +
-- r for running jobs  +
-- qw for queued and waiting jobs +
-</​code>​ +
-Jobs that encountered a problem : +
-<​code>​ +
-- Eqw : job in error state in the queue +
-- Rr : running jobs that have been rerunned  +
-</​code>​ +
-For more information on ressources consumption for running jobs only +
-<​code>​ +
-qstat -s r -ext +
-</​code>​ +
-For more informtaion on declared ressources for running jobs only +
-<​code>​ +
-qstat -r +
-</​code>​ +
-The other possible states are : +
-^Category ^State ​                                        ^SGE Letter Code                            ^ +
-|Pending ​ |                                              |                                           | +
-|         ​|pending ​                                      ​|qw ​                                        | +
-|         ​|pending,​ user hold                            |hqw                                        | +
-|         ​|pending,​ system hold                          |hqw                                        | +
-|         ​|pending,​ user and system hold                 ​|hqw ​                                       | +
-|         ​|pending,​ user hold, re-queue ​                 |hRwq                                       | +
-|         ​|pending,​ system hold, re-queue ​               |hRwq                                       | +
-|         ​|pending,​ user and system hold, re-queue ​      ​|hRwq ​                                      | +
-|Running ​ |                                              |                                           | +
-|         ​|running ​                                      ​|r ​                                         | +
-|         ​|transferring ​                                 |t                                          | +
-|         ​|running,​ re-submit ​                           |Rr                                         | +
-|         ​|transferring,​ re-submit ​                      ​|Rt ​                                        | +
-|Suspended| ​                                             |                                           | +
-|         |job suspended ​                                |s, ts                                      | +
-|         ​|queue suspended ​                              |S, tS                                      | +
-|         ​|queue suspended by alarm                      |T, tT                                      | +
-|         |all suspended with re-submit ​                 |Rs, Rts, RS, RtS, RT, RtT                  | +
-|Error ​   |                                              |                                           | +
-|         |all pending states with errors ​               |Eqw, Ehqw, EhRqw                           | +
-|Deleted ​ |                                              |                                           | +
-|         |all running and suspended states with deletion|dr,​ dt, dRr, dRt, ds, dS, dT, dRs, dRS, dRT| +
- +
- +
-====  Information about a given job  ==== +
- +
-When using qstat for a given job it is recommended to use the option -nenv in order\\ +
-to reduce the high verbosity of the output : +
-<​code>​ +
-qstat -nenv -j <​jobid>​  +
-</​code>​ +
- +
-====  Information about jobs of a group  ==== +
- +
-For more information on running jobs for a given group : +
-<​code>​ +
-qstat -s r -u @<​groupname>​ +
-</​code>​ +
- +
-====  Information about ended jobs  ==== +
- +
-The qacct command provides information on completed tasks.\\ +
-\\ +
-For instance to see details for all jobs since 1 day : qacct -j \* -d 1.\\ +
-\\ +
-To display all available options and filters : qacct -help\\ +
-\\ +
-qacct has by default access to the accounting up to one week back.\\ +
-\\ +
-The accounting files for past months are available with the option -f of qacct in /​opt/​sge/​ccin2p3/​common/​ with the names accounting.YYYY.MM.\\ +
-\\ +
-For instance : qacct -j <​jobnumber>​ -f /​opt/​sge/​ccin2p3/​common/​accounting.2013.05\\ +
-\\ +
-Older accounting files are available in : /​afs/​in2p3.fr/​common/​sge/​prod/​common\\ +
-\\ +
-The definition of the used terms can be obtained with : man accounting\\ +
-\\ +
-Note that from Feb. 7th 2012 the cpu obtained with qacct is directly expressed in the normalized units : HS06 . secondes\\ +
-\\ +
-If a soft limit is exceeded, the following signals are sent : SIGXCPU for CPU (ct) or memory (s_rss), SIGUSR1 for elapsed time, XFSZ for Disk (fsize). If a hard limit is exceeded a signal SIGKILL is sent.\\ +
-\\ +
-For a given jobs these signals are given by the exist_status of qacct. +
-<​code>​ +
-exit_status ​ < 128 exit status set by user if lower than 128              +
-exit_status ​ 137  corresponds to SIGKILL ​            137 = 9 (SIGKILL) + 128  +
-exit_status ​ 152  corresponds to SIGXCPU ​            152 = 24 (SIGXCPU) + 128  +
-exit_status ​ 138  corresponds to SIGUSR1 ​            138 = 10 (SIGUSR1) + 128  +
-exit_status ​ 153  corresponds to XFSZ                153 = 25 (XFSZ) + 128  +
-</​code>​ +
-The failed values in qacct are the following : +
-<​code>​ +
-1 : assumedly before job  Job could not be started +
-7 : before prolog ​ Job could not be started +
-8 : in prolog ​ Job could not be started +
-10 : in pestart ​ Job could not be started +
-19 : before writing exit_status   +
-21 : in recognizing job   +
-25 : rescheduling ​ Job ran, job will be rescheduled +
-26 : opening input/​output le  Job could not be started, stderr/​stdout le could not be opened +
-28 : changing into working directory ​ Job could not be started, error changing to start directory  +
-29 : invalid execution state   +
-37 : qmaster enforced h rt limit   +
-100 : assumedly after job  Job ran, job killed by a signal +
-</​code>​ +
-**Environmental variables**\\ +
-\\ +
-- $BATCH_SYSTEM is set to GE\\ +
-- $ENVIRONMENT can take the values :\\ +
-PARALLEL_BATCH , SEQUENTIEL_BATCH or INTERACTIVE_BATCH\\ +
-- Grid engine set additional variables into the job's environment as listed in the last section of the 'man qsub'​. +
-=====  CPU consumption of a job  ===== +
- +
-The wallclock time of a job is the time elapsed between the beginning and the end of the job.\\ +
-The cpu time usage of a job should be almost equal to the wallclock time (times the number of processors\\ +
-for parallel jobs) if the job is computing efficiently and is not limited for instance by I/O access.\\ +
-\\ +
-In order to evaluate the cpu consumption in HS06 x hours you need to multiply the cpu time\\ +
-usage by the power of the processor expressed in HS06. HS06 x hours is the units in which the cpu accounting\\ +
-is expressed at cc-in2p3.\\ +
-\\ +
-Example : If the cpu time usage of a job is 10 000 secondes , the consumption of the job\\ +
-is 10 000 secondes x 9.5 HS06 (power of some processor) = 26,4 HS06 x hours\\ +
-\\ +
-Note that from Feb. 7th 2012 the cpu obtained with qacct is directly expressed in the normalized units : HS06 . secondes +
-=====  More information ​ ===== +
- +
-To get help, you can type : +
-<​code>​ +
-> man qsub +
-</​code>​ +
-or: +
-<​code>​ +
-> qsub -help +
-</​code>​ +
  • en/ge_submit_a_job_qsub.txt
  • Last modified: 2018/01/17 15:15
  • by Quentin LE BOULC'H