Storage areas
CC-IN2P3 provides two distributed filesystems to accommodate personal and group storage for supported experiences.
To know the allowed quotas on the different areas from an interactive server:
% df -h /pbs/home/<u>/<username>
Filesystem Size Used Avail Use% Mounted on
ccsisi.in2p3.fr:/pbs/home 20G 19G 1.2G 95% /pbs/home
% df -h /pbs/throng/<group>
% df -h /sps/<group>
% spsquotalist
spsquotalist usage examples:
...
Attention
Each collaboration must provide a management plan describing the lifecycle of their data. This document will subsequently be discussed with the collaboration contacts once a year during the resource requests period.
To have a general view of the storage services, you may refer to the documents used during the trainings provided by CC-IN2P3.
Personal directory (HOME)
The user HOME directory is dedicated to the storage of personal data, and is is backed-up every day (see Backup policy). This space is associated to the Unix environment variable HOME:
% echo $HOME
/pbs/home/<u>/<username>
By default, HOME directories have by default a quota of 20 GiB and are not accessible by other users, but access rights may be modified (see Manage ACL).
Important
Although “personal” by default, this directory remains subject to professional use; the data can therefore be transmitted to the collaboration contact upon request, in particular when the account is closed.
Attention
A full HOME directory can cause errors, e.g. if jobs are trying to write on it.
Group storage
Group storage areas are shared by the users belonging to the same Unix group. Such users are expected to be working on common topics and likely to need share data.
THRONG directory
The THRONG directory is a common area where users may share with eachother important data that need to be backed-up regularly. CC-IN2P3 allocates the THRONG directory upon request to the user support. The default quota is 100 GiB.
The THRONG directory is backed-up every day (see Backup policy). It is associated with the Unix environment variable THRONG_DIR:
% echo $THRONG_DIR
/pbs/throng/<group>
When created, this space can be managed indifferently by all the users belonging to the Unix group. Users can also organize and delegate authorizations to other users (see Manage ACL).
GROUP directory
Unlike THRONG, the GROUP directory is not backed-up. It is designed to host larger data volumes, like non-critical experimental data that have to be accessed regularly by several colleagues.
CC-IN2P3 allocates the GROUP directory upon request to the user support. The default quota is 5 TiB.
This space is located on the SPS service, under the path /sps/<group>. For more information, please check the dedicated page.
When created, this space can be managed indifferently by all the users belonging to the Unix group. Users can also organize and delegate authorizations to other users.
Working areas
In addition to the HOME, THRONG et GROUP directories, you can take advantage of the /scratch directory from any interactive server to store transient data.
The /scratch directory is shared between all the users able to access the server. To use it, you can create it if it doesn’t exist yet:
% mkdir /scratch/$USER
and to store your files there.
Attention
This working area is not permanent, the data is automatically and regularly cleaned-up.
From a compute server, your job runs inside a directory in the /scratch space, which path is defined by the bact system. You can get this path value through the environment variable TMPDIR.
Advanced information
For more information on the two distributed file systems available features, please refer to the following pages.
Data Management Plan
A Data Management Plan (DMP) describes data lifecycle. It is not an administrative form to fill out, but a planning process to be managed by the collaboration contacts with the contribution of any user data owner.
- A DMP should include:
general description (purpose of the data),
information on data processing during and after the end of the project (responsibility),
data usage scenarios,
request of resources,
legal and ethical aspects,
how data will be curated and stored (including after the end of the project).
To help create/edit the document and make it available for revision throughout the project activity, CC-IN2P3 provides a form following the standards of the RDA format to be filled in through the DMP portal. To obtain the necessary credentials, please contact user support.
Attention
A number of questions have been designated as carrying essential information for CC-IN2P3 storage management. Without this information, the DMP cannot be validated for the current year. The essential questions will be presented and explained in the following section.