The High Performance Storage System (HPSS) service is a system capable of storing very large quantities of data with a relatively good performance for write and read operations.
HPSS is not an archival system, and stores only single copies of files.
File access and management¶
File management on HPSS is based on the RFIO API. To source the RFIO environment, please load “shift”:
% ccenv shift
Here are the most common commands to manage files with RFIO:
rfcp # remote file copy rfchmod # change file access permission rfdir # remote directory list rfmkdir # make remote directories rfrename # rename remote file rfrm # remove remote files and directories rfstat # get information about a file or directory
These commands should be used with the RFIO server
cchpss0, in the following way:
% rfdir cchpss0:/hpss/in2p3.fr/group/alice
It is possible to use local symbolic links pointing to remote HPSS directories, like:
% ln -s cchpss0:/hpss/in2p3.fr/group . % rfdir group drwxr-xr-x 79 root root 1536 Jun 03 10:38 . drwxr-xr-x 4 root root 512 Mar 27 2017 .. drwxrwxr-x 7 tmaterna mnm 512 Mar 27 10:28 mnm drwxrwxr-x 2 aubourg sdss 512 Apr 14 2009 sdss .....
Finding a file in the HPSS tree can be slow. In case of frequent calls to find a file in HPSS, we recommand the use of a dedicated database or catalogue.
File permissions on HPSS work in the same way as standard Unix permissions. Keep in mind that
rfcp propagates the original file permissions to the new copy.
Access from jobs¶
To access HPSS from a job run on the computing platform, please use the
-l option at job submission:
% qsub -l hpss=1 script.sh
HPSS supports a maximum file size of 64 GB. In order to benefit from better I/O performance, users are invited to store large files (> 1GB) in HPSS, the optimum being 4 GB.
I case you need to store a large number of small files, please contact the user support.