This is an old revision of the document!


RFIO: rfiod servers



The RFIO server, rfiod, supports the RFIO64 interface. There are two kinds of servers: ordinary servers that support a UNIX file system and HPSS servers. In both cases the module is called rfiod. The ordinary server is pointed to /usr/local/bin/rfiod which is a link to /usr/local/products/shift/prod/bin/rfiod. It is available on all platforms supported at CC-IN2P3. The RFIO server must run as a root user. However, an RFIO server launched as a normal user can process the requests from that user. Access to the man of the rfiod command: rfiod(1)

If the use of RFIOD is supported, it is recommended to use a local file system or, failing this, use a subtree to be defined in /var/spool/rfio. In /var/spool/rfio/bin we set the binary rfiod to be independent from AFS. Warning: This prevents server updates during an RFIO version upgrade. Logs will be written in /var/spool/rfio/log. The name specification allows to automatically create a log file per day. But a manual storage management is needed!

shift.conf : the rfio configuration file is /etc/shift.conf. A template may be found in /usr/local/products/shift/prod/etc/shift.conf. If you need to recover the compatibility you need to add:

#
# Accounting ON
#
ACCT       RFIO             YES

services : the file /etc/services must specify the rfio port if it is not the IN2P3 default (5001).

rfio            5001/tcp                 # CERN Remote File I/O  System

Please note that the official RFIO port is not 5001! inittab : you may add an entry in inittab or rc.local to automatically start rfiod. If you need an automatic restart in case of abend, define an entry in the file /etc/inittab with a respawn and specify the option -n:

rfio:2:respawn:/var/spool/rfio/bin/rfiod -nsl -D /var/spool/rfio > /dev/null 2>&1

On Linux machines, do not add an identifier more than 4 characters long and specify all the possible run levels:

rfio:2345:respawn:/var/spool/rfio/bin/rfiod -nsl -D /var/spool/rfio > /dev/null 2>&1

An HPSS tool is available to handle HPSS RFIO accounting. This tool must work with other servers. In order for the accounting to work, you must declare “ACCT RFIO YES” in the /etc/shift.conf file and the /var/spool/sacct file must exist. For the shift.conf file, see: shift.conf. At IN2P3 we define /var/spool/sacct as a link to /var/spool/rfio/sacct which must exist if we want the accounting.

DCE

The RFIOD HPSS server must run on a machine defined in the DCE HPSS cell. The DCE rfiod account under which the RFIO server is running must be defined. The rfiod user must have “rc” access rights on the core server. These are the same privileges as the NFS server. It is necessary to generate a file of keys to allow the connection as rfiod (/krb5/rfiod.keytabs).

Generation

The HPSS server must be generated on an HPSS server with the DCE development environment. It is necessary to obtain the code of CASTOR from CVS and modify the file config/site.def to generate a server RFIOD HPSS. The variables to be modified in site.def in relation to non-HPSS clients and non-HPSS servers are:

#define       HPSS_HomeDir            /opt/hpss
#define       BuildHpssClient         NO
#define       BuildHpssServer         YES
#define       BuildRfioClient         NO

These options are used to generate an RFIO-HPSS server in rfio/rfiod. There is no need to generate a client part that only runs under DCE. Virtually, rfiod HPSS is generated on a HPSS server core and copied into ~hpss/bin/rfiod for broadcast.

Environment variables

HPSS API environment variables improve performance: ${HPSS_HOSTNAME} : specifies the quick interface name for transfers when there are multiple network interfaces on the same server (useful for the rfio API). ${HPSS_API_HOSTNAME} : specifies the quick interface name for transfers when there are multiple network interfaces on the same server in HPSS 5.1 (useful for the rfio API). ${HPSS_REUSE_CONNECTIONS} : setting the value to 1 (true) improves the rfio_read() and rfio_write() performance.

shift.conf configuration file

The shift.conf file contains specific values for HPSS. Here is an example:

###############################################################################
# File: shift.conf                                                             #
# Purpose: configuration file for CERN's SHIFT software, mainly RFIO.          #
# Author: phg and jon                                                          #
# Date: 20 Mar 2000                                                            #
# Last Modification: 20 Mar 2000                                               #
###############################################################################
#
# Number of simultaneous threads
#
RFIO       MAX_THREADS      200     # Default=10
#
# I/O buffer size to be used by client and server
#
RFIO       IOBUFSIZE        1048576   # I/O buffer size (1*1024*1024)
#
# Read and write buffer size for stream mode
#
RFIO       DAEMONV3_RDSIZE  1048576   # Read buffer size
RFIO       DAEMONV3_WRSIZE  1048576   # Write buffer size
#
# Keep alive socket connection in RFIO server
#
RFIOD      KEEPALIVE        YES       # Keep it alive
#
# Accounting ON
#
ACCT       RFIO             YES

Start up

You can start rfiod manually (rfiod -sl) or use the /etc/rc.rfio utility. For an automatic start with monitoring, you must put an entry in /etc/inittab. Here is for example the entry in intab of the ccmcrs12 server:

rfiod:2:respawn:/usr/bin/env HPSS_HOSTNAME=ccmdrs12 HPSS_API_HOSTNAME=ccmdrs12 HPSS_REUSE_CONNECTIONS=1 /var/spool/rfio/bin/rfiod -nsl -D /var/spool/rfio > /dev/null 2>&1

Here is the list of RFIO documentation and associated products:

  • en/rfio_les_serveurs_rfiod.1507025109.txt.gz
  • Last modified: 2017/10/03 12:05
  • by Gino MARCHETTI