release notes | Book: 1.9.5, 1.9.12 (opt, FHS), 2.7 (FHS), 2.8 (FHS), 2.9 (FHS), 2.10 (FHS), 2.11 (FHS), 2.12 (FHS), 2.13 (FHS), | Wiki | Q&A black_bg
Web: Multi-page, Single page | PDF: A4-size, Letter-size | eBook: epub black_bg

Details on the TSS-support executable

[return to top]

Summary of command line options

This part explains the syntax of calling the executable that supports STORE FILE, FETCH FILE and REMOVE FILE operations.

put <pnfsID> <filename> -si=<storage-information> [<other-options>...]

get <pnfsID> <filename> -si=<storage-information> -uri=<storage-uri> [<other-options>...]

remove -uri=<storage-uri> [<other-options>...]

  • put / get / remove: these keywords indicate the operation to be performed.
    • put: copy file from disk to TSS.
    • get: copy file back from TSS to disk.
    • remove: remove the file from TSS.
  • <pnfsID>: The internal identifier (i-node) of the file within dCache. The <pnfsID> is unique within a single dCache instance and globally unique with a very high probability.
  • <filename>: is the full path of the local file to be copied to the TSS (for put) and respectively into which the file from the TSS should be copied (for get).
  • <storage-information>: the storage information of the file, as explained below.
  • <storage-uri>: the URI, which was returned by the executable, after the file was written to tertiary storage. In order to get the file back from the TSS the information of the URI is preferred over the information in the <storage-information>.
  • <other-options>: -<key> = <value> pairs taken from the TSS configuration commands of the pool 'setup' file. One of the options, always provided is the option -command=<full path of this executable>.

[return to top]

Storage Information

The <storage-information> is a string in the format

-si=size=<bytes>;new=<true/false>;stored=<true/false>;sClass=<StorageClass>;\
cClass0<CacheClass>;hsm=<StorageType>;<key>=<value>;[<key>=<value>;[...]]

Example:

-si=size=1048576000;new=true;stored=false;sClass=desy:cms-sc3;cClass=-;hsm=osm;Host=desy;

Mandatory storage information’s keys

  • <size>: Size of the file in bytes
  • <new>: False if file already in the dCache; True otherwise
  • <stored>: True if file already stored in the TSS; False otherwise
  • <sClass>: HSM depended, is used by the poolmanager for pool attraction.
  • <cClass>: Parent directory tag (cacheClass). Used by the poolmanager for pool attraction. May be '-'.
  • <hsm>: Storage manager name (enstore/osm). Can be overwritten by the parent directory tag (hsmType).

OSM specific storage information’s keys

  • <group>: The storage group of the file to be stored as specified in the ".(tag)(sGroup)" tag of the parent directory of the file to be stored.
  • <store>: The store name of the file to be stored as specified in the ".(tag)(OSMTemplate)" tag of the parent directory of the file to be stored.
  • <bfid>: Bitfile ID (get and remove only) (e.g. 000451243.2542452542.25424524)

Enstore specific storage information’s keys

  • <group>: The storage group (e.g. cdf, cms ...)
  • <family>: The file family (e.g. sgi2test, h6nxl8, ...)
  • <bfid>: Bitfile ID (get only) (e.g. B0MS105746894100000)
  • <volume>: Tape Volume (get only) (e.g. IA6912)
  • <location>: Location on tape (get only) (e.g. : 0000_000000000_0000117)

There might be more key values pairs which are used by the dCache internally and which should not affect the behaviour of the executable.

[return to top]

Storage URI

The <storage-uri> is formatted as follows:

hsmType://hsmInstance/?store=<storename>&group=<groupname>&bfid=<bfid>

  • <hsmType>: The type of the Tertiary Storage System
  • <hsmInstance>: The name of the instance
  • <storename> and <groupname> : The store and group name of the file as provided by the arguments to this executable.
  • <bfid>: The unique identifier needed to restore or remove the file if necessary.

Example:

A storage-uri:

osm://osm/?store=sql&group=chimera&bfid=3434.0.994.1188400818542

[return to top]

Summary of return codes

Return CodeMeaningBehaviour for PUT FILEBehaviour for GET FILE
30 <= rc < 40 User defined Deactivates request Reports problem to poolmanager
41 No space left on device Pool retries Disables pool and reports problem to poolmanager
42 Disk read I/O error Pool retries Disables pool and reports problem to poolmanager
43 Disk write I/O error Pool retries Disables pool and reports problem to poolmanager
other - Pool retries Reports problem to poolmanager

[return to top]

The executable and the STORE FILE operation

Whenever a disk file needs to be copied to a Tertiary Storage System dCache automatically launches an executable on the pool containing the file to be copied. Exactly one instance of the executable is started for each file. Multiple instances of the executable may run concurrently for different files. The maximum number of concurrent instances of the executables per pool as well as the full path of the executable can be configured in the ’setup’ file of the pool as described in the section called “The pool ’setup’ file”.

The following arguments are given to the executable of a STORE FILE operation on startup.

put <pnfsID> <filename> -si=<storage-information> <more options>

Details on the meaning of certain arguments are described in the section called “Summary of command line options”.

With the arguments provided the executable is supposed to copy the file into the Tertiary Storage System. The executable must not terminate before the transfer of the file was either successful or failed.

Success must be indicated by a 0 return of the executable. All non-zero values are interpreted as a failure which means, dCache assumes that the file has not been copied to tape.

In case of a 0 return code the executable has to return a valid storage URI to dCache in formate:

hsmType://hsmInstance/?store=<storename>&group=<groupname>&bfid=<bfid>

Details on the meaning of certain parameters are described above.

The <bfid> can either be provided by the TSS as result of the STORE FILE operation or the pnfsID may be used. The latter assumes that the file has to be stored with exactly that pnfsID within the TSS. Whatever URI is chosen, it must allow to uniquely identify the file within the Tertiary Storage System.

Note

Only the URI must be printed to stdout by the executable. Additional information printed either before or after the URI will result in an error. stderr can be used for additional debug information or error messages.

[return to top]

The executable and the FETCH FILE operation

Whenever a disk file needs to be restored from a Tertiary Storage System dCache automatically launches an executable on the pool containing the file to be copied. Exactly one instance of the executable is started for each file. Multiple instances of the executable may run concurrently for different files. The maximum number of concurrent instances of the executable per pool as well as the full path of the executable can be configured in the ’setup’ file of the pool as described in the section called “The pool ’setup’ file”.

The following arguments are given to the executable of a FETCH FILE operation on startup:

get <pnfsID> <filename> -si=<storage-information> -uri=<storage-uri> <more options>

Details on the meaning of certain arguments are described in the section called “Summary of command line options”. For return codes see the section called “Summary of return codes”.

[return to top]

The executable and the REMOVE FILE operation

Whenever a file is removed from the dCache namespace (file system) a process inside dCache makes sure that all copies of the file are removed from all internal and external media. The pool which is connected to the TSS which stores the file is activating the executable with the following command line options:

remove -uri=<storage-uri> <more options>

Details on the meaning of certain arguments are described in the section called “Summary of command line options”. For return codes see the section called “Summary of return codes”.

The executable is supposed to remove the file from the TSS and report a zero return code. If a non-zero error code is returned, the dCache will call the script again at a later point in time.