In order to install dCache the following requirements must be met:

The RPM packages may be installed right away, for example using the command:

[root] # rpm -ivh dcache-2.12.0-1.noarch.rpm

The actual sources lie at http://www.dcache.org/downloads/IAgree.shtml. To install for example Version 2.12.0-1 you would use this:

[root] # rpm -ivh http://www.dcache.org/downloads/1.9/repo/2.12/dcache-2.12.0-1.noarch.rpm

The client can be found in the download-section of the above url, too.

Using a PostgreSQL server with dCache places a number of requirements on the database. You must configure PostgreSQL for use by dCache and create the necessary PostgreSQL user accounts and database structure. This section describes how to do this.

[root] # service postgresql-9.2 initdb
Initializing database:                                     [  OK  ]
[root] # service postgresql-9.2 start
Starting postgresql-9.2 service:                           [  OK  ]
[root] # chkconfig postgresql-9.2 on

# TYPE  DATABASE        USER            ADDRESS                 METHOD

# "local" is for Unix domain socket connections only
local   all             all                                     trust
# IPv4 local connections:
host    all             all             127.0.0.1/32            trust
# IPv6 local connections:
host    all             all             ::1/128                 trust

[root] # service postgresql-9.2 restart
Stopping postgresql-9.2 service:                           [  OK  ]
Starting postgresql-9.2 service:                           [  OK  ]

Chimera is a library providing a hierarchical name space with associated meta data. Where pools in dCache store the content of files, Chimera stores the names and meta data of those files. Chimera itself stores the data in a relational database. We will use PostgreSQL in this tutorial. The properties of Chimera are defined in /usr/share/dcache/defaults/chimera.properties. See Chapter 4, Chimera for more information.

[root] # createdb -U postgres chimera
CREATE DATABASE
[root] # createuser -U postgres --no-superuser --no-createrole --createdb --pwprompt chimera
Enter password for new role:
Enter it again:
You do not need to enter a password.

[root] # createuser -U postgres --no-superuser --no-createrole --createdb --pwprompt srmdcache
Enter password for new role:
Enter it again:
You do not need to enter a password.

[root] # createdb -U srmdcache dcache

[root] # createdb -O srmdcache -U postgres billing

[root] # dcache database update
PnfsManager@dCacheDomain:
INFO  - Successfully acquired change log lock
INFO  - Creating database history table with name: databasechangelog
INFO  - Reading from databasechangelog
many more like this...
      

Before the first start of dCache replace the file /etc/dcache/gplazma.conf with an empty file.

[root] # mv /etc/dcache/gplazma.conf /etc/dcache/gplazma.conf.bak
[root] # touch /etc/dcache/gplazma.conf

dCache can be started now.

[root] # dcache start
Starting dCacheDomain done

So far, no configuration of dCache is done, so only the predefined domain is started.

pnfsmanager.default-retention-policy=REPLICA
pnfsmanager.default-access-latency=ONLINE

As the files in /usr/share/dcache/defaults/ do serve as succinct documentation for all available configuration parameters and their default values it is quite useful to have a look at them.

dcache.layout=mylayout

dcache.broker.scheme=none

[dCacheDomain]
[dCacheDomain/admin]
[dCacheDomain/broadcast]
[dCacheDomain/poolmanager]

[dCacheDomain]
[dCacheDomain/broadcast]
[dCacheDomain/loginbroker]
[dCacheDomain/topo]
[dCacheDomain/info]

[namespaceDomain]
[namespaceDomain/pnfsmanager]
[namespaceDomain/cleaner]
[namespaceDomain/dir]

[poolmanagerDomain]
[poolmanagerDomain/poolmanager]

[adminDoorDomain]
[adminDoorDomain/admin]

[httpdDomain]
[httpdDomain/httpd]
[httpdDomain/billing]
[httpdDomain/srm-loginbroker]

[gPlazmaDomain]
[gPlazmaDomain/gplazma]

[<domainname>/pool]
name=<poolname>
path=/path/to/pool
pool.wait-for-files=${path}/data

[root] # dcache pool create /srv/dcache/p1 pool1 poolDomain
Created a pool in /srv/dcache/p1. The pool was added to poolDomain in
file:/etc/dcache/layouts/mylayout.conf.

(local) admin > cd <poolname>
(<poolname>) admin > set gap 2G
(<poolname>) admin > save

[root] # dcache restart
Stopping dCacheDomain 0 1 done
Starting dCacheDomain done
Starting namespaceDomain done
Starting poolmanagerDomain done
Starting adminDoorDomain done
Starting httpdDomain done
Starting gPlazmaDomain done
Starting poolDomain done
[root] # dcache status
DOMAIN            STATUS  PID   USER
dCacheDomain      running 17466 dcache
namespaceDomain   running 17522 dcache
poolmanagerDomain running 17575 dcache
adminDoorDomain   running 17625 dcache
httpdDomain       running 17682 dcache
gPlazmaDomain     running 17744 dcache
poolDomain        running 17798 dcache

Now you can have a look at your dCache via The Web Interface, see the section called “The Web Interface for Monitoring dCache”: http://<httpd.example.org>:2288/, where <httpd.example.org> is the node on which your httpd service is running. For a single node dCache this is the machine on which your dCache is running.

dcache.java.memory.heap=512m
dcache.java.memory.direct=512m

[dCacheDomain]
dcache.java.memory.heap=2048m
dcache.java.memory.direct=0m
...
[utilityDomain]
dcache.java.memory.heap=384m
dcache.java.memory.direct=16m

Installing dCache on several nodes is not much more complicated than installing it on a single node. Think about how dCache should be organised regarding services and domains. Then adapt the layout files, as described in the section called “Defining domains and services”, to the layout that you have in mind. The files /etc/dcache/layouts/head.conf and /etc/dcache/layouts/pool.conf contain examples for a dCache head-node and a dCache pool respectively.

On any other nodes than the head node, the property dcache.broker.host has to be added to the file /etc/dcache/dcache.conf. This property should point to the host containing the special domain dCacheDomain, because that domain acts implicitly as a broker.

Create the root of the Chimera namespace and a world-writable directory by

[root] # /usr/bin/chimera mkdir /data
[root] # /usr/bin/chimera mkdir /data/world-writable
[root] # /usr/bin/chimera chmod 777 /data/world-writable

To use WebDAV you need to define a WebDAV service in your layout file. You can define this service in an extra domain, e.g. [webdavDomain] or add it to another domain.

[webdavDomain]
[webdavDomain/webdav]
webdav.authz.anonymous-operations=FULL

to the file /etc/dcache/layouts/mylayout.conf.

#  ---- Whether to redirect GET requests to a pool
#
#   If true, WebDAV doors will respond with a 302 redirect pointing to
#   a pool holding the file. This requires that a pool can accept
#   incoming TCP connections and that the client follows the
#   redirect. If false, data is relayed through the door. The door
#   will establish a TCP connection to the pool.
#
(one-of?true|false)webdav.redirect.on-read=true

#  ---- Whether to redirect PUT requests to a pool
#
#   If true, WebDAV doors will respond with a 307 redirect pointing to
#   a pool to which to upload the file. This requires that a pool can
#   accept incoming TCP connections and that the client follows the
#   redirect. If false, data is relayed through the door. The door
#   will establish a TCP connection to the pool. Only clients that send
#   a Expect: 100-Continue header will be redirected - other requests
#   will always be proxied through the door.
#
(one-of?true|false)webdav.redirect.on-write=true

Now you can start the WebDAV domain

[root] # dcache start webdavDomain

and access your files via http://<webdav-door.example.org>:2880 with your browser.

You can connect the webdav server to your file manager and copy a file into your dCache.

To use curl to copy a file into your dCache you will need to set webdav.redirect.on-write=false.

[root] # curl -T test.txt http://webdav-door.example.org:2880/data/world-writable/curl-testfile.txt

[root] # curl http://webdav-door.example.org:2880/data/world-writable/curl-testfile.txt

To be able to use dCap you need to have the dCap door running in a domain.

[dCacheDomain]
[dCacheDomain/dcap]

For anonymous access you need to set the property dcap.authz.anonymous-operations to FULL.

[dCacheDomain]
[dCacheDomain/dcap]
      dcap.authz.anonymous-operations=FULL

For this tutorial install dCap on your worker node. This can be the machine where your dCache is running.

Get the gLite repository (which contains dCap) and install dCap using yum.

[root] # cd /etc/yum.repos.d/
[root] # wget http://grid-deployment.web.cern.ch/grid-deployment/glite/repos/3.2/glite-UI.repo
[root] # yum install dcap

Create the root of the Chimera namespace and a world-writable directory for dCap to write into as described above.

Copy the data (here /bin/sh is used as example data) using the dccp command and the dCap protocol describing the location of the file using a URL, where <dcache.example.org> is the host on which the dCache is running

[root] # dccp -H /bin/sh dcap://<dcache.example.org>/data/world-writable/my-test-file-1
[##########################################################################################] 100% 718 kiB
735004 bytes (718 kiB) in 0 seconds

and copy the file back.

[root] # dccp -H dcap://<dcache.example.org>/data/world-writable/my-test-file-1 /tmp/mytestfile1
[##########################################################################################] 100% 718 kiB
735004 bytes (718 kiB) in 0 seconds

To remove the file you will need to mount the namespace.

dCache has a powerful administration interface. It can be accessed with the ssh1 or with the ssh2 protocol. The server is part of the adminDoor domain.

It is useful to define the admin service in a seperate domain. This allowes to restart the admin service seperatly from other services. In the example in the section called “Installing a dCache instance” this domain was called adminDoorDomain.

[adminDoorDomain]
[adminDoorDomain/admin]

[adminDoorDomain]
[adminDoorDomain/ssh1]

There are two ways of authorizing administrators to access the dCache ssh2 admin interface. The preferred method authorizes users through their public key. The second method employs gPlazma2 and the dcache.kpwd file. Thereby authorization mechanisms can be added later by deploying another gPlazma2 plugin. The configuration of both authorization mechanisms is described in the following.

ssh-dss AAAAB3....GWvM= /Users/JohnDoe/.ssh/id_dsa

[user] $ ssh -l admin -p 22224 headnode.example.org

    dCache Admin (VII) (user=admin)


(local) admin >

[<gplazma-${host.name}>Domain]
[<gplazma-${host.name}>Domain/gplazma]

# This is the main configuration file of dCache.
#
...
#
# use gPlazma2
gplazma.version=2

auth optional kpwd "kpwd=/etc/dcache/dcache.kpwd"
map optional kpwd "kpwd=/etc/dcache/dcache.kpwd"
session optional kpwd "kpwd=/etc/dcache/dcache.kpwd"

[user] $ dcache kpwd dcuseradd admin -u 12345 -g 1000 -h / -r / -f / -w read-write -p password
writing to /etc/dcache/dcache.kpwd :

done writing to /etc/dcache/dcache.kpwd :

[user] $

# set pwd
passwd admin 4091aba7 read-write 12345 1000 / /

[user] $ ssh -l admin -p 22224 headnode.example.org
admin@headnode.example.org's password:

    dCache Admin (VII) (user=admin)


(local) admin > 

Connect to the server using ssh1 with:

[user] $ ssh -c blowfish -p 22223 -l admin headnode.example.org

The initial password is “dickerelch” (which is German for “fat elk”) and you will be greeted by the prompt

   dCache Admin (VII) (user=admin)


(local) admin >

The password can now be changed with

(local) admin > cd acm
(acm) admin > create user admin
(acm) admin > set passwd -user=admin <newPasswd> <newPasswd>
(acm) admin > ..
(local) admin > logoff

The command help lists all commands the cell knows and their parameters. However, many of the commands are only used for debugging and development purposes.

Starting from the local prompt ((local) admin >) the command cd takes you to the specified cell. In general the address of a cell is a concatenation of cell name @ symbol and the domain name. cd to a cell by:

(local) admin > cd <cellName>@<domainName>

The domains that are running on the dCache-instance, can be viewed in the layout-configuration (see Chapter 2, Installing dCache). Additionally, there is the topo cell, which keeps track of the instance’s domain topology. If it is running, it can be used to obtain the list of domains the following way:

(local) admin > cd topo
(topo) admin > ls
adminDoorDomain
gsidcapDomain
dcapDomain
utilityDomain
gPlazmaDomain
webdavDomain
gridftpDomain
srmDomain
dCacheDomain
httpdDomain
namespaceDomain
poolDomain
(topo) admin > ..
(local) admin >

The escape sequence .. takes you back to the local prompt.

The command logoff exits the admin shell.

If you want to find out which cells are running on a certain domain, you can issue the command ps in the System cell of the domain.

(local) admin > cd System@poolDomain
(System@poolDomain) admin > ps
  Cell List
------------------
c-dCacheDomain-101-102
System
pool_2
c-dCacheDomain-101
pool_1
RoutingMgr
lm

The cells in the domain can be accessed using cd together with the cell-name scoped by the domain-name. So first, one has to get back to the local prompt, as the cd command will not work otherwise.

(local) admin > cd nonsense
java.lang.IllegalArgumentException: Cannot cd to this cell as it doesn't exist

Login to the routing manager of the dCacheDomain to get a list of all well-known cells you can directly cd to without having to add the domain.

(System@poolDomain) admin > ..
(local) admin > cd RoutingMgr@dCacheDomain
(RoutingMgr@dCacheDoorDomain) admin > ls
Our routing knowledge :
 Local : [PoolManager, topo, broadcast, LoginBroker, info]
 adminDoorDomain : [pam]
 gsidcapDomain : [DCap-gsi-example.dcache.org]
 dcapDomain : [DCap-example.dcache.org]
 utilityDomain : [gsi-pam, PinManager]
 gPlazmaDomain : [gPlazma]
 webdavDomain : [WebDAV-example.dcache.org]
 gridftpDomain : [GFTP-example.dcache.org]
 srmDomain : [RemoteTransferManager, CopyManager, SrmSpaceManager, SRM-example.dcache.org]
 httpdDomain : [billing, srm-LoginBroker, TransferObserver]
 poolDomain : [pool_2, pool_1]
 namespaceDomain : [PnfsManager, dirLookupPool, cleaner]

All cells know the commands info for general information about the cell and show pinboard for listing the last lines of the pinboard of the cell. The output of these commands contains useful information for solving problems.

It is a good idea to get aquainted with the normal output in the following cells: PoolManager, PnfsManager, and the pool cells (e.g., <poolHostname>_1).

The most useful command of the pool cells is rep ls. To execute this command cd into the pool. It lists the files which are stored in the pool by their pnfs IDs:

(RoutingMgr@dCacheDoorDomain) admin > ..
(pool_1) admin > rep ls
000100000000000000001120 <-P---------(0)[0]> 485212 si={myStore:STRING}
000100000000000000001230 <C----------(0)[0]> 1222287360 si={myStore:STRING}

See the section called “How to Store-/Restore files via the Admin Interface” for more information about rep ls.

The most important commands in the PoolManager are: rc ls and cm ls -r.

rc ls lists the requests currently handled by the PoolManager. A typical line of output for a read request with an error condition is (all in one line):

(pool_1) admin > ..
(local) admin > cd PoolManager
(PoolManager) admin > rc ls
000100000000000000001230@0.0.0.0/0.0.0.0 m=1 r=1 [<unknown>]
[Waiting 08.28 19:14:16]
{149,No pool candidates available or configured for 'staging'}

See the section called “Obtain information via the dCache Command Line Admin Interface” for more information about the command rc ls

Finally, cm ls with the option -r gives the information about the pools currently stored in the cost module of the pool manager. A typical output is:

(PoolManager) admin > cm ls -r
pool_1={R={a=0;m=2;q=0};S={a=0;m=2;q=0};M={a=0;m=100;q=0};PS={a=0;m=20;q=0};PC={a=0;m=20;q=0};
    (...continues...)   SP={t=2147483648;f=924711076;p=1222772572;r=0;lru=0;{g=20000000;b=0.5}}}
pool_1={Tag={{hostname=example.org}};size=0;SC=0.16221282938326134;CC=0.0;}
pool_2={R={a=0;m=2;q=0};S={a=0;m=2;q=0};M={a=0;m=100;q=0};PS={a=0;m=20;q=0};PC={a=0;m=20;q=0};
    (...continues...)   SP={t=2147483648;f=2147483648;p=0;r=0;lru=0;{g=4294967296;b=250.0}}}
pool_2={Tag={{hostname=example.org}};size=0;SC=2.7939677238464355E-4;CC=0.0;}

While the first line for each pool gives the information stored in the cache of the cost module, the second line gives the costs (SC: space cost, CC: performance cost) calculated for a (hypothetical) file of zero size. For details on how these are calculated and their meaning, see the section called “Classic Partitions”.

To create a new user, <new-user> and set a new password for the user cd from the local prompt ((local) admin >) to the acm, the access control manager, and run following command sequence:

(local) admin > cd acm
(acm) admin > create user <new-user>
(acm) admin > set passwd -user=<new-user> <newPasswd> <newPasswd>

For the new created users there will be an entry in the directory /etc/dcache/admin/users/meta.

Give the new user access to a particular cell:

(acm) admin > create acl cell.<cellName>.execute
(acm) admin > add access -allowed cell.<cellName>.execute <new-user>

(acm) admin > create acl cell.PnfsManager.execute
(acm) admin > add access -allowed cell.PnfsManager.execute <new-user>

(acm) admin > check cell.PnfsManager.execute <new-user>
Allowed
(acm) admin > show acl cell.PnfsManager.execute
<noinheritance>
<new-user> -> true

The following commands allow access to every cell for a user <new-user>:

(acm) admin > create acl cell.*.execute
(acm) admin > add access -allowed cell.*.execute <new-user>

The following command makes a user as powerful as admin (dCache’s equivalent to the root user):

(acm) admin > create acl *.*.*
(acm) admin > add access -allowed *.*.* <new-user>

The ssh admin interface can be used non-interactively by scripts. For this the dCache-internal ssh server uses public/private key pairs.

The file /etc/dcache/authorized_keys contains one line per user. The file has the same format as ~/.ssh/authorized_keys which is used by sshd. The keys in /etc/dcache/authorized_keys have to be of type RSA1 as dCache only supports SSH protocol 1. Such a key is generated with

[user] $ ssh-keygen -t rsa1 -C 'SSH1 key of <user>'
Generating public/private rsa1 key pair.
Enter file in which to save the key (/home/<user>/.ssh/identity):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/<user>/.ssh/identity.
Your public key has been saved in /home/<user>/.ssh/identity.pub.
The key fingerprint is:
c1:95:03:6a:66:21:3c:f3:ee:1b:8d:cb:46:f4:29:6a SSH1 key of <user>

The passphrase is used to encrypt the private key (now stored in /home/<user>/.ssh/identity). If you do not want to enter the passphrase every time the private key is used, you can use ssh-add to add it to a running ssh-agent. If no agent is running start it with

[user] $ if [ -S $SSH_AUTH_SOCK ] ; then echo "Already running" ; else eval `ssh-agent` ; fi

and add the key to it with

[user] $ ssh-add
Enter passphrase for SSH1 key of <user>:
Identity added: /home/<user>/.ssh/identity (SSH1 key of <user>)

Now, insert the public key ~/.ssh/identity.pub as a separate line into /etc/dcache/authorized_keys. The comment field in this line “SSH1 key of <user>” has to be changed to the dCache user name. An example file is:

1024 35 141939124(... many more numbers ...)15331 admin

Using ssh-add -L >> /etc/dcache/authorized_keys will not work, because the line added is not correct. The key manager within dCache will read this file every minute.

Now, the ssh program should not ask for a password anymore. This is still quite secure, since the unencrypted private key is only held in the memory of the ssh-agent. It can be removed from it with

[user] $ ssh-add -d
Identity removed: /home/<user>/.ssh/identity (RSA1 key of <user>)

In scripts, one can use a “Here Document” to list the commands, or supply them to ssh as standard-input (stdin). The following demonstrates using a Here Document:

#!/bin/sh
#
#  Script to automate dCache administrative activity

outfile=/tmp/$(basename $0).$$.out

ssh -c blowfish -p 22223 admin@<adminNode> > $outfile << EOF
cd PoolManager
cm ls -r
(more commands here)
logoff
EOF

or, the equivalent as stdin.

#!/bin/bash
#
#   Script to automate dCache administrative activity.

echo -e 'cd <pool_1>\nrep ls\n(more commands here)\nlogoff' \
  | ssh -c blowfish -p 22223 admin@<adminNode> \
  | tr -d '\r' > rep_ls.out

To use GSIdCap you must run a GSIdCap door. This is achieved by including the gsidcap service in your layout file on the machine you wish to host the door.

[gsidcapDomain]
[gsidcapDomain/dcap]
dcap.authn.protocol=gsi

In addition, you need to have libdcap-tunnel-gsi installed on your worker node, which is contained in the gLite-UI.

[root] # yum install libdcap-tunnel-gsi

It is also available on the dCap downloads page.

[root] # rpm -i http://www.dcache.org/repository/yum/sl5/x86_64/RPMS.stable//libdcap-tunnel-gsi-2.47.5-0.x86_64.rpm

The machine running the GSIdCap door needs to have a host certificate and you need to have a valid user certificate. In addition, you should have created a voms proxy as mentioned above.

Now you can copy a file into your dCache using GSIdCap

[user] $ dccp /bin/sh gsidcap://<dcache.example.org>:22128/data/world-writable/my-test-file3
801512 bytes in 0 seconds

and copy it back

[user] $ dccp gsidcap://<dcache.example.org>:22128/data/world-writable/my-test-file3 /tmp/mytestfile3.tmp
801512 bytes in 0 seconds

To use the SRM you need to define the srm service in your layout file.

[srmDomain]
[srmDomain/srm]

In addition, the user needs to install an SRM client for example the dcache-srmclient, which is contained in the gLite-UI, on the worker node and set the PATH environment variable.

[root] # yum install dcache-srmclient

You can now copy a file into your dCache using the SRM,

[user] $ srmcp -2 file:////bin/sh srm://<dcache.example.org>:8443/data/world-writable/my-test-file4

copy it back

[user] $ srmcp -2 srm://<dcache.example.org>:8443/data/world-writable/my-test-file4 file:////tmp/mytestfile4.tmp

and delete it

[user] $ srmrm -2 srm://<dcache.example.org>:8443/data/world-writable/my-test-file4

If the grid functionality is not required the file can be deleted with the NFS mount of the Chimera namespace:

[user] $ rm /data/world-writable/my-test-file4

To use WebDAV with certificates you change the entry in /etc/dcache/layouts/mylayout.conf from

[webdavDomain]
[webdavDomain/webdav]
webdav.authz.anonymous-operations=FULL
webdav.root=/data/world-writable

to

[webdavDomain]
[webdavDomain/webdav]
webdav.authz.anonymous-operations=NONE
webdav.root=/data/world-writable
webdav.authn.protocol=https

Then you will need to import the host certificate into the dCache keystore using the command

[root] # dcache import hostcert

and initialise your truststore by

[root] # dcache import cacerts

Now you need to restart the WebDAV domain

[root] # dcache restart webdavDomain

and access your files via https://<dcache.example.org>:2880 with your browser.

[webdavDomain]
[webdavDomain/webdav]
webdav.root=/data/world-writable
webdav.authz.anonymous-operations=READONLY
port=2880
webdav.authn.protocol=https

If you plan to use dCap with a mounted file system instead of the URL-syntax (e.g. dccp /data/file1 /tmp/file1), you need to mount the root of Chimera locally (remote mounts are not allowed yet). This will allow us to establish wormhole files so dCap clients can discover the dCap doors.

[root] # mount localhost:/ /mnt
[root] # mkdir /mnt/admin/etc/config/dCache
[root] # touch /mnt/admin/etc/config/dCache/dcache.conf
[root] # touch /mnt/admin/etc/config/dCache/'.(fset)(dcache.conf)(io)(on)'
[root] # echo "<door host>:<port>" > /mnt/admin/etc/config/dCache/dcache.conf

The default values for ports can be found in Chapter 29, dCache Default Port Values (for dCap the default port is 22125) and in the file /usr/share/dcache/defaults/dcache.properties. They can be altered in /etc/dcache/dcache.conf

Create the directory in which the users are going to store their data and change to this directory.

[root] # mkdir -p /mnt/data
[root] # cd /mnt/data

Now you can copy a file into your dCache

[root] # dccp /bin/sh test-file
735004 bytes (718 kiB) in 0 seconds

and copy the data back using the dccp command.

[root] # dccp test-file /tmp/testfile
735004 bytes (718 kiB) in 0 seconds

The file has been transferred succesfully.

Now remove the file from the dCache.

[root] # rm  test-file

When the configuration is complete you can unmount Chimera:

[root] # umount /mnt

You can create tags with

[user] $ /usr/bin/chimera writetag <directory> <tagName> "<content>"

list tags with

[user] $ /usr/bin/chimera lstag <directory> 

and read tags with

[user] $ /usr/bin/chimera readtag <directory> <tagName>

[user] $ /usr/bin/chimera writetag /data sGroup "myGroup"
[user] $ /usr/bin/chimera writetag /data OSMTemplate "StoreName myStore"

[user] $ /usr/bin/chimera lstag /data
Total: 2
OSMTemplate
sGroup

[user] $ /usr/bin/chimera readtag /data OSMTemplate
StoreName myStore
[user] $ /usr/bin/chimera readtag /data sGroup
myGroup

If the namespace is mounted, change to the directory for which the tag should be set and create a tag with

[user] $ cd <directory>
[user] $ echo '<content1>' > '.(tag)(<tagName1>)'
[user] $ echo '<content2>' > '.(tag)(<tagName2>)'

Then the existing tags may be listed with

[user] $ cat '.(tags)()'
.(tag)(<tagname1>)
.(tag)(<tagname2>)

and the content of a tag can be read with

[user] $ cat '.(tag)(<tagname1>)'
<content1>
[user] $ cat '.(tag)(<tagName2>)'
<content2>

[user] $ cd data
[user] $ echo 'StoreName myStore' > '.(tag)(OSMTemplate)'
[user] $ echo 'myGroup' > '.(tag)(sGroup)'

[user] $ cat '.(tags)()'
.(tag)(OSMTemplate)
.(tag)(sGroup)

[user] $ cat '.(tag)(OSMTemplate)'
StoreName myStore
[user] $ cat '.(tag)(sGroup)'
 myGroup

[user] $ grep "" $(cat  ".(tags)()")
.(tag)(OSMTemplate):StoreName myStore
.(tag)(sGroup):myGroup

When creating or changing directory tags by writing to the command file as in

[user] $ echo '<content>' > '.(tag)(<tagName>)'

one has to take care not to treat the command files in the same way as regular files, because tags are different from files in the following aspects:

The following directory tags appear in the dCache context:

The storage class is a string of the form <StoreName>:<StorageGroup>@<hsm-type>, where <StoreName> is given by the OSMTemplate tag, <StorageGroup> by the sGroup tag and <hsm-type> by the HSMType tag. As mentioned above the HSMType tag is assumed to be osm if the tag OSMTemplate exists.

In the examples above two tags have been created.

[user] $ /usr/bin/chimera lstag /data
Total: 2
OSMTemplate
sGroup

If directory tags are used to control the behaviour of dCache and/or a tertiary storage system, it is a good idea to plan the directory structure in advance, thereby considering the necessary tags and how they should be set up. Moving directories should be done with great care or even not at all. Inherited tags can only be created by creating a new directory.

[user] $ /usr/bin/chimera writetag /data/experiment-a OSMTemplate "StoreName exp-a"
[user] $ /usr/bin/chimera writetag /data/experiment-b OSMTemplate "StoreName exp-b"

[user] $ /usr/bin/chimera writetag /data/experiment-a/2010 sGroup "run2010"
[user] $ /usr/bin/chimera writetag /data/experiment-a/2011 sGroup "run2011"
[user] $ /usr/bin/chimera writetag /data/experiment-b sGroup "alldata"

[user] $ /usr/bin/chimera readtag /data/experiment-a/2010 OSMTemplate
StoreName exp-a
[user] $ /usr/bin/chimera readtag /data/experiment-a/2010 sGroup
run2010
[user] $ /usr/bin/chimera readtag /data/experiment-a/2011 OSMTemplate
StoreName exp-a
[user] $ /usr/bin/chimera readtag /data/experiment-a/2011 sGroup
run2011
[user] $ /usr/bin/chimera readtag /data/experiment-b/2011 OSMTemplate
StoreName exp-b
[user] $ /usr/bin/chimera readtag /data/experiment-b/2011 sGroup
alldata

There are more tags used by dCache if the HSMType is enstore.

For more complex installations of dCache you might want to define a pool group for the resilient pools.

Define the resilient pool group in the /var/lib/dcache/config/poolmanager.conf file on the host running the poolmanager service. Only pools defined in the resilient pool group will be managed by the replica service.

(local) admin > cd PoolManager
(PoolManager) admin > psu create pgroup ResilientPools
(PoolManager) admin > psu create pool  pool3
(PoolManager) admin > psu create pool  pool4
(PoolManager) admin > psu addto pgroup ResilientPools pool3
(PoolManager) admin > psu addto pgroup ResilientPools pool4
(PoolManager) admin > save

To use another pool group defined in /var/lib/dcache/config/poolmanager.conf for replication, please specify the group name in the etc/dcache.conf file.

replica.poolgroup=<NameOfResilientPoolGroup>.

The possible states of a pool are online, down, offline, offline-prepare and drainoff. They can be set by the admin through the admin interface. (See the section called “Commands for the admin interface”.)

Figure 6.1. Pool State Diagram

When the replica service starts it cleans up the database. Then it waits for some time to give a chance to most of the pools in the system to connect. Otherwise unnecessary massive replication would start. Currently this is implemented by some delay to start adjustments to give the pools a chance to connect.

For security reasons you might want to spread your replicas such that they are not on the same host, or in the same building or even in the same town. To configure this you need to set the tag.hostname label for your pools and check the properties replica.enable.check-pool-host and replica.enable.same-host-replica.

[poolDomain]
[poolDomain/pool1]
name=pool1
path=/srv/dcache/p1
pool.size=500G
pool.wait-for-files=${path}/data
tag.hostname=Hamburg

[poolDomain]
[poolDomain/pool2]
name=pool2
path=/srv/dcache/p2
pool.size=500G
pool.wait-for-files=${path}/data
tag.hostname=Berlin

By default the property replica.enable.check-pool-host is true and replica.enable.same-host-replica is false. This means that the tag.hostname will be checked and the replication to a pool with the same tag.hostname is not allowed.

A hybrid dCache operates on a combination of pools (maybe connected to tape) which are not in a resilient pool group and the set of resilient pools. The replica service takes care only of the subset of pools configured in the pool group for resilient pools and ignores all other pools.

If you are an advanced user, have proper privileges and you know how to issue a command to the admin interface you may connect to the ReplicaManager cell and issue the following commands. You may find more commands in online help which are for debug only — do not use them as they can stop replica service operating properly.

A link consists of a set of unit groups and a list of pools. If all the unit groups are matched, the pools belonging to the link are added to the list of allowable pools.

A link is defined in the file /var/lib/dcache/config/poolmanager.conf by

For the preference values see the section called “Preference Values for Type of Transfer”.

The main task is to understand how the unit groups in a link are defined. After we have dealt with that, the preference values will be discussed and a few examples will follow.

The four properties of a transfer request, which are relevant for the PSU, are the following:

Each link contains one or more unit groups, all of which have to be matched by the transfer request. Each unit group in turn contains several units. The unit group is matched if at least one of the units is matched.

psu create pgroup <name-of-poolgroup>
psu create pool <name-of-pool>
psu addto pgroup <name-of-poolgroup> <name-of-pool>

psu create pgroup normal-pools
psu create pool pool1_1
psu addto pgroup normal-pools pool1_1
psu create pool pool1_2
psu addto pgroup normal-pools pool1_2
psu create pool pool2_1
psu addto pgroup normal-pools pool2_1

psu removefrom pgroup normal-pools pool1_2
psu create pgroup special-pools
psu addto pgroup special-pools pool1_2

[root] # /usr/bin/chimera lstag /data/experiment-a
Total: 2
OSMTemplate
sGroup
[root] # /usr/bin/chimera readtag /data/experiment-a OSMTemplate
StoreName myStore
[root] # /usr/bin/chimera readtag /data/experiment-a sGroup
STRING

[root] # /usr/bin/chimera writetag /data/experiment-a OSMTemplate "StoreName exp-a"
[root] # /usr/bin/chimera writetag /data/experiment-a sGroup "run2010"

[root] # /usr/bin/chimera writetag /data/experiment-a cacheClass "metaData"

[root] # /usr/bin/chimera lstag /path/to/directory
Total: numberOfTags
tag1
tag2
...
[root] # /usr/bin/chimera readtag /path/to/directory tag1
contentOfTag1

psu create ugroup <name-of-unitgroup>
psu create unit - <type> <unit>
psu addto ugroup <name-of-unitgroup> <unit>

psu create pgroup <poolgroup>
psu create pool <pool>
psu addto pgroup <poolgroup> <pool>

psu create link <link> <name-of-unitgroup>
psu set link <link> -readpref=<10> -writepref=<0> -cachepref=<10>-p2ppref=<-1>
psu add link <link>  <poolgroup>

Find some examples for the configuration of the PSU below.

psu create pgroup read-pools
psu create pool pool1
psu addto pgroup read-pools pool1
psu create pgroup write-pools
psu create pool pool2
psu addto pgroup write-pools pool2

psu create unit -net 0.0.0.0/0.0.0.0
psu create ugroup allnet-cond
psu addto ugroup allnet-cond 0.0.0.0/0.0.0.0

psu create link read-link allnet-cond
psu set link read-link -readpref=10 -writepref=0 -cachepref=10
psu add link read-link read-pools

psu create link write-link allnet-cond
psu set link write-link -readpref=0 -writepref=10 -cachepref=0
psu add link write-link write-pools

psu create pgroup read-pools
psu create pool pool1
psu addto pgroup read-pools pool1
psu create pgroup write-pools
psu create pool pool2
psu addto pgroup write-pools pool2

psu create unit -store *@*

psu create unit -net 111.111.111.0/255.255.255.0
psu create unit -net 111.111.111.201/255.255.255.255
psu create unit -net 111.111.111.202/255.255.255.255
psu create unit -net 111.111.111.203/255.255.255.255

psu create ugroup write-cond
psu addto ugroup write-cond 111.111.111.201/255.255.255.255
psu addto ugroup write-cond 111.111.111.202/255.255.255.255
psu addto ugroup write-cond 111.111.111.203/255.255.255.255

psu create ugroup read-cond
psu addto ugroup read-cond 111.111.111.0/255.255.255.0
psu addto ugroup read-cond 111.111.111.201/255.255.255.255
psu addto ugroup read-cond 111.111.111.202/255.255.255.255
psu addto ugroup read-cond 111.111.111.203/255.255.255.255

psu create link read-link read-cond
psu set link read-link -readpref=10 -writepref=0 -cachepref=10
psu add link read-link read-pools

psu create link write-link write-cond
psu set link write-link -readpref=0 -writepref=10 -cachepref=0
psu add link write-link write-pools

psu create pgroup exp-a-pools
psu create pool pool1
psu addto pgroup exp-a-pools pool1

psu create pgroup exp-b-pools
psu create pool pool2
psu addto pgroup exp-b-pools pool2

psu create pgroup exp-b-imp-pools
psu create pool pool3
psu addto pgroup exp-b-imp-pools pool3

psu create unit -net 111.111.111.0/255.255.255.0
psu create ugroup allnet-cond
psu addto ugroup allnet-cond 111.111.111.0/255.255.255.0

psu create ugroup exp-a-cond
psu create unit -store exp-a:run2011@osm
psu addto ugroup exp-a-cond exp-a:run2011@osm
psu create unit -store exp-a:run2010@osm
psu addto ugroup exp-a-cond exp-a:run2010@osm

psu create link exp-a-link allnet-cond exp-a-cond
psu set link exp-a-link -readpref=10 -writepref=10 -cachepref=10
psu add link exp-a-link exp-a-pools

psu create ugroup exp-b-cond
psu create unit -store exp-b:alldata@osm
psu addto ugroup exp-b-cond exp-b:alldata@osm

psu create ugroup imp-cond
psu create unit -dcache important
psu addto ugroup imp-cond important

psu create link exp-b-link allnet-cond exp-b-cond
psu set link exp-b-link -readpref=10 -writepref=10 -cachepref=10
psu add link exp-b-link exp-b-pools

psu create link exp-b-imp-link allnet-cond exp-b-cond imp-cond
psu set link exp-b-imp-link -readpref=20 -writepref=20 -cachepref=20
psu add link exp-b-link exp-b-imp-pools

psu create pgroup it-pools
psu create pool pool_it
psu addto pgroup it-pools pool_it

psu create link fallback-link allnet-cond
psu set link fallback-link -readpref=5 -writepref=5 -cachepref=5
psu add link fallback-link it-pools

There are various parameters that affect the load balancing policy. Some of them are generic and apply to any load balancing policy, but many are specific to a particular policy. To avoid limiting the complete dCache instance to a single configuration, the choice of load balancing policy and the various parameters apply to partitions of the instance. The load balancing algorithm and the available parameters is determined by the partition type.

Each PSU link can be associated with a different partion and the policy and parameters of that partition will be used to choose a pool from the set of candidate pools. The only partition that exists without being explicitly created is the partition called default. This partition is used by all links that do not explicitly identify a partition. Other partitions can be created or modified as needed.

The default partition has a hard-coded partition type called classic. This type implements the one load balancing policy that was available in dCache before version 2.0. The classic partition type is described later. Other partitions have one of a number of available types. The system is pluggable, meaning that third party plugins can be loaded at runtime and add additional partition types, thus providing the ultimate control over load balancing in dCache. Documentation on how to develop plugins is beyond the scope of this chapter.

To ease the management of partition parameters, a common set of shared parameters can be defined outside all partitions. Any parameter not explicitly set on a partition inherits the value from the common set. If not defined in the common set, a default value determined by the partition type is used. Currently, the common set of parameters happens to be the same as the parameters of the default partition, however this is only due to compatibility constraints and may change in future versions.

For each partition you can choose the load balancing policy. You do this by chosing the type of the partition.

Currently four different partition types are supported:

Commands related to dCache partitioning:

A partition, so far, is just a set of parameters which may or may not differ from the default set. To let a partition relate to a part of the dCache, links are used. Each link may be assigned to exactly one partition. If not set, or the assigned partition doesn’t exist, the link defaults to the default partition.

Whenever this link is chosen for pool selection, the associated parameters of the assigned partition will become active for further processing.

In the Web Interface you can find a web page listing partitions and more information. You will find a page summarizing the partition status of the system. This is essentially the output of the command pm ls -l.

#
# define the units
#
psu create unit -protocol   */*
psu create unit -protocol   xrootd/*
psu create unit -net        0.0.0.0/0.0.0.0
psu create unit -net        131.169.0.0/255.255.0.0
psu create unit -store      *@*
#
#  define unit groups
#
psu create ugroup  any-protocol
psu create ugroup  any-store
psu create ugroup  world-net
psu create ugroup  xrootd
#
psu addto ugroup any-protocol */*
psu addto ugroup any-store    *@*
psu addto ugroup world-net    0.0.0.0/0.0.0.0
psu addto ugroup desy-net     131.169.0.0/255.255.0.0
psu addto ugroup xrootd       xrootd/*
#
#  define the pools
#
psu create pool pool1
psu create pool pool2
psu create pool pool3
psu create pool pool4

#
#  define the pool groups
#
psu create pgroup default-pools
psu create pgroup special-pools
#
psu addto pgroup default-pools pool1
psu addto pgroup default-pools pool2
#
psu addto pgroup special-pools pool3
psu addto pgroup special-pools pool4
#

#
pm create xrootd-section
#
pm set default        -p2p=0.4
pm set xrootd-section -p2p=0.0
#
psu create link default-link any-protocol any-store world-net
psu add    link default-link default-pools
psu set    link default-link -readpref=10 -cachepref=10 -writepref=0
#
psu create link xrootd-link xrootd any-store world-net
psu add    link xrootd-link special-pools
psu set    link xrootd-link -readpref=10 -cachepref=10 -writepref=0
psu set    link xrootd-link -section=xrootd-section
#        

#
pm create incoming-section
#
pm set default          -cpucostfactor=0.2 -spacecostfactor=1.0
pm set incoming-section -cpucostfactor=0.0 -spacecostfactor=0.0
#
psu create link default-link any-protocol any-store desy-net
psu add    link default-link default-pools
psu set    link default-link -readpref=10 -cachepref=10 -writepref=10
#
psu create link default-link any-protocol any-store world-net
psu add    link default-link default-pools
psu set    link default-link -readpref=10 -cachepref=10 -writepref=0
#
psu create link incoming-link any-protocol any-store world-net
psu add    link incoming-link default-pools
psu set    link incoming-link -readpref=10 -cachepref=10 -writepref=10
psu set    link incoming-link -section=incoming-section
#

The classic partition type implements the load balancing policy known from dCache releases before version 2.0. This partition type is still the default. This section describes this load balancing policy and the available configuration parameters.

(PoolManager) admin > set pool decision -spacecostfactor=3 -cpucostfactor=1

                  pm set default -spacecostfactor=0.3
              

Most migrating storage systems provide a regular POSIX file system interface. Based on rules, data is migrated from primary to tertiary storage (mostly tape systems). Examples for migrating storage systems are:

Some tape systems provide a simple PUT, GET, REMOVE interface. Typically, a copy-like application writes a disk file into the TSS and returns an identifier which uniquely identifies the written file within the Tertiary Storage System. The identifier is sufficient to get the file back to disk or to remove the file from the TSS. Examples are:

This part explains the syntax of calling the executable that supports STORE FILE, FETCH FILE and REMOVE FILE operations.

-si=size=<bytes>;new=<true/false>;stored=<true/false>;sClass=<StorageClass>;\
cClass0<CacheClass>;hsm=<StorageType>;<key>=<value>;[<key>=<value>;[...]]

-si=size=1048576000;new=true;stored=false;sClass=desy:cms-sc3;cClass=-;hsm=osm;Host=desy;

hsmType://hsmInstance/?store=<storename>&group=<groupname>&bfid=<bfid>

osm://osm/?store=sql&group=chimera&bfid=3434.0.994.1188400818542

Whenever a disk file needs to be copied to a Tertiary Storage System dCache automatically launches an executable on the pool containing the file to be copied. Exactly one instance of the executable is started for each file. Multiple instances of the executable may run concurrently for different files. The maximum number of concurrent instances of the executables per pool as well as the full path of the executable can be configured in the ’setup’ file of the pool as described in the section called “The pool ’setup’ file”.

The following arguments are given to the executable of a STORE FILE operation on startup.

Details on the meaning of certain arguments are described in the section called “Summary of command line options”.

With the arguments provided the executable is supposed to copy the file into the Tertiary Storage System. The executable must not terminate before the transfer of the file was either successful or failed.

Success must be indicated by a 0 return of the executable. All non-zero values are interpreted as a failure which means, dCache assumes that the file has not been copied to tape.

In case of a 0 return code the executable has to return a valid storage URI to dCache in formate:

hsmType://hsmInstance/?store=<storename>&group=<groupname>&bfid=<bfid>

Details on the meaning of certain parameters are described above.

The <bfid> can either be provided by the TSS as result of the STORE FILE operation or the pnfsID may be used. The latter assumes that the file has to be stored with exactly that pnfsID within the TSS. Whatever URI is chosen, it must allow to uniquely identify the file within the Tertiary Storage System.

Whenever a disk file needs to be restored from a Tertiary Storage System dCache automatically launches an executable on the pool containing the file to be copied. Exactly one instance of the executable is started for each file. Multiple instances of the executable may run concurrently for different files. The maximum number of concurrent instances of the executable per pool as well as the full path of the executable can be configured in the ’setup’ file of the pool as described in the section called “The pool ’setup’ file”.

The following arguments are given to the executable of a FETCH FILE operation on startup:

Details on the meaning of certain arguments are described in the section called “Summary of command line options”. For return codes see the section called “Summary of return codes”.

Whenever a file is removed from the dCache namespace (file system) a process inside dCache makes sure that all copies of the file are removed from all internal and external media. The pool which is connected to the TSS which stores the file is activating the executable with the following command line options:

Details on the meaning of certain arguments are described in the section called “Summary of command line options”. For return codes see the section called “Summary of return codes”.

The executable is supposed to remove the file from the TSS and report a zero return code. If a non-zero error code is returned, the dCache will call the script again at a later point in time.

After restarting the necessary dCache domains, pools, already containing files, will start transferring them into the TSS as those files only have a disk copy so far. The number of transfers is determined by the configuration in the pool ’setup’ file as described above in the section called “The pool ’setup’ file”.

By default dCache is configured to only log information if something unexpected happens. However, to get familiar with Tertiary Storage System interactions you might be interested in more details. This section provides advice on how to obtain this kind of information.

<threshold>
     <appender>stdout</appender>
     <logger>root</logger>
     <level>warn</level>
   </threshold>

<threshold>
     <appender>stdout</appender>
     <logger>root</logger>
     <level>info</level>
   </threshold>

[example.dcache.org] (local) admin > cd PoolManager
[example.dcache.org] (PoolManager) admin > log set stdout ROOT INFO
[example.dcache.org] (PoolManager) admin > log ls
stdout:
  ROOT=INFO
  dmg.cells.nucleus=WARN*
  logger.org.dcache.cells.messages=ERROR*
.....

The dCache Command Line Admin Interface gives access to information describing the process of storing and fetching files to and from the TSS, as there are:

For evaluation purposes, the pinboard of each component can be used to track down dCache behavior. The pinboard only keeps the most recent 200 lines of log information but reports not only errors but informational messages as well.

[example.dcache.org] (local) admin > cd PoolManager
[example.dcache.org] (PoolManager) admin > show pinboard 100
08.30.45  [Thread-7] [pool_1 PoolManagerPoolUp] sendPoolStatusRelay: ...
08.30.59  [writeHandler] [NFSv41-dcachetogo PoolMgrSelectWritePool ...
....

[example.dcache.org] (pool_1) admin > rep rm  00002A9282C2D7A147C68A327208173B81A6

[user] $ dccp dcap://example.dcache.org:/data/test.root test.root

[example.dcache.org] (local) admin > cd PoolManager
[example.dcache.org] (PoolManager) admin > rc ls
0000AB1260F474554142BA976D0ADAF78C6C@0.0.0.0/0.0.0.0-*/* m=1 r=0 [pool_1] [Staging 08.15 17:52:16] {0,}

[example.dcache.org] (local) admin > cd pool_1
[example.dcache.org] (pool_1) admin > queue ls -l queue
                   Name: chimera:alpha
              Class@Hsm: chimera:alpha@osm
 Expiration rest/defined: -39 / 0   seconds
 Pending   rest/defined: 1 / 0
 Size      rest/defined: 877480 / 0
 Active Store Procs.   :  0
  00001BC6D76570A74534969FD72220C31D5D


[example.dcache.org] (pool_1) admin > flush ls
Class                 Active   Error  Last/min  Requests    Failed
dteam:STATIC@osm           0       0         0         1         0

[example.dcache.org] (local) admin > cd pool_1
[example.dcache.org] (pool_1) admin > st ls
0000EC3A4BFCA8E14755AE4E3B5639B155F9  1   Fri Aug 12 15:35:58 CEST 2011

[example.dcache.org] (local) admin > cd pool_1
[example.dcache.org] (pool_1) admin >  rh ls
0000B56B7AFE71C14BDA9426BBF1384CA4B0  0   Fri Aug 12 15:38:33 CEST 2011

To check the repository on the pools run the command rep ls that is described in the beginning of the section called “How to Store-/Restore files via the Admin Interface”.