[user] $ ssh -c blowfish -p 22223 -l admin headnode.example.org
Connection closed by 192.0.2.11

dCache has a powerful administration interface. It is accessed with the ssh protocol. The server is part of the adminDoor domain. Connect to it with

[user] $ ssh -c blowfish -p 22223 -l admin headnode.example.org

The initial password is “dickerelch” (which is German for “fat elk”) and you will be greeted by the prompt

   dCache Admin (VII) (user=admin)


(local) admin >

The password can now be changed with

(local) admin > cd acm
(acm) admin > create user admin
(acm) admin > set passwd -user=admin <newPasswd> <newPasswd>
(acm) admin > ..
(local) admin > logoff

This already illustrates how to navigate within the administration interface: Starting from the local prompt ((local) admin >) the command cd takes you to the specified cell (here acm, the access control manager). There two commands are executed. The escape sequence .. takes you back to the local prompt and logoff exits the admin shell.

Note that cd only works from the local prompt. If the cell you are trying to access does not exist, the cd command will not complain. However, trying to execute any command subsequently will result in an error message “No Route to cell...”. Type .. to return to the (local) admin > prompt.

To create a new user, <new-user>, set a new password and to give him/her an access to a particular cell (for example to the PoolManager) run following command sequence:

(local) admin > cd acm
(acm) admin > create user <new-user>
(acm) admin > set passwd -user=<new-user> <newPasswd> <newPasswd>
(acm) admin > create acl cell.PoolManager.execute
(acm) admin > add access -allowed cell.PnfsManager.execute 

Now you can check the permissions by:

(acm) admin > check cell.PnfsManager.execute <new-user>
Allowed
(acm) admin > show acl cell.PnfsManager.execute <new-user>
<noinheritance>
<new-user> -> true

Following commands allow to a particular user an access to every cell:

(acm) admin >  create acl cell.*.execute
(acm) admin > add access -allowed cell.*.execute <new-user>

To make an user as powerful as admin (dCache’s equivalent to the root user):

(acm) admin > create acl *.*.*
(acm) admin > add access -allowed *.*.* <new-user>

All cells know the commands info for general information about the cell and show pinboard for listing the last lines of the pinboard of the cell. The output of these commands contains useful information for solving problems. It is a good idea to get aquainted with the normal output in the following cells: PoolManager, PnfsManager, and the pool cells (e.g., <poolHostname>_1).

If you want to find out which cells are running on a certain domain, you can issue the command ps in the System cell of the domain. For example, if you want to list the cells running on the adminDoor, cd to its System cell and issue the ps command.

(local) admin > cd System@adminDoorDomain
(System@adminDoorDomain) admin > ps
  Cell List
------------------
acm
alm
skm
c-dCacheDomain-101-102
System
c-dCacheDomain-101
RoutingMgr
alm-admin-103
pam
lm

The cells in the domain can be accessed using cd together with the cell-name scoped by the domain-name. So first, one has to get back to the local prompt, as the cd command will not work otherwise.

(System@adminDoorDomain) admin > ..
(local) admin > cd skm@adminDoorDomain
(skm) admin >

The domains that are running on the dCache-instance, can be viewed in the layout-configuration (see Chapter 2, Installing dCache). Additionally, there is the topo cell, which keeps track of the instance’s domain topology. If it is running, it can be used to obtain the list of domains the following way:

(local) admin > cd topo
(topo) admin > ls
dirDomain
infoDomain
adminDoorDomain
spacemanagerDomain
utilityDomain
gPlazmaDomain
nfsDomain
dCacheDomain
httpdDomain
statisticsDomain
namespaceDomain

There also is the command help for listing all commands the cell knows and their parameters. However, many of the commands are only used for debugging and development purposes. Only commands described in this documentation should be used for the administration of a dCache system.

The most useful command of the pool cells is rep ls. It lists the files which are stored in the pool by their pnfs IDs:

000100000000000000001120 <-P---------(0)[0]> 485212 si={myStore:STRING}
000100000000000000001230 <C----------(0)[0]> 1222287360 si={myStore:STRING}

Each file in a pool has one of the 4 primary states: “cached” (<C---), “precious” (<-P--), “from client” (<--C-), and “from store” (<---S).

Two commands in the pool manager are quite useful: rc ls lists the requests currently handled by the pool manager. A typical line of output for a read request with an error condition is (all in one line):

000100000000000000001230@0.0.0.0/0.0.0.0 m=1 r=1 [<unknown>]
  [Waiting 08.28 19:14:16]
  {149,No pool candidates available or configured for 'staging'}

As the error message at the end of the line indicates, no pool was found containing the file and no pool could be used for staging the file from a tertiary storage system.

Finally, cm ls with the option -r gives the information about the pools currently stored in the cost module of the pool manager. A typical output is:

(PoolManager) admin > cm ls -r
<poolName1>={R={a=0;m=2;q=0};S={a=0;m=2;q=0};M={a=0;m=100;q=0};PS={a=0;m=20;q=0};PC={a=0;m=20;q=0};
    (...continues...)   SP={t=2147483648;f=924711076;p=1222772572;r=0;lru=0;{g=20000000;b=0.5}}}
<poolName1>={Tag={{hostname=<hostname>}};size=0;SC=0.16221282938326134;CC=0.0;}
<poolName2>={R={a=0;m=2;q=0};S={a=0;m=2;q=0};M={a=0;m=100;q=0};PS={a=0;m=20;q=0};PC={a=0;m=20;q=0};
    (...continues...)   SP={t=2147483648;f=2147483648;p=0;r=0;lru=0;{g=4294967296;b=250.0}}}
<poolName2>microcebus_2={Tag={{hostname=<hostname>}};size=0;SC=2.7939677238464355E-4;CC=0.0;}

While the first line for each pool gives the information stored in the cache of the cost module, the second line gives the costs (SC: space cost, CC: performance cost) calculated for a (hypothetical) file of zero size. For details on how these are calculated and their meaning, see the section called “The Cost Module”.

The ssh admin interface can be used non-interactively by scripts. For this the dCache-internal ssh server uses public/private key pairs.

The file config/authorized_keys contains one line per user. The file has the same format as ~/.ssh/authorized_keys which is used by sshd. The keys in config/authorized_keys have to be of type RSA1 as dCache only supports SSH protocol 1. Such a key is generated with

[user] $ ssh-keygen -t rsa1 -C 'SSH1 key of <user>'
Generating public/private rsa1 key pair.
Enter file in which to save the key (/home/<user>/.ssh/identity):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/<user>/.ssh/identity.
Your public key has been saved in /home/<user>/.ssh/identity.pub.
The key fingerprint is:
c1:95:03:6a:66:21:3c:f3:ee:1b:8d:cb:46:f4:29:6a SSH1 key of <user>

The passphrase is used to encrypt the private key (now stored in /home/<user>/.ssh/identity). If you do not want to enter the passphrase every time the private key is used, you can use ssh-add to add it to a running ssh-agent. If no agent is running start it with

[user] $ if [ -S $SSH_AUTH_SOCK ] ; then echo "Already running" ; else eval `ssh-agent` ; fi

and add the key to it with

[user] $ ssh-add
Enter passphrase for SSH1 key of <user>:
Identity added: /home/<user>/.ssh/identity (SSH1 key of <user>)

Now, insert the public key ~/.ssh/identity.pub as a separate line into config/authorized_keys. The comment field in this line “SSH1 key of <user>” has to be changed to the dCache user name. An example file is:

1024 35 141939124(... many more numbers ...)15331 admin

The key manager within dCache will read this file every minute.

Now, the ssh program should not ask for a password anymore. This is still quite secure, since the unencrypted private key is only held in the memory of the ssh-agent. It can be removed from it with

[user] $ ssh-add -d
Identity removed: /home/<user>/.ssh/identity (RSA1 key of <user>)

In scripts, one can use a “Here Document” to list the commands, or supply them to ssh as standard-input (stdin). The following demonstrates using a Here Document:

#!/bin/sh
#
#  Script to automate dCache administrative activity

outfile=/tmp/$(basename $0).$$.out

ssh -c blowfish -p 22223 admin@<adminNode> > $outfile << EOF
cd PoolManager
cm ls -r
(more commands here)
logoff
EOF

or, the equivalient as stdin.

#!/bin/bash
#
#   Script to automate dCache administrative activity.

echo -e 'cd <pool_1>\nrep ls\n(more commands here)\nlogoff' \
  | ssh -c blowfish -p 22223 admin@<adminNode> \
  | tr -d '\r' > rep_ls.out