release notes | Book: 1.9.5, 1.9.12 (opt, FHS), 2.11 (FHS), 2.12 (FHS), 2.13 (FHS), 2.14 (FHS), | Wiki | Q&A black_bg
Web: Multi-page, Single page | PDF: A4-size, Letter-size | eBook: epub black_bg

Installing a dCache instance

In the following the installation of a dCache instance will be described. The Chimera name space provider, some management components, and the SRM need a PostgreSQL server installed. We recommend running this PostgreSQL on the local node. The first section describes the configuration of a PostgreSQL server. After that the installation of Chimera and of the dCache components will follow. During the whole installation process root access is required.

[return to top]

Prerequisites

In order to install dCache the following requirements must be met:

  • An RPM-based Linux distribution is required for the following procedure. For Debian derived systems the RPM may be converted to a DEB using alien. Solaris is supported using either the Solaris package or the tarball.

  • dCache requires Java 1.6 JRE. Please use Sun JVM at the latest patch-level (at the moment of writing this is 1.6.0_25) and make sure to upgrade to the newest version frequently. It is recommendet to use JDK as dCache scripts can make use of some extra features that JDK provides to gather more diagnostic information (heap-dump, etc). This helps when tracking down bugs.

  • PostgreSQL must be installed and running. See the section called “Installing a PostgreSQL Server” for more details. PostgreSQL version 8.3 or later is required.

[return to top]

Installation of the dCache Software

The RPM packages may be installed right away, for example using the command:

[root] # rpm -ivh dcache-server-<version>.noarch.rpm

The actual sources lie at http://www.dcache.org/downloads.shtml. To install for example Version 1.9.12-1 of the server you would use this:

[root] # rpm -ivh http://www.dcache.org/downloads/1.9/dcache-server-1.9.12-1.noarch.rpm

The client can be found in the download-section of the above url, too.

[return to top]

Readying the PostgreSQL server for the use with dCache

Using a PostgreSQL server with dCache places a number of requirements on the database. You must configure PostgreSQL for use by dCache and create the necessary PostgreSQL user accounts and database structure. This section describes how to do this.

[return to top]

Enabling local trust

Perhaps the simplest configuration is to allow password-less access to the database and the following documentation assumes this is so.

To allow local users to access PostgreSQL without requiring a password, ensure the file pg_hba.conf, usually located in /var/lib/pgsql/data, contains the following lines.

local   all         all                        trust
host    all         all         127.0.0.1/32   trust
host    all         all         ::1/128        trust

Note

Please note it is also possible to run dCache with all PostgreSQL accounts requiring passwords. See the section called “Configuring Access to PostgreSQL for more advice on the configuration of PostgreSQL.

Restarting PostgreSQL

If you have edited PostgreSQL configuration files, you must restart PostgreSQL for those changes to take effect. On many systems, this can be done with the following command:

[root] # /etc/init.d/postgresql restart
Stopping postgresql service:                               [  OK  ]
Starting postgresql service:                               [  OK  ]

[return to top]

Configuring Chimera

Chimera is a library providing a hierarchical name space with associated meta data. Where pools in dCache store the content of files, Chimera stores the names and meta data of those files. Chimera itself stores the data in a relational database. We will use PostgreSQL in this tutorial. The properties of Chimera are defined in /usr/share/dcache/defaults/chimera.properties. See Chapter 4, Chimera for more information.

[return to top]

Initialize the database

Create the Chimera user and database and add the Chimera-specific tables and stored procedures:

[root] # createdb -U postgres chimera
CREATE DATABASE

[root] # createuser -U postgres --no-superuser --no-createrole --createdb --pwprompt chimera
Enter password for new role:
Enter it again:
CREATE ROLE
You do not need to enter a password.

[root] # psql -U chimera chimera -f /usr/share/dcache/chimera/sql/create.sql
psql:/usr/share/dcache/chimera/sql/create.sql:23: NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "t_inodes_pkey" for table "t_inodes"
CREATE TABLE
psql:/usr/share/dcache/chimera/sql/create.sql:35: NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "t_dirs_pkey" for table "t_dirs"
CREATE TABLE
psql:/usr/share/dcache/chimera/sql/create.sql:45: NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "t_inodes_data_pkey" for table "t_inodes_data"
many more like this...
INSERT 0 1
many more like this...
INSERT 0 1
CREATE INDEX
CREATE INDEX
psql:/usr/share/dcache/chimera/sql/create.sql:256: NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "t_storageinfo_pkey" for table "t_storageinfo"
CREATE TABLE
psql:/usr/share/dcache/chimera/sql/create.sql:263: NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "t_access_latency_pkey" for table "t_access_latency"
CREATE TABLE
psql:/usr/share/dcache/chimera/sql/create.sql:270: NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "t_retention_policy_pkey" for table "t_retention_policy"
CREATE TABLE
psql:/usr/share/dcache/chimera/sql/create.sql:295: NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "t_locationinfo_pkey" for table "t_locationinfo"
CREATE TABLE
psql:/usr/share/dcache/chimera/sql/create.sql:311: NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "t_locationinfo_trash_pkey" for table "t_locationinfo_trash"
CREATE TABLE
CREATE INDEX
psql:/usr/share/dcache/chimera/sql/create.sql:332: NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "t_acl_pkey" for table "t_acl"
CREATE TABLE
CREATE INDEX

[root] # createlang -U postgres plpgsql chimera
[root] # psql -U chimera chimera -f /usr/share/dcache/chimera/sql/pgsql-procedures.sql
CREATE FUNCTION
CREATE FUNCTION
CREATE FUNCTION
CREATE TRIGGER
CREATE FUNCTION
CREATE TRIGGER
CREATE SEQUENCE
CREATE FUNCTION
CREATE TRIGGER

[return to top]

Creating users and databases for dCache

The dCache components will access the database server with the user srmdcache which can be created with the createuser; for example:

[root] # createuser -U postgres --no-superuser --no-createrole --createdb --pwprompt srmdcache
You do not need to enter a password

Several management components running on the head node as well as the SRM will use the database dcache for storing their state information:

[root] # createdb -U srmdcache dcache

There might be several of these on several hosts. Each is used by the dCache components running on the respective host.

Now the configuration of PostgreSQL is done.

[return to top]

Generating ssh-keys

In order to use the Admin Interface of dCache (see the section called “The Admin Interface”) you will need to generate ssh-keys.

[root] # ssh-keygen -b 768 -t rsa1 -f /etc/dcache/server_key -N ""
Generating public/private rsa1 key pair.
Your identification has been saved in /etc/dcache/server_key.
Your public key has been saved in /etc/dcache/server_key.pub.
The key fingerprint is:
33:ac:91:4c:21:4e:63:aa:2d:90:58:4d:72:e6:b5:88 root@example.org
[root] # ssh-keygen -b 1024 -t rsa1 -f /etc/dcache/host_key -N ""
Generating public/private rsa1 key pair.
Your identification has been saved in /etc/dcache/host_key.
Your public key has been saved in /etc/dcache/host_key.pub.
The key fingerprint is:
75:a4:2b:24:cc:75:ad:b7:bd:8b:dd:1a:3e:96:3f:9d root@example.org

Now you can start dCache.

[root] # dcache start
Starting dCacheDomain done

So far, no configuration is done, so only one predefined domain is started.

[return to top]

Configuring dCache

[return to top]

Terminology

dCache consists of one or more domains. A domain in dCache is a Java Virtual Machine hosting one or more dCache cells. Each domain must have a name which is unique throughout the dCache instance and a cell must have a unique name within the domain hosting the cell.

A service is an abstraction used in the dCache configuration to describe atomic units to add to a domain. It is typically implemented through one or more cells. dCache keeps lists of the domains and the services that are to be run within these domains in the layout files. The layout file may contain domain- and service- specific configuration values. A pool is a cell providing physical data storage services.

[return to top]

Configuration files

In the setup of dCache, there are three main places for configuration files:

  • /usr/share/dcache/defaults
  • /etc/dcache/dcache.conf
  • /etc/dcache/layouts

The folder /usr/share/dcache/defaults contains the default settings of the dCache. If one of the default configuration values needs to be changed, copy the default setting of this value from one of the files in /usr/share/dcache/defaults to the file /etc/dcache/dcache.conf, which initially is empty and update the value.

Note

In this first installation of dCache your dCache will not be connected to a tape sytem. Therefore please change the values for DefaultRetentionPolicy and DefaultAccessLatency in the file /etc/dcache/dcache.conf.

DefaultRetentionPolicy=REPLICA
DefaultAccessLatency=ONLINE

Layouts describe which domains to run on a host and which services to run in each domain. For the customized configuration of your dCache you will create a mylayout.conf file in /etc/dcache/layouts.

Important

Do not update configuration values in the files in the defaults folder, since changes to these files will be overwritten by updates.

As the files in /usr/share/dcache/defaults/ do serve as succinct documentation for all available configuration parameters and their default values it is quite useful to have a look at them.

[return to top]

Defining domains and services

Domains and services are defined in the layout files. Depending on your site, you may have requirements upon the doors that you want to configure and domains within which you want to organise them.

A domain must be defined if services are to run in that domain. Services will be started in the order in which they are defined.

Every domain is a Java Virtual Machine that can be started and stopped separately. You might want to define several domains for the different services depending on the necessity of restarting the services separately.

The layout files define which domains to start and which services to put in which domain. Configuration can be done per domain and per service.

A name in square brackets, without a forward-slash (/) defines a domain. A name in square brackets with a forward slash defines a service that is to run in a domain. Lines starting with a hash-symbol (#) are comments and will be ignored by dCache.

There may be several layout files in the layout directory, but only one of them is read by dCache when starting up. By default it is the single.conf. If the dCache should be started with another layout file you will have to make this configuration in /etc/dcache/dcache.conf.

Example:

dcache.layout=mylayout
This entry in /etc/dcache/dcache.conf will instruct dCache to read the layout file /etc/dcache/layouts/mylayout.conf when starting up.

Example:

These are the first lines of /etc/dcache/layouts/single.conf:
broker.scheme=none

[dCacheDomain]
[dCacheDomain/admin]
[dCacheDomain/broadcast]
[dCacheDomain/poolmanager]

[dCacheDomain] defines a domain called dCacheDomain. In this example only one domain is defined. All the services are running in that domain. Therefore no messagebroker is needed, which is the meaning of the entry messageBroker=none.

[dCacheDomain/admin] declares that the admin service is to be run in the dCacheDomain domain.

Example:

Here is an example for the first lines of the mylayout.conf of a single node dCache with several domains.

[dCacheDomain]
[dCacheDomain/poolmanager]
[dCacheDomain/dummy-prestager]
[dCacheDomain/broadcast]
[dCacheDomain/loginbroker]
#[dCacheDomain/topo]

[namespaceDomain]
[namespaceDomain/pnfsmanager]
[namespaceDomain/cleaner]
[namespaceDomain/acl]
[namespaceDomain/nfsv3]

[gPlazmaDomain]
[gPlazmaDomain/gplazma]

[srmDomain]
[srmDomain/srm]
[srmDomain/spacemanager]
[srmDomain/transfermanagers]

Important

Having defined more than one domain, a messagebroker is needed. This is because the defined domains need to be able to communicate with each other. The default value is messageBroker=cells, as defined in the defaults /usr/share/dcache/defaults/dcache.properties.

[return to top]

Creating and configuring pools

dCache will need to write the files it keeps in pools. These pools are defined as services within dCache. Hence, they are added to the layout file of your dCache instance, like all other services.

The best way to create a pool, is to use the dcache script and restart the domain the pool runs in. The pool will be added to your layout file.

[<domainname>/pool]
name=<poolname>
path=/path/to/pool
waitForFiles=${path}/data

The property waitForFiles instructs the pool not to start up until the specified file or directory is available. This prevents problems should the underlying storage be unavailable (e.g., if a RAID device is offline).

Note

Please restart dCache if your pool is created in a domain that did not exist before.

Example:

[root] # dcache pool create /srv/dcache/p1 pool1 poolDomain
Created a pool in /srv/dcache/p1. The pool was added to poolDomain in
file:/etc/dcache/layouts/mylayout.conf.

In this example we create a pool called pool1 in the directory /srv/dcache/p1. The created pool will be running in the domain poolDomain.

Now you need to change the ownership of the directories where your data is to be stored to the user running dCache which by default is dcache.

[root] # chown -R dcache /srv/dcache/p1

Mind the Gap!

The default gap for poolsizes is 4GiB. This means you should make a bigger pool than 4GiB otherwise you would have to change this gap in the dCache admin tool. See the example below. See also the section called “The Admin Interface”.

(local) admin > cd <poolname>
(<poolname>) admin > set gap 2G
(<poolname>) admin > save

Adding a pool to a configuration does not modify the pool or the data in it and can thus safely be undone or repeated.

[return to top]

Java heap size

By default the Java heap size and the maximum direct buffer size are defined as

dcache.java.memory.heap=512m
dcache.java.memory.direct=512m

Again, these values can be changed in /etc/dcache/dcache.conf.

For optimization of your dCache you can define the Java heap size in the layout file separately for every domain.

Example:

[dCacheDomain]
dcache.java.memory.heap=2048m
dcache.java.memory.direct=0m
...
[utilityDomain]
dcache.java.memory.heap=384m
dcache.java.memory.direct=16m

Note

dCache uses Java to parse the configuration files and will search for Java on the system path first; if it is found there, no further action is needed. If Java is not on the system path, the environment variable JAVA_HOME defines the location of the Java installation directory. Alternatively, the environment variable JAVA can be used to point to the Java executable directly.

If JAVA_HOME or JAVA cannot be defined as global environment variables in the operating system, then they can be defined in either /etc/default/dcache or /etc/dcache.env. These two files are sourced by the init script and allow JAVA_HOME, JAVA and DCACHE_HOME to be defined.

[return to top]

Starting dCache

All configured components can now be started with dcache start.

Example:

[root] # dcache start
Starting dCacheDomain  Done (pid=7574)
Starting namespaceDomain  Done (pid=7647)
Starting gPlazmaDomain  Done (pid=8002)
Starting srmDomain  Done (pid=8081)

Now you can have a look at your dCache via The Web Interface, see the section called “The Web Interface for Monitoring dCache: http://<httpd.example.org>:2288/, where <httpd.example.org> is the node on which your httpd service is running. For a single node dCache this is the machine on which your dCache is running.

[return to top]

Running dCache as a non-root user

For security reasons you might want to run dCache as a non-root user. This needs to be configured in /etc/dcache/dcache.conf and the ownership of the pools and the billing directories need to be changed.

Example:

In this example we explain how to configure dCache to run as user dcache.

First of all create the user dcache and a new directory /var/log/dcache, owned by dcache

[root] #useradd dcache
[root] #mkdir /var/log/dcache
[root] #chown dcache /var/log/dcache

and add two lines to /etc/dcache/dcache.conf.

dcache.user=dcache
dcache.paths.billing=/var/log/dcache

Change the ownership of the files /etc/grid-security/hostcert.pem and /etc/grid-security/hostkey.pem

[root] #chown dcache /etc/grid-security/hostcert.pem
[root] #chown dcache /etc/grid-security/hostkey.pem

In the layout file /etc/dcache/layouts/mylayout.conf a pool was defined.

[${host.name}Domain/pool]
name=pool1
path=/path/to/pool1
maxDiskSpace=2T

Change the ownership of the directory of the pool.

[root] #chown -R dcache /path/to/pool1

Moreover, if dCache has been running as root the files /tmp/createDDL.jdbc and /tmp/dropDDL.jdbc need to be removed as they are owned by the user root.

[root] #rm -f /tmp/createDDL.jdbc /tmp/dropDDL.jdbc

Important

If you switched the portmap daemon off the domain in which the NFS server is running needs to be running as root. This needs to be set in the layout file.

[namespaceDomain]
dcache.user=root
[namespaceDomain/pnfsmanager]
[namespaceDomain/cleaner]
[namespaceDomain/acl]
[namespaceDomain/nfsv3]

Now you can start dCache as user dcache

[root] #dcache start
Starting dCacheDomain done
Starting namespaceDomain done
Starting gPlazmaDomain done
Starting srmDomain done

and verifiy that dCache is running as dcache and the NFS server is running as root.

[root] #dcache status
DOMAIN          STATUS  PID   USER
dCacheDomain    running 11040 dcache
namespaceDomain running 11117 root
gPlazmaDomain   running 11125 dcache
srmDomain       running 11182 dcache

[return to top]

Installing dCache on several nodes

Installing dCache on several nodes is not much more complicated than installing it on a single node. Think about how dCache should be organised regarding services and domains. Then adapt the layout files, as described in the section called “Defining domains and services”, to the layout that you have in mind. The files /etc/dcache/layouts/head.conf and /etc/dcache/layouts/pool.conf contain examples for a dCache head-node and a dCache pool respectively.

Important

You must configure a domain called dCacheDomain but the other domain names can be chosen freely.

Please make sure that the domain names that you choose are unique. Having the same domain names in different layout files on different nodes may result in an error.

On any other nodes than the head node, the property broker.host has to be added to the file /etc/dcache/dcache.conf. broker.host should point to the host running your dCache broker. Usually that is the host containing the special domain dCacheDomain, because that domain acts implicitly as a broker.

Tip

On dCache nodes running only pool services you do not need to install PostgreSQL. If your current node hosts only these services, the installation of PostgreSQL can be skipped.