In the following the installation of a dCache
instance will be described. The Chimera name space provider,
some management components, and the SRM need a PostgreSQL server
installed. We recommend running this PostgreSQL on the local
node. The first section describes the configuration of a PostgreSQL
server. After that the installation of Chimera and of the
dCache components will follow. During the whole installation
process root access is required.
In order to install dCache the following requirements must be met:
An RPM-based Linux distribution is required for the following procedure. For Debian derived systems the RPM may be converted to a DEB using alien. Solaris is supported using either the Solaris package or the tarball.
dCache requires Java 1.6 JRE. Please use Sun JVM at the latest patch-level (at the moment of writing this is 1.6.0_25) and make sure to upgrade to the newest version frequently. It is recommendet to use JDK as dCache scripts can make use of some extra features that JDK provides to gather more diagnostic information (heap-dump, etc). This helps when tracking down bugs.
PostgreSQL must be installed and running. See the section called “Installing a PostgreSQL Server” for more details. PostgreSQL version 8.3 or later is required.
The RPM packages may be installed right away, for example using the command:
[root] #rpm -ivh dcache-server-<version>.noarch.rpm
The actual sources lie at http://www.dcache.org/downloads.shtml. To install for example Version 1.9.12-1 of the server you would use this:
[root] #rpm -ivh http://www.dcache.org/downloads/1.9/dcache-server-1.9.12-1.noarch.rpm
The client can be found in the download-section of the above url, too.
Using a PostgreSQL server with dCache places a number of requirements on the database. You must configure PostgreSQL for use by dCache and create the necessary PostgreSQL user accounts and database structure. This section describes how to do this.
Perhaps the simplest configuration is to allow password-less access to the database and the following documentation assumes this is so.
To allow local users to access PostgreSQL without requiring a
password, ensure the file
pg_hba.conf, usually located in
/var/lib/pgsql/data,
contains the following lines.
local all all trust host all all 127.0.0.1/32 trust host all all ::1/128 trust
Note
Please note it is also possible to run dCache with all PostgreSQL accounts requiring passwords. See the section called “Configuring Access to PostgreSQL” for more advice on the configuration of PostgreSQL.
Restarting PostgreSQL
If you have edited PostgreSQL configuration files, you must restart PostgreSQL for those changes to take effect. On many systems, this can be done with the following command:
[root] #/etc/init.d/postgresql restartStopping postgresql service: [ OK ] Starting postgresql service: [ OK ]
Chimera is a library providing a hierarchical name space
with associated meta data. Where pools in dCache store the
content of files, Chimera stores the names and meta data
of those files. Chimera itself stores the data in a
relational database. We will use PostgreSQL in this tutorial.
The properties of Chimera are defined in
/opt/d-cache/share/defaults/chimera.properties.
See Chapter 4, Chimera for more information.
Create the Chimera user and database and add the Chimera-specific tables and stored procedures:
[root] #createdb -U postgres chimeraCREATE DATABASE[root] #createuser -U postgres --no-superuser --no-createrole --createdb --pwprompt chimeraEnter password for new role: Enter it again: CREATE ROLE You do not need to enter a password.[root] #psql -U chimera chimera -f /opt/d-cache/libexec/chimera/sql/create.sqlpsql:/opt/d-cache/libexec/chimera/sql/create.sql:23: NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "t_inodes_pkey" for table "t_inodes" CREATE TABLE psql:/opt/d-cache/libexec/chimera/sql/create.sql:35: NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "t_dirs_pkey" for table "t_dirs" CREATE TABLE psql:/opt/d-cache/libexec/chimera/sql/create.sql:45: NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "t_inodes_data_pkey" for table "t_inodes_data" many more like this... INSERT 0 1 many more like this... INSERT 0 1 CREATE INDEX CREATE INDEX psql:/opt/d-cache/libexec/chimera/sql/create.sql:256: NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "t_storageinfo_pkey" for table "t_storageinfo" CREATE TABLE psql:/opt/d-cache/libexec/chimera/sql/create.sql:263: NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "t_access_latency_pkey" for table "t_access_latency" CREATE TABLE psql:/opt/d-cache/libexec/chimera/sql/create.sql:270: NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "t_retention_policy_pkey" for table "t_retention_policy" CREATE TABLE psql:/opt/d-cache/libexec/chimera/sql/create.sql:295: NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "t_locationinfo_pkey" for table "t_locationinfo" CREATE TABLE psql:/opt/d-cache/libexec/chimera/sql/create.sql:311: NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "t_locationinfo_trash_pkey" for table "t_locationinfo_trash" CREATE TABLE CREATE INDEX psql:/opt/d-cache/libexec/chimera/sql/create.sql:332: NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "t_acl_pkey" for table "t_acl" CREATE TABLE CREATE INDEX[root] #createlang -U postgres plpgsql chimera[root] #psql -U chimera chimera -f /opt/d-cache/libexec/chimera/sql/pgsql-procedures.sqlCREATE FUNCTION CREATE FUNCTION CREATE FUNCTION CREATE TRIGGER CREATE FUNCTION CREATE TRIGGER CREATE SEQUENCE CREATE FUNCTION CREATE TRIGGER
The dCache components will access the database server with the user srmdcache which can be created with the createuser; for example:
[root] #createuser -U postgres --no-superuser --no-createrole --createdb --pwprompt srmdcacheYou do not need to enter a password
Several management components running on the head node as
well as the SRM will use the database
dcache for storing their state
information:
[root] #createdb -U srmdcache dcache
There might be several of these on several hosts. Each is used by the dCache components running on the respective host.
Now the configuration of PostgreSQL is done.
In order to use the Admin Interface of dCache (see the section called “The Admin Interface”) you will need to generate ssh-keys.
[root] #ssh-keygen -b 768 -t rsa1 -f /opt/d-cache/etc/server_key -N ""Generating public/private rsa1 key pair. Your identification has been saved in /opt/d-cache/etc/server_key. Your public key has been saved in /opt/d-cache/etc/server_key.pub. The key fingerprint is: 33:ac:91:4c:21:4e:63:aa:2d:90:58:4d:72:e6:b5:88 root@example.org[root] #ssh-keygen -b 1024 -t rsa1 -f /opt/d-cache/etc/host_key -N ""Generating public/private rsa1 key pair. Your identification has been saved in /opt/d-cache/etc/host_key. Your public key has been saved in /opt/d-cache/etc/host_key.pub. The key fingerprint is: 75:a4:2b:24:cc:75:ad:b7:bd:8b:dd:1a:3e:96:3f:9d root@example.org
Now you can start dCache.
[root] #/opt/d-cache/bin/dcache startStarting dCacheDomain done
So far, no configuration is done, so only one predefined domain is started.
dCache consists of one or more domains. A domain in dCache is a Java Virtual Machine hosting one or more dCache cells. Each domain must have a name which is unique throughout the dCache instance and a cell must have a unique name within the domain hosting the cell.
A service is an abstraction used in the dCache configuration to describe atomic units to add to a domain. It is typically implemented through one or more cells. dCache keeps lists of the domains and the services that are to be run within these domains in the layout files. The layout file may contain domain- and service- specific configuration values. A pool is a cell providing physical data storage services.
In the setup of dCache, there are three main places for configuration files:
/opt/d-cache/share/defaults/opt/d-cache/etc/dcache.conf/opt/d-cache/etc/layouts
The folder /opt/d-cache/share/defaults
contains the default
settings of the dCache.
If one of the default configuration values needs to be changed, copy the default
setting of this value from one of the files in
/opt/d-cache/share/defaults
to the file /opt/d-cache/etc/dcache.conf, which initially is
empty and update the value.
Note
In this first installation of dCache your dCache will not be connected to a tape sytem. Therefore please change the values for DefaultRetentionPolicy and DefaultAccessLatency in the file /opt/d-cache/etc/dcache.conf.
DefaultRetentionPolicy=REPLICA DefaultAccessLatency=ONLINE
Layouts describe which domains to run on a host and which services to run in each domain. For the customized configuration of your dCache you will create a mylayout.conf file in /opt/d-cache/etc/layouts.
Important
Do not update configuration values in the files in the defaults folder, since changes to these files will be overwritten by updates.
As the files in /opt/d-cache/share/defaults/ do serve
as succinct documentation for all available configuration parameters and their
default values it is quite useful to have a look at them.
Domains and services are defined in the layout files. Depending on your site, you may have requirements upon the doors that you want to configure and domains within which you want to organise them.
A domain must be defined if services are to run in that domain. Services will be started in the order in which they are defined.
Every domain is a Java Virtual Machine that can be started and stopped separately. You might want to define several domains for the different services depending on the necessity of restarting the services separately.
The layout files define which domains to start and which services to put in which domain. Configuration can be done per domain and per service.
A name in square brackets, without a
forward-slash (/) defines a domain. A
name in square brackets with a forward
slash defines a service that is to run in a domain. Lines
starting with a hash-symbol (#) are
comments and will be ignored by dCache.
There may be several layout files in the layout directory, but only one
of them is read by dCache when starting up. By default it is the
single.conf.
If the dCache should be started with another layout file you will have to make
this configuration in /opt/d-cache/etc/dcache.conf.
Example:
dcache.layout=mylayoutThis entry in
/opt/d-cache/etc/dcache.conf
will instruct dCache to read the layout file
/opt/d-cache/etc/layouts/mylayout.conf when starting up.
Example:
/opt/d-cache/etc/layouts/single.conf:
broker.scheme=none [dCacheDomain] [dCacheDomain/admin] [dCacheDomain/broadcast] [dCacheDomain/poolmanager]
[ defines a domain
called dCacheDomain]dCacheDomain. In this example only one domain is defined.
All the services are running in that domain. Therefore
no messagebroker is needed, which is the meaning of the entry
messageBroker=none.
[ declares that
the dCacheDomain/admin]admin service is to be run in the dCacheDomain domain.
Example:
Here is an example for the first lines of the mylayout.conf
of a single node dCache with several domains.
[dCacheDomain] [dCacheDomain/poolmanager] [dCacheDomain/dummy-prestager] [dCacheDomain/broadcast] [dCacheDomain/loginbroker] #[dCacheDomain/topo] [namespaceDomain] [namespaceDomain/pnfsmanager] [namespaceDomain/cleaner] [namespaceDomain/acl] [namespaceDomain/nfsv3] [gPlazmaDomain] [gPlazmaDomain/gplazma] [srmDomain] [srmDomain/srm] [srmDomain/spacemanager] [srmDomain/transfermanagers]
Important
Having defined more than one domain, a messagebroker is needed. This is because the defined domains need to be able to communicate with each other. The default value ismessageBroker=cells, as defined in the defaults
/opt/d-cache/share/defaults/dcache.properties.
dCache will need to write the files it keeps in pools. These pools are defined as services within dCache. Hence, they are added to the layout file of your dCache instance, like all other services.
The best way to create a pool, is to use the dcache script and restart the domain the pool runs in. The pool will be added to your layout file.
[<domainname>/pool]
name=<poolname>
path=/path/to/pool
waitForFiles=${path}/data
The property waitForFiles instructs the
pool not to start up until the specified file or directory
is available. This prevents problems should the underlying storage be unavailable (e.g., if a
RAID device is offline).
Note
Please restart dCache if your pool is created in a domain that did not exist before.
Example:
[root] #/opt/d-cache/bin/dcache pool create /srv/dcache/p1 pool1 poolDomainCreated a pool in /srv/dcache/p1. The pool was added to poolDomain in file:/etc/dcache/layouts/mylayout.conf.
In this example we create a pool called pool1 in the directory /srv/dcache/p1.
The created pool will be running in the domain poolDomain.
Now you need to change the ownership of the directories where your data is to be stored to the user running dCache which by default is dcache.
[root] #chown -R dcache /srv/dcache/p1
Mind the Gap!
The default gap for poolsizes is 4GiB. This means you should make a bigger pool than 4GiB otherwise you would have to change this gap in the dCache admin tool. See the example below. See also the section called “The Admin Interface”.
(local) admin >cd <poolname>(<poolname>) admin >set gap 2G(<poolname>) admin >save
Adding a pool to a configuration does not modify the pool or the data in it and can thus safely be undone or repeated.
By default the Java heap size and the maximum direct buffer size are defined as
dcache.java.memory.heap=512m dcache.java.memory.direct=512m
Again, these values can be changed in
/opt/d-cache/etc/dcache.conf.
For optimization of your dCache you can define the Java heap size in the layout file separately for every domain.
Example:
[dCacheDomain] dcache.java.memory.heap=2048m dcache.java.memory.direct=0m ... [utilityDomain] dcache.java.memory.heap=384m dcache.java.memory.direct=16m
Note
dCache uses Java to parse the configuration files and
will search for Java on the system path first; if it is
found there, no further action is needed. If Java is not
on the system path, the environment variable
JAVA_HOME defines the location of the
Java installation directory. Alternatively, the
environment variable JAVA can be used to
point to the Java executable directly.
If JAVA_HOME or JAVA cannot
be defined as global environment variables in the
operating system, then they can be defined in either
/etc/default/dcache or
/etc/dcache.env. These two files are
sourced by the init script and allow
JAVA_HOME, JAVA and
DCACHE_HOME to be defined.
All configured components can now be started with
/opt/d-cache/bin/dcache start.
Example:
[root] #/opt/d-cache/bin/dcache startStarting dCacheDomain Done (pid=7574) Starting namespaceDomain Done (pid=7647) Starting gPlazmaDomain Done (pid=8002) Starting srmDomain Done (pid=8081)
Now you can have a look at your dCache via The Web
Interface, see the section called “The Web Interface for Monitoring dCache”:
http://<httpd.example.org>:2288/,
where <httpd.example.org> is the node on which your
httpd service is running.
For a single node dCache this is the machine on which your dCache is running.
For security reasons you might want to run dCache as a
non-root user. This needs to be configured in
/opt/d-cache/etc/dcache.conf and the
ownership of the pools and the billing directories need to
be changed.
Example:
In this example we explain how to configure dCache to
run as user dcache.
First of all create the user dcache and a new
directory /var/log/dcache, owned by
dcache
[root] #useradd dcache[root] #mkdir /var/log/dcache[root] #chown dcache /var/log/dcache
and add two lines to
/opt/d-cache/etc/dcache.conf.
dcache.user=dcache dcache.paths.billing=/var/log/dcache
Change the ownership of the files
/etc/grid-security/hostcert.pem and
/etc/grid-security/hostkey.pem
[root] #chown dcache /etc/grid-security/hostcert.pem[root] #chown dcache /etc/grid-security/hostkey.pem
In the layout file
/opt/d-cache/etc/layouts/mylayout.conf
a pool was defined.
[${host.name}Domain/pool]
name=pool1
path=/path/to/pool1
maxDiskSpace=2TChange the ownership of the directory of the pool.
[root] #chown -R dcache /path/to/pool1
Moreover, if dCache has been running as root the
files /tmp/createDDL.jdbc and
/tmp/dropDDL.jdbc need to be removed
as they are owned by the user root.
[root] #rm -f /tmp/createDDL.jdbc /tmp/dropDDL.jdbc
Important
If you switched the portmap daemon off the domain in which
the NFS server is running needs to be running as
root. This needs to be set in the layout file.
[namespaceDomain] dcache.user=root [namespaceDomain/pnfsmanager] [namespaceDomain/cleaner] [namespaceDomain/acl] [namespaceDomain/nfsv3]
Now you can start dCache as user dcache
[root] #/opt/d-cache/bin/dcache startStarting dCacheDomain done Starting namespaceDomain done Starting gPlazmaDomain done Starting srmDomain done
and verifiy that dCache is running as dcache and
the NFS server is running as root.
[root] #/opt/d-cache/bin/dcache statusDOMAIN STATUS PID USER dCacheDomain running 11040 dcache namespaceDomain running 11117 root gPlazmaDomain running 11125 dcache srmDomain running 11182 dcache
Installing dCache on several nodes is not much more
complicated than installing it on a single node. Think about
how dCache should be organised regarding services and
domains. Then adapt the layout files, as described in the section called “Defining domains and services”, to the layout that you have in
mind. The files
/opt/d-cache/etc/layouts/head.conf and
/opt/d-cache/etc/layouts/pool.conf contain
examples for a dCache head-node and a dCache pool
respectively.
Important
You must configure a domain called dCacheDomain but the
other domain names can be chosen freely.
Please make sure that the domain names that you choose are unique. Having the same domain names in different layout files on different nodes may result in an error.
On any other nodes than the head node, the property
broker.host has to be added to the file
/opt/d-cache/etc/dcache.conf.
broker.host should point to the host running
your dCache broker. Usually that is the host containing the
special domain dCacheDomain, because that domain acts
implicitly as a broker.
Tip
On dCache nodes running only pool services you do not need to install PostgreSQL. If your current node hosts only these services, the installation of PostgreSQL can be skipped.