How to get from 1.9.5 to 1.9.12
By Gerd Behrmann <behrmann@ndgf.org>
dCache 1.9.12 is the second long term support release (aka golden release). It is the result of 18 months of development since the release of dCache 1.9.5, the previous long term support release. During these 18 months 7 feature releases (1.9.6 to 1.9.12) were made at regular intervals. This document compiles the most important information from these seven feature releases.
dCache 1.9.12 will be supported with regular bug fix releases at least until primo 2013. dCache 1.9.5 will be supported until primo 2012, however one should expect the release frequency to drop of as only the most critical bugs and security issues will be fixed. While the upgrade path from 1.9.5 to 1.9.12 is relatively easy, it should be expected that no direct upgrade path from releases prior to 1.9.12 to a future (third) long term support release is provided.
Many things have changed between 1.9.5 and 1.9.12 and this document does not attempt to describe every little change. The focus is on changes that affect the upgrade process and on new features. Minor feature changes and bug fixes are often exluded. There is more information scattered in the release notes of each individual release.
The last section of this document contains useful reference material that should be consulted while reading this document. The reference material also includes a proposed checklist that may be used while planning an upgrade to 1.9.12.
The filesystem hierarchy standard (FHS) provides a set of requirements and guidelines for file and directory placement under UNIX-like operating systems.
dCache has traditionally been installed in /opt/d-cache. Although this directory is specified by the FHS, dCache did not follow the recommendations for the installation in /opt.
As part of the European Middleware Initiative (EMI) release 1 dCache is now provided in two different layouts:
The FHS compliant layout installs the bulk of the static files in /usr/share/dcache, with a few files in /usr/bin, /usr/sbin and /usr/share/man. Configuration files are stored under /etc/dcache and non-static files are stored under /var/lib/dcache with log files in /var/log/dcache.
The FHS compliant packages automatically create a user account (dcache) during installation and dCache will drop to this account during startup. An init script and a logrotation configuration is automatically installed. Admin door ssh keys are auto created during installation.
The FHS layout allows software and users to predict the location of installed files and directories. The dCache packages available from EMI follow the FHS layout.
Upgrading from 1.9.5 to the FHS packages of 1.9.12 is possible, but requires more manual work than upgrading to the classic layout.
The classic layout on the other hand is not FHS compliant and is similar to the layout used by dCache 1.9.5. The packages at www.dcache.org use the classic layout and this guide assumes that the classic layout is used.
We expect that all users will transition to the FHS packages and www.dcache.org will in the future host both layouts. Eventually we will stop distributing packages for /opt.
Head nodes of 1.9.12 are compatible with pools from any release between 1.9.5 and 1.9.12 (both included). Pools from any release between 1.9.5 and 1.9.12 can be mixed. Beginning with the release of 1.9.13 (primo July 2011) head nodes will only be compatible with pools belonging to release 1.9.12 and newer.
Because new head nodes are compatible with old pools, it is possible to do a staged upgrade by first updating all head nodes (PNFS manager, pool manager, pin manager, space manager, SRM, all doors, monitoring services, etc) while leaving pools on the 1.9.5 release. Once the head nodes are online again and confirmed to be working, pools can be upgraded one by one while dCache is online. Obviously the service will be slightly degraded as files on a pool that is being upgraded will be momentarily unavailable. Please notice that staged upgrade is not possible if pool nodes run any other dCache service (such as doors) besides the pool service.
The alternative to a staged upgrade is to shutdown all dCache services and upgrade all nodes.
In either case an inplace upgrade is possible and recommended: Most configuration files have changed format and quite a number of configuration properties have been renamed, yet a migration script provided with dCache 1.9.12 takes care of most conversions. dCache will refuse to start if it detects the old configuration files. You can read more about these changes in the following section.
The first time dCache is started after the upgrade several databases will be updated: This includes the Berkeley DB Java Edition files on pools, the pin manager, the space manager, and a couple of schema tracking tables used by the SRM. The upgrade should be transparent, although the first start may be slightly slower than usual. Due to these schema changes, a downgrade from 1.9.12 is not possible without expert support and should not be attempted.
Lots of components have been modified to improve consistency, robustness and add new features. In several cases this has affected the semantics of common operations in dCache. We recommend reading through the following sections, paying attention to issues like authorization, file ownership, multihoming, obsolete and forbidden configuration properties, and init scripts.
The /opt/d-cache/libexec/migrate-from-1.9.5.sh script is provided to ease migration of an existing setup. This script creates /opt/d-cache/etc/dcache.conf, /opt/d-cache/etc/layouts/hostname.conf and /etc/default/dcache based on the /opt/d-cache/dCacheSetup, /opt/d-cache/etc/node_config and /opt/d-cache/config/*.poollist files. It also relocates dcache.local.sh and dcache.local.run.sh from /opt/d-cache/jobs/ to /opt/d-cache/bin/, and admin door SSH keys from /opt/d-cache/config/ to /opt/d-cache/etc/. The script is able to handle most installations, however the resulting dcache.conf file must always be adjusted as some properties have been obsoleted.
Notice that the conversion will be incomplete if custom changes have been made to any batch file. Also custom Java settings (like heap size) will not be migrated. Such changes must be added, by hand, either to /opt/d-cache/etc/dcache.conf or /opt/d-cache/etc/layouts/hostname.conf.
The most visible change is the move to a new format for configuration files. After thorough considerations we concluded that the old configuration format was too restrictive to cover the full range of possible deployments of dCache, ranging from the smallest single process installations to the largest distributed installations running on thousands of hosts. At the same time the old format was in many places inconsistent, illogical, and hard to parse.
The new format is not backwards compatible. Upgrading from 1.9.5 to 1.9.12 requires that old configuration files are migrated to the new format. Care has been taken to provide an easy migration path. The process is assisted by a script supplied with dCache, however some manual steps are required to replace deprecated and obsolete configuration properties.
Third party deployment scripts will have to be updated. The migration script provided with dCache may be of little use if such third party scripts are used.
The main configuration file is /opt/d-cache/etc/dcache.conf and replaces /opt/d-cache/config/dCacheSetup. The file is optional. Most sites will want a non-empty /opt/d-cache/etc/dcache.conf that, at least, defines the dcache.layout parameter. This parameter identifies which layout file is to be used (the next section explains what a layout is).
Most of the service specific configuration parameters known from the old /opt/d-cache/config/dCacheSetup file can still be defined in /opt/d-cache/etc/dcache.conf. Please consult the defaults in /opt/d-cache/share/defaults/ for a complete list of parameters.
The file follows the syntax of Java properties files. Values must not be put in qoutes. Any single or double qoute will become part of the value. A trailing backslash acts as a line continuation character. For technical information about the file format, see the Java documentation.
Layouts describe which domains to run on a host and which services to run in each domain. Layouts replace /opt/d-cache/etc/node_config and are defined as files in the /opt/d-cache/etc/layout/ directory. When starting, dCache will read a single layout file. The layout file is chosen by defining the dcache.layout property in /opt/d-cache/etc/dcache.conf; for example, specifying dcache.layout=single in /opt/d-cache/etc/dcache.conf will instruct dCache to read the layout file /opt/d-cache/etc/layouts/single.conf when starting.
There are no longer any hard coded naming conventions regarding what a particular domain does. It is completely up to the layout to define which domains to run, what they are called and which services to run in them. The exception to the rule is dCacheDomain, which acts as the central message communication hub that all other domains connect to.
Besides defining domains and the services to run in a domain, a layout defines domain and service specific configuration parameters. Any property that can be assigned in /opt/d-cache/etc/dcache.conf can be assigned a new value per domain or per service in the layout file.
The syntax is easy to explain by example:
[dCacheDomain]
[dCacheDomain/poolmanager]
[dCacheDomain/broadcast]
[dCacheDomain/loginbroker]
[dCacheDomain/topo]
[adminDoorDomain]
[adminDoorDomain/admin]
[spacemanagerDomain]
[spacemanagerDomain/spacemanager]
[namespaceDomain]
[namespaceDomain/pnfsmanager]
[namespaceDomain/cleaner]
[namespaceDomain/acl]
[dirDomain]
[dirDomain/dir]
[infoDomain]
[infoDomain/info]
[statisticsDomain]
[statisticsDomain/statistics]
[httpdDomain]
[httpdDomain/billing]
[httpdDomain/srm-loginbroker]
[httpdDomain/httpd]
[gPlazmaDomain]
[gPlazmaDomain/gplazma]
[utilityDomain]
[utilityDomain/gsi-pam]
[utilityDomain/pinmanager]
# [${host.name}Domain]
# [${host.name}Domain/pool]
# name=pool1
# path=/path/to/pool1
# [${host.name}Domain/pool]
# name=pool2
# path=/path/to/pool2
# [replicaDomain]
# [replicaDomain/replica]
# [dcap-${host.name}Domain]
# [dcap-${host.name}Domain/dcap]
# [xrootd-${host.name}Domain]
# [xrootd-${host.name}Domain/xrootd]
# [gridftp-${host.name}Domain]
# [gridftp-${host.name}Domain/gridftp]
# [webdav-${host.name}Domain]
# [webdav-${host.name}Domain/webdav]
# [gsidcap-{host.name}Domain]
# [gsidcap-{host.name}Domain/gsidcap]
# [srm-${host.name}Domain]
# [srm-${host.name}Domain/srm]
# [transfermanagersDomain]
# [transfermanagersDomain/transfermanagers]
Lines in which the first non-whitespace character is a hash symbol are comment lines and are ignored. A name in square brackets, without a forward-slash defines a domain; for example, [gPlazmaDomain] defines a domain called gPlazmaDomain. A name in square brackets with a forward slash defines a service that is to run in a domain; for example, [gPlazmaDomain/gplazma] declares that the gplazma service is to be run in the gPlazmaDomain domain. A domain must be defined if services are to run in that domain. The order that services are defined in determines in which order the services are started; however, this order should not matter.
The following illustrates how to start two GridFTP doors on the same host:
[gridftp-${host.name}-1Domain]
[gridftp-${host.name}-1Domain/gridftp]
port=2811
cell.name=GFTP-${host.name}-1
[gridftp-${host.name}-2Domain]
[gridftp-${host.name}-2Domain/gridftp]
port=2812
cell.name=GFTP-${host.name}-2
This defines two domains, each containing a GridFTP door. Each door uses its own TCP port and each door has a unique cell name. Both domain and cell names are parameterised using the host name. Notice the dCache does not enforce any fixed scheme for domain naming. Neither does dCache enforce any fixed scheme for naming door cells.
We recommend using the host name as the name of the layout. Often the main configuration file can be shared among all hosts in the dCache installation, with host specific settings being defined in the layout file. Using the host name for the layout allows dCache to be installed in a shared directory in, for instance, NFS or AFS. The layout reference in the main configuration file can be parameterized, for instance, dcache.layout=${host.name}.
Default values specified by dCache.org are supplied as properties files in /opt/d-cache/share/defaults/. Those files should not be modified as any modification will be lost when upgrading. These files do however serve as succinct documentation for all available configuration parameters and their default values.
Properties defined in the default files may have been tagged. The tags appear in parentheses as prefixes or property names (eg (deprecated,not-for-services)logArea). These tags affect how properties are interpreted by dCache and the tags should not be included when defining properties in configuration files. The tags used are:
Some properties are scoped for a specific service. Such properties are prefixed by the service name and a forward slash (eg dcap/port). These properties only take effect for instances of the specific service. When defining the properties for a particular service instance the prefix should not be repeated. If the properties are defined globally or for a domain, then the prefix must be included (otherwise the default value will take precedence).
The /opt/d-cache/bin/dcache check-config command provides diagnostic information about obsolete and deprecated properties and other problems in the configuration and layout files.
The /opt/d-cache/config/hostname.domains and /opt/d-cache/config/*.poollist files are no longer used to define pools. Instead pools are defined in the layout like any other service:
[myPoolDomain/pool]
name=pool1
path=/san/pool1
lfs=precious
tags=hostname=${host.name}
waitForFiles=${path}/data
maxDiskSpace=2T
The only required parameters are name and path. The path property defines in which directory files are to be stored. The data/ and control/ subdirectories appear under that path (/san/pool1/data/ and /san/pool1/control/ for the above example). The name property defines the name of the pool; pool1 in the above example.
Other parameters that may be set include lfs (the example pool has LFS mode precious), tag (the example pool has the tag hostname set to the host name) and maxDiskSpace (the example pool has a size of 2 Tebibytes).
The /opt/d-cache/bin/dcache pool create command updates the layout file when creating a pool. In versions before dCache 1.9.7 this functionallity was available as the separate dcache pool add command. As a consequence of this consolidation, the parameters accepted by dcache pool create have changed; please consult the help output of the script or the man page for details.
If pools are created on a SAN or cluster file system and there is a risk that a pool is not mounted at the time dCache starts, the waitForFiles parameter may be used to delay pool startup until particular paths are available. This prevents that a new pool is accidentally created in the mount point. In the example above the pool waits for the data directory (${path}/data) to become available.
Notice that the setup file in the pool directory is no longer required. It is parsed if present, but if not present then default values will be assumed. If neither the configuration file nor the setup file specify a value for maxDiskSpace, then a limit is derived from the size of the file system.
As dCache no longer parses any configuration files from shell scripts, the location of the Java binary cannot be defined in the main configuration file. dCache relies on the standard environment variable JAVA_HOME to define the location of Java. If Java is in the shell's search path, then JAVA_HOME is not required.
If JAVA_HOME cannot be defined as a global environment variable in the operating system, then it can be defined in either /etc/default/dcache or /etc/dcache.env. These two files are sourced by the init script and allow JAVA_HOME and DCACHE_HOME to be defined.
As an alternative to defining JAVA_HOME one can define JAVA to point directly to the Java executable.
dCache no longer mounts the name space. For Chimera this is not needed anyway, and for PNFS it is only needed for PnfsManager. As PnfsManager normally runs on the same host as PNFS itself, the PNFS init script will already have mounted the name space.
The script /opt/d-cache/install/install.sh, which used to initialize PNFS among other things, is no longer shipped with dCache. This functionality is now available in /opt/d-cache/install/preparePnfs.sh and only needs to be run once on initial install. There is no reason to run this script on upgrade. The scrit only applies to PNFS. The initialization procedure for Chimera is described in the dCache book.
The Chimera NFS3 name space daemon has been embedded into dCache. It is now available as the service nfsv3 and can be added to the dCache layout file. The domain that is configured to run the nfsv3 service may be started and stopped with /opt/d-cache/bin/dcache, in keeping with all other dCache domains. The old Chimera init script (chimera-nfs-run.sh) is no longer provided.
The Chimera client utilities have been moved to /opt/d-cache/bin/.
The Chimera configuration file has been replaced by properties in /opt/d-cache/etc/dcache.conf. Available properties and documentation can be found in /opt/d-cache/share/defaults/chimera.properties. The default database user of Chimera has changed from postgres to chimera. Please update your database or set chimera.db.user when upgrading from a previous version of dCache.
The Chimera name space provider has been updated such that the OSMTemplate and sGroup directory tags are optional.
For performance reasons dCache with PNFS only verified the lookup
permissions of the directory containing the file system entry
corresponding to the path. I.e., only the lookup permissions for the
last parent dirctory of the path were enforced. For compatibility
reasons Chimera inherited these semantics.
Starting with dCache 1.9.12-22 the new
property pnfsVerifyAllLookups has been introduced to enable
the verification of the lookup permissions of all directories of a
path. When this property is set to true, Chimera will verify
the lookup permissions of all directories of a path.
This change requires that chimera users recreate the stored
procedures by psql -U
Initial support for using Enstore with Chimera was added. Please contact support@dcache.org for details.
As before, dCache uses gPlazma as a user mapping and black listing service. Authorization decisions are performed by several components, in particular in PNFS manager and space manager, but also in pool manager, pin manager and in doors.
Although the overall design of user mapping and authorization hasn't changed, the details have been completely redesigned. This was done to get rid of inconsistencies and limitations of the old scheme. Due to the inconsistencies in earlier versions, it may be hard to predict when existing installations need to be adjusted. Instead we try to describe the process used in 1.9.12.
Users authenticate with dCache doors. Typically this will be via GSI or SSL, but may also be through Kerberos or user name and password. The door presents the user identity and credentials to gPlazma, which will complete the authentication step, consult blacklistings, and map the user identity to a UID and one or more GIDs, one of which will be the primary or effective GID. gPlazma also associates the request with session attributes such as the user's home directory, root directory and whether the user is limitted to read only access. How these steps are perform depends on the plugins used and whether gPlazma 1 or 2 is used.
The outcome of gPlazma is given back to the door. The door will include the UID, the GIDs, and possibly the DN, FQANs and mapped user name when talking to other services in dCache. These services use the principals when making authorization decisions. PNFS manager authorizes file system access by UID and GID. Space manager authorizes writes to link groups by mapped user name or FQAN. Pool manager authorizes staging by DN, FQAN or mapped user name. Pin manager authorizes by UID and GID.
Although a single GID and a single FQAN is considered primary, all GIDs and all FQANs are consulted when using these for authorization decisions. For instance, a user that presented FQANs from several VOs is authorized to write to the link groups of both VOs regardless of which FQAN was the primary FQAN. The same user is also authorized to stage files for both VOs regardless of the order of FQANs (assuming the FQANs are authorized to stage files).
The primary group (whether identified by GID or FQAN) is however used for defining ownership for objects created by the user. Files or directories created by the user receive the UID and primary GID of the request (althogh PNFS manager can be forced to inherit ownership from the parent directory).
Notice that the mapping to UID and GID are defined entirely by gPlazma plugins. Traditionally these plugins supported rather flexible, although possibly illogical, mappings. For instance, FQANs can be mapped to UID and GID pairs. Since a proxy certificate likely contains several FQANs this could correspond to multiple UIDs, yet dCache only allows a single UID. dCache still supports these mappings. In case of conflicts, the first mapping obtained controls the UID that is chosen, and also provides the home and root directories and whether the session is a read only session. For mappings controlled by FQANs the first mapping is that of the primary FQAN if that FQAN has a mapping. Otherwise an arbitrary FQAN with a mapping is chosen.
The above description applies to all doors (DCAP, FTP, HTTP/WebDAV, xrootd, SRM, NFS). We are aware that this change will possibly force configuration changes during upgrade. We are however convinced that the more consistent authentication, mapping and authorization logic is worth the effort.
Many users have been dissatisfied with the complexity of adjusting log levels at runtime. For some time dCache has relied on log4j as its logging framework. Log4j is no longer maintained and there was no way for us to simplify the logging user interface within the constraints of log4j. Therefore dCache has switched from log4j to logback.
The first consequence is that the log configuration file has changed. Now logging is configured in /opt/d-cache/etc/logback.xml. For further details see the instructions embedded in the packaged version of that file and chapter 3 of the logback manual. dCache configuration properties are accessible using ${...} placeholders in logback.xml.
Note that log level configuration in dCache is different from what is described in the logback manual. Pure logback allows log levels to be configured on loggers and appenders, similar to log4j. This has proven to be too restrictive for dCache. We ship dCache with a custom log filter that implements a log level for each triple of logger, appender and cell. This provides very fine grained control over which types of messages from which cell are logged where. Detailed instructions can be found in the instructions embedded in the default /opt/d-cache/etc/logback.xml file.
The second consequence of switching to logback is that the cell admin interface for changing log levels on-the-fly has changed. The legacy set printout commands no longer have any effect. The commands that were prefixed with log4j are no longer supported. Instead a new set of commands prefixed with log have been introduced. Each cell exposes these commands to adjust the log level of logger hierarchies for specific appenders within that cell. In the simplest case one can adjust the log level of all loggers for a particular appender with:
log set stdout DEBUG
or
log set pinboard DEBUG
Notice that these commands only affect the cell and its subcells in which they are executed. In previous versions of dCache, adjusting the log levels affected all cells.
The system cell has additional commands that allow appenders to be attached and detached as needed, although it is unlikely that these will need to be adjusted at runtime.
We recommend consulting the logback manual for details about available appenders. Logback can be configured to perform log rotation, log to syslog, log to SMTP, log to a remote logger, etc. Feel free to contact us for details about specific setups.
The minimum version of PostgreSQL support by dCache 1.9.12 is PostgreSQL 8.4. If you use an older version of PostgreSQL then please upgrade it before upgrading dCache.
The JPOX ORM has been replaced by DataNucleus. This affects the database used by the SRM transfer managers, the pin manager and SRM. The tables used by the transfer managers and the SRM have not changed, however the tables used by the ORM to track schema changes have been changed. Upon upgrade the schema will be updated by DataNucleus.
The pin manager was rewritten and uses a new schema. The schema is maintained with the LiquiBase schema management library. LiquiBase itself adds two tables to track schema changes.
Both changes affect the ability to downgrade to previous versions of dCache. Please contact support@dcache.org for details on how to downgrade from dCache 1.9.12 if needed.
The /opt/d-cache/bin/dcache script has been updated with several new commands related to databases used by various services. The subcommand database ls lists the databases used by services on this host and is supported by all services that use an external database.
Other database subcommands only apply to services using the LiquiBase schema management library. Currently pin manager is the only service that uses this library. Please consult the man page for details about these commands.
It is quite common to deploy dCache on hosts with multiple network interfaces. Typically there is an internal network for communicating with the CPU farm and for internal dCache communication, and an external network for communicating with the rest of the world.
In various scenarios dCache services (specifically doors and pools) have to provide a network address to a client or to another service in dCache. To avoid unnecssary DNS lookups and reverse lookups and to avoid inconsistencies, dCache now always uses the IP address of an interface internally. In earlier versions dCache sometimes performed a reverse lookup on the IP address and provided the DNS name. Besides being slow, this could lead to inconsistencies. The algorithm for selecting a network interface was changed and unified. This section describes when dCache needs to provide a network address, and describes how dCache selects the interface.
The network address of a door is needed:
The network address of a pool is needed:
Whenever possible, dCache services try to select an interface that faces the client. This requires that dCache knows the address of the client that is supposed to connect to the service. dCache uses the routing table to determine which interface the host would use to talk to the client. It is assumed that communication is symmetric and traffic from the client would be received on the same interface.
Automatically selecting the correct interface is for instance possible for passive data transfer protocols in which a pool provides an address to the door, which then provides that address to the client. In this case the door knows the remote address of the client and it is assumed that the data connection to the pool is performed from the same network. That is, a client connected to the door from the external network would also connect to the pool from the external network.
The benefit of the above scheme is that the correct interface is provided no matter whether a local client or an external client transfers a file. Care must however be taken with third party transfers. For instance, a local client which initiates a third party transfer from an external storage system to the local dCache should contact the door over the external network. This is because the data connection to the pool is created from the external storage system rather than the local client, and it is thus paramount that the pool selects its external interface.
In some cases an interface cannot be selected automatically. This is for instance the case when transferring the file from one pool to another: Pools have no knowledge about the addresses of other pools and thus have no means to select the correct interface. The same problem arrises when doors act as proxies and provide their address to a pool. In such cases the service resolves the fully qualified domain name of the host and provides that address. This means that the content of /etc/hosts and the hostname influences which interface is selected.
Many service allow the interface to use to be specified in the configuration. Pools provide the pp set listen command to specify the interface on which to accept pool to pool transfers, and most doors have a configuration option to specify the interface to provide to doors when acting as proxies.
The Berkeley DB library has been updated. The new version uses a new file format. The library is backwards compatible, meaning it can read data from previous versions. However old versions are not forward compatible with the new version. Thus once a pool has been upgraded to 1.9.12, it is not possible to downgrade to a previous version without restoring the meta data information. This only affects pools using the Berkeley DB backend.
Pools now remove stale meta data entries on startup. This will affect pool startup speed, but ensures that stale entries are removed from the control/ and meta/ directories.
Under high load, pools in 1.9.5 have been reported to become unresponsive to interactive commands. This is most likely to happen if the Berkeley DB used for meta data is on the same physical disks as the pool data. Heavy I/O can cause meta data queries to starve. To improve responsiveness in these situations, message processing is now performed by a pool of threads rather than by a single thread.
In dCache 1.9.1 the meta data store of pools was rewritten. The rewrite solved some longstanding issues with space accounting in pools. At the time we decided on a single global lock on any meta data operation, as that made it significantly easier to design the component. Since the meta data store is backed by an on-disk representation, the single lock has proven to be a bottleneck during high load. In dCache 1.9.10 the single lock was replaced by more fine grained locking. This improves throughput to the meta data store.
One immediate benefit of the more fine grained locking is that a pool is enabled in read-only mode while the meta data is still being read. This is in particular of benefit when the meta data has been destroyed and needs to be recovered from PNFS: Computing the checksum during such recovery takes a while and would in previous releases block access to all files on the pool. With the above change, clients will be able to read a file as soon as the meta data of that particular file was recovered. Any write operation is blocked until all meta data has been verified.
The implementation of volatile pools has been refactored: The LFS policy now only applies to client uploads and when recovering meta data. In other cases the state of the replica will be set as usual. For instance when migrating files to a volatile pool using the migration module, those files would previously be marked CACHED no matter what was requested by the migration task. On the other hand, volatile pools now correctly ignore the access latency and retention policy of files. In earlier versions files written with access latency ONLINE would be marked sticky even when written to volatile pools.
The code base for queue management in pools has been cleaned up. As a consequence the definition of default queues has changed. In previous releases the default queue happened to be the first queue defined. Starting with 1.9.9, a queue called 'regular' is always added and the list of custom queues must not contain a default queue. All pools that define the poolIoQueue configuration parameters must be reconfigured when upgrading to 1.9.9.
Since the poolIoQueue parameter no longer contains the default queue, it is not possible to configure the default queue in LIFO mode. This is a regression that will be resolved in a future release.
The queue used for pool to pool transfers is now a regular queue called p2p. It can be managed using the mover commands in the admin shell. The commands p2p remove and p2p kill are obsolete. Third party scripts that use these commands have to be updated.
Mover states have changed. This affects the output of the mover ls command as well as the Active Transfers page in the monitoring interface. Third party scripts may need to be adapted.
The checksum module was extended with an onflush switch. This forces the pool to compute checksums on files before flushing them to tape. The computed checksum is compared to a known checksum and the flush fails in case of mismatch. If a checksum is not known already, then the computed checksum is stored in the name space.
The migration module was significantly enhanced. A new expression language allows for more flexibility in pool selection, job termination, and job suspension. Use cases include suspending a job during high load, excluding target pools with high load, and terminating a job after a certain amount of data has been moved. Please consult the help output of the migration copy command, or the dCache book.
The migration module has been extended with the -order option. This option makes it possible to specify a sort order for migration jobs, e.g. oldest files first.
The migration module was extended to reduce interference between multiple migration jobs running on the same pool. The migration module now respects the pool mode on the target pool.
A common problem when adding new pools to dCache is that the new pools have significantly more free space than full pools. They naturally attract more data and thus are subject to overload during transfer peaks. Since data is often read back in the same batches in which they were written, this problem persists even after the new pool has been filled with data.
To address this issue many sites have moved files from old pools to new pools to even out the load. Before the introduction of the migration module, this task required manually moving files behind the back of dCache (e.g. using scp or rsync). With the introduction of the migration module in dCache 1.9.1 this task became simpler and safer, but still required that migration jobs were defined by hand on the old pools - and cancelled once enough data had been moved.
In dCache 1.9.10 a couple of new commands were added to pool manager to rebalance pool groups. These commands automate generating migration jobs. The migration jobs generated this way rely heavily on the new features added to the migration module in 1.9.10 and will only work with new pools.
Pool manager maintains no state about the migration jobs started by the rebalance commands. All such jobs however share a command job id, rebalance, that makes them easy to identify. Pool manager will cancel existing rebalance jobs on a pool before starting a new job. Pool manager also provides a command to cancel all such jobs within a pool group. Please read the help of the rebalance pgroup command for details or consult the dCache book.
DISCLAIMER: This feature is experimental. We urge all sites to carefully test this feature before using it on a production system, as well as monitoring the system while rebalancing is in progress. The task of moving files between pools is handled by the migration module, which has been used successfully by many sites since its introduction in dCache 1.9.1. The rebalancer will however stress the migration module much more than a typical migration job.
SRM is available as a service called srm and can be enabled in the layout file.
In contrast to dCache 1.9.5, the SRM service uses Jetty as a container rather than Tomcat. Jetty provides a much more smooth integration of the SRM with the rest of dCache. Some of the advantages are:
Jetty has the option of using synchronous and asynchronous IO. In previous versions, dCache always used synchronous IO for the SRM. The consequence was that each TCP connection required its own thread. This limited the maximum number of TCP connections the SRM could efficiently maintain. Since dCache 1.9.10 the SRM can optionally use asynchronous IO. This will only make a difference if clients use HTTP keep alive to keep the TCP connection open for some time while the connection is idle. Whether to use synchronous or asynchronous IO can be configured using the srmJettyConnectorType property.
Traditionally the SRM door failed to reload the host certificate and CA certificates and required a restart whenever these were updated. Since dCache 1.9.10 both host certificates and CA certificates can be reloaded periodically. The refresh period can be configured using the hostCertificateRefreshPeriod and trustAnchorRefreshPeriod properties. The default is to reload every 12 hours.
The SRM protocol usually relies on HTTP over GSI as a transport. In dCache 1.9.11 HTTPS was added as a transport option for the SRM. The main benefits of using HTTPS rather than HTTP over GSI is that HTTPS is a standard protocol and has support for sessions, improving latency in case a client needs to connect to the same server several times. Our current implementation does not offer a delegation service. Hence srmCopy will not work with SRM over HTTPS. We are working on adding a separate delegation service in a later release.
There are currently no clients with SRM over HTTPS support available for download, but both the dCache SRM clients and the ARC clients are in the process of being updated to support the new transport.
To enable HTTPS as a transport, set srmGssMode to SSL. In the current version the SRM can only run in either SSL or GSI mode. To support both, two SRM instances have to be created. For such a deployment it is essential that the two SRM instances use two different databases.
The subject dicussed below is unrelated to the use of synchronous or asynchronous IO discussed in the previous section.
The SRM protocol allows most requests to be treated either synchronously or asynchronously. In the former case the client will not receive a reply until the request was processed by the server. In the latter case the server will reply that the request is accepted and then the client periodically asks (polls) the server about the status of the request.
Traditionally dCache would treat list requests synchronously, and put, get and bring online requests asynchronously. Starting with dCache 1.9.3 the behaviour of lists was configurable. Since dCache 1.9.10 the behaviour for both list, get, put and bring online is configurable. One does not have to choose between either synchronous or asynchronous operation. dCache can be configured to use synchronous replies for fast requests, and switch to asynchronous processing for slow requests. The switch over period is configurable. When configured to 0 all replies are asynchronous. When configured to infinity all replies are synchronous. The upside of synchronous replies is lower CPU overhead since this avoids repeated authentication, and lower latency from the point of view of the user (an asynchronous request will always take at least the time between the initial request and the first status poll - typically four seconds). The downside of synchronous replies is that the request occupies a thread in the SRM door. A conservative switch over time of a few seconds is unlikely to cause problems on the server. The period before the SRM switches to asynchronous mode can be configured with the srm*SwitchToASynchronousModeDelay parameters in /opt/d-cache/etc/dcache.conf. The default value is one second.
The SRM uses a database to persist all SRM requests. This has the advantage that the SRM door can be restarted and requests can be restored from the database. It also means that one can extract information about already completed SRM requests, which is great for monitoring and debugging. The aggressive updates of the database may however be a bottleneck for SRM scalability. Continous persistence can now be disabled using the parameters srmDatabaseEnabled (defaults to true) and srmStoreCompletedRequestsOnly (defaults to false).
Most of the persistence setting are now configurable per request type. This allows, for instance, the period for how long requests are kept in the database to be configured seperately for uploads, downloads and list requests.
The SRM used to have the ability to periodically submit vacuum commands to the database. This command is PostgreSQL specific and thus tied the SRM to PostgreSQL. PostgreSQL has for quite a while supported auto vacuuming, and manually executing the vacuum command is no longer necessary. We have therfore removed the submission of vacuum commands from dCache. The configuration parameters srmVacuum and srmVacuumPeriod are obsolete. PostgreSQL 8.4 is the minimum version of PostgreSQL supported by dCache.
The SRM protocol allows the client to provide the expected file size before actually uploading the file. dCache uses this file size to reserve sufficient space for the file. Since dCache 1.9.11 the file size is also propagated to PoolManager and the pool receiving the upload. PoolManager uses the file size in its pool selection. The pool uses the file size to verify the integrity of the file after upload. If the file size does not match, the pool will flag the transfer as having failed and will delete the file.
The RemoteHttpTransferManager cell has been removed. The functionality has been merged into the RemoteGsiftpTransferManager cell, which in turn has been renamed to RemoteTransferManager. The SRM was extended to support third party srmCopy transfers from HTTP resources. Third party upload to HTTP resources and support for HTTPS is not available yet in the SRM.
The pin manager is responsible for staging and pinning files on behalf of SRM and DCAP doors. In this release the pin manager has been reimplemented from scratch. The design has been significantly simplified to increase performance and robustness of this critical component.
The database schema is separate from the old implementation's schema. The new schema is created the first time the pin manager is started. Existing pins are automatically imported if the same database is used. We strongly recommend using the same database. The tables of the old pin manager are not deleted, however downgrading is not possible without loosing the pins that were created with the new pin manager.
The command line interface of the pin manager has changed slightly and third party scripts may have to be updated.
DCAP supports prestaging from tape. The DCAP door used to rely on a stager service to perform the staging. In dCache 1.9.12 the DCAP door instead uses the pin manager to trigger staging. The dummy-prestager service is obsolete and can be deleted from the layout.
Since version 1.9.5 dCache has had the option to do authorization in the door or in PNFS manager. File access authorization is now always done in PNFS manager. Code to perform permission checks in doors has been removed. The configuration parameter permissionPolicyEnforcementPoint is obsolete.
One consequence of the above change is that ACL support only needs to be enabled on the node hosting the PNFS manager. All other services ignore the ACL settings. If ACL checks are enabled, please double check that ACL checking is enabled and configured on the node hosting the PNFS manager. Another consequence is that ACLs are now enforced for all supported protocols, including all SRM operations.
dCache has traditionally placed a lot of logic in protocol specific doors. Over time this has caused the semantics of various operations to diverge slightly, depending on which protocol is used. To counter this, the logic that decides the ownership of a new name space entry was moved into PNFS manager. This change altered the behaviour of some doors. The new behaviour is closer to POSIX: the default ownership of a new name space entry is the UID and primary GID of the user creating the entry. There are two exceptions: if the entry was created by an anonymous user then the ownership is inherited from the parent directory; if pnfsInheritFileOwnership is set to true in /opt/d-cache/etc/dcache.conf then entries created by authenticated users will also have ownership inherited from the parent directory.
dCache no longer ships with /opt/d-cache/config/PoolManager.conf. The equivalent default values are now embedded in a batch file. As before, the PoolManager uses /opt/d-cache/config/PoolManager.conf to store its setup. The ONLY difference is that dCache no longer ships with a /opt/d-cache/config/PoolManager.conf file. This was changed to avoid the risk of the default PoolManager.conf overwriting a custom PoolManager.conf file on upgrade.
IMPORTANT: Before restarting PoolManager after upgrade, please verify that your custom PoolManager.conf is in place. The RPM contains instructions to preserve custom PoolManager.conf files, however we recommend to verify the presence of the file before starting dCache after upgrading.
One task of space manager is to select the link group to use for writes. Space manager uses a link group authorization file to determine which link groups a user is allowed to write to.
One challenge space manager has is when a user is authorized to write to multiple link groups. This can for instance be the case when the user has multiple FQANs. In previous releases the space manager would choose the first link group the user is authorized to write to. There is however no guarantee that the link group selected by the space manager contains links serving the given path. If this happened the write would fail because no write pools were configured for this file in the link group chosen by space manager.
Starting with dCache 1.9.9 space manager queries pool manager to select the proper link group. The query guarantees that space manager selects a LinkGroup with a link that allows the given file to be written. To reduce the cost of the new query, it is only generated for writes with implicit space reservation and only if the user is authorized to more than one link group.
The link group authorization file was extended to support wild card matches for FQANs.
The gPlazma cell was rewritten. The admin interface of the gPlazma cell has been replaced. The output of the info command has changed. Third party scripts that query the gPlazma cell through the admin interface will have to be updated.
gPlazma request timeouts are now defined by the door making the request. As a consequence the configuration parameter gPlazmaRequestTimeout is now obsolete.
The gPlazma module has been replaced by an embedded instance of the new gPlazma cell running in the same domain as the doors. The configuration parameters have not changed. If gPlazma is configured to run as a module then the cell will automatically be started in every domain hosting a door.
Despite its name, gPlazma 1 was not actually a pluggable framework. gPlazma 2 tries to address this issue. It is rewritten from scratch with a well defined plugin interfaces and a dynamic plugin loader mechanism.
gPlazma 2 is enabled by defining gplazma.version=2 in dcache.conf and is configured through a new configuration file: /opt/d-cache/etc/gplazma.conf. The file follows a PAM like syntax. The following is an example of a gPlazma 2 configuration file:
auth optional voms auth optional x509 map optional vorolemap map optional gridmap map requisite authzdb session requisite authzdb
The format of a line in gplazma.conf is
(auth|map|account|session|identity) (optional|sufficient|required|requisite) PLUGIN [KEY=VALUE]...
where PLUGIN is the name of a plugin and KEY is the name of a configuration property. Plugin configuration is integrated with dCache's configuration system. Therefore configuration defaults can be found in /opt/d-cache/share/defaults/ and can be redefined in dcache.conf, in the layout file, and in gplazma.conf.
When a user connects to a dCache door, the door sends the login information to gPlazma. gPlazma processes the information in four steps:
gPlazma invokes the plugins of each phase in order. Any plugin may succeed or fail and the gPlazma configuration specifies how to process a success or failure:
Identity plugins are the fith type of plugin and are not used for processing login requests. Identity plugins are used by services that have to map between external and internal user identities independent of whether such mappings would be authorized or not. For instance FTP directory listing needs to map UIDs and GIDs used for file ownership to user identities that make sense to an external user. Likewise NFS4 ACL support has to map an external user identity to the corresponding UID and GID in order to update the internal ACLs.
A list of available plugins is provided in the reference section.
From Wikipedia: "Web-based Distributed Authoring and Versioning, or WebDAV, is a set of extensions to the Hypertext Transfer Protocol (HTTP) that allows computer-users to edit and manage files collaboratively on remote World Wide Web servers. RFC 4918 defines the extensions." Wikipedia further notes: "The WebDAV protocol allows interactivity, making the Web a readable and writable medium, in line with Tim Berners-Lee's original vision. It allows users to create, change and move documents on a remote server (typically a web server or "web share"). This has obvious uses when authoring the documents that a web server serves, but it can also be used for storing files on the web, so that the files can be accessed from anywhere."
WebDAV is supported by all modern operating systems out of the box. This includes Windows XP, Windows Vista, Windows 7, Mac OS X, and Gnome and KDE shells for Linux and Unixes. Third part clients for iOS and Android are available.
The WebDAV door in dCache replaces the HTTP door. To enable the WebDAV door add the webdav service to the layout. You may want to adjust the default parameters for the WebDAV door. Consult /opt/d-cache/share/defaults/webdav.properties for available parameters and documentation on those parameters.
The door supports both unauthenticated HTTP and HTTPS and client certificate based authentication over HTTPS. To enable HTTPS, set webdavProtocol=https or webdavProtocol=https-jglobus. For the former the host and CA certificates need to be imported to be readable by the Java SSL libraries. To do this, run the commands /opt/d-cache/bin/dcache import hostcert and /opt/d-cache/bin/dcache import cacerts. For the latter host and CA certificates are read from /etc/grid-security/. The latter option also enables support for proxy certificates, including those generated by VOMS.
The HTML rendering of a directory listing is customizable through a template file. Please read the document embedded in the default template file /opt/d-cache/share/webdav/templates/list.stg for details on customizing the look and feel.
The WebDAV door can be configured to proxy PUT or GET requests through the door. This is useful when pools are not accessible by clients or if clients do not support the redirect reply. The property webdav.redirect.on-read controls whether GET requests are redirected to pools or proxied through doors. The default is to redirect the client. PUT requests are currently always proxied through the door.
RFC 2518 specifies that a PUT performed on an existing resource replaces the GET response entity of the resource. That means a PUT overwrites existing files. However by default dCache refuse to overwrite existing files. The behaviour is controlled by the webdav.overwrite parameter.
The HTTP mover has been reimplemented and now relies on the Netty asynchronous event-driven network application framework. Whereas the old mover allocated a TCP port for each transfer, the new mover uses a shared TCP port for all HTTP transfers. The mover generates a UUID, which is included in the HTTP redirect generated by the door. The new mover supports HTTP/1.1 and HTTP keep alive.
The interpretation of xrootdAllowedPaths has changed. The parameter now applies to both reading and writing. The parameters xrootdReadPaths and xrootdWritePaths have been added to allow reading and writing to be configured separately.
The interpretation of the empty string for xrootdAllowedPaths has changed: An empty string used to allow writing to all paths. Now it disallows access to any path. To allow access to all paths the property has to be set to / (which is the default).
The permission check on write has changed. In previous releases the parent directory had to have write and execute permissions set for the owner of the directory. Starting with dCache 1.9.8 both reads and writes are subject to regular authorisation checks. For anonymous transfer the authorization happens against the user specified by the xrootdUser property. The default is nobody, meaning that only world readable/writable files can be read/written. The xrootdUser property also controls the ownership of files and directories created by anonymous clients.
Notice transfers authorized by the TokenAuthorizationFactory security plugin are anonymous transfers and are subject to file system authorization checks for the user specified by the xrootdUser property.
The XROOTD protocol has a plugable authentication mechanism. dCache 1.9.11 added support for GSI authentication. The XROOTD door integrates with gPlazma for VOMS authorization and identity mapping. Regular XROOTD clients and Root support the GSI authentication mechanism out of the box. Notice that the proxy certificate generated by the XROOTD client is not supported by dCache: JGlobus rejects the proxy certificate because it lacks key usage extensions. A regular proxy certificate generated with grid-proxy-init or voms-proxy-init works just fine.
Be aware that the XROOTD protocol only uses GSI for purposes of authentication. Neither the connection to the door (the xrootd redirector) nor the pool (the xrootd dataserver) are encrypted. No confidentiality or consistency guarantees are provided and thus man-in-the-middle attacks are possible. We recommend restricting access to a GSI xrootd door to trusted clients, to particular directories, or to restrict write access. This is different from GSIDCAP and GridFTP: Although the pool connection in those protocols is unauthenticated, the control channel to the door is GSI encrypted.
The xrootd door was extended to support the following new xrootd requests: kXR_rm (file removal), kXR_rmdir (directory removal), kXR_mkdir (directory creation), kXR_mv (move/rename), kXR_dirlist (directory listing) and kXR_prepare. Support for the option kXR_delete in the kXR_open was added. This option allows existing files to be overwritten.
The xrootd mover has been restructured such that it only needs to listen to a single TCP port. Previous versions opened a new port for each transfer. Since dCache 1.9.10 all xrootd mover instances within a single pool domain share a TCP port. For this to work the client must present a transfer specific UUID to the pool. The UUID is generated by the door when redirecting the client. This all happens within the bounds of the existing xrootd protocol and should work with all clients.
Configuration parameters for controlling FTP proxy behaviour have changed. The FTP proxy allows FTP doors to act as proxies, relaying data between pools and clients. The proxy behaviour is now controlled by the parameters ftp.proxy.on-active and ftp.proxy.on-passive. The parameter gsiftpAllowPassivePool is no longer supported and is flagged as a forbidden parameter preventing dCache from starting. The new parameters allow the proxy to be used for both active and passive transfers, whereas the old parameter only controlled passive transfers. The default behaviour has changed. The proxy now defaults to off for both passive and active transfers. Previously it defaulted to on for passive and was not used for active transfers.
Authorization for anonymous users in the weak FTP door has changed: Anonymous users are now limited to operations with world permissions. This behaviour is consistent with other doors allowing anonymous access in dCache.
The FTP door has been refactored and shares more code with the xrootd, NFS4, and WebDAV doors. One consequence of the refactored code is that a number of deadlocks have been fixed. These deadlocks are not easy to resolve in stable releases without significant code changes, so will not be backported. The deadlocks are triggered by restarting a pool at a particular moment during a GridFTP 2 transfer.
The code base of the DCAP door was cleaned up. Error handling, robustness and efficiency should be improved. Several inconsistencies in how authorization were handled were removed. In particular how client overrides for uid, gid, pid and role are handled is affected. The role client option was broken for all late 1.8 releases and all 1.9 releases up to 1.9.9; those releases interpreted role as an FQAN whereas earlier releases interpreted it as a user name. The behaviour of 1.7 and early 1.8 releases has been restored.
DCAP supports prestaging from tape. The DCAP door used to rely on a stager service to perform the staging. In dCache 1.9.12 the DCAP door instead uses the pin manager to trigger staging. The dummy-prestager service is obsolete and can be deleted from the layout.
NFS 4.1 support was introduced in dCache 1.9.3. Since then the robustness, performance and standard compliance has been significantly improved and the NFS 4.1 door is now considered production ready.
The nfsv4 service supports both NFS 4.1 and NFS 3. NFS 3 support has to be explicitly enabled by setting nfs.v3 to true.
The old info provider has been removed. To publish GLUE one has to use the info service and the XSLT based info provider.
The info provider is an XSLT script that transforms the output of the info service into GLUE compliant LDIF. The XSLT script used to be stored in /opt/d-cache/etc/glue-1.3.xml and used to contain both the transformation logic and local configuration properties.
In dCache 1.9.12 the info provider has been extended to support GLUE 2.0 in addition to GLUE 1.3, and it has been restructured to separate configuration properties from the transformation logic.
The old info provider file, /opt/d-cache/etc/glue-1.3.xml, is obsolete. All site specific configuration is done within the file /opt/d-cache/etc/info-provider.xml. The configuration properties httpHost, xsltProcessor, xylophoneConfigurationDir, xylophoneConfigurationFile, xylophoneXSLTDir, and saxonDir are deprecated and have been replaced by other properties. The property httpPort is obsolete. The property info-provider.publish specifies whether to publish GLUE 1.3, 2.0, or both. See /opt/d-cache/share/defaults/info-provider.properties for details.
Persistent command history in the admin shell now has to be explicitly enabled by defining the adminHistoryFile property.
The behaviour of the cd command in the admin interface has changed. Previously, if the command's target cell is not running then the cd command will succeed; however, subsequent commands would fail will a NoRouteToCellException error message. The new behaviour is that the cd command will check that the target cell is running. If the cell is running then the cd command will behave as before; if the cell isn't running then the cd command will fail and an error message is returned.
The location of the ssh keys used by the admin shell has changed from /opt/d-cache/config/ to /opt/d-cache/etc/.
The HTTP monitoring interface of dCache contains a pool group view. This view contained bugs that prevented pools that were shut down, disabled or removed from pool groups from being remove from the pool group view. In dCache 1.9.11 the pool group view has been reimplemented to solve this problem. The new implementation lacks some little used features of the old implementation. Please consult the release notes of 1.9.12 for further details.
Java options have been changed to request a memory dump whenever the Java virtual machine suffers from an out of memory condition. By default the memory dumps are stored in /var/log/, although the location can be changed by setting dcache.java.oom.location.
The init script generates a warning if any memory dumps are detected.
Use this checklist to plan the upgrade to 1.9.12. The checklist focuses on upgrading a single node and focuses on a manual upgrade. Upgrades involving installation scripts like dCacheConfigure, YAIM, or site specific deployment scripts are not covered by this process.
Sites that have modified batch files can still use the procedure below, however the procedure will be incomplete. We strongly recommend against modifying batch files provided by dcache.org; almost all settings can be modified in the main configuration file. If a batch must be modified we recommend to make a copy of the batch file, giving it a new name. This introduces a new service name and avoids that the batch file gets overwritten by upgrades. Notice that one has to copy service specific (scoped) defaults into the main configuration file and rename the scope to match the new service name.
All head nodes, that is, all services besides pools, have to be upgraded at the same time. Pools (but not doors) of releases 1.9.5 to 1.9.12 can be mixed with 1.9.12 head nodes, giving an opportunity to perform a staged upgrade.
[nfs-${host.name}Domain]
[nfs-${host.name}Domain/nfsv3]
| Term | Description |
|---|---|
| cell | A component of dCache. dCache consists of many cells. A cell must have a name which is unique within the domain hosting the cell. |
| domain | A container hosting one or more dCache cells. A domain runs within its own process. A domain must have a name which is unique throughout the dCache instance. |
| service | An abstraction used in dCache configuration files to describe atomic units to add to a domain. A service is typically implemented through one or more cells. |
| layout | Definition of domains and services on a given host. The layout is specified in the layout file. The layout file may contain both domain- and service- specific configuration values. |
| pool | A service providing physical data storage. |
This section lists all supported services. Those marked with a * are services that dCache requires to function correctly.
| Name | Decscription |
|---|---|
| broadcast∗ | Internal message broadcast service. |
| cleaner∗ | Service to remove files from pools and tapes when the name space entry is deleted. |
| pnfsmanager∗ | Gateway to name space (either PNFS or Chimera). |
| pool∗ | Provides physical data storage. |
| poolmanager∗ | Central registry of all pools. Routes transfers to pools, triggers staging from tape, performs hot spot detection. |
| dir | Directory listing support for DCAP. |
| gplazma | Authorization cell |
| loginbroker | Central registry of all doors. Provides data to SRM for load balancing. |
| srm-loginbroker | Central registry of all SRM doors. |
| pinmanager | Pinning and staging support for SRM and DCAP. |
| replica | Manages file replication for Resilient dCache. |
| spacemanager | Space reservation support for SRM. |
| Name | Decscription |
|---|---|
| acl | Administrative interface for ACLs. |
| admin | SSH based admin shell. |
| billing∗ | Service for logging to billing files or the billing database. |
| httpd | Monitoring portal. Depends on: loginbroker, topo. |
| info | Info service that collects information about the dCache instance. Recommends: httpd |
| statistics | Collects usage statistics from all pools and generates reports in HTML. |
| topo | Builds a topology map of all domains and cells in the dCache instance. |
| webadmin | Next generation web admin portal. Depends on: info |
| Name | Decscription |
|---|---|
| authdcap | Authenticated DCAP door. Depends on: dir. Recommends: pinmanager. |
| dcap | dCap door. Depends on: dir. Recommends: pinmanager. |
| gsidcap | GSI dCap door. Depends on: dir. Recommends: pinmanager. |
| kerberosdcap | Kerberized dCap door. Depends on: dir. Recommends: pinmanager. |
| ftp | Regular FTP door without strong authentication. |
| gridftp | GridFTP door. |
| kerberosftp | Kerberized FTP door. |
| nfsv3 | NFS 3 name space export (only works with Chimera). |
| nfsv41 | NFS 4.1 door (only works with Chimera). |
| srm | SRM door. Depends on: pinmanager, loginbroker, srm-loginbroker. Recommends: transfermanagers, spacemanager. |
| transfermanagers | Server side srmCopy support for SRM. |
| webdav | HTTP and WebDAV door. |
| xrootd | XROOT door. |
The following gPlazma 2 plugins are shipped with dCache and can be used in gplazma.conf. Note that several plugins implement more than one type. Usually such plugins should be added to all phases supported by the plugin.
| Name | Type | Description |
|---|---|---|
| x509 | auth | Extracts the DN from an X.509 certificate. The certificate chain is not validated (it is assumed that the door already validated the chain). The plugin fails if no certificate chain is provided. |
| voms | auth | Validates any VOMS attributes in an X.509 certificate and extracts all valid FQANs. Requires that a vomsdir is configured. Fails if no valid FQAN is found. |
| kpwd | auth | Verifies username and password credentials using the kpwd file. Fails if no username or password is provided, if the username is not defined in the kpwd file, if the password is invalid, or if the entry has been disabled in the kpwd file. |
| gridmap | map | Maps DN principals to user name principals according to a grid-mapfile. Fails if no DN was provided or no mapping is found. |
| vorolemap | map | Maps FQAN principals to group name principals according to a grid-vorolemap file. Each FQAN is mapped to the first entry that is a prefix of the FQAN. The primary FQAN (the first in the certificate) is mapped to the primary group name. Fails if no FQAN was provided or no mapping was found. |
| kpwd | map | Maps user names, DNs and Kerberos principals according to the kpwd file. Only user names verified by the kpwd auth plugin are mapped. Fails if nothing was mapped or if the kpwd entry has been disabled. Maps to user name, UID and GID principals. |
| authzdb | map | Maps user and group name principals to UID and GID principals according to a storage authzdb file. The file format does not distinguish between user names and group names and hence each entry in the file maps to both a UID and one or more GIDs. Therefore the UID and the primary GID are determined by the mapping for the primary group name or user name. The name of that mapping is kept as the user name of the login and may be used for a session plugin or for authorization in space manager. Remaining GIDs are collected from other mappings of available group names. |
| argus | account | |
| kpwd | account | Fails if the kpwd entry used during the map has been disabled. |
| authzdb | session | Associates a user name with root and home directory and read-only status according to a storage authzdb file. |
| kpwd | session | Adds home and root directories and read-only status to the session. Only applies to mappings generated by the kpwd map plugin. |
Please consult /opt/d-cache/share/defaults/gplazma.properties for details about available configuration properties.
Most of the service specific configuration parameters known from the old /opt/d-cache/config/dCacheSetup file can still be defined in /opt/d-cache/etc/dcache.conf. Some have however been removed or replaced and others have been added. The following tables provide an overview of the properties that may need to be changed when upgrading from dCache 1.9.5 to 1.9.12.
| Property | Alternative | Description |
|---|---|---|
| user | dcache.user | Defines the user account to run dCache under. |
| pidDir | dcache.pid.dir | The directory containing PID files. |
| logArea | dcache.log.dir | The directory containing log files. |
| logMode | dcache.log.mode | Whether to rename log files on every restart. |
| classpath | dcache.java.classpath | Classpath for add-on JARs |
| ourHomeDir | dcache.home | Automatically set to value of DCACHE_HOME environment variable |
| librarypath | dcache.paths.lib | Load path for add-on shared libraries |
| config | dcache.paths.config | Location of dCache's config directory |
| classesDir | dcache.paths.classes | Location of dCache's classes directory |
| keyBase | dcache.paths.ssh-keys | Location of ssh keys for admin door |
| messageBroker | broker.scheme | Message broker implementation |
| serviceLocatorHost | broker.host | Message broker host name |
| serviceLocatorPort | broker.port | Message broker TCP port |
| amqHost | broker.amq.host | Active MQ broker host |
| amqPort | broker.amq.port | Active MQ broker TCP port |
| amqUrl | broker.amq.url | Active MQ connection URL |
| portBase | Use the per door setings | |
| aclConnDriver | chimera.db.driver | DB driver for ACL access |
| aclConnUser | chimera.db.user | DB user name for ACL access |
| aclConnPswd | chimera.db.password | DB password for ACL access |
| cleanerArchive | cleaner.archive | Cleaner archive mode |
| cleanerDB | cleaner.book-keeping.dir | Cleaner book keeping directory |
| cleanerPoolTimeout | cleaner.pool-reply-timeout | Cleaner pool timeout |
| cleanerProcessFilePerRun | cleaner.max-files-in-message | Cleaner message size |
| cleanerRecover | cleaner.pool-retry | Cleaner pool retry time |
| cleanerRefresh | cleaner.period | Cleaner frequency |
| hsmCleaner | cleaner.hsm | Cleaner support for HSM |
| hsmCleanerFlush | cleaner.hsm.flush.period | Cleaner failure flush frequency |
| hsmCleanerRecover | cleaner.pool-retry | Cleaner pool retry time |
| hsmCleanerRepository | cleaner.hsm.repository.dir | Cleaner HSM failure directory |
| hsmCleanerRequest | cleaner.hsm.max-file-in-message | Cleaner HSM message size |
| hsmCleanerScan | cleaner.period | Cleaner frequency |
| hsmCleanerTimeout | cleaner.hsm.pool-reply-timeout | Cleaner pool timeout |
| hsmCleanerTrash | cleaner.hsm.trash.dir | Cleaner HSM trash directory |
| hsmCleanerQueue | cleaner.hsm.max-current-requests | Cleaner HSM request limit |
| trash | cleaner.trash.dir | Cleaner trash directory |
| gsiftpDefaultStreamsPerClient | Breaks protocol compliance | |
| httpHost | info-provider.http.host | Host running the httpd service |
| xsltProcessor | info-provider.processor | XSLT implementation to use |
| xylophoneConfigurationDir | info-provider.configuration.dir | |
| xylophoneConfigurationFile | info-provider.configuration.file | |
| xylophoneXSLTDir | info-provider.xylophone.dir | |
| saxonDir | info-provider.saxon.dir | |
| SpaceManagerDefaultAccessLatency | DefaulAccessLatencyForSpaceReservation | |
| srmDbHost | srmDatabaseHost | Host providing the SRM database |
| srmPnfsManager | pnfsmanager | Cell name of PNFS manager |
| srmPoolManager | poolmanager | Cell name of pool manager |
| srmNumberOfDaysInDatabaseHistory | srmKeepRequestHistoryPeriod | Days before old transfers are removed |
| srmOldRequestRemovalPeriodSeconds | srmExpiredRequestRemovalPeriod | Seconds between removing old transfers |
| srmJdbcMonitoringLogEnabled | srmRequestHistoryDatabaseEnabled | Enables SRM request transition history |
| srmJdbcSaveCompletedRequestsOnly | srmStoreCompletedRequestsOnly | |
| srmJdbcEnabled | srmDatabaseEnabled | |
| httpPortNumber | webdavPort | WebDAV TCP port |
| httpRootPath | webdavRootPath | WebDAV name space root |
| httpAllowedPaths | webdavAllowedPaths | WebDAV path based authorization |
| kerberosRealm | kerberos.realm | |
| kerberosKdcList | kerberos.key-distribution-center-list | |
| authLoginConfig | kerberos.jaas.config | |
| kerberosScvPrincipal | kerberos.service-principle-name | |
| images | httpd.static-content.images | Location of images for httpd service |
| styles | httpd.static-content.styles | Location of style sheets for httpd service |
| Property | Description |
|---|---|
| dcache.namespace | Whether to use Chimera or PNFS. Replaces the namespace Parameter of etc/node_config. |
| dcache.layout | The layout to use; The layout filename is derived from this value. |
| pnfsVerifyAllLookups | verify lookup permissions of the entire path (starting with 1.9.12-22) |
| Name | Description |
|---|---|
| useFileSystem | No longer relevant |
| srmVacuum | Use PostgreSQL auto vacuuming instead |
| srmVacuumPeriod | Use PostgreSQL auto vacuuming instead |
| bufferSize | Had no effect |
| tcpBufferSize | Had no effect |
| maintenanceLibPath | Module has been removed |
| maintananceLibAutogeneraetPaths | Module has been removed |
| maintenanceLogoutTime | Module has been removed |
| srmVersion | No longer relevant |
| permissionHandler | No longer relevant |
| PermissionHandlerDataSource | No longer relevant |
| gsiftpPoolProxy | Determined automatically |
| ftpBase | Derived from gPlazma values |
| spaceReservation | Legacy space manager was removed |
| spaceReservationStrict | Legacy space manager was removed |
| gPlazmaRequestTimeout | Timeout is now controlled by the door |
| delegateToGPlazma | Functionality was removed |
| httpPort | Use httpdPort instead |
| httpdEnablePoolCollector | Functionality is now always enabled |
| srmAsynchronousLs | Use srmLsRequestSwitchToAsynchronousModeDelay |
| nostrongauthorization | Functionality was removed |
| removeUnexistingEntriesOnFlush | The logic was improved and orphaned files will automatically be removed if the file system entry no longer exist |
| Property | Alternative | Description |
|---|---|---|
| java | Define JAVA_HOME in /etc/dcache.env or /etc/default/dcache | |
| java_options | dcache.java.memory.heap | The Java heap size, For instance, 512m. |
| dcache.java.memory.direct | The Java maximum direct buffer size, For instance, 128m. | |
| net.wan.port.min | Lower bound on TCP port range for WAN protocols. | |
| net.wan.port.max | Upper bound on TCP port range for WAN protocols. | |
| net.lan.port.min | Lower bound on TCP port range for LAN protocols. | |
| net.lan.port.max | Upper bound on TCP port range for LAN protocols. | |
| net.inetaddr.lifetime | DNS cache time | |
| pool.dcap.port | TCP port for DCAP mover | |
| gsi.delegation.cache.lifetime | GSI delegation cache time | |
| gsi.crl.cache.lifetime | CRL cache time | |
| dcache.java.options.extra | Additional options to provide to the Java VM | |
| gsiftpAllowPassivePool | ftp.proxy.on-passive | Whether to proxy passive transfers. |
| waitForRepositoryReady | waitForFiles | Delay startup until file system was mounted |