1.9.10 Release Notes

In addition to the usual improvements and bug fixes, dCache 1.9.10 focuses on:

Upgrade Instructions

Incompatibilities

Please consider the following changes when upgrading from a version before 1.9.10-1:

Compatibility

It is safe to mix pools of releases 1.9.5, 1.9.6, 1.9.7, 1.9.8, 1.9.9, and 1.9.10. Head nodes and doors must be upgraded to 1.9.10 together and cannot be mixed with head nodes or doors of releases before 1.9.10. Components of different 1.9.10 releases can be mixed freely.

Compatibility Matrix

We distinguish between pool components and head nodes. Any component which is not a pool is considered a head node, including doors. The following table shows compatibility between different releases.

1.9.10-5 Head1.9.10-5 Pool
Head 1.9.5nono
1.9.7nono
1.9.8nono
1.9.9nono
1.9.10yesyes
Pool 1.9.5yesyes
1.9.7yesyes
1.9.8yesyes
1.9.9yesyes
1.9.10yesyes

Changes in 1.9.10-5

FTP

Improved RFC 3659 compliance.

Pool

Implemented waitForFiles configuration parameter (the parameter was documented before, but it didn't actually work).

Fixed a bug in the cancellation logic for pool to pool transfers. The symptoms of this problem was that a transfer would stay in the Failed state until the pool was restarted.

Fixed an issue in which a pool would attempt to update the file size and checksum when staging a file from tape or copying it from another pool.

Fixed a bug in the migration module that could cause migration jobs to never terminate. When the transfer queue became empty it would stay in the RUNNING state indefinitely.

Avoid unnecessary DNS reverse lookups in xrootd. Fixed a null pointer exception in the xrootd mover.

Fixed a deadlock in space allocation.

PoolManager

Restore compatibility with pools older than 1.9.10. Earlier versions of 1.9.10 only worked with 1.9.10 pools.

Deployment

Fixed configuration file parsing in various shell scripts. The symptom was that the scripts simply failed to work in earlier releases. This includes the script to migrate the configuration from 1.9.5.

Fixed Solaris compatibility.

Removed the transactionlog appender in the default log configuration. This appender caused a file called transaction.log to be created in the dCache installation directory.

Chimera

Fixed a bug that could cause name space entries to change type.

SRM

Fixed an issue that caused the SRM not to provide the client IP to PoolManager when pinning files. Thus the file could potentially be pinned on the wrong pool.

Fixed an issue with implicit space reservation. The symptom of the problem was that the error message Record(s) Already Exist was generated.

Fixed an IllegalArgumentException in case of clock shift.

gPlazma

Added the get mapping command to the gPlazma cell. This command was present in earlier releases, but was dropped when the gPlazma cell was reimplemented in 1.9.9.

Changed how gPlazma processes requests with multiple FQANs. Before gPlazma would invoke each plugin for the first FQAN, then each plugin for the second FQAN, and so on. This could in some cases lead to a request being authorized by a plugin late in the chain even though an earlier pluging could have performed the authorization.

gPlazma now processes all FQANs with the first plugin, then again with the second plugin, and so on. Thus the next plugin is only activated if the previous plugin is not able to process the complete request.

Xrootd door

Avoid unnecessary DNS reverse lookups. Fixed implementation of the kXR_mkpath option (the option creates parent directories on upload).

Downgraded the reported protocol version to 2.89. Recent versions of the protocol specification document the kXR_locate messages. This message doesn't fit well into the logical separation of doors and pools in dCache. Thus there is no way we can support this message without fundamental changes in dCache. Advertising an older protocol version causes client to use the kXR_open message followed by a redirect. This restores the behaviour of the xrootd door in dCache 1.9.5.

Added an implementation of the kXR_prepare message. The current implementation confirms the request, but ignores it otherwise. This is allowed according to the spec. A complete implementation will be added in future versions of dCache. The present implementation restores compatibility with the latest CMS software release.

Added logging support for the Netty IO framework on which the xrootd door is implemented.

Cells

Fixed a messaging loop. The issue was observed between space manager and pool manager and would cause a request to bounce back and forth between the two components.

Detailed changelog 1.9.10-4 to 1.9.10-5

Changes in 1.9.10-4

DCAP Door

Fixed a bug in prestage command that causes requests to get stuck.

Xrootd

Fixed interoperability with ROOT and encoding of opaque information. The fix requires upgrade of both the xrootd door and the pools.

WebDAV Door

Fixed HTTP compliance of the HEAD command.

Authorization

Fixed handling of secondary groups in gPlazma and FTP doors.

Detailed changelog 1.9.10-3 to 1.9.10-4

Changes in 1.9.10-3

Pool

Fixed two regressions that affected pool to pool transfers and performance cost computation.

Fixed partial retrieval of files when using FTP extended block mode.

FTP

Fixed NullPointerException in info command.

Detailed changelog 1.9.10-2 to 1.9.10-3

Changes in 1.9.10-2

Xrootd Door

Fixed retry logic on write. The bug would cause empty files to be left behind on write while the correct file with the same ID would uploaded to another pool.

Logging

Fixed a bug that would cause some log messages to be lost. This particularly affected log messages generated during startup. The bug affects all components in dCache.

Pools

Implemented the rep ls -l=c command. This command was mentioned in the documentation, but was never implemented.

Fixed the migration set concurrency command. In 1.9.10-1 the command caused an error to be reported even though the command did correctly change the concurrency of the migration job.

SRM

Upgraded to JGlobus 1.8.0. This resolves a compatibility issue with VOMS RFC proxy certificates.

Fixed a minor security bug.

Fixed performance regression in directory listing.

Temporary TopLink related files are now written to /tmp rather than /opt/d-cache/.

Messaging

Upgraded to ActiveMQ 4.5.1. Only affects deployments with JMS.

PinManager

Fixed a bug that caused timeout handling to fail. The observable effect was that stage requests that didn't complete within 1 hour would hang.

Detailed changelog 1.9.10-1 to 1.9.10-2

Changes in 1.9.10-1

Name Space

dCache has traditionally placed a lot of logic in protocol specific doors. Over time this has caused the semantics over various operations to diverge slightly, depending on which protocol is used. To counter this, we have moved some of the name space logic from the doors into PnfsManager. This is in keeping with previous consolidation where, in earlier releases, we moved file access authorization to PnfsManager.

In dCache 1.9.10 we moved the logic that decides the ownership of a new name space entry into PnfsManager. In doing this, we altered the behaviour of some doors. The new behaviour is closer to POSIX: the default ownership of a new name space entry is the UID and primary GID of the user creating the entry. There are two exceptions: if the entry was created by an anonymous user then the ownership is inherited from the parent directory; if pnfsInheritFileOwnership is set to true in etc/dcache.conf then entries created by authenticated users will also have ownership inherited from the parent directory.

Logging

Many of our users have been dissatisfied with the relative complexity of adjusting log levels at runtime. For some time, dCache has relied on log4j as its logging framework. Log4j is no longer maintained and there was no way for us to simplify the logging user interface within the constraints of log4j. Therefore, with 1.9.10, dCache has switched from log4j to logback.

The first consequence is that the log configuration file has changed. Now logging is configured in etc/logback.xml. We refer to the instructions embedded in the packaged version of that file and to chapter 3 of the logback manual.

Notice that log level configuration in dCache is different from what is described in the logback manual. Pure logback allows log levels to be configured on loggers and appenders, similar to log4j. This has proven to be too inflexible for dCache. We ship dCache with a custom log filter that implements a log level for each triple of logger, appender and cell. This provides very fine grained control over which types of messages from which cell are logged where. Detailed instructions can be found in the instructions embedded in the default etc/logback.xml file.

The second consequence of switching to logback is that the cell admin interface to changing log levels on-the-fly has changed. The legacy set printout commands no longer have any effect. The commands that were prefixed with log4j are no longer supported. Instead a new set of commands prefixed with log have been introduced. Each cell exposes these commands to adjust the log level of logger hierarchies for specific appenders within that cell. In the simplest case one can adjust the log level of all loggers for a particular appender with:

log set stdout DEBUG

or

log set pinboard DEBUG

Notice that these commands only affect the cell and its subcells in which they are executed. In previous versions of dCache, adjusting the log levels affected all cells.

The system cell has additional commands that allow appenders to be attached and detached as needed, although it is unlikely that these will need to be adjusted at runtime.

We recommend consulting the logback manual for details about available appenders. Logback can be configured to perform log rotation, log to syslog, log to SMTP, log to a remote logger, etc. Feel free to contact us for details about specific setups.

SRM

In dCache 1.9.7 we switched from using Tomcat to Jetty as our preferred servlet container for the SRM. In dCache 1.9.9 we stopped shipping Tomcat with dCache. In dCache 1.9.10 we have unified the Jetty hosting cell and the SRM cell in a single cell. The main difference is that logging and log configuration is now perform in one cell rather than two. The SRM startup time also improved significantly.

The SRM protocol allows most requests to be treated either synchronously or asynchronously. In the former case the client will not receive a reply until the request was processed by the server. In the latter case the server will reply that the request is accepted and then the client periodically asks (polls) the server about the status of the request. Traditionally dCache would treat list requests synchronously, and put, get and bring online requests asynchronously. Starting with dCache 1.9.3 the behaviour of lists was configurable. In dCache 1.9.10 one can configure the behaviour for both list, get, put and bring online. Even more, one does not have to choose between either synchronous or asynchronous operation. dCache can be configured to use synchronous replies for fast requests, and switch to asynchronous processing for slow requests. The switch over period is configurable. When configured to 0 all replies are asynchronous. When configured to infinity all replies are synchronous. The upside of synchronous replies is lower CPU overhead since this avoids repeated authentication, and lower latency from the point of view of the user (an asynchronous request will always take at least the time between the initial request and the first status poll - typically four seconds). The downside of synchronous replies is that the request occupies a thread in the SRM door. A conservative switch over time of a few seconds is unlikely to cause problems on the server. The period before the SRM switches to asynchronous mode can be configured with the srm*SwitchToASynchronousModeDelay parameters in etc/dcache.conf.

The SRM uses a database to persist all SRM requests. This has the advantage that the SRM door can be restarted and requests can be restored from the database. It also means that one can extract information about already completed SRM requests, which is great for monitoring and debugging. The aggressive updates of the database may however be a bottleneck for SRM scalability. In dCache 1.9.10, continous persistence can be disabled using the parameters srmJdbcEnabled (defaults to true) and srmJdbcSaveCompletedRequestsOnly (defaults to false).

Jetty has the option of using synchronous and asynchronous IO (this is unrelated to synchronous and asynchronous request processing in the SRM protocol). In previous versions dCache always used synchronous IO for the SRM. The consequence was that each TCP connection required its own thread. This limited the maximum number of TCP connections the SRM could efficiently maintain. In dCache 1.9.10 the SRM can optionally use asynchronous IO. This will only make a difference if clients use HTTP keep alive to keep the TCP connection open for a long time while the connection is idle. Whether to use synchronous or asynchronous IO can be configured using the srmJettyConnectorType parameter.

Traditionally the SRM door has failed to reload the host certificate and CA certificates and has required a restart whenever these were updated. Starting with dCache 1.9.10 both host certificates and CA certificates can be reloaded periodically. The refresh period can be configured using the hostCertificateRefreshPeriod and trustAnchorRefreshPeriod parameters. The default is to reload every 12 hours.

Space manager

Database access was optimized to make better use of indexes.

WebAdmin

WebAdmin is a replacement service for the httpd admin/monitoring interface of dCache. The service is still work-in-progress, but we invite sites to have a look and provide feedback. The service is enabled like any other service in dCache: Simply add the webadmin service to your layout file. The default port of WebAdmin is 8080.

In contrast to the old httpd service, the webadmin:

Most information exposed through the WebAdmin interface is pulled from the info service. Hence the info service must be running for WebAdmin to function.

Notice that there are known issues in running WebAdmin in a system with a JMS message broker.

Pool Rebalancing

A common problem when adding new pools to dCache is that the new pools have significantly more free space than full pools. They naturally attract more data and thus are subject to overload during transfer peaks. Since data is often read back in the same batches in which they were written, this problem persists even after the new pool has been filled with data.

To address this issue many sites have moved files from old pools to new pools to even out the load. Before the introduction of the migration module, this task required manually moving files behind the back of dCache (e.g. using scp or rsync). With the introduction of the migration module in dCache 1.9.1 this task became simpler and safer, but still required that migration jobs were defined by hand on the old pools - and cancelled once enough data had been moved.

In dCache 1.9.10 we added a couple of new commands to PoolManager to rebalance pool groups. These commands automate generating migration jobs. The migration jobs generated this way rely heavily on the new features added to the migration module in 1.9.10 and will only work with 1.9.10 pools.

PoolManager maintains no state about the migration jobs started by the rebalance commands. All such jobs however share a command job id, rebalance, that makes them easy to identify. PoolManager will cancel existing rebalance jobs on a pool before starting a new job. PoolManager also provides a command to cancel all such jobs within a pool group. Please read the help of the rebalance pgroup command for details or consult the dCache book.

We plan to add additional features in 1.9.11, such as the ability to query the progress of the rebalance jobs and to rebalance a custom set of pools rather than a complete pool group.

DISCLAIMER: This feature is experimental. We urge all sites to carefully test this feature before using it on a production system, as well as monitoring the system while rebalancing is in progress. The task of moving files between pools is handled by the migration module, which has been used successfully by many sites since its introduction in dCache 1.9.1. The rebalancer will however stress the migration module much more than a typical migration job.

Pool

The migration module was significantly enhanced. A new expression language allows for more flexibility in pool selection, job termination, and job suspension. Use cases include suspending a job during high load, excluding target pools with high load, and terminating a job after a certain amount of data has been moved. Please consult the help output of the migration copy command, or the dCache book.

In dCache 1.9.1 the meta data store of pools was rewritten. The rewrite solved some longstanding issues with space accounting in pools. At the time we decided on a single global lock on any meta data operation, as that made it significantly easier to design the component. Since the meta data store is backed by an on-disk representation, the single lock has proven to be a bottleneck during high load. In dCache 1.9.10 the single lock is replaced by more fine grained locking. This improves throughput to the meta data store. Such a change however comes with a risk of introducing deadlocks and race conditions and we urge sites to stress test the pools before upgrading.

The checksum module was extended with an onflush switch. This forces the pool to compute checksums on files before flushing them to tape. The computed checksum is compared to a known checksum and the flush fails in case of mismatch. If a checksum is not known already, then the computed checksum is stored in the name space.

xrootd

In dCache 1.9.4 the xrootd door in dCache was replaced by a reimplementation based on the Netty network application framework. This door has proven more stable and scalable than the old door. In dCache 1.9.5 the xrootd mover was replaced with a new mover based on Netty. In dCache 1.9.9 the new xrootd door was extended to support new xrootd primitives. In dCache 1.9.10 we have removed the old xrootd door and xrootd mover completely. Sites that have used custom batch files to start the old door or mover are requested to use the new implementations instead.

The xrootd mover has been restructured such that it only needs to listen to a single TCP port. Previous versions opened a new port for each transfer. In dCache 1.9.10 all xrootd mover instances within a single pool domain share a TCP port. For this to work the client must present a transfer specific UUID to the pool. The UUID is generated by the door when redirecting the client. This all happens within the bounds of the existing xrootd protocol and should work with all clients.

The xrootd mover will now redirect clients back to the door for requests it cannot handle itself.

Scripts

The init scripts have been subject to plenty of changes in dCache 1.9.7, 1.9.8 and 1.9.9. dCache 1.9.10 is no exception to this trend. The script is now much faster and we have reduced the number of invocations of the Java interpreter.

FTP

The FTP door has been refactored and now shares more code with the xrootd, NFS4, and WebDAV doors.

One consequence of the refactored code is that a number of deadlocks have been fixed. These deadlocks are not easy to resolve in stable releases without significant code changes, so will not be backported. The deadlocks are triggered by restarting a pool at the wrong moment during a GridFTP 2 transfer.

HTTP/WebDAV

dCache 1.9.6 introduced the WebDAV door. WebDAV is a superset of HTTP and consequently there is no need for a seperate HTTP door in dCache. In dCache 1.9.10 we have removed the HTTP door. Please use the WebDAV door instead.

NFS 4.1

Compatibility with latest Linux kernels has been fixed. NFS transfers are now visible in the "Active Transfers" page. NFS transfers now generate billing transfers. Greatly improved performance.

Access to dot-files now works through NFS 4.1. This means NFS 4.1 can be used as a replacement for the NFS 3 name space export.

External libraries

A large number of external libraries have been updated in this release: Spring Framework 3.0.4, Active MQ 5.4.0, slf4j 1.6.1, Jetty 7.1.6, Netty 3.2.2, and Berkeley DB Java Edition 4.0.103.

Detailed changelog 1.9.9-1 to 1.9.10-1