Executive summary

  • Reduces chimera overhead on file creation and file open.

  • New database connection pool for more robust database interaction.

  • Supports NFS 4 (without parallel NFS)

  • DCAP clients can now be forced to do I/O through a mounted NFS 4 file system.

  • Much improved performance and standard compliance for NFS 4.

  • Improves support for meta data in FTP.

  • Improves FTP client compatibility.

  • Implements FTP delayed passive, allowing IPv6 FTP transfers without proxying.

  • New pool manager partition for transfer buffers (e.g. import and export pools).

  • Improves ROOT compatibility for HTTP access.

  • GridSite delegation support for SRM.

  • Unique TURLs for SRM upload for improved error handling.

  • Improves transfer protocol negotiation for SRM.

  • Reduces impact of SRM restart.

  • Improves protocol support for xrootd.

  • More flexible tape integration.


  • The spacemanager database schema has changed. Changes are automatically applied after upgrade. Downgrade is not possible without rolling back the schema changes. See the section on the spacemanager service for details.

  • The chimera database schema has changed. Changes are automatically applied during upgrade. Downgrade is not possible without rolling back the schema changes. See the section on chimera for details.

  • The new property dcache.upload-directory may have to be defined on upgrade. Read the section on the srm service for further details.

  • Third party plugins that provide custom service definitions may have to be updated to be compatible with dCache 2.9.

  • SRM uploads that are not finalized by submitting a srmPutDone will be deleted. The srmPutDone message is required by the SRM standard and standard compliant clients will not have a problem with this. The file is not available for reading until srmPutDone has been issued.

  • IPv6 support is enabled by default.

  • Tape integration was rewritten. Classic HSM scripts continue to be supported, however the pool setup file should be regenerated after upgrade. See the section on the pool service for details.

  • The HSM cleaner is enabled by default.

Release 2.9.27

Changes affecting multiple services

dCache uses a standard format to monitor the performance of various components: in the srm door to record how quickly SRM requests are processed (print srm counters command), the generic cell message monitoring (monitoring info command), and pnfsmanager service (the “Statistics” section in info). This release fixes a rounding error that prevents these statistics from including long-lived requests.


The pinmanager service has the ls command that allows the admin to limit the results to a specific pin or all pins against some PNFS-ID. This release fixes listing by pin id.


This release fixes the srm service so it cancels corresponding pinmanager requests when an SRM client aborts a bring-online request.

Changelog 2.9.26..2.9.27

[maven-release-plugin] prepare release 2.9.27
common: Fix division by zero regression in gauges
common: Fix rounding error in request gauge
srm: Abort pinning when cancelling bring-online requests
pinmanager: Fix listing by id
system-test: add missing dCache disposible CA certificate
system-test: add regenerated host and user credentials
[maven-release-plugin] prepare for next development iteration

Release 2.9.26


Update dcap door to avoid possible NullPointerException when describing an error to the client.

Changelog 2.9.25..2.9.26

[maven-release-plugin] prepare release 2.9.26
dcap: do not call toString() on error object
[maven-release-plugin] prepare for next development iteration

Release 2.9.25


When a client reads a file, the pool reads blocks of data from the local filesystem. When reading such a block, the pool could receive fewer bytes than requested. Previously, the pool assumed that this only happens when the end-of-file is reached; however, this is not guaranteed. Should this assumption be violated then the data sent to the client will be corrupt. In practise, the pool’s assumption is true for Linux and local filesystems; however, the code has been updated to remove this theoretical cause of corruption.

Changelog 2.9.24..2.9.25

[maven-release-plugin] prepare release 2.9.25
pool: Fix read corruption in HTTP mover
[maven-release-plugin] prepare for next development iteration

Release 2.9.24

Changes affecting multiple services

If a bug is found in dCache then it should be logged. For the older admin commands, an uninformative message was logged.

Previously, if a bug was discovered when starting up a service dCache would abort starting up with a non-informative log message. Now, the stack-trace is logged.


When writing into dCache with SRM, the Access Latency (AL) and Retential Policy (RP) may be specified or omitted. Additionally, the client may specify a space reservation into which the file should be written. If the client specifies both, they must match.

In dCache, there are three mechanisms to support a client that specifies neither AL/RP nor space reservation: the directory can have AccessLatency and RetentialPolicy tags, the directory can have the WriteToken tag, there are a system-wide default AL/RP values.

Previously, dCache would reject uploads where the user-supplied AL/RP information does not match the AccessLatency/RetentialPolicy tags, despite the latter being intended as default values.

With this release, if the client specifies neither space token, AL or RP then the directory tags will be used. If a directory specifies both WriteToken and AccessLatency/RetentionPolicy tags, then these have to be consistent. If the directory conains a WriteToken tag and the client specifies AL/RP, then the client specified values have to be consistent with the WriteToken tag.


When attempting to upgrade non-precious and non-cached files (e.g. a file marked broken), the receiving end of the migration module would answer twice: first (correctly) with a failure and then (incorrectly) with a success. This is now fixed.

Changelog 2.9.23..2.9.24

[maven-release-plugin] prepare release 2.9.24
cells: log bugs found by CellShell
cells: fix how bugs are reported from ac_ command.
pool: Fix bug in migration module upgrade logic
pnfsmanager: Fix upload to space token that conflicts with AL and RP tags
[maven-release-plugin] prepare for next development iteration

Release 2.9.23

Changes affecting multiple services

Various scripts, including the dcache command, invoke the java command with a list of directories in which Java should look for support libraries. Previously, the current working directory was (mistakenly) included in that list. This could lead to odd behaviour; one particular example is running a dcache database command from the /etc/dcache directory. This release fixes this problem by excluding the current directory.

When there is some problem in the communication between domains and error message is logged. Previously the explanation for the problem was logged as “null”. With this release, a more descriptive explanation is provided.


Fixed the ls spaces and ls files commands so they do not fail if there is a reservation without an owner.

Changelog 2.9.22..2.9.23

[maven-release-plugin] prepare release 2.9.23
spacemanager: Fix NPE in listing space reservations
tunnel: use toString if IOException#getMessage returns null
Exclude cwd from classpath
[maven-release-plugin] prepare for next development iteration

Release 2.9.22

Changes affecting multiple services

This patch fixes a race condition in Chimera that affects the nfs door and the pnfsmanager service. The effect is that, if two clients attempt to delete the same target (a file, link or directory) at the same time then the nlink count for the parent directory is decreased twice. “At the same time” means within the time taken to process the deletion; this is instance-specific but should be much less than 1 ms for well-configured systems. Sites can repare any incorrect nlinks with the following SQL:

UPDATE t_inodes SET inlink = (
    SELECT COUNT(*) FROM t_dirs  WHERE t_inodes.ipnfsid = t_dirs.iparent
) WHERE itype = 16384;

This is safe to run on a running production instance, but may take some time and will affect dCache’s responsiveness while running.


The description for how to migrate away from using the forbidden useGPlazmaAuthorizationModule and useGPlazmaAuthorizationCell properties had caused confusion. The description has now been updated to be more explicit.


Fix filtering boxes and sorting on Pool Admin, Pool Usage, Poolgroups, Space Tokens and Tape Transfer Queue.


Uploading a file with SRM involves three steps: preparing for the upload (srmPrepareToPut), uploading the file, marking the upload finished (srmPutDone). The third step can fail but previously the response from dCache is always Upload failed.. With this release, a meaningful error message is returned.

Changelog 2.9.21..2.9.22

[maven-release-plugin] prepare release 2.9.22
gplazma: update error message for forbidden properties
(2.9) webadmin: make jquery selector specific to individual tables
(2.9) webadmin: restore missing components to respect jquery script options
srm: include the reason why upload failed
chimera: fix race condition on remove
webadmin: ensure unique id attributes for all (currently) tested UI elements
[maven-release-plugin] prepare for next development iteration

Release 2.9.21


Fix default value for ftp.authz.readonly for plain (unencrypted) doors. This restores the default value to the dCache v2.6 default value of true.

The response from the plain (unencrypted) ftp door if the user specifies the wrong password is badly formed. Althogh it is possible that some clients are robust against such incorrect responses, with this release the ftp door responds correctly.


Fixes a bug where, if a double-slash is present, all parts of the path leading up to the double-slash are ignored; for example, with the bug, a path like /a/b//c/d is handled as if /c/d was specified. With this release, double-slashes are treated like single slashes; the above example is handled as if /a/b/c/d was specified.

Fix the NullPointerException triggered if client attempts to upload a file as a child of some existing file.

Changelog 2.9.20..2.9.21

[maven-release-plugin] prepare release 2.9.21
ftp: fix response if user fails to authenticate to weak FTP door
ftp: fix invalid default for ftp.authz.readonly property
webdav: fix double-slash bug by upgrading to patched milton
webdav: fix NullPointerException when PUT as a child of a file
[maven-release-plugin] prepare for next development iteration

Release 2.9.20

Changes affecting multiple services

Previously releases of dCache claimed to support Java–8; however, this is not true. For Java–8 support, sites must upgrade to dCache v2.10 or newer.


Fix two minor issues when authenticating with the webadmin interface: “unauthorised access” and being redirected to the home page. The unauthorised access error can occur when selecting “Login” under the bird logo (top right corner); this is now fixed. The redirection problem occurs when selecting a tab that requires administrative privileges while not logged in; this redirects the browser to the login page. Previously, after a successful login, the browser was redirected to the home page. Now the browser is redirected to the selected tab.


A previous bug-fix release fixed how dCache responds when the client attempts to DELETE a non-existent file. Unfortunately, this triggered a different problem where such activity results in a stack-trace that starts java.lang.ClassCastException: java.lang.String cannot be cast to javax.security.auth.Subject. This second problem is now fixed.

Changelog 2.9.19..2.9.20

[maven-release-plugin] prepare release 2.9.20
Disallow Java 8
webdav: Alternative to fixing return code of DELETE of absent file
(2.9) webadmin: fix login redirect bug
[maven-release-plugin] prepare for next development iteration

Release 2.9.19

Changes affecting multiple services

It is possible that a sudden burst of activity from many clients exceeds dCache capacity to queue such requests. Although dCache is designed to degrade gracefully under such circumstances, there existed the possibility of certain requests becoming stuck or memory leaking. This is now fixed.


In previous releases, the info-provider assumed the broker domain is dCacheDomain. This assumption has been removed.


Upgrade to nfs4j v0.8.5. This fixes export file parsing if a host is mentioned multiple times; note localhost must now have an explicit entry in the exports file. This also fixes a deadlock when closing a file on a busy dCache instance.


Fix NullPointerException when a file is stored in a directory with an empty tag.


Upon reloading the pool configuration an error is produced when an nearline storage was already defined in the existing configuration. This is now fixed.

A recent release fixed a bug that caused pool.mover.ftp.allow-incoming-connections to be ignored. Fixing that bug revealed another that caused the property to have the opposite effect. This is now fixed.

dCache pool configuration allows passing (fixed) arguments to the HSM script. In earlier releases, these arguments were supplied to the script in an arbitrary order. While HSM scripts should not depend on the order, they might; so, with this release, the order is preserved.


Although dCache behaves correctly if the client interrupts a proxied transfer; however, this is logged as a bug. This is now fixed.

Changelog 2.9.18..2.9.19

[maven-release-plugin] prepare release 2.9.19
webdav: don’t log a stack-track when proxy transfer is interrupted
info-provider: remove dCacheDomain assumption.
Fix compilation with Java 7
cells: Fix message timeout in case of thread pool overflows
libs: pull grizzly-framework–2.3.12 for nfs4j
chimera: handle NULL field of directory tags
pool: Fix regression preventing configuration to be reloaded
(2.9) gitignore additions for IntelliJ
libs: update to nfs4j–0.8.5
pool: Fix regression causing FTP movers to default to proxy mode
pool: Preserve HSM option ordering
[maven-release-plugin] prepare for next development iteration

Release 2.9.18

Changes affecting multiple services

Although dCache system configuration property names do not contain spaces, it is possible to define such properties. Previously, doing so breaks the dcache command. This is now fixed.

If the number of concurrent connections is set to -1 then the ftp and dcap doors will leak memory, eventually triggering an out-of-memory error that will restart the domain. This is now fixed.


Fix erronous reporting of bugs when the user supplies incorrect arguments to an admin command.


Fix regression against v2.6 and earlier dCache in how gsidcap doors are known to SRM and how they are published in BDII/GLUE.


The ftp door supports the HELP command as some clients use this commands output to discover if certain optional functionality is available. This release fixes the output from this command.


Improve which interface the ftp mover selects when the client is redirected to the pool. In particular, the door will only redirect the client if the IP protocol matches (IPv4 vs IPv6); for example, if a client connects with IPv6 and the pool has no IPv6 address then the door will now proxy the data connection.

The pool.mover.ftp.allow-incoming-connections property had no effect. This is now fixed.

Improve shutdown of nearline storage subsystem; now dCache will attempt to cancel all ongoing activity before shutting down the pool. The previous asynchronous approach could lead to IllegalStateException being logged if the pool was busy at that time.


Adjust spacemanager schema migration to be aware of earlier dCache bugs and to work-around site-local indexes that clash with new indexes that dCache needs.


Previously, if the SRM client requests listing a directory, specifies a non-zero offset and does not limit the response size then dCache would fail this request with an IllegalArgumentException. This is now fixed.


With the recent upgrade of the Milton library some new behaviour was introduced. One example is that, under certain circumstances (and to support certain clients) the Milton library returns a 401 (not authorised) when attempting to delete a non-existing file. Unfortunately, this change then broke ATLAS clients. This patch updates dCache so it returns 404 (not found) under these circumstances.

Fix the response when a client requests a byte-range beyond the end of a file. This is necessary for compatibly with ARC clients.

Changelog 2.9.17 to 2.9.18

[maven-release-plugin] prepare release 2.9.18
webdav: Fix reported content length for partial GETs
webdav: Fix return code on DELETE of absent file
srm: fix semi-infinite ls range with non-zero offset
pool: Partially fix interface selection for FTP mover
pool: Fix typo that breaks pool.mover.ftp.allow-incoming-connections
shell: fix shell oracle for configuration keys with a space
ftp: fix help output
dcap: Fix regression in published protocol family
loginmanager: Fix leak caused by absent child limit
pool: Fix shutdown of nearline storage subsystem
spacemanager: Fix null constraints and other schema migration issues
admin: Fix error reporting
[maven-release-plugin] prepare for next development iteration

Release 2.9.17


The NFS specification allows the server to specify multiple addresses when telling the client where to connect; for example, specifying both an IPv4 and an IPv6 address, or both addresses for multi-homed machines. This requires the client to choose the appropriate interface. For Scientific Linux 6, the kernel client will always use the first supplied address in the list and fail if it cannot access the pool with that address. With this release, pools will order the list, using heuristics to select which IPv4 address is “correct” and list it first.


With previous dCache versions, the WAAS selection algorithm had a bug where it could (mistakenly) consider pools full if all pools had very fresh files. This is fixed with this release.


Fix various issues to improve the robustness of spacemanager: handle failed uploads correctly, handle files deleted during upload correctly, increase robustness against dCache loosing (internal) messages.


Upgrade to Milton v2.6. This fixes the buffering problem where a proxied vector read request results in the entire file being written to a tmp directory and not deleted. With this release, requests for 100 kiB or less data result in no data being written to disk; requests for more than 100 kiB are still written to disk, but only the data needed to satisfy the request is stored and the file is deleted once the response has been sent. Some issues persist: data isn’t deleted if there is a failure sending it to the client and the whole file is requested from the pool.

Changelog 2.9.16 to 2.9.17

[maven-release-plugin] prepare release 2.9.17
poolmanager: Fix full pool detection for WASS
Upgrade to Milton 2.6
pool: reorder ip addresses returned to NFS client
spacemanager: Fix various error recovery scenarios
[maven-release-plugin] prepare for next development iteration

Release 2.9.16

Changes affecting multiple services

Restarting a domain within an active dCache instance can lead to a domain receiving messages for a cell as it is starting. Strict control is require to avoid the cell attempting to process messages before it is ready. This release fixes one place where this control was missed, which could lead to a NullPointerException. While this problem can affect any core dCache service, it was noticed with the spacemanager service.


dCache versions including and after 2.11.0, 2.10.9, 2.9.12, 2.8.16, 2.7.21 and 2.6.36 required sites to delete existing RRD files when upgrading; i.e., run the command rm -f /var/lib/dcache/plots/*.rrd when the domain hosting the httpd service is stopped. This release reverts that change, but requires sites that have already upgraded to repeat the rm command. Sites upgrading from an earlier dCache version do not need to delete anything.

Fix a potential NullPointerException in poolCollector.


Protect against a NullPointerException if the client attempts to read the contents of a file’s level where that level exists in the database but contains a Nil value. This does not happen under normal circumstances.


The Berkeley DB, which may be used to store file metadata on the pool, does not like being interrupted. The pool tries hard to avoid interrupting reading or writing; this release fixes one place that slipped through.

Fix regression in the output from the info command: it did not include statistics about the number of HSM requests and HSM timeouts. Also fixes how active HSM jobs are counted: cancelled jobs are still active until the underlying job as ended.

Changelog 2.9.15 to 2.9.16

[maven-release-plugin] prepare release 2.9.16
imera: protect against NPE in FsSqlDriver#read
pool: Restore flush and stage stats in info
Fix NPE in cell initialization
Fix NPE in httpd service
pool: Avoid interrupting Berkeley DB in migration module server
dcache-webadmin: revert rrd data source names
[maven-release-plugin] prepare for next development iteration

Release 2.9.15


Fix minor corruption at beginning of the alarms defaults file.


In previous versions of dCache, the webadmin war file is automatically unpacked. This has been problematic as a dCache upgrade did not always trigger updating the unpacked webadmin, resulting in dCache running the older webadmin. With this release, dCache no longer unpacks the war file; the webadminWarunpackdir and httpd.container.webapps.tmp-dir properties are now obsolete.

Changelog from 2.9.14 to 2.9.15

[maven-release-plugin] prepare release 2.9.15
alarms.properties fix accidental header corruption
[maven-release-plugin] prepare for next development iteration
(2.9) dcache-webadmin: change Jetty setting so .war is not unpacked

Release 2.9.14

Changes affecting multiple services

A bug was discovered that resulted in the nlink count becoming negative. While wrong, this had a knock-on effect that prevented pnfsmanager and nfs services from listing the affected directory. With this release, directory listing are robust against such problems.


Fix alarm definition that triggers when the pool discovers a file with the wrong checksum. Previously it mistakenly triggered if an upload was incomplete.


Update nfs4j to v0.8.3. This brings some small performance benefits when listing a directory and for reading data through NFS v3.

Fix bug where, when a client read activity is proxied, the corresponding dcap mover wasn’t removed if client didn’t wait for the queued mover to start. Previously, such movers would accumulate on the pool.


Prevent attempts to remove the . and .. directories.


Display the command’s output in the cell admin page to use a monospace font.

Remove a javascript error due to missing clojure dependency.

Sending commands in cell admin page is fixed.

Changelog from 2.9.13 to 2.9.14

[maven-release-plugin] prepare release 2.9.14
webadmin: Fix ClassCastException in cell admin
alarms: fix regex for checksum alarm
webadmin: remove redundant head element in alarms panel html
dcache-webadmin: change output field of cell admin page to monospace font
(2.11) dcache-webadmin: eliminate clojure dependency
nfs-proxy: kill mover if we get a timeout on redirect
libs: update to nfs4j–0.8.3
chimera: protect list initialization from FS inconsistencies
webadmin: tidy up unavailable page slightly
chimera: prevent attempts to remove ‘.’ and ‘..’
[maven-release-plugin] prepare for next development iteration

Release 2.9.13

Fixes affecting multiple services

Update dcache check-config to print an error if site-local configuration contains scoped properties: one that contain a ‘/’ character.


Fix pool queue plots in webadmin.


Performance improvements.


Fix the message reported when moving (or renaming) a file or directory and certain problems were found: the destination directory isn’t a directory, the source doesn’t exist, or overwriting with different types (e.g., overwriting a directory with a file).

Fail requests earlier that plan to upload a file if that file’s path already exists as a directory. Previously, dCache would fail such uploads after the file was upload (during srmPutDone). With this release, dCache will fail during srmPrepareToPut instead.


If a site using WAAS specifies too large a Space Cost Factor then the algorithm used for write pool selection breaks down, logging “Unreachable statement.” Starting with this release, a warning is logged indicating the cause of the problem.


The srm.enable.space-reservation.implicit property is now marked forbidden. This means a dCache instance with this property configured will not start. Consider the spacemanager.enable.unreserved-uploads-to-linkgroups property instead.

Changelog from 2.9.12 to 2.9.13

[maven-release-plugin] prepare release 2.9.13
chimera: Add messages to JdbcFs#move exceptions
poolmanager: Warn when spacecostfactor is too big
pnfsmanager: Check file type before overwrite
check-config: Produce error when using scoped properties
srm: Mark srm.enable.space-reservation.implicit forbidden
(2.9) pool queue plots - fix partial package refactoring and neglected removal of jndi arguments
pom: update to nfs4j–0.8.2
[maven-release-plugin] prepare for next development iteration

Release 2.9.12

Fixes affecting multiple services

Running the eval admin command so it returns a non-zero return-code would respond indicating that a bug had been found. This has been fixed.

An named environment may be executed and can contain a reference to itself. If this recursion is unchecked, eventually memory is exhausted and the domain would restart. Now an error is logged and the domain is not killed.

Previously, some exceptions were logged with the wrong cell context or without any context. This has been fixed.


Billing records issued by doors never included the file’s size. This is now fixed for all doors except NFS.


The xacml plugin now supports the gums server returning GID values.


Fix internal copying of files (triggered by the SRM copy command) so they respect the LAN port range.

Fix migration module’s random pool selection so that it does not select pools that are full.

Fix NullPointerException triggered when using migration module’s proportional pool selection and all pools are full.

The rep ls command can calculate per-storage-class statistics. This release fixes a problem where files without a storage-info would trigger an IllegalStateException when calculating these statistics.

Fix sweeping of files without a storage-info. Previously an IllegalStateException was thrown; this could be triggered by the admin interface or by the sweeper.

Fix a problem where the pool would register free space before actually deleting the file from the file system; for a brief moment, the pool would appear to have more free space that is actually available.

Periodically, pools check that dCache’s internal accounting of total and free capacity does not exceed the OS supplied values for the partition (i.e., the output from the df command); if they do then the dCache internal accounting is adjusted to match. The dCache.org team have observed that, after a file is deleted, some seconds may elapse before the corresponding extra free capacity is reported by the OS. To protect pools from this effect, the total and free capacity check is suppressed for 60 seconds after a file was deleted.


If a transient deadlock resolution error occurs, unconditionally retry the operation and do not log it. Such errors are part of RDBMS design when resolving concurrent updates.


Fix the srm service’s admin interface ls command so that listed jobs can optionally include only those that have failed, have completed or were cancelled.


Webadmin periodically creates billing graphs by querying the billing service for the information it needs. Previously, if this query failed (e.g., the billing service was restarted or failed to start first) then no further billing graphs are generated and a domain restart is needed. As a result of this work, the properties poolqplots.refresh-interval, poolqplots.refresh-interval-unit, httpd.plots.pool-queue.refresh and httpd.plots.pool-queue.refresh.unit are no longer supported. The httpd.plots.pool-queue.min-time-step and httpd.plots.pool-queue.min-time-step.unit properties now also cover this configuration.

Changelog from 2.9.11 to 2.9.12

[maven-release-plugin] prepare release 2.9.12
gplazma-xacml: Add handling of GID returned by GUMS in gPlazma2 XACML plugin
chimera: fix unit tests
pool: Fix pool size health check in case of asynchronous release of space
spacemanager: Suppress transient deadlock resolution errors
pool: Fix race leading to false positices in pool size health checks
pool: Fix ISE in CacheEntryImpl#toString
pool: Fix ISE in ‘rep ls’
Made PoolQueuePlotData enum compatible with java 8.
(2.9) webadmin: fix exit login in billing refresh loop
Marked refresh properties obsolete.
cell: Log exceptions within the correct cell context
pool: Fix pool selection bugs in migration module
billing: Add file size to request records
(2.9) webadmin: minor improvements to rrd4j-based pool-queue plots
pool: Respect LAN port range for internal srmcp transfers
srm: Fix listing of failed, done and cancelled jobs
cell: Prevent interpreter stack overflow from killing the domain
cell: Declassify eval failure as a bug
[maven-release-plugin] prepare for next development iteration

Release 2.9.11

Fixes affecting multiple services

The nfs and srm both cache replies from gPlazma to increase the speed of authenticating and identifying the user. Previously transient errors when communicating with gPlazma were also cached, delaying the recovery time from such errors. Now, such errors are not cached.

Update the comments within the configuration property files to reflect the new configuration property names.

Fix OpenMQ communications to use new configuration property names.


Fix opening a file when space-manager enabled.


No longer trigger an NullPointerException when opening the .(parent)() dot-command file in Chimera’s root directory.

The dcache ports command now shows the NFS port based on the new configuration properties.


Fix a race-condition when a user starts uploading a file that should go to tape, deletes the file before the write is completed, then cancels the file upload. Previously, there was a risk that the pool fails to delete the uploaded data.

Fix Berkley DB usage when an HSM operation is cancelled to avoid an error logged as:

InterruptedException may cause incorrect internal state, unable to
continue. Environment is invalid and must be closed.

Fix xrootd support for vector read. The problem was discovered with ROOT v6, which reports errors like Single readv transfer is too large.

Fix the dcap mover to use the new pool.mover.dcap.port configuration property.


Fix cleaning of upload directories for expired uploads after service restart.


Fix problems when calculating a response to the kXR_set request. If encountered, the problem results in the following being logged, where ‘nnn’ in ‘xrootd-disk-nnn’ is some integer number:

Uncaught exception in thread xrootd-disk-nnn java.lang.IllegalStateException: null

Changelog from 2.9.10 to 2.9.11

[maven-release-plugin] prepare release 2.9.11
ftp: Fix root path validation for upload directory
dcap: fix interaction with Spacemanager
xrootd: Upgrade to xrootd4j 1.3.5
pool: Prevent interruption of replica deletion during HSM flush
pool: Fix HSM cancellation
The gPlazma cache used by NFS and SRM caches both positive and negative replies. Unfortunately it also caches failures to communicate with gPlazma. This means that a transient timeout would be cached too, thus increasing the effect of the transient error.
pool: Fix xrootd vector read limits
chimera: fix NPE on ‘.(parent)()’ for root inode
srm-client: Fix logback configuration
srm: Fix asynchroneous job storage leak
configuration: adjust references to deprecated properties
unittests: increase timeouts and fix race in DiskSpaceAllocatorTest
[maven-release-plugin] prepare for next development iteration

Release 2.9.10

Fixes affecting multiple services

In some cases, dCache would log the wrong cause of an internal failure. This has been fixed.

Two problems are fixed that affected those services that make use of the grid trust store (/etc/grid-security/certificates): gplazma, srm, gsiftp, webdav and xrootd:

  • When a CA is found not to have a signing-policy file, subsequent attempts to discover a signing-policy file will automatically fail for one hour. Previously, if any CA was removed since the service started then certain CAs (those using UTF–8 encoded subject RDNs) would be placed in this not-found cache by mistake. This resulted users with certificates from such CAs would succeed the first time they used dCache but all subsequent attempts would fail until the cache expires or the trust-store is refreshed.

  • A minor memory leak is fixed. The amount of memory leaked was proportial to the number of different CAs in use.

Two problems are fixed that affected services providing GSI or plain SSL authentication using the jGlobus library:

  • dCache now supports TLS v1.2 clients when using GSI or plain SSL using the jGlobus library.

  • Signal end-of-stream when remote sends a CLOSE notification. The remote party normally sends a CLOSE notification before terminating the TCP connection, typically send from server to client. Previously, receiving such a notification was treated as an error.

Fixed thread safetly of timestamp formatting, principally used in logging and generating the output of admin commands.


Fixes a bug where the bytes read graph can show inflated values.


Fix permission check when client requests a list of some directory’s contents.

Add support for mounting when the server-side mount-point is a sym-link.


Fix the lifetime of the response when client makes a successful request for a reservation with infinite lifetime.

Fix incomplete restore of “ready queued” jobs. After restarting the srm service, any jobs in this state RQUEUED would not be handled correctly. This has been fixed.

Allow srm to start if there were active bring-online requests when the service was stopped.

Fix the set max ready get command in the admin interface.

Fix a problem where, if a client aborts an upload shortly after initiating the upload then the server blocks all activity for a timeout period. The SRM will eventually recover but, during this period, it appears deadlocked.


The chimera shell failed to interpret the creation time flag when choosing which time should be displayed in directory listing.

Changelog from 2.9.9 to 2.9.10

[maven-release-plugin] prepare release 2.9.10
common: Fix race condition in AtomicCounter unit test
Add missing import
srm: Fix deadlock like bug
billing plots: use ‘transferred’ rather than ‘size’ for bytes
srm: Fix reporting of infinite space lifetime
chimera: Fix creation time flag in ls command
Fix CacheException references in error messages
libs: update jglobus to 2.0.6-rc8.d
srm: Fix incomplete restore of “ready queued” jobs
Avoid thread-unsafe use of SimpleDateFormatter
srm: Fix set max ready get command
libs: update to jglobus–2.0.6-rc7.d
libs: update to nfs4j–0.8.1
[maven-release-plugin] prepare for next development iteration
build: Update to findbugs 3

Release 2.9.9


Previously, the automatic schema management for Chimera failed to create an index on the iparent column of t_dirs table. The lack of this index results in deletions becoming progressively slower as dCache stores more files. If your dCache instance was created with dCache v2.3.0 or newer then it is likely affected by this problem. Instances created with an earlier version are not affected. The pnfsmanager service will check for the index when it starts and add the index if it is missing. Alternatively, the check may be performed manually using the dcache database update command.


This release fixes several issues with the sweeper:

  • Previously, the sweeper consumed considerable amount of memory while freeing up space; in the worse case, this could result in the domain running out of memory and restarting; this has been fixed.

  • The sweeper ls command would make the pool unresponsive while the output is generated; now, running this command no longer prevents the pool from taking on more work.

  • As the sweeper purge, sweeper free and rep rmclass commands can take a long time to finish, the admin interface does not wait for them to finish; instead, a quick response is returned and the command continues in the background. Previously there was no indication when such commands had finished; now, the pool will log when the command has completed.

  • The pool logs regular sweeper operations at lower priority; this makes the pool logs less noisy when operating under normal conditions.


In previous versions, poolmanager would log a stack-trace if a user attempts to open a file for reading and no read pool could be found because a unit failed to match. Now the failure is logged as a normal message.


The xrootd door now informs billing when a user uses the xrootd protocol to delete a file.


The ldap plugin allows a door to request converting a uid or gid back to a username; currently, only the nfs door makes use of this functionality, when generating a directory list. In previous dCache releases, the plugin behaved incorrectly if the uid or gid is unknown; this is now fixed.

Changelog from 2.9.8 to 2.9.9

[maven-release-plugin] prepare release 2.9.9
chimera: mark as-run if i_dirs_ipnfsid exists
solaris: fix solaris package script and add work-around for pkgmk bug
chimera: create missing index i_dirs_iparent
pool: Resolve high memory usage and other issues in sweeper
info: Fix HashMap ordering assumption in unit test
Make JDK byte code verification bug workaround Java 8 compatible
gplazma: Fix JVM implementation dependency in unit test
poolmanager: Suppress stack trace in case of unmatch units
xrootd: Add billing entry on delete
bugfix: fix reverseMap not throwing NoSuchPrincipalExceptions
[maven-release-plugin] prepare for next development iteration

Release 2.9.8

Cell startup

Fix potential dead-lock when a dCache domain is starting. The problem is triggered when a cell receives a message while initialising. If dCache is started “from cold” then this cannot happen. However, if a domain that contains well-known cell restarts while the rest of dCache is operational then the problem could be triggered. For example, if a door domain ran out of memory then there is a small risk that the domain would not restart correctly. The problem was present since dCache v2.8.0 and is fixed with this release.

Changelog from 2.9.7 to 2.9.8

[maven-release-plugin] prepare release 2.9.8
cells: Fix deadlock during startup
[maven-release-plugin] prepare for next development iteration

Release 2.9.7


Add python-psycopg2 as recommended dependency to debian package; it’s needed by the dcache-star command.


The alert should be generated if a pool’s background checksum scanning (“scrubbing”) discovers a broken file. With this release, such alerts are now generated.


When the client authenticates via GSI, the dcap door will check the user certificate (generally, a voms-proxy certificate) to extract the DN and primary FQAN. To do this, it needs to know where the trusted CAs and trusted VOMS server identities are stored; these are typically /etc/grid-security/certificates and /etc/grid-security/vomsdir respectively. Previously, dcap ignored the dcap.authn.capath and dcap.authn.vomsdir properties. With this release, these properties are honoured.


If a stage requests has been forwarded to the nearline subsystem and the file is deleted then the stage request is now cancelled.

Nearline storage configuration is now applied atomically rather than one attribute at a time.

The maximum number of concurrent nearline operations may be adjusted using the hsm set admin command. If the concurrency is increased then previous versions would only start additional activity as more requests are received. This has been fixed so increasing the concurrency limit will immediately start more activity if there is any backlog of requests.


Stage and pool-to-pool transfers, triggered by poolmanager, are monitored by poolmanager querying the source pool to check if the request is still active. In previous versions of dCache, for n transfers from a pool, this monitoring generated O(n*n) of network traffic. In particular, poolmanager’s monitoring of bulk staging requests can result in sufficient network traffic between the pool and poolmanager that other dCache activity is impacted. With this release, the monitoring traffic is reduced in general, most prominently for large concurrent activity, such as bulk stage requests.

The rc destroy command has been removed. The implementation was incomplete and the operation was dangerous.

If the door resubmits a request to poolmanager that triggers staging of a file, poolmanager uses additional memory while waiting for the stage to complete. For bulk staging, this additional memory usage can be significant, potentially resulting in poolmanager exhausting the available memory. This release updates poolmanager so a door resubmitting a stage request does not increase the memory footprint of poolmanager.

The poolmanager has an incomplete feature called ‘clumping’. This is where poolmanager can handle multiple select-pool-for-read requests for the same file as a single request. There is a hard-coded clumping limit that prevents too many files from being handled together; this allows poolmanager to choose an alternative pool when staging or replicating to make the file available. Once this limit is reached, subsequent open requests for the same file will fail in poolmanager. Doors handle this failure differently; the xrootd and webdav doors propagates this failure back to the client, while the dcap and ftp doors will resubmit the request to poolmanager. This release increases the clumping limit from 1 to 20; it also changes the behaviour of the doors so all will retry the pool-selection if the clumping limit is reached. This fixes the ‘request clumping limit reached’ failures.

This release fixes a race-condition between two clients opening the same file. Previously, if poolmanager starts processing the second open request as it is finishing processing the first then there is a tiny chance that the second request is lost.


The srm service will check the user certificate (generally a voms-proxy certificate) to extract the DN and primary FQAN. To do this, the srm service needs to know where the trusted CAs and trusted VOMS server identities are stored; these are typically /etc/grid-security/certificates and /etc/grid-security/vomsdir respectively. Previously the SRM ignored the srm.authn.capath property and there was no way to configure the trusted VOMS server identity path. With this release, the srm.authn.capath is honoured and the srm.authn.vomsdir property is introduced.

When the SRM server sends an asychronous response to the client, the client must make subsequent queries to check if the request has completed. Part of each response is a hint suggesting for how long the client should wait before querying for a progress update. Prior to dCache v2.9, the srm service provided this hint where subsequent responses suggested a longer wait time. With v2.9, this feature was lost, resulting in the SRM client hammering the srm service. This has been fixed.

Changelog from 2.9.6 to 2.9.7

[maven-release-plugin] prepare release 2.9.7
build: add work-around for JDK bug and PowerMock
pool: Fix concurrency settings for script nearline storage
poolmanager: Fix ‘request clumping limit reached’ failures
poolmanager: Avoid request leak
poolmanager: Fix excessive p2p and stage alive checks
srm: Restore estimated wait time update
pool: Fix several minor issues in nearline storage subsystem
srm,dcap: Use configured vomsdir and capath
srm-client: by default only delegate for srmcp with 3rd-party copies
alarms: Fix checksum alarm filter.
deb: Add python-psycopg2 as a recommended dependency
delegation-shell: generate delegated credential of correct type
[maven-release-plugin] prepare for next development iteration

Release 2.9.6


Fixes a regression that prevented the admin from deleting alerts in the alarm system.

Improve esthetics of the pool queue plot grid.


Fixes a problem where, under specific circumstances, a pool with a file’s data will erroneously claim it does not have that file’s data. This problem occurs only if an HTTP or xrootd client attempts to read a file immediately after uploading it; it does not affect other protocols. If a client triggers the problem then PnfsManager will remove the pool as the location of the file’s data with two consequences: first, the file cannot be read; second, the orphaned entry will not be removed if the file is deleted. With this release, the problem is fixed; we recommend running the ‘pnfs register’ command on all pools as a precaution against orphaned entries.


Suppress publishing NFS door endpoints. Publishing these endpoints can result in sites failing functional tests since dCache running an NFS endpoint does not mean that files are accessible from the test machine.

Changelog from 2.9.5 to 2.9.6

[maven-release-plugin] prepare release 2.9.6
Suppress nfs’ file protocol from being published
pool: Fix race condition that leads to orphaned files
dcache-webadmin: fix pool queue plot grid placeholder image
(2.9) webadmin: fix regression in alarm deletion filter
cells: remove System time dependency in unit-test
[maven-release-plugin] prepare for next development iteration

Release 2.9.5


Fix globus-url-copy hanging on proxied passive-mode connections when the mover is killed. In addition to reported the error on the control channel, the ftp door now also closes the data channel to force globus-url-copy to fail such transfers.

Fix the control channel responses for LIST, NLST and MLSD commands; the broken response prevented Globus (formally GlobusOnline) from listing directories.


The background checksum scanner now treats files that cannot be accessed as an error rather than skipping them.

Update the compatibility script-based HSM module to be more compatible with the HSM interface in earlier versions of dCache.

Fix the response headers when HTTP clients (such as Davix) make a request for multiple fragments of a file.


Remove a race-condition where aborting an upload at the wrong moment resulted in dCache failing to remove the temporary upload directory.

Fix issue where pools can appear to be offline on lightly loaded systems.

Increase scalability of SRM by allowing work to spread over more CPU cores during file upload and download.


For systems using enstore, fix restoring of files with filenames that contain a ‘+’ in the filename.


Honour the WriteToken directory tag and implicit reservation when writing data.

Fix issue where pools can appear to be offline on lightly loaded systems.


Honour the WriteToken directory tag and implicit reservation when writing data.


Fix issue where pools can appear to be offline on lightly loaded systems.


Fix issue where pools can appear to be offline on lightly loaded systems.

Changelog from 2.9.4 to 2.9.5

pool: log with error if checksum scanner gets an IO error
ftp: Close passive port on failure
srm: Cancel upload path if request was aborted
ftp: fix LIST, NLST and MLSD cmd-channel output
pool: Fix regressions in new HSM subsystem
Fix ArrayIndexOutOfBoundsException when processing multiple surls invoking srmget.
Fix issue with incorrect determination of whether or not a string is encoded by replacing URLDecoder with import org.springframework.web.util.UriUtils as the fo
http: fix multipart response size
xrootd,nfs: Add support for space management
pinmanager,srm,spacemanager,xrootd: Fix pool monitor expiration
srm: Fix lock congestion
[maven-release-plugin] prepare for next development iteration

Release 2.9.4


Fix redirection to IPv6 pools.

Report read requests that suffer a permission denied failure as “Read permission denied” rather than “Write permission denied”.

To allow xrootd plugins to intercept and translate errors.

Update dCache responses to avoid intermittent failures with xrdfs.


Fix login bug that results in Access Denied page.


Provide more robust cleanup if there is a problem after the client finishes reading or writing data. The symptom is an increasing number of “active transfers” reported in the web monitoring pages. Eventually, the httpd and webadmin services consume all available memory and the domain(s) hosting them will die with an OutOfMemoryError.


Update dCache scripts to allow running with Java 8 runtime.

Changelog from 2.9.3 to 2.9.4

[maven-release-plugin] prepare release 2.9.4
xrootd: Quote IPv6 addresses in kXR_redirect replies
xrootd: Fix error message for read patch check
dcache-webadmin: fix login redirect bug which misleadingly results in Access Denied Page
xrootd: Upgrade to xrootd4j 1.3.3
scripts: allow java8 runtime
dcap: ensure that we remove session on DoorTransferFinishedMessage
[maven-release-plugin] prepare for next development iteration

Release 2.9.3


Fix the column formatting for various admin commands should one of the numerical values be negative.


Fix a null-pointer exception triggered by a client querying a request that survived an SRM restart.

Update the SRM to use the SRM-specific configuration property (‘srm.enable.space-reservation’) rather than the generic property (‘dcache.enable.space-reservation’). As the default value for srm.enable.space-reservation is the value of dcache.enable.space-reservation, this change should not affect sites.


Fix the shutdown sequence of cleaner to ensure quick and clean shutdown.


Don’t export child doors. When a client connects, the ftp door creates a child cell that handles the client requests. Changes in 2.7 meant that these child cells mistakenly became well-known cells. This is now fixed.


Don’t export child doors. When a client connects, the dcap door creates a child cell that handles the client requests. Changes in 2.7 meant that these child cells mistakenly became well-known cells. This is now fixed.


Update the use of xmllint to include the ‘–noent’ option; mitigation against CVE–2014–0191 resulted in xmllint behaving differently when ‘–noent’ is omitted.


Update jGlobus version. This results in better error reporting for FTP transfers and X.509 certificates with a comma in the DN are handled correctly.

Changelog from 2.9.2 to 2.9.3

[maven-release-plugin] prepare release 2.9.3
info-provider: fix works-by-accent bug when importing dCache config
pom: update jglobus library
srm: Use srm.enable.space-reservation rather than dcache.enable.space-reservation
Fix rendering of negative numbers in columm layout
srm: Fix NPE
ftp,dcap: Don’t export child doors
cleaner: Shut down timer thread
[maven-release-plugin] prepare for next development iteration

Release 2.9.2


Prevent reading of configuration files.


Fix how the status of requests that can contain multiple files (Get, Put, Ls, BringOnline, Copy) are calculated and avoid “illegal state transition” errors that are sometimes logged.

Clean up correctly when a client deletes a file that is being uploaded (thereby cancelling any concurrent uploads) by deleting the temporary upload directory.

Fix how a request is reported to the client after it times out, so the value follows the SRM specification. This also fixes some of the logged “illegal state transition” errors.

Ensure that request history information is listed chronologically.

Avoid deadlocking should either a client tries to pin files or if the SRM attempts to retry pinning a file, while the SRM is updating its knowledge of the pools.


When logging a transfer, place a colon between the version and the client IP address.


Don’t log “duplicate key” errors if the error is understood.


Report correct statistics about messages even after a cell’s message queue overflows.


Make pools more robust against the unexpected. Files that appear unexpectedly (e.g., an HSM script delivers a file after dCache has cancelled the request) do not trigger the pool from disabling itself but are deleted; requests to create a file that already exists trigger re-registration of that file in pnfsmanager.

Log a stack-trace for pool-to-pool transfers that encounter an extreme problem with the virtual machine.

Fix the ordering of arguments when invoking an HSM script to be compatible with pre–2.9 versions of dCache.


Work-around bug in StringTemplate library that can result in NullPointerException being logged when generating an error response for the client.


Fix uploads through SRM if the client supplies the file’s expected size and fix an internal cache that allows database-overhead to be avoided.

Changelog from 2.9.1 to 2.9.2

[maven-release-plugin] prepare release 2.9.2
httpd: Fix a couple of bugs
srm: Avoid duplicate status code update on status query
xrootd: split protocol name form the client IP in XrootdProtocolInfo
chimera: Fix duplicate key supression
Revert “srm: Drop fail fast for protocol checks”
srm: Fix request cancellation due to SURL deletion
srm: Fix request expiration
cells: Fix message counting in case of queue overflow
pool: Improve how we deal with unexpected files in pools
pool: Fix error handling in staging code
pool: Log P2P fatal errors with stack trace
srm: Fix transition ordering for restored requests
hsm scripts: fix command and options order
webdav: work-around race-condition in StringTemplate library
systemtest: regenerate host credentials, add user credentials
srm: Fix deadlock in pinning
srm: Schedule resubmit of pin request to avoid deadlock
srm: Drop fail fast for protocol checks
pnfsmanager: Fix expected size extraction and tag caching
fhs: Use gzip compression for deb package
[maven-release-plugin] prepare for next development iteration

Release 2.9.1


Add missing configuration elements and documentation to support sending an email when an alert is received.


Fix NullPointerException in statistics service

Changelog from 2.9.0 to 2.9.1

[maven-release-plugin] prepare release 2.9.1
dcache alarms: add missing configuration elements and documentation for SMTP appender
statistics: Fix NPE
[maven-release-plugin] prepare for next development iteration

Release 2.9.0

Changes affecting several services


Removed the saxon library from the dCache classpath. Removed the xalan XSLT library from dCache as the library is included in all Java 7 JVMs.

Upgraded Netty from version 3.6.5 to 3.9.0. This I/O library is used by xrootd doors and to support xrootd and http data transfers in pools. Consult the Netty news archive for information about Netty bug fixes.

Upgraded Jetty from version 8.1.13 to 8.1.14. This embedded webserver is used by the webdav and srm doors, and the httpd service.

Upgraded the PostgreSQL JDBC driver from version 9.3–1100 to 9.3–1101.

Upgraded the Berkeley DB library from version 5.0.97 to 5.0.103. This embedded database library is used by pools to store meta data. Please consult the change log for details about bugs fixed in this release (you can ignore the information about disk format changes - those only apply when upgrading from older versions of the library).

Upgraded the Bouncy Castle Security Provider from version 1.43 to 1.45. This library is used by most certificate related code in dCache.


A schema change to the t_inodes table will cause the first start of pnfsmanager to take a while. Large sites are encouraged to time the schema change on a clone of the database before upgrading the production system. Due to these changes, dCache cannot be downgraded without rolling back the schema changes using one of the dcache database rollback commands with a 2.9 dCache and only then downgrading dCache. The schema change was necessary to fix issues with inode cache invalidation.

File creation and file open overhead has been reduced.

Fixed error reporting for hard link creation.

Changed registration of file locations to avoid integrity constraint warnings in the PostgreSQL log file.

Fixed a bug that would cause sockets to be interpreted as directories.

The chimera command line interface was extended with a mv command and new time stamp options for the ls command.


The cell message passing system has been refactored. This is an internal change and mostly invisible. If messages are monitored using the events log files, you will notice that the blocking mode is no longer used.

Plugin compatibility

An internal change to how we implement dCache services in the Spring framework breaks compatibility with third party plugins providing custom services implemented with Spring. Plugin providers should update their plugin to match the changes in b7234ee.


The BoneCP database connection pool library has been replaced with HikariCP. HikariCP is more maintained, smaller and faster. We also hope it is more robust.

HikariCP has slightly different controls for adjusting the database connection pool. While BoneCP partitioned connections, HikariCP has no such concept. HikariCP also maintains a minimum number of idle connections; this is in contrast to the minimum number of connections that BoneCP maintains. Sites that have tuned their data connections should note these differences and adjust their configuration accordingly.

The *.db.driver configuration properties have been marked obsolete. The proper driver is now automatically discovered from the JDBC URI. Only JDBC 4 compliant database drivers are supported (this is only relevant if you install a custom JDBC driver).

The liquibase database schema management library has been updated to version 3.1.1. Version 3.1 adds support for generating SQL scripts without a live database connection. Details about this feature can be found in the Liquibase documentation. There is no wrapper for this functionality in dCache, so this is to be considered for expert use only.

Added the dcache database showUpdateSQL command to output the SQL that would be executed if dcache database update was executed. The emitted SQL may be applied manually to the database, so updating the schema.


Tarball packages no longer have a trailing -1 in the file name or base directory name.

Command line tools

Startup speed of several command line tools has been improved.


dCache now supports IPv6 by default. It still prefers using IPv4 addresses on dual-stack machines (those with both IPv4 and IPv6 addresses) so the impact should be minor. However, the change does mean that services will now also listen on IPv6 interfaces. To disable this, define


in your configuration.

Changes to services


Loginbroker now supports registering the root directory of doors. As a consequence SRM no longer needs to be configured with the root paths of each door and the properties srm.service.webdav.root and srm.service.xrootd.root are obsolete.


Billing information as published by several services has been completed to reduce the number of values marked as Unknown in the billing log files. The fix was made in the services publishing billing information. Thus only services updated to 2.9 publish complete information.

The billing indexer was extended with the -f option to the find command. It allows search strings to be read from a file, enabling efficient billing entry lookup for a collection of files.


Add additional attributes to the domain.<domain-name>.static node, including information on the OS and the JVM.


Added new export options dcap and no_dcap. The former is the default while the latter hides the .(get)(cursor) dot command that DCAP clients use to detect an NFS-mounted dCache. Using no_dcap will force DCAP clients to read data through NFS instead of through DCAP.

A new command has been added to the NFS dot-command set to allow users to pin and unpin files. Files which are not on disk will be brought on line when pinned. The command takes the following form:

touch ".(fset)(filename)(pin|stage|bringonline)(duration)([SECONDS|MINUTES|HOURS|DAYS])"

The variants of the third argument are equivalent in effect. The last argument is optional and defaults to SECONDS. A duration value of 0 will unpin the file. The command does not allow the user to pin the file indefinitely as the duration value must be a positive integer.

Three new properties associated with the configuration of the pinmanager have been added to nfs.properties: nfs.service.pinmanager, nfs.service.pinmanager.timeout and nfs.service.pinmanager.timeout.unit.

Server side attribute caching drastically improves performance for some workloads. Three new attributes have been added to configure the cache: nfs.namespace-cache.time, nfs.namespace-cache.time.unit and nfs.namespace-cache.size.

A large number of fixes to improve logging, protocol compliance, error handling and performance have been implemented.

Adds support for NFS v4.0. Note that there is no parallel NFS (pNFS) extension for NFS v4.0; therefore, the NFS door will act as a proxy. This means that all data will flow through the door for clients using NFS v4.0.


Improved logging and handling of configuration errors.


Added support for the RFC 3659 CREATE fact to report file creation time in directory listings.

Added support for the MFMT, MFCT and MFF commands as described in this internet draft.

Added support for the UNIX.ctime and UNIX.atime facts.

Improved compatibility with the ncftp client by adding a leading zero to the value of the mode fact.

Added support for the HELP and SITE HELP commands as defined in RFC 959. This improves compatibility with some clients.

dCache user accounts can be configured with a personal root and home directory. How these are interpreted depends on the door and protocol used. For FTP, the account root directory forms the root of the name space and the home directory is the initial directory of the FTP session. For other doors, a configuration property usually defines a directory to export as the root of that door, and the account root directory is used as an additional authorization check to prevent users from accessing any directory outside their root. This behaviour is now configurable for the FTP door using the new ftp.root property. If left empty, the per account root is used as before. If set to a path, that path is exported as the root of the door.

Added FTP protocol request logging to the access log files. The access logs were introduced in dCache 2.8, but initial support was limited to SRM. In dCache 2.9 the FTP door logs every command interaction to these log files. The access logs are automatically compressed each night and deleted after one month.

Added a workaround that selects the MD5 checksum algorithm when a file is written by Globus Online. This allows such transfers to succeed with integrety checking.

Added support for Globus delayed passive. This is an alternative to using GridFTP v2 GETPUT. It allows the data channel to be created directly between the client and the pool. In contrast to GridFTP v2, delayed passive supports IPv6. Note that current clients often default to disabling delayed passive, even when delayed passive is implemented. If the client connects to the door using IPv4 and the pool does not support IPv4, then the data connection is proxied through the door. Similarly, if the client connects to the door using IPv6 and the pool does not support IPv6, then the data connection is proxied through the door. This is the case even when the client is dual stacked. Like with GridFTP v2, use of direct data channels is subject to the ftp.proxy.on-passive and ftp.proxy.on-active properties.


A new partition type, buffer, was added that is intended for pools used as transfer buffers. This new partition type chooses a pool based on a weighted random selection, with weights derived from the pools’ load only. Free space has no effect as all pools with sufficient space to store the file are considered. This partition type is suitable for pools which do not hold files permanently, such as import and export pools.

The wass partition type is not suited for this use case, as wass prioritizes balancing free space over load.

The buffer partition type respects the mover cost factor and performance cost factor settings, using an interpretation similar to wass.


Disabled statistics files for Berkeley DB meta data backend. These statistics files consumed disk space, which would periodically prompt the pool to generate a warning as it adjusted for the unexpected loss of storage capacity.

The time format of the sweeper ls -l command has been changed to conform to ISO 8601.

Improved thread termination during pool shutdown.

Modified permanent migration jobs to react to access time and sticky flag changes. Thus migration jobs that filter by access time or sticky flags may now pick up files when access time or sticky flags change.

HTTP mover

Implemented limited support for the HEAD HTTP method.

FTP mover

The listening socket of a passive FTP data channel is now bound to a single address rather than the wildcard address. The FTP protocol only allows a single address to be returned to the client, so there is no point in binding to the wildcard address. Binding to the wildcard address had the problem that it could succeed even if the port on the specific address returned to the client was already open. The change does have observable effects in that previously the pool would submit a hostname back to the door and the door would do a DNS lookup to find the address to send to the client. Now the pool sends an IP address.

The TCP accept backlog for FTP passive data channels has been increased. This reduces the risk of rejected connections during a surge of activity.

Nearline storage

The nearline storage support has been rewritten. Nearline storages are supported through drivers. These drivers can be packaged and shipped as dCache plugins. A driver that supports the classic HSM scripts is shipped with dCache and will be used by default for existing setups.

Nearline storage drivers solve many issues of the classic HSM scripts:

  • A callout to an external script for every invocation was a performance issue, as such a callout to an external interpreter was associated with considerable overhead.

  • A callout to an external script was a tape efficiency issue, as it would limit the number of concurrent callouts due to allocating a process and four Java threads for every file. Thus one had to balance how many processes and threads the host could handle with how large batches one could submit to the tape system.

  • The stage queue was limited to available disk capacity, as the pool had to reserve space for the file before calling the script. Thus the tape reodering queue was limited by disk space.

A nearline storage driver is written in JVM-supported language (for example, Java, Scala or Groovy) and is executed within dCache. This avoids the overhead from calling out to an external script. Request queuing is dealt with by the driver, which allows the driver to reorder requests in a way that optimizes tape access.

Since nearline storage request queuing is now no longer part of dCache, the maximum active requests limits are no longer reported to poolmanager. For some drivers, such a limit may not even exist. Pool selection has been updated to no longer rely on these values.

The hsm family of commands in the pool service have been updated to allow the nearline driver to be specified when defining a new nearline storage. The new hsm create command is used to define the nearline storage, while the hsm set command has been updated to allow the driver to be configured.

dCache ships with a number of nearline storage drivers. These are explained in the following sections.

script driver

dCache ships with a driver called script. This driver performs a callout to the classic HSM integration script. The driver accepts a number of arguments, settable through the hsm set command:

The path to the script.
Maximum number of concurrent reads.
Maximum number of concurrent writes.
Maximum number of concurrent removes.

To provide backwards compatibility with existing setups, the script driver is the default. During intialization, the rh set max active, st set max active and rm set max active commands are translated to configuration parameters for instances of the script driver. Support for these commands is however deprecated and it is recommended that the pool setup file is regenerated using the save command.

copy driver

The copy driver is similar to the classic hsmcp script. It copies files between the pool and some other directory. It accepts a single configuration property:

The path of the directory into which files are copied.
link driver

Similar to the copy driver, but creates hard links rather than copying files. The directory in which hard links are created must be on the same file system as the pool’s data directory. Like the copy driver, the link driver only accepts a single configuration property:

The path of the directory within which hard links are created.
tar driver

This is an experimental driver, provided mostly as a demonstration right now. The driver bundles files into tarballs and stores those in a configurable directory. The driver does currently not support removing files.

The path of the directory within which tarball files are created.



BringOnline requests, as well as get and put requests for which a transfer URL (TURL) was already prepared, now survive an srm restart.

The database code was updated to avoid warnings about duplicate key violations in the PostgreSQL log file during startup.

The SRM list response has been fixed such that the status of failed or unprocessed SURLs is included.

Failure reporting for SRM v1 requests has been improved.

srm shutdown was improved to avoid errors about unterminated threads in log files. Additionally, the service now waits for up to three seconds for database updates to finish before shutting down.

Culling of jobs in the database is now limited to a single thread to reduce the impact on regular operation.

The ThreadManager cell has been eliminated. The following new properties have been introduced instead: srm.cell.limits.message.threads.min, srm.cell.limits.message.threads.max, srm.cell.limits.message.threads.max-idle-time, srm.cell.limits.message.threads.max-idle-time.unit, srm.cell.limits.message.queue.max. The defaults will suffice for most deployments.

The output of the ls command in the srm admin interface has been cleaned up.

Absolute FTP TURLs

The SRM protocol allows the client to negotiate a transfer URL (TURL) for uploading or downloading a file. TURLs for FTP transfers are defined in RFC 1738. According to the standard, they are to be interpreted relative to the user’s login directory (home directory) and not relative to the root directory.

The ‘srm’ service in previous versions of dCache generated TURLs relative to the root directory. This will not work as expected for RFC 1738-compliant clients if the user has a home directory other than ‘/’.

This problem has now been fixed and the returned TURL will work for users with a non-root home directory. Unfortuantely neither JGlobus nor Globus interpret RFC 1738 correctly. As a workaround we now generate TURLs with an extra leading / in the path. This provides the correct behaviour with FTP clients typically used for SRM, but would fail if a standard compliant client would be used.

Root path aware transfer protocol negotiation

When negotiating a TURL, the srm service must choose from a list of protocols provided by the client. dCache will use the list of doors registered with the loginbroker to determine supported protocols. If doors are configured to use different root directories, not all doors may be able to serve a file. This is now taken into account when choosing the transfer protocol to use.

GridSite compatible delegation

Support for GridSite delegation v2.0.0 was added. The endpoint is run as part of the SRM service and has the path /srm/delegation. The GridSite delegation endpoint allows a user to delegate a credential, discover the remaining lifetime of any delegated credential and destroy delegated credentials. Those credentials delegated via GSI (typically as part of an SRM operation) will have the delegated ID of gsi. Using this, a client may query the lifetime of any already delegated credential and delete credentials delegated via SRM operations.

A delegation client that allows scripted and console-based interaction with GridSite endpoints has been implemented. The client is released independently, as part of the dCache srm client package.

Unique write TURLs

Until version 2.9, the dCache srm service has acted as a redirector for regular dCache doors. That means that the TURLs generated by the srm service have been the same URLs you would have used to access those files without SRM.

With dCache 2.9 this has changed and, for uploads, each TURL is now unique. The uploaded file remains invisible to users using non-SRM protocols until the upload is committed by the SRM client calling srmPutDone.

SRM upload requests have a finite lifetime; if the SRM client does not call srmPutDone in time, the request will timeout. Should this happen, any file uploaded using the corresponding TURL will be deleted and any subsequent attempt to upload to the TURL will fail.

There are many reasons for making this change. To name a few:

  • The expected file size, access latency and retention policy provided by the SRM client could not be supported without enabling the spacemanager service. These parameters are now respected even without space management.

  • Failure recovery was difficult, as dCache could never be certain whether a file uploaded through a door happened as part of the SRM upload or was done by another non-SRM client that incidentally uploaded a file to the same path at the same time.

  • If an SRM upload was aborted and retried, we could not prevent the first client from completing the second upload as we always generated the same TURL. Thus we couldn’t properly isolate the two uploads from each other.

  • Since the only way to bind file size, space token, access latency and retention policy to a file before upload was through the spacemanager service, space reservations were bound to paths. Such reservations would prevent the srm from creating concurrent uploads to the same SURL, but to work correctly, the two services had to agree on the state. Wherever there is distributed state, there is a risk that the state is inconsistent. This problem was at the heart of the “Already have 1 record(s)” errors.

To generate unique write TURLs, the srm creates a unique upload directory for each upload. After the transfer completes, the file is moved to the target path and the upload directory is deleted. Parameters like expected file size, access latency, retention policy and space token are bound to the directory using Chimera directory tags.

By default, these temporary upload directories are created under the /upload directory. Chimera automatically creates this directory the first time it is needed. Ownership and permissions are set such that nobody can list the directory and thus nobody can discover the temporary upload directories.

IMPORTANT: If the transfer doors used for srm do no export the /upload directory, the dcache.upload-directory property has to be altered to a path that is exported by the doors. A typical site value would be /pnfs/example.org/data/upload. This property has to be set for pnfsmanger and all doors used for SRM transfers. In the unlikely event that you cannot find a directory visible through all doors used by the srm, you have to introduce special door instances only used for SRM transfers. Such doors can be configured to only export the upload base directory.

The srm.enable.space-reservation.implicit and srmImplicitSpaceManagerEnabled properties are obsolete. To allow uploads to links in link groups outside a space reservation, enable spacemanager.enable.unreserved-uploads-to-linkgroups in spacemanager. That property deprecates spacemanager.enable.reserve-space-for-non-srm-transfers. The property srm.enable.space-reservation.strict is obsolete too as it did not have any effect.


Due to the changes to the srm service, several features of spacemanager are no longer needed and have been removed. The relevant options have been removed from the admin shell interface.

Upon upgrade, the space manager database schema is changed: The pnfspath, expirationtime and deleted columns are dropped from the srmspacefile table. State value 0 is no longer used in the srmspacefile table. The freespaceinbytes column is replaced by availablespaceinbytes in the srmlinkgroup table.

Due to these changes, dCache cannot be downgraded without rolling back the schema changes using one of the dcache database rollback commands with a 2.9 dCache and only then downgrading dCache.


The service was extended with a feature to generate temporary upload paths. This is an internal change that is used by the srm service to generate unique write TURLs.


The HSM cleaner is enabled by default. It used to be disabled by default because traditionally PNFS setups all had custom solutions for how to remove files from tape. Now that cleaning from tape is an integral part of dCache and PNFS is no longer supported, having the HSM cleaner work out of the box is a sensible default. It can be disabled by setting:

cleaner.enable.hsm = false


dCache is in the middle of a transition between two frameworks for implementing the command line commands available through the admin service shell.

The newer of these frameworks provides much nicer help output. This output has been improved to include a description of the arguments accepted by a command and the default values for arguments and options. This information is only available for commands implemented using the new command framework.

Many of the commands shared between services have been ported to the new command framework.

The say and exec context commands are no longer available from cells other than the System cell.

Several admin commands primarily intended for internal use by dCache return array data. Executing these commands through the admin interface resulted in only an internal object reference. Now the array content is output.


Added suport for the kXR_locate, kXR_query, and kXR_set xrootd commands.

Improved thread pool termination on door shutdown.


There is no longer a requirement that this service is hosted in the same domain as the srm service.

Error log messages have been simplified.



The LDAP plugin now allows the connection URL to defined by setting gplazma.ldap.url. The properties gplazma.ldap.server and gplazma.ldap.port are deprecated.

Changelog from 2.8.0 to 2.9.0

pool: Report correct client IP to billing for passive FTP transfers
srm: Fix race condition in listing
poolmanager: Fix infinite loop in lru partition type
gplazma-nis: pulgin must fail if no matching principal found
nfs: do not use CDC as try-with-resource
databases: update hikari minimum connection configuration
nfs4: do not log stacktrace on IO errors
poolmanager,pool: Fix precision underflow in WASS
scripts: fix host-credential generation
pool: Rewrite of HSM interface
Generify CDC executor decorators
pool: Let HsmStorageHandler2#store accept more than one file
Update libraries to the latest releases
ftp: Implement delayed passive and enable redirect FTP for IPv6
Enable IPv6 stack by default
TransferManager: fix error-handling
nfs: use vfs cache
libs: use nfs 0.8.0
chimera: fix behavior on create of ‘.(fset)’ files
srm: tidy up toString methods
info-provider: fix capitalisation of SRM in ControlProtocol
libs: update to nfs4j–0.7.5
scripts: fix shell-specific expansion
srm/transfermanager: make errors less noisy
srm/transfermanager: make errors less noisy
srm: tidy up SRM CopyRequest and CopyFileRequest classes
ftp: fix unit tests so they run on IPv4-only machine
Kill workers when calling BoundedExecutor#shutdownNow
Merge remote-tracking branch ‘upstream/master’ into ldap-use-url/master
srm: fix stored diff so that the Solaris ‘patch’ command works
gplazma2-ldap: use url rather than host+port
chimera: introduce inode generation number
scripts: add database updateSQL command
admin: Fix help on error
ftp: add work-around for GlobusOnline uploads
ftp: fix CKSM command for files with white-space
info: add potentially useful static information
info: remove emi.version from non-EMI dCache releases
pool: Let permanent migration jobs react to access and sticky changes
admin: Add array output
cells: Add help command to System cell
transfermanagers: Fix multi-domain issue
gplazma-nis: suppress annoying error message
ftp: add logging when client disconnects
cells: Fix message timeout
admin: Fix cd to System cell
pkg: fix solaris packaging
ftp: add support for updating atime
Fix CellStub callback.
pnfsmanager: Fix tag handling in upload path creation
billing: Add -f option to dcache billing command
chimera: do not explicitly update parent’s mtime on create/remove
spacemanager: Another hsqldb schema management fix
Fix executor flaws
Make RemotePoolMonitor asynchronous
srm: Avoid loading the same job multiple times
spacemanager: Fix DB trigger migration
nfs4: add support for WRITE with proxy IO
nfs4: refactor proxy io adapter
nfs4: implement write for DcapChannel
nfs4: reuse existing mover if client comes with the same state id
srm: fix bring-online rescheduling on restart
srm: fix unit-tests after recent update
srm: fix restore so jobs for other schedulers are not restored
srm: Added missing import
srm: Break cyclic dependencies
xrootd: Shutdown channel factory on shutdown
srm: Fix scheduling period for expiration task
srm: allow jobs in RQUEUED and READY states to survive restart
chimera: fix NPE if mkdir fails with permission denied
srm: add description of job states
srm: fix restore from database
srm: Improve database initialization and shutdown
srm: Add error message to an IllegalArgumentException
pool: Controlled termination of thread pools during pool shutdown
poolmanager: Get rid of ThreadPool interface
Get rid of callback executor in CellNucleus
srm: Get rid of ThreadManager
nfs4: update stateid on read with proxy adapter
Revert “poolmanager: Get rid of ThreadPool interface”
poolmanager: Get rid of ThreadPool interface
srm: Terminate db threads on shutdown
billing: Make indexer case insensitive
srm: Restore RESTORED state
pool: Avoid memory exhaustion problems on write with xrootd and http
pool: Fix lock problems in xrootd and http movers
ftp: add read-only support for ctime and atime
logback: add comment about bugs in 1.1.1 to poms
pnfsmanager: Clean up cleaner by making use of new CellStub features
Mock CellEndpoint rather than CellAdapter in HsmRestoreTest
Get rid of support for synchronous messages in CellNucleus
loginbroker: Fix login broker registration retry
chimera: allow updating the creation time
remote logging: add empty string defense to remote server wrapper
logback: correct MDC dependency in appender
srm: remove unused state
srm: refactor SchedulerFactory
delegation: add support for GridSite
libraries: update bouncy-castle
srm: fix race condition if stop is called too soon after start
Get rid of more uses of CellNucleus#send methods
Avoid direct use of CellAdapter#sendMessage
Remove direct calls of CellEndpoint#sendAndWait outside cells module
Layer synchronous CellStub use on top of async code
Delete obsolete code
Add transparent retry support to async use of CellStub
Eliminate some direct use of AbstractCellComponent#send
Refactor CellStub
Rename one-way CellStuf#send to notify
srm: Replace pin and unpin callbacks with ListenableFuture
http: add minimal HEAD support in the mover
spacemanager: Remove obsolete features
srm: Generate unique write TURLs
Auto indented file to clean up inconsistencies
pnfsmanager: Add support for temporary upload paths
http: adjust unit test to multirage changes
nfs: return NFSERR_IO is we detect broken file on tape
ftp: add support for MFMT, MFCT and MFF commands
alarms: add logging and peaceful exit for logback startup errors
http: fix multipart reply
srm: Fix compatibility with PostgreSQL 8.4
ftp: fix printing of UNIX.Mode fact
ftp: add logging to access log file
ftp: add support for HELP and SITE HELP commands
logback: revert to 1.0.12
http: fix mocking of Netty in unit-tests
nfs: do not label pnfsid and client ip in NDC
core: update PoolPassiveIoFileMessage to provide boot verifier
Upgrade third party libraries
xrootd,httpd,pool: Upgrade to Netty 3.9.0
system-test: Explicitly disable GridFTP 2 in tests
pool: Fix pf command regression
spacemanager: Fix database sequence initialization
srm: Make FTP TURLs absolute
ftp: Make root path behaviour configurable
Allow retry on transient transfer failures
srm-client: Do not retry on permanent FTP failures
pnfsmanager: Fix compatibility with 2.6 pools
ftp: Fix race condition in command dispatching
ftp: Give io thread pool a name
chimera: Avoid integrity constraint violations on registering locations
srm: Avoid duplicate key violations on startup
ssh1: Fix session instantiation
ssh1: Fix typo in service batch file
ftp: add support for the CREATE fact
system-test: Enabled tiered compilation
Improve startup speed of tools
srm-client: Improve startup time
Convert tabs to spaces
billing: Initialize all fields in billing records
pool: Fix logging context for thread pool
pool: Use FireAndForgetTask instead of custom class
chimera: fix inode type detection
srm: Include failed or unprocessed surls in list response
Add missing newline when quitting shell applications
Upgrade to HikariCP 1.3.2
xrootd: Upgrade to xrootd4j 1.3.1
srm: Add SRM status code to SRMException
srm: Various minor refactorings
srm: Take root path into account when selecting protocol
chimera: fix create link on existing object
packaging: remove ‘–1’ from tarball
srm: simply job.setState
gss: tidy up FQAN-extraction
nfs: fix NPE on door reboot
nfs: remove unused exception declaration
Convert tabs to space
chimera: Minor refactoring of tag reading code
loginbroker: Register root path of doors
Update to liquibase 3.1.1
chimera: Add mv to chimera shell
srm,srm-client: Report failure for failed SRM v1 transfers
poolmanager: Add partition for transfer buffers
use ISO 8601 time format in sweeper ls -l output
srm-client: fix adler32
pool: fix lost space messages (and cause)
srmclient: fix support for URLs with IPv6 address
chimera: Optimize read of file and directory attributes
system-test: Fix wrong path in hostCredentials script
pool: Bind FTP data channel to single interface
spacemanager: Replace freespaceinbytes with availablespaceinbytes in link group table
spacemanager: Set file to TRANSFERRING on accept file request message
src: split nfs code info a dedicated module
chimera: change FileSystemProvider#move() return value meaning
chimera: fix transaction leak
libs: update nfs to 0.7.4
chimera: fix rename into a file
cells: move command support into commons-cli and create skeleton CLI client
libraries: update Hikari to v1.3.0
nfs4: remove dead class
nfs-proxy: fix logging CDC
nfs4: clean failed request from pending queue
nfs-proxy: use sessionId provided by Transfer class
nfs-proxy: fix logging
system-test: Add support for compatibility testing releases
pool: Increase TCP backlog for FTP mover
ftp: Auto indent data channel code
nfs-proxy: add support for open-less read requests
Revert “nfs-proxy: add support for open-less read requests”
nfs-proxy: add support for open-less read requests
Log fatal errors when fetching pool monitor
gplazma-xacml: remove dependency on org.apache.xalan
Revert “Revert ”packages, fhs: fix saxon includes/excludes expression“”
gplazma2-xacml: revert version of opensaml
xrootd: Add support for kXR_locate, kXR_query and kXR_set
dcache-webadmin, alarms: remove required annotations from datastore setters
dcache-core, alarms: restore xml connector to initialization properties
Revert “packages, fhs: fix saxon includes/excludes expression”
packages, fhs: fix saxon includes/excludes expression
pool: Fix access time update
xrootd: Upgrade to xrootd4j v1.2.4
chimera: Add time options to ls command
chimera: Add help command in shell
chimera: Fix option parsing in FsFactory
core, billing: remove unit test
libs: update to bugfix nfs4j–0.7.3
billing,webadmin,alarms: Use HikariCP
Use HikariCP connection pool in most services
dcache-core, alarms: remove postgresql precondition on schema
webadmin: Inject service configuration into Spring environment
Use Spring’s Environment and PropertySource feature to inject configuration
Drop JDBC driver properties
Add alarms and transfermanager database to system-test setup
libs: update nfs code to 0.7.2
httpd: revert datanucleus-xml version (pom.xml)
chimera-nfs: add bringonline command to “.(fset)”
system-test: Add utility to start HSQLDB Swing Console
Use annotated command syntax for most shared commands
Include arguments and default values in admin command help output
chimera: Optimized creation time column changeset
dcache-webadmin: fix ArrayIndexOutOfBounds exception in ListView (Tape Transfers page)
ftp: Reenable support for RFC 2428
cells: remove conflict between CellAdapter and CellShell
srm-client: switch to common Args
utils: move Args class into commons
Update branch ready for the 2.9 branch