dCache 2.15 Release Notes

Executive summary

Highlights from this release:

Improved glob patterns.
More robust handling of over load situations.
fsck for Chimera on upgrade.
Reduced storage requirements for Chimera.
Chimera can store multiple checksum for a file.
Admin shell documentation updates.
gPlazma restriction attributes.
Set ACLs through DCAP.
Pool startup improvements.

Incompatibilities

Some configuration properties have been deprecated or made obsolete. dcache check-config can tell you if you are affected.
Several Chimera schema changes are applied upon upgrade. To downgrade those schema changes have to be rolled back prior to downgrade.
Several bugs have been fixed. In the unlikely event that you relied on these bugs, upgrading may break your installation.
X.509 proxy certificate source address restrictions are now enforced. Clients that relied on these not being enforced will fail after upgrade.

Release 2.15.40

admin

While migration move tasks on pools were working correctly, for migration info command an error occurred, that the current user (root) wasn’t allowed to execute anything (due to missing ACLs). This is now fixed.

Changelog 2.15.39..2.15.40

ab2e71c: [maven-release-plugin] prepare release 2.15.40
a4577fd: admin: Fix Inconsistent ACL enforcement, RT 9207
90ff687: [maven-release-plugin] prepare for next development iteration

Release 2.15.39

httpd

The “Disk Space Usage” webpage (/usageInfo) contains a table showing information about each pool in the dCache cluster. The “Layout” column showed the capacity usage graphically, with different colours showing how much of that pool’s capacity is being used for different tasks. This release fixes the Layout heading to describe a previously undocumented colour.

Changelog 2.15.38..2.15.39

1ae416b: [maven-release-plugin] prepare release 2.15.39
3d03714: [maven-release-plugin] prepare for next development iteration
609a15d: httpd: Fixed table headers in usageInfo

Release 2.15.38

pool

If the communication between a pool and PnfsManager times out, the error message is not well suited to diagnosing the problem: Failed to instantiate mover due to unsupported checksum type: Request to [>PnfsManager@local] timed out. The checksum type is not playing an important role here. Hence, this patch updates the error message.

Changelog 2.15.37..2.15.38

8e94730851: [maven-release-plugin] prepare release 2.15.38
611976e77f: pool: fix error message on timeout
5475651cb9: [maven-release-plugin] prepare for next development iteration

Release 2.15.37

webdav

A recent update, commit 5abc0e1c, improved the behaviour of the Milton WebDAV libraries if an IOException occurs during an upload. That patch, unfortunately, did not address all issues, and when non-spec-conformant clients are used against dCache, stacktraces can be triggered.

This patch corrects that behaviour. Also, in case of errors, the error code returned in case of any problems was changed from 400 to 500, which should signal cliens that they are free to retry the transfer after a timeout.

Changelog 2.15.36..2.15.37

d670934319: [maven-release-plugin] prepare release 2.15.37
a632339cf6: webdav: make Milton work-around more robust
30a90184cc: [maven-release-plugin] prepare for next development iteration

Release 2.15.36

webdav

File transfers through WebDAV doors could potentially bypass any Restrictions checks in PnfsManager. This patch ensures that Restrictions are always checked and observed, and improves PnfsManager’s logging to give information in case a Restrictions check is not posssible.

Changelog 2.15.35..2.15.36

ed212800a2: [maven-release-plugin] prepare release 2.15.36
a7049777c3: webdav: Fix restriction check when downloading a file
edecb45a81: [maven-release-plugin] prepare for next development iteration

Release 2.15.35

Changes affecting multiple services

The version of the PostgreSQL driver used by dCache internally was brought up to 9.4.1212. This fixes the issue described in liquibase bug 2939.

system-test

The system-test module, used for demonstration or testing purposes, comes with a built-in X.509 infrastructure. With this release, expired certificates are replaced by new ones.

Changelog 2.15.34..2.15.35

8fec0c7a80: [maven-release-plugin] prepare release 2.15.35
d092cb863f: system-test: update disposable-CA generated credentials
7dcb06876e: postgresql driver: update version to 9.4.1212
8ec36faeb1: Revert “libs: update to nfs4j–0.12.4”
ff0f9afb66: libs: use nfs4j–0.12.4
a6b500057d: libs: update to nfs4j–0.12.4
1f10c223cf: [maven-release-plugin] prepare for next development iteration

Release 2.15.34

chimera

There was an issue with a symbolic link to a directory where destination where destination contained trailing slash. This is now fixed.

Changelog 2.15.33..2.15.34

74554b5: [maven-release-plugin] prepare release 2.15.34
758b40b: [maven-release-plugin] prepare for next development iteration
a676ef6: chimera : handle empty paths elements path2inumber stored procedure

Release 2.15.33

chimera

The current release fixed database query for storing multiple checksums for a file.

ftp

The Socket read method may return zero to indicate that no bytes were read. Although this is not an error, such occurances will result in a transfer failing.

This is now fixed.

Changelog 2.15.32..2.15.33

fe749c0: [maven-release-plugin] prepare release 2.15.33
921d6a7: ftp: prevent execution of most commands when unwrapped
766d2e3: chimera: fixed database query for storing multiple checksums for a file.
d9fe757: ftp: do not fail proxy transfer if read returns zero bytes
7af7a9b: [maven-release-plugin] prepare for next development iteration

Release 2.15.32

ftp

The current release added implementation of MLSC. As a result Globus is able to query the contents of dCache directories using FTP and without creating additional TCP connections.

Changelog 2.15.31..2.15.32

952cd67: [maven-release-plugin] prepare release 2.15.32
52fd7e0: ftp: implement the MLSC command
0fa0eec: [maven-release-plugin] prepare for next development iteration

Release 2.15.31

ftp

The current release improves compatibility between dCache FTP client and Globus GridFTP server.

srm

During an ATLAS stress-test of tape recalls, it was discovered that various sites had relatively short request lifetimes. However, the SRM spec provides the opportunity for the server to inform the client (FTS, in this case) of what lifetime a request actually has. The current release includes the requests remaining lifetime in the response from the server.

The current release improves the documentation to help Admins to have a better understanding how to configure dCache correctly.

Changelog 2.15.30..2.15.31

332aad0: [maven-release-plugin] prepare release 2.15.31
aa7bbfa: ftp: add support for paths relative to home directory
ac60896: ftp: Add support for SITE WHOAMI command
1eb8a63: ftp: update parsing of CLIENTINFO command
e809fa5: srm: include remaining request lifetime in various responses
93bce13: srm: update srm request.*.lifetime configuration properties documentation
6081b01: ftp: modify facts describing namespace ownership
daad892: ftp: add support for SITE TASKID command
6883e11: ftp: add initial support for checksum performance markers
85ef56e: ftp: show SIZE facts for directories
c94decf: ftp: add support in OPTS RETR for specifying performance marker frequency
04fadfb: [maven-release-plugin] prepare for next development iteration

Release 2.15.30

xrootd

In https://github.com/xrootd/xrootd/issues/459, it became apparent that dCache could improve xrdcp compatibility by sending checksum information in lower case. This release contains this change, which should improve xrootd operations.

Changelog 2.15.29..2.15.30

5e8913d: [maven-release-plugin] prepare release 2.15.30
97518c0: xrootd : use lower case for checksum algorithm names when replying to checksum queries.
1d4f183: [maven-release-plugin] prepare for next development iteration

Release 2.15.29

srm

The SRM code has been made more robust against races between file deletions and copies.

systemtest

The ‘system-test’ script was updated to ensure anonymous dcap tests succeed.

Changelog 2.15.28..2.15.29

23c0fbb: [maven-release-plugin] prepare release 2.15.29
12f712c: systemtest: allow anonymous dcap activity
8eebdd4: srm: fix recovery procedure in internal copy if source is deleted
a1c5bea: [maven-release-plugin] prepare for next development iteration

Release 2.15.28

chimera-enstore

Several improvements added to db chimera so that sql statements work as expected.

Changelog 2.15.27..2.15.28

8a235a3: [maven-release-plugin] prepare release 2.15.28
0a2d5cf: chimera-enstore: specify fields order on insert
fc69f5d: [maven-release-plugin] prepare for next development iteration

Release 2.15.27

cleaner

Users reported that they wanted to see space freed up by cleaner processes to be reported as free as soon as possible. This patch sends notifications about freed up space more often, resulting in quicker status updates.

pool

A problem was fixed that could cause the csm check to fail on pools containing broken files.

Changelog 2.15.26..2.15.27

a31dd00: [maven-release-plugin] prepare release 2.15.27
03bf013: cleaner: Send notification more often
0dade8e: pool: Fix csm check command in the pressence of broken files
63d0647: [maven-release-plugin] prepare for next development iteration

Release 2.15.26

ftpclient

The current release improves compatibility between dCache FTP client and Globus GridFTP server.

Changelog 2.15.25..2.15.26

1cf0d51: [maven-release-plugin] prepare release 2.15.26
1328e3b: ftpclient: fix multiline ADAT reponses
9c2d939: ftpclient: encrypt SITE CLIENTINFO command
e564036: [maven-release-plugin] prepare for next development iteration

Release 2.15.25

dcap

Connections of non-DCAP clients to a dCache no longer result in stack-traces in the logs.

poolmanager

PoolManager was updated to properly handle the dcache.authz.staging.pep and dcache.authz.staging parameters. This allows to enable stage protection properly.

Changelog 2.15.24..2.15.25

073a1d5: [maven-release-plugin] prepare release 2.15.25
477c6c0: dcap: don’t create stack-trace if tunnel fails due to bad client
07b4153: PoolManager : stage protection, fix error in stage.fragment
194d914: [maven-release-plugin] prepare for next development iteration

Release 2.15.24

Changes affecting multiple services

Fixed a bug in admin and in the pool manager rebalancer component that caused these to not detect certain error replies from other components.

Internal notification processing between cleaners and Pin Manager, Replica Manager and Space Manager was improved and runs quicker now.

billing

If a dCache instance was shut down while the billing service was in the middle of a refresh, an exception was logged and shutdown was delayed. A change in exception handling fixes this rare scenario, ensuring a quick shutdown and no unnecessary log entries.

dcap

This release makes it possible for admins to ban outdated, problematic versions of dcap clients. Some old client versions contain a bug that causes the client to make unsatisfiable requests to a pool with no way for dCache to reject the request: the client will simply retry.

The client version limits are exposed using the new configuration property dcap.limits.client-version. The default is to allow all dcap client versions, unchanged from the previous behaviour.

This release fixes a regression through which Kerberos dcap would not work for host principals containing a ‘-’ character. GSI dcap was not affected.

doors

Fixed a bug in the lb set tags command in doors that prevented setting an empty list of tags.

pool

A bug that caused non-critical stack traces to be logged on the pool after stage or deletion failure from nearline storage has been fixed.

Changelog 2.15.23..2.15.24

4d9233d: [maven-release-plugin] prepare release 2.15.24
b59b563: dcap: expose dcap client version limit
a3e7434: dcap: fix Kerberos dcap if principal contains a ‘-’
7d39b2a: dcap: fix regression in handling old version
20927d4: pool: Suppress two stack traces in nearline storage handling
1330036: cleaner: Send notifications concurrently
b7a0b4e: billing: fix stacktrace and slow shutdown if in refresh
96682a6: doors: Allow setting an empty list of login broker tags
b8b0388: dcache: Fix detection of message errors in poolmanager and admin
77d05f7: [maven-release-plugin] prepare for next development iteration

Release 2.15.23

doors

Fixed a bug in which information on stage pool and number of attempts were lost when retrying pool selection requests.

Changelog 2.15.22..2.15.23

6849760: [maven-release-plugin] prepare release 2.15.23
60b1402: doors: Ensure that pool selection context survives between retries
9b0d23d: [maven-release-plugin] prepare for next development iteration

Release 2.15.22

billing

Fixed a problem in the output of the dcache billing command using JSON or YAML when the billing format includes a custom date format.

dcap

Add support for the dcap client supplying additional version information.

gplazma

Fix explain login and test login commands so they are able to test logging in with username and password.

Add examples to explain login command help. Our thanks to Onno Zweers for this change.

httpd

Fix regression in the transfers.txt output format.

Update the transfers.html page so it no longer includes <unknown> for default/unknown protocols. With this version of dCache, these are represented by a ? character. The webadmin page is updated to give consistent output.

scripts

Fix the billing indexer to ignore the format string, if present.

Changelog 2.15.21..2.15.22

a83ec70: [maven-release-plugin] prepare release 2.15.22
f349a39: Active Transfers: substitute ? for <unknown> on html pages
3e91415: common: add support for UserNamePrincipal as user:<name>
8091f5f: Added ‘explain login’ examples to help text in Gplazma2LoginStrategy.java
45259b1: billing: Strip format string from attribute name
0664a22: transferObserverV1: replace Args with Joiner to construct transfers.txt linesMotivation:
0812c1e: billing: Make billing indexer work with custom format strings
fc802eb: [maven-release-plugin] prepare for next development iteration
7ea0b69: dcap: add support for clients presenting more version metadata

Release 2.15.21

commons

If dCache’s internal ShellApplication framework detects a critical behaviour that might indicate a bug in the application, error messages now include an explicit request to send a mail to the developers and more relevant information for assessing and reproducing the situation.

gplazma2

This release fixes a small bug in GPlazma which inappropriately tried to handle non-DN subjects in x509 certificates. These will usually fail in gPlazma anyway, but the reported error was confusing.

srm

Handling of DNS names without trailing dots in certificates has been made more robust and universal.

Changelog 2.15.20..2.15.21

b1fa6b1: [maven-release-plugin] prepare release 2.15.21
0ad1442: commons: log bugs with stack-trace and instructions
3972ba3: gplazma2-xacml: remove erroneous creation of placeholder extensions
bf793d3: [maven-release-plugin] prepare for next development iteration
3855287: srm: remove trailing dot from reverse lookup result

Release 2.15.20

cells

In rare cases, an interrupt needed to cleanly shut down the location manager connector would not arrive. This issue was corrected, ensuring more reliable behaviour on cell shutdown.

Fixed a problem in which threads could inappropriately be created as daemon threads, causing problems in killing those threads when the cell shuts down.

Changelog 2.15.19..2.15.20

df65480: [maven-release-plugin] prepare release 2.15.20
17a58ad: cells: Fix lost interrupt exception
4192d80: cells: Ensure that newly created threads are non-daemon normal priority threads
9af1181: [maven-release-plugin] prepare for next development iteration

Release 2.15.19

commons

The \s admin command uses the toString method to serialise the requested so that the remote cell may correctly parse it. This did not always work: ‘=’ characters in arguments were escaped but not unescaped; arguments that start with a ‘-’ character were not escaped; empty words were lost. In the current release this is fixed and \s command works as expected.

dcache

The current release fixes a regression in which the exit code of check-config would always be zero even when errors were detected.

Changelog 2.15.18..2.15.19

77cf0d0: [maven-release-plugin] prepare release 2.15.19
ebaaa55: dcache: Generate proper exit code for check-config command
1bde593: commons: fix Args string parsing and toString method
ab401cb: [maven-release-plugin] prepare for next development iteration

Release 2.15.18

alarms

Alarm email notifications are now sent only on the first occurrence of given alarm (i.e., for that alarm instance’s unique ID).

If an alarm has been closed and not deleted, but then occurs again, the counter for receiving that alarm is now reset to 1, in order to treat this as a new (set of) occurrences, and to guarantee a new notification will be sent.

webadmin

Due to an implementation detail in a library used for the webadmin pages, filtering tables was a bit unintuitive until now: A filter that was set in a certain table column would reappear on tables in other browser windows if they had similar columns.

This behaviour was corrected, and tables on different pages exposed simultaneously in different browser tabs are now filtered independently. However, the fix also has the side effect that now with page reloads and form submissions the filters are cleared. Any commands, however, will always be issued correctly.

Filtering rows in webadmin tables could occasionally lead to unintuitive behaviour with regard to selections: Filtering a table and hiding rows may be included in a “select all” or “deselect all” operation. This was fixed, and selection/Deselection of hidden rows is now prevented.

Issues with filter boxes disappearing or filters resetting have been solved by disabling AJAX auto refresh for the affected pages.

Changelog 2.15.17..2.15.18

f4ef1a5: [maven-release-plugin] prepare release 2.15.18
6de5710: dcache-webadmin: synchronize client-side filtering with server-side selection of rows on pages using picnet table filters
d41dda5: alarms: reset count history on reopened alarm
8bc6532: dcache-webadmin: disable saving table filter settings to browser cookies
65e3ae2: dcache-webadmin: disable AJAX autorefresh on pages using picnet table filter library
d9c9f9a: alarms: only send email on first alarm occurrence
3f6240d: [maven-release-plugin] prepare for next development iteration

Release 2.15.17

chimera

Chimera occasionally suffered from (operationally irrelevant) IllegalStateExceptions. Those are now avoided.

doors

Doors could get stuck temporarily if a file was deleted during pool selection. This has been fixed, and in such cases, transfers are now aborted properly.

Changelog 2.15.16..2.15.17

1990b83: [maven-release-plugin] prepare release 2.15.17
f6d6640: doors: Abort transfer if file is deleted during pool selection
e739915: chimera: Fix IllegalStateException in inode cache
212287f: [maven-release-plugin] prepare for next development iteration

Release 2.15.16

billing

When using the dcache billing command with a non-default date format in the billing file, an unneccessary stack trace was printed. This has been corrected.

srm

The SRM should periodically (by default every 10 minutes) delete obsolete historic data (older than 10 days by default) from the database. For cases where there are problems with that process, error logging and robustness against temporary database problems have been improved.

webadmin

The Wicket library used by dCache internally issued warnings about upcoming naming changes. Those cluttered the log files, and are silenced from the current version on.

Changelog 2.15.15..2.15.16

319b6b6: [maven-release-plugin] prepare release 2.15.16
f7573ab: srm: make out-of-date historic data deletion more robust
c6b5c0d: webadmin: silence warning about future change in wicket
a578916: [maven-release-plugin] prepare for next development iteration
f60fb98: billing: Removing erroneous stack trace output

Release 2.15.15

ftp

The Apache Commons FtpClient can issue the LIST command with the non-standard -a option. Which was causing dCache to switch output format from the long (ls -l-like) to the short (ls-like) response. This is fixed now and dCache is more compatible with Apache Commons FtpClient.

Changelog 2.15.14..2.15.15

60e7939: [maven-release-plugin] prepare release 2.15.15
689d9bb: ftp: improve compatibility with Apache Commons FtpClient
19724ba: [maven-release-plugin] prepare for next development iteration

Release 2.15.14

cells

If the create command in CellShell fails because of unreadable setup files, it throws an IOException. This was incorrectly reported as a bug. Reporting has been corrected now.

In rare cases, active transfers would show up with an incorrect state in the active transfers page of the admin backend. This was fixed, so that ransfers which are staging from non-DCAP doors are correctly indicated (in yellow) on the active transfers page, instead of showing up as “No Mover found” (in red).

common

If there is an IOException when trying to read a setup file, the corresponding file name is now listed in the error message.

poolmanager

An issue with PoolManager prevented it from delivering correct cost estimates. This was fixed, resulting in improved estimations of pool load.

Changelog 2.15.13..2.15.14

c2bee26: [maven-release-plugin] prepare release 2.15.14
ecbdabc: cells: IOException is not a bug in create command
c73bfee: common: include filename in error message
b1120b1: poolmanager: Fix incorrect correction of pool cost
ac5b994: cells: handle empty string pool value on staging in TransferObserver
949d236: [maven-release-plugin] prepare for next development iteration

Release 2.15.13

alarms

Log entries that were promoted to alarm status and that show up in the webadmin table can now contain more detailed information.

Changelog 2.15.12..2.15.13

28777b7: [maven-release-plugin] prepare release 2.15.13
372a696: alarms: add ndc info to alarm info
c663871: [maven-release-plugin] prepare for next development iteration

Release 2.15.12

gplazma2-argus

Fixed a problem with the gPlazma argus plugin that caused it to fail with a ClassNotFoundException.

Changelog 2.15.11..2.15.12

9ec1ba2: [maven-release-plugin] prepare release 2.15.12
825fef5: gplazma2-argus: Update to Argus client 2.2.0 to fix dependency on VOMS library
e877671: [maven-release-plugin] prepare for next development iteration

Release 2.15.11

alarms

A change to the alarms system improves handling of alarms with unset types.

nfs

A race condition in the NFS door that could result in the creation of multiple inconsistent copies of a file being uploaded has been fixed.

pool

A regression was fixed that caused the jtm go command to occasionally not work.

When a transfer’s status is queried before the transfer is initiated, which can occasionally happen for queued requests, Exceptions were logged. This behaviour has now been corrected, providing more robust operation.

Several race conditions in the pool’s migration module are fixed now.

A regression prevented queues to be set to not handle any jobs at all. The fix allows to pass a limit of 0 to mover set max active.

A regression was fixed that caused pools to “leak” movers if those were cancelled while still being queued.

spacemanager

Fixes a compatibility problem with NFS in which space manager would fail with a duplicate key error.

srm

A hint to describe the necessity to include escaping has been added.

The current release fixes issues in which the use of SRM third party copy operations could cause the SRM cell to become unresponsive, possibly even run out of memory.

Changelog 2.15.10..2.15.11

917ff9e: [maven-release-plugin] prepare release 2.15.11
ac1b1df: srm: add hint to escape IDs
2999fee: nfs: Fix race condition in transfer startup
02dffa3: common-cli: Fix compatibility with Java 7
8f1884e: info: fix broken unit-test
7e9f7d6: srm: Resolve message thead blocking issues with SRM third party copy
2e2bedf: spacemanager: Work around for doors resubmitting PoolAcceptFileMessage
54bfb03: pool: Fix several race conditions in migration module
7b848b8: pool: Fix regression in mover set max active command
d63fc4e: pool: Fix mover leak
d762d37: pool: Fix synchronization regression in jtm
001ce03: pool: avoid NPE when querying status of a 3rd-party HTTP transfer
52d0e2a: [maven-release-plugin] prepare for next development iteration
de6004c: alarms: fix NPE in type setter
17f17c6: chimera: update unit-test to log ChimeraFsExceptions

Release 2.15.10

Changes affecting multiple services

Fix security vulnerability. Affects following services: dcap, ftp, srm, transfermanagers, webdav and xrootd.

Release 2.15.9

admin

The admin door now generates SSH keys to ensure compatibility with OpenSSH 7. Additionally, a new property admin.paths.host-keys was introduced in the admin.properties file, allowing to specify the location of keys.

pool

A minor documentation error for the rep ls command was fixed.

script

When a pool’s metadata conversion operations would fail, an error caused a script to report successful conversions. This error has been fixed now.

webdav

There are WebDAV clients that do not send a User-Agent header along with their requests. dCache’s WebDAV code has been updated to avoid NullPointerExceptions occuring in those cases.

Changelog 2.15.8..2.15.9

29c349e: [maven-release-plugin] prepare release 2.15.9
5370f74: webdav: avoid NPE if client fails to send a User-Agent header
90cba25: admin: Fix compatibility with OpenSSH 7
0f98c97: pool: Fix documentation error for -storage option of rep ls command
bb0a6aa: script: Do not claim success if meta data conversion failed
de3207f: [maven-release-plugin] prepare for next development iteration

Release 2.15.8

admin

First observed on Ubuntu Xenial, dCache fails to install on modern Linux distributions due to the short key length of the SSH 1 keys generated in the post install script. This patch removes those keys and their generation code. dCache has been supporting only modern key formats for quite some time now, so this change should not have any impact on users.

pool

When creating movers, some error conditions are expected to occur and dCache is designed to transparently recover from these. Consequently, this patch lowers the log level for the related error messages to reflect that their causes are harmless.

Fixed a staging problem that would lead to failures in nearline COPY operations.

srm

Some race conditions during SRM startup were fixed. Those race conditions could potentially have lead to failures to expire jobs and to wrong job counts in the SRM schedulers.

webdav

Until now, trying to access a file for which the client was not authorized would generate a reply with a status code 200 OK, but an empty body, rather than an error page. This patch corrects that behaviour and also improves exception handling for that case.

Changelog 2.15.7..2.15.8

025b55f: [maven-release-plugin] prepare release 2.15.8
9d057c8: admin: Drop old ssh 1 keys
bb3fa82: pool: Lower log level of certain failures to create mover
bc9ff2a: pool: fix staging for CopyNearlineStorage
451f627: webdav: Fix error reporting when client is unauthorized
c39599a: srm: Fix job expiration during service startup
b966e09: [maven-release-plugin] prepare for next development iteration

Release 2.15.7

billing

The data used to create the 24 hour billing overviews is aggregated in hourly intervals before creating the plots. However, if there is very high activity on the system during an entire 24 hour period, there have occasionally been timeouts when querying the database for this aggregate data. This patch makes the data aggregation more robust against such situations, resulting in lower latency for histogram generation and no more timeouts.

core

The Online Certificate Status Protocol (OCSP) is used to query status information about certificates during authentication. dCache supports this protocol, but relies on a functioning OCSP server for it to work properly. This patch changes the default OCSP mode for dCache to IGNORE, effectively disabling it, which is helpful for sites without a working OCSP server in place.

gplazma

Previously, attempts to authenticate users against an htpasswd entry that was malformed resulted in a stack trace. This patch modifies the error handling so that only a detailed error message (“Bad entry in file: hash does not start ‘$1$’ or ’$apr1$”) is logged.

Changelog 2.15.6..2.15.7

3eb5e44: [maven-release-plugin] prepare release 2.15.7
a420be7: (2.15) billing: use in-memory buffer for hourly aggregate data
ec99c04: Disable OCSP by default
7ada35e: [maven-release-plugin] prepare for next development iteration
d9674a0: gplazma: don’t generate a stack-trace if htaccess is malformed

Release 2.15.6

Changes affecting multiple services

Fixed an issue with the dcache heap dump command when called with a simple file name as the output path. In this case the dump could in some cases be written to a different directory while the script claimed the dump had failed. The dcache dump heap command has a --force option for cases in which the JVM is unresponsive. This option was ignored for processes not running as root. This is fixed now.

cells

Fixed a problem causing FTP and DCAP per connection instances to subscribe to topics they should not subscribe to. This reduces overhead caused by routing updates.

Fixes a problem during shutdown in which communication tunnels between domains were shut down too early.

Fixed an issue that would cause log messages in which placeholders had not been replaced with actual values.

A bouncing message bug in System cell is fixed.

chimera

Sql statement is fixed to use correct field.

pnfsmanager

Reduces transaction length of flush processing in pnfs manager.

Setting atime-gap to –1 (default value) should disable file’s last access time updates. Nevertheless, this was not the case and atime update was always enabled. This is fixed now and file’s last access time can be disabled as described in the documentation.

This release reintroduces multiple pnfs manager queues - one per thread - to avoid possible lock contention in chimera. Please note, that this realease reintroduces as well request folding.

pool

Fixes an issue with pools becoming unresponsive in case of slow DNS reverse lookups.

Fix race condition in request scheduler.

Fixes a couple of regressions in the pool meta data utilities. The dcache pool convert command now opens the source database in read-only mode, preventing that the source is modified. Both the dcache pool yaml and dcache pool convert commands now work without accessing the data files, making them faster and allowing them to be used without the data files being present.

If FTP clients disconnect mid-transfer, pools log a DoorTransferFinished delivery failure as the door is gone. This is fixed now and log messages like Failed to deliver DoorTransferFinishedMessage message are suppressed in pools.

When cancelling a job in a state that doesn’t allow cancellation, an illegal state exception is thrown. This was logged as a bug. This is now fixed.

Pool to pool transfers are supposed to be cancellable, but was not working as the HttpURLConnection does not appear to react to thread interrupt. This could lead to migration job and rebalance job cancellation appearing to hang. This is now fixed.

Fixed a race condition in pool startup that could lead to invalid link counts, resulting in premature deletion of files still open.

Fixes several performance regressions in pools that reduced mover creation rate and could cause pools to become unresponsive due to lock contention.

Fixed a problem in which the meta data store was not closed properly in the dcache pool convert and dcache pool yaml utilities.

poolmanager

The rebalancer commands used blocking sequential messaging to all pools in a pool group from the main pool manager message thread. This blocked all pool manager cell communication for the duration of the rebalancer commands during the start and cancellation. This is fixed now.

srm

Fixed a bug that caused delivery failures of credential service announcements to be logged. The ls -completed=n command has been observed to fail with SRMInvalidRequestException. This is fixed now and the output format of listing list requests has been changed to match that of other requests.

srmclient

The SRM client scripts support discovering necessary paths by navigating relative to the script’s path. This technique does not work with RHEL-7 since they have /bin as a symbolic link to /usr/bin. This is fixed in this release and SRM client scripts work on SL-7, CENTOS-7, RHEL-7.

webdav

Http Basic Authentication was broken and was not triggering a browser login prompt. This resulted in authorization denied. This is fixed now.

Changelog 2.15.5..2.15.6

7461fea: [maven-release-plugin] prepare release 2.15.6
020054f: pool: Fix race in pool initialization
9fff749: pool: Close stores after use in meta data utilities
54feaf3: pool: Fix regressions in meta data utilities
5736f46: srm: Fix listing of completed list requests
f4f8a53: chimera: fix writing into file level
73ba449: poolmanager: Don’t block message thread in rebalancer
2673e38: dcache: fix heap dump to simple file names
e9bddf0: script: Make dump heap –force work for non-root processes
5905cee: pnfsmanager: Revert “Use a single shared request queue for threads”
885a2a1: pnfsmanager: change the timing of PnfsModifyCacheLocationMessage relays to follow reply to pools
9d74ea8: script: Add missing she-bang
b408c41: pnfsmanager: Move flush notification out of chimera transaction
0536efa: pool: Fix p2p cancellation
743e0db: srm: Do not expose TURL before request is ready
7e94e30: pool: Don’t log illegal state exception on migration job cancellation as a bug
69ad050: pool: Reduce lock contention on mover creation
a6aefa7: pool: Fix race condition in request scheduler
15f5646: pool: Avoid reverse DNS lookup in HTTP mover
8a27396: srmclient: fix execution on SL–7 / CENTOS–7 / RHEL–7
b93b5d4: billing: additional fixes to insert triggers
ae337c6: webdav: fixed broken HTTP Basic Authentication
9cc6090: pool: Suppress logging of delivery failure of DoorTransferFinished
7c7f007: system-test: update disposable-CA generated credentials
5fd4a3d: pnfsmanager: fix atime update regression
0ef2be2: cells: Fix logging formatting string
4c1ed93: cells: Avoid bouncing message on no-route errors in System cell
e13ca9c: srm: Suppress message delivery failures for credential service announcements
4ceeabe: cells: Fix tunnel shutdown order
6b63ac8: cells: Do not subscribe to topics in per-session door instances
6f2e5b6: [maven-release-plugin] prepare for next development iteration

Release 2.15.5

chimera

An internal database trigger was updated to insert data in the correct table, thus fixing a problem with the Enstore client.

When accessing a file for reading, the atime value must be updated. Previously, due to an error, the ctime (intended to reflect the time of changes to file attributes) was also changed. This update corrects that problem.

http

In order to increase the performance of the Billing system, reverse DNS lookups were removed from the code. While this will result in IP addresses representing hosts in the billing file, DNS performance no longer impacts overall system performance.

many

When representing checksums in the admin interface and configuration files, checksums are now presented in an improved format.

nfs

This change fixes a NullPointerException that could occur upon file removals when using NFSv3.

Using NFS, a client will poll actively while a file is not available. In order to increase overall system performance, this change introduces a fail-fast behaviour in case the requested file is not available on disk. This will result in more responsive NFS doors from a users’ perspective.

pool

The nearline storage subsystem uses thread pools to manage its workload. Since some tasks are blocking, very high activity can cause these thread pools to grow beyond effective sizes. This may even lead to the pool becoming unresponsive.

This change introduces a new configuration property, pool.limits.nearline-threads, which limits the thread pool size. The default value, 30, is chosen to be sufficient for almost all imaginable use cases while at the same time avoiding potential problems with resource exhaustion.

srm

This update fixes a regression that caused the SRM to hang during startup. The root cause was a problem in cell lifecycle communications.

Due to a timing issue, an initial service announcement in the SRM was sent before any listeners could register for those announcements. Thus, upon startup, a delivery error would be logged. With this patch, sending of the initial message is delayed until after the registration of listeners, and the irrelevant error messages are avoided.

Changelog 2.15.4..2.15.5

ca0b125: [maven-release-plugin] prepare release 2.15.5
397284e: nfs: fix NPE when remove sent to billing
877f954: nfs: fail quickly if we know that file is offline or lost
bd6538f: srm: Fix regression in SRM startup
9417720: chimera : fix trigger that populates data in t_locationinfo and t_inodes on insert or update of t_level_4
0da59ab: common: fix ChecksumType.toString()
835dc43: http: avoid dns reverse lookup on HttpProtocolInfo#toString()
73de63d: chimera: do not update ctime on atime only attribute update
e416dc0: [maven-release-plugin] prepare for next development iteration
87162fd: srm: Delay announcing credential service after cell start
e2642dc: pool: Improve scalability of nearline storage subsystem

Release 2.15.4

doors

Fixes a bug in which a host name set in *.net.listen properties was not preserved when publishing a door or generating SURLs.

Changes affecting multiple services

When building rpm files a package which is now explicitly required as a dependency.

cells

This change fixes a bug in routing manager that would leave orphaned topic routes in dCache domain.

Changelog 2.15.3..2.15.4

5592581: [maven-release-plugin] prepare release 2.15.4
0aa6954: pool: Fix regression breaking hopping mananger
6470425: fix regression from 859431218e3d7cdddcf906144c1b8928bb625fed
11f8a33: cells: Fix route removal in routing manager
4e8a009: rpm: explicitly require which package
b912cba: doors: Preserve name when publishing the address of doors
63824c7: [maven-release-plugin] prepare for next development iteration

Release 2.15.3

cells

LoginManager would occasionally generate error messages similar to “Discarding listening on $LOCATION 53684’ because its age of 18721640 ms exceeds its time to live of 4500 ms.”. This was due to erroneous reuse of old message envelopes in the internal messaging. This change fixes that problem.

This change addresses a potential problem in which messages sent between cells in the same domain could appear older than they are and thus would risk being discarded due to the time-to-live being expired.

Contains corrections to cells logging. The routing manager pinboard now shows information previously logged to various other cells.

srm

A Tier–1 site reported problems with a major WLCG VO’s read requests. Investigating the source of the problems showed that the srm_ifce library, used by the (outdated) GFAL v1 and the (supported) GFAL v2 SRM libraries, drastically limits the permitted lifetime of requests without providing admins any way to configure this.

For sites seeing errors related to desiredTotalRequestTime being exceeded, this change provides the new configuration option srm.request.maximum-client-assumed-bandwidth in srm.properties as a work-around.

Sites not observing such errors do not need to change anything with regard to this value.

Changelog 2.15.2..2.15.3

a2362e7: [maven-release-plugin] prepare release 2.15.3
8e0537a: srm: add short request lifetime work-around
e2e0441: cells: Set correct logging context in cell callbacks
f0dc019: cells: Improve robustness of message time to live
ec33e37: cells: Fix erroneous reuse of message envelope in location manager registration
277a9c3: [maven-release-plugin] prepare for next development iteration

Release 2.15.2

admin

A fix of a bug which caused the set breakeven command not to accept zero.

chimera

Fix a regression causing directories to inherit ACLs as if they were files rather than directories. Soon we will provide a procedure to clean-up already existing wrongly inherited directories.

ftp

Several problems with restrictions are fixed.

pnfsmanager

Directory items with the restriction preventing READ_METADATA activity were not omitted in a directory listing. This is now fixed.

srm

Previous version of dCache could not reload information about previous or ongoing SRM transactions; this prevented the ls command from functioning and prevented SRM from starting if there were ongoing activity when dCache was shut down. This is now fixed.

Changelog 2.15.1..2.15.2

ed2fd16: [maven-release-plugin] prepare release 2.15.2
4e372a5: chimera: Fix regression in inheriting ACLs on directory creation
efe2293: pools: fix bug in set breakeven command
ce09f29: authorisation: reintroduce ReadOnly class
75a735a: ftp: fix several problems with restrictions
9737ecc: PnfsManager: fix restriction to prevent directory items
c82a1ba: srm-client: Fix type mismatch in third party copy client
1f08d34: srm-client: Fix type error when aborting copy requests
871d15a: [maven-release-plugin] prepare for next development iteration

Release 2.15.1

ftp

On very short transfers, two internal messages could occasionally arrive in the wrong order. This would cause clients to see the “226 Transfer complete.” message without the “150 Opening BINARY data connection for” immediate reply. This rare issue is now fixed.

pnfsmanager

Billing entries for SRM uploads recently lost the storage class part of the entry. This update fixes that issue. We observed an error caused by the parallel execution of two uploads, both trying to create the same (previously non-existing) directory). This modification fixes the underlying race condition, allowing such transfers to succeed.

pool

Fixed a regression introduced in 2.13 in which internal options like -c:puts were erroneously included to the call out to the HSM script.

poolmanager

This modification fixes a race condition in pool manager that could theoretically have provided erroneous data to pin manager, space manager, srm, xrootd and webdav. The pool selection unit allows multiple links with different priorities to be defined. By default, fallback to a lower priority link is only allowed when pools with higher priority links are inaccessible. For read an option exists to allow fallback on high load too, but until now, there was no option to fallback on write when target pools have no free space. Pool manager wass partitions now support the -fallback-onspace option to enable fallback to lower priority links when all available pools in higher priority links are full.

srm

When writing to a path /a/b/c, if b exists and is a file, SRM currently returns SRM_INTERNAL_ERROR. This modification changes that behaviour so that the more appropriate status SRM_INVALID_PATH is returned. During SRM-based operations, some sites reported problems with the delegation of user credentials. This modification remedies those problems, while at the same time reducing the CPU usage of establishing 3rd-party SRM connections.

Changelog 2.15.0..2.15.1

6bb347b: [maven-release-plugin] prepare release 2.15.1
931ace3: poolmanager: Fixed typo
28c1ddf: pnfsmanager: Fix regression in SRM billing entries
c8ba614: poolmanager: Allow fallback on write when pools are full
dd01e17: pool: Fix filtering of options in script HSM driver
c0f3612: srm: Disable delegation on srmCopy to or from other SRMs
1687e21: srm: Return SRM_INVALID_PATH when target directory is a file
1b8e541: pnfsmanager: Fix race leading to transaction failures in Chimera
c396f2d: [maven-release-plugin] prepare for next development iteration
fa8fc98: ftp: Fix race on short transfers
c0af0f1: poolmanager: Acquire read lock when serializing pool selection unit

Release 2.15.0

Glob patterns gain alternation lists

Adds support for {} alternation lists in many places were globs are supported. I.e. 1{foo,bar}2 matches both 1foo2 and 1bar2. Alternation lists can be nested. Among others these are used for matching pool names in pool manager, admin shell and migration module, patterns for file name matching in directory listings, and in gPlazma.

Cell communication gets fail fast path on overload

dCache services communicate through message passing. For request-reply style interactions messages have a time to live field. If messages are delivered late due to overload, messages are discarded to avoid inducing even more load on the receiving service. When this happens the requesting service will wait for the reply until the request times out and thus an overloaded system will give the appearance to hang.

This release adds a fail fast path in the receiving service in which it immediately generates a timeout reply if the expected queuing time exceeds the time to live of the message. Thus with this release services may generate timeout responses quickly when another service approaches an overload situation.

Cell message thread pool configuration is harmonized

dCache services communicate through message passing. Messages are delivered to a service from a pool of threads. For some services, the number of threads and the maximum message queue length are configurable. Keeping with dCache tradition, the configuration properties were different for different services. Trying to establish a new tradition, these properties have now been harmonized.

The following are the relevant configuration properties (copied verbatim from the default property files).

#  ---- Maximum number of concurrent requests to process.
#
#  The number of login requests that gPlazma will process
#  concurrently.  Setting this number too high may result in large
#  spikes of CPU activity and the potential to run out of memory.
#  Setting the number too lower results in potentially slow login
#  activity.
#
(deprecated)gplazma.cell.limits.threads=30
gplazma.cell.max-message-threads=${gplazma.cell.limits.threads}

#  ---- Maximum number of requests to queue.
#
#  The number of login requests that gPlazma will queue before
#  rejecting requests. Unlimited if left empty.
#
gplazma.cell.max-messages-queued=


#
# NFS door message processing thread pool configuration
#
(deprecated)nfs.cell.limits.message.threads.max=8
(deprecated)nfs.cell.limits.message.queue.max=1000
nfs.cell.max-message-threads=${nfs.cell.limits.message.threads.max}
nfs.cell.max-messages-queued=${nfs.cell.limits.message.queue.max}

#  ---- Cell message processing parameters
#
#  Settings for the request processing thread pool.
#
#  The thread pool will stay at the minimum number of threads until the
#  maximum request queue length has been reached. At that point the number
#  of threads is increased up to the maximum, after which further requests
#  will be dropped. Idle threads are terminated after the idle time until
#  the number of threads drops back to the minimum.
#
#  Except for database operations, the pin manager operates in an
#  asynchronous fashion. The minimum number of threads should be chosen
#  such that the database can be saturated under load. If network latency
#  between the pin manager and the database is high, then the minimum
#  number of threads could be increased to hide this latency. The default
#  setting is likely fine for most cases.
#
#  The maximum number of threads should be below the database connection
#  limit - otherwise threads end up blocking for idle connections and may
#  potentially deadlock in case the same thread needs more than one
#  connection (e.g. for nested transactions).
(obsolete)pinmanager.cell.threads.min=
(obsolete)pinmanager.cell.threads.max-idle-time=
(obsolete)pinmanager.cell.threads.max-idle-time.unit=

(deprecated)pinmanager.cell.threads.max=45
(deprecated)pinmanager.cell.queue.max=10000

pinmanager.cell.max-message-threads=${pinmanager.cell.threads.max}
pinmanager.cell.max-messages-queued=${pinmanager.cell.queue.max}

# Cell message processing limits
(obsolete)pool.cell.limits.message.threads.min=
(obsolete)pool.cell.limits.message.threads.max-idle-time=
(obsolete)pool.cell.limits.message.threads.max-idle-time.unit=

(deprecated)pool.cell.limits.message.threads.max=50
(deprecated)pool.cell.limits.message.queue.max=1000

pool.cell.max-message-threads=${pool.cell.limits.message.threads.max}
pool.cell.max-messages-queued=${pool.cell.limits.message.queue.max}

# Cell message processing limits
(deprecated)srm.cell.limits.message.threads.max = 10
(deprecated)srm.cell.limits.message.queue.max = 100

srm.cell.max-message-threads = ${srm.cell.limits.message.threads.max}
srm.cell.max-messages-queued = ${srm.cell.limits.message.queue.max}

Several obsolete admin shell commands are dropped

The following deprecated admin shell commands have been removed or marked obsolete: load cellprinter, load interpreter, set classloader, and show classloader. If you used these (you didn’t) then stop doing so.

Chimera gets new compact primary key

The Chimera database schema has been heavily updated. The schema has traditionally used the variable length and relatively long PNFS ID as a primary and foreign key. In this release a 64 bit inumber is introduced to replace the PNFS ID as a key in the schema. The PNFS ID is still an essential concept to dCache as it is used as a persistent identifier for files on pools, however in Chimera it is just another file attribute.

Upon upgrade all Chimera tables will be rewritten. One has to expect that for large instances the schema update will take many hours. We recommend testing the schema migration on a clone of the database prior to deploying dCache 2.15. The schema can be rolled back to the previous structure using the dcache database rollbackToDate command, but this must be done before downgrading dCache.

Custom Chimera queries and views will likely have to be updated. If custom views or triggers have been installed, we recommend extra attention to testing the migration prior to upgrade to ensure that the custom modifications do not cause the upgrade to fail.

The new schema uses less disk space which means more data can be cached in available memory on the RDBMS server. Depending on hardware, this may provide a significant performance improvement. On the other hand, hardware with plenty of RAM and SSDs may only observe a minor improvement.

Chimera can store several checksums for a file

Chimera can now store several checksums of different types for a file.

Chimera performs consistency check upon upgrade

Previous dCache releases have fixed several bugs in Chimera. These bugs may however have introduced inconsistencies such as wrong link counts or unlinked files.

Upon upgrade, the Chimera schema migration will fix these inconsistencies. It will create a lost+found directory in the root of the name space and link unreachable files and directories from here. After upgrade the content should be inspected and either deleted or moved out of the lost+found directory. Running this consistency check takes significant time and is one of the reasons why the Chimera schema upgrade in this release is time consuming. The check cannot be triggered manually. The inconsistencies observed were caused by bugs that should be fixed now. If they are not, that’s a bug.

Admin shell commands gain better documentation

Many admin shell commands have been updated with additional documentation. Most commands should be functionally unchanged. Affected services are cleaner, dir, pool, pnfsmanager, and poolmanager.

gPlazma gains new feature to restrict file access

gPlazma plugins now have access to a new powerful abstraction called a restriction. A restriction is invoked upon any name space operation and the restriction can block the access. dCache itself currently uses restrictions to implement read-only accounts, but third party session plugins can generate custom restrictions as part of a client login.

Developers should consult the JavaDoc of the Restriction class and contact the dCache team.

DCAP can accept ACLs on file or directory creation

At dCache.org we love standards, but DCAP is still around and utilized by many users. Some of those users like to use ACLs too, and this release adds support for specifying ACLs when creating files or directories. Programmatically this can be done like so:

dc_setExtraOpotion("-acl=\"<ace1> ... <aceN>\"");

From the command line you would do something like:

dccp -X-acl="\"A::3750:rwatTnNcCoy A::EVERYONE@:rtncy\"" .....

The first one to decipher that ACE without consulting the documentation gets a free license!

FTP AUTH return codes are now RFC 2228 compliant

Return codes for error results of the AUTH command have been changed to be more in compliance with RFC 2228. In the unlikely event that your client relied on dCache not being compliant with RFC 2228, your client is now broken.

gPlazma produces better diagnostic information

Proxy certificates include an embedded attribute certificate when they are a voms proxy. This attribute certificate contains group-membership information and is signed by the VOMS server’s credential.

An .lsc file associates a trusted VOMS server with some already trusted certificate authority. This requires that the CA that signed the VOMS server’s certificate matches the DN in the .lsc file, otherwise the VOMS server is not trusted and no FQANs are extracted. Therefore, when diagnosing a problem where the voms plugin fails to extract any FQANs it is sometimes useful to know which CA signed the VOMS server’s certificate.

This release extends the output logged upon failure to login and by the gPlazma explain login command to show the attribute certificate extensions. One VOMS-specific extension includes the certificate chain of the VOMS server. With this release, the various issuers of this chain are shown.

Additionally, the output of the DN of the VOMS certificate is updated to RFC 2253 format. Yay standards!

gPlazma authdb can be sufficient

gPlazma uses a PAM style structure with plugins being marked optional, sufficient, required or prerequisite. sufficient indicates that if a plugin succeeds then that is sufficient (a very descriptive name, it seems) to complete the current phase. For this to work it requires that a plugin can fail too - otherwise it would always be sufficient.

authzdb was a plugin that would not fail in the session phase if no session information was found - now it does.

Pool `rep ls` command gains filter on storage info

The rep ls admin shell command lists the files on a pool. It now accepts the -storage option to filter by storage class and thus list only a subset of the files. The option even accepts a glob pattern.

Pool startup has been improved

The pool startup logic has been heavily refactored in this release. A pool will now immediately switch to read-only mode when it starts. Like before, the pool will remain read-only until a full inventory has been build and any inconsistencies have been addressed.

Lock contention during pool initialization has been reduced. In particular this avoids a period of unresponsiveness just prior to switching to read-write mode.

NFS becomes more robust against mover timeouts

If the NFS door times out while waiting for a write mover to be activated, it would retry the creation and erroneously create a read mover instead. It no longer does this.

ANONYMOUS and AUTHENTICATED ACEs gain consistent semantics

ANONYMOUS and AUTHENTICATED ACEs in name space ACLs in previous releases was inconsistent and erroneous and in some situations both ACEs were applied simultaneously despite RFC 3530 stating that they are mutually exclusive. The interpretation is now no longer erroneous nor inconsistent.

SRM `info` command provides cache statistics

The SRM maintains several internal caches. Statistics about the awesome effectiveness of these caches are now included in the output of the info admin shell command:

--- storage (dCache plugin for SRM) ---
Custom reverse DNS lookup cache: CacheStats{hitCount=0, missCount=0, loadSuccessCount=0, loadExceptionCount=0, totalLoadTime=0, evictionCount=0}
Space token by owner cache: CacheStats{hitCount=0, missCount=0, loadSuccessCount=0, loadExceptionCount=0, totalLoadTime=0, evictionCount=0}
Space by token cache: CacheStats{hitCount=0, missCount=0, loadSuccessCount=0, loadExceptionCount=0, totalLoadTime=0, evictionCount=0}

Admin shell gets configurable history length

The admin shell can store a persistent command history. This release increases the default and adds a configuration property for how many commands to store. Gone are the days of forgetting that brilliant one-liner.

#  ---- Admin door history size
#
#   The size of the history in lines and thereby the history can be
#   limited by setting admin.history.size to an integer value. The
#   standard size is 500.
#
admin.history.size = 500

gPlazma produces better log messages for VOMS failures

FQANs in a VOMS proxy are only trusted if signed by a VOMS server for which an .lsc file is present. Other FQANs are silently dropped (you don’t want to reject a user just because he signed up with an unknown VOMS server). In previous releases it was really hard to distinguish the situation in which a user didn’t have any VOMS extensions in the certificate or if they had all been rejected due to validation failures. It was also really hard to know why validation failed. This is now really easy as gPlazma logs this information to the log file.

WebDAV logs `X-Forwarded-For` header to reveal clients behind proxies

The WebDAV door produces an access log detailing who has been using it and how. Among the information logged is the client IP address, however if a user cowardly hides behind a proxy only the proxy’s IP address was logged. Now the WebDAV door logs the client IP address as reported by the proxy, too. Obviously, this information is only as trustworthy as the proxy.

Enforce source address in proxy certificates

An X.509 proxy certificate is a certificate signed by an end user certificate or another proxy certificate. Whoever got the proxy certificate can act upon the original end user’s behalf. The proxy can however be limited in various ways. One of these is by adding a policy element that restricts from which IP addresses the certificate can be used. Whoever got the proxy can still do bad things, but at least we limit them to doing it from a particular node. In the past dCache ignored this optional policy - now it no longer does.

gPlazma xacml plugin uses updated XACML client library

The xacml gPlazma plugin integrates dCache with GUMS serves. The plugin relies on a third party XACML client library. In this release we upgraded that library to a new version with awesome new features. Among other things it no longer relies on JGlobus, which means dCache 2.15 is the first release in decades without JGlobus (yay!). It also understands HTTP keep alive now, so we don’t have to reconnect to the GUMS server all the time.

To not just make this about hidden internal things, we obsoleted two configuration properties and introduced a new one:

#  ---- Path to the vomsdir directory
gplazma.xacml.vomsdir=${dcache.authn.vomsdir}
(obsolete)gplazma.xacml.vomsdir.dir = Use gplazma.xacml.vomsdir
(obsolete)gplazma.xacml.vomsdir.ca = Use gplazma.xacml.ca

Changelog 2.14.0..2.15.0

6df3b7e: [maven-release-plugin] prepare release 2.15.0
8042240: nfs: notify billing on file removes
7863afc: Fix compatibility issues with ctlcluster
6a8a59f: poolmanager: Acquire read lock when serializing cost module and partition manager
7cfe9ea: poolmanager: Fix race condition in pool selection unit
d181c22: webdav: fix 404 error if attempting to delete a nonexistent file
a18b99a: poolmanager: Make pm set accept all options
81c3be3: pool: Fix obsoletion of pp set pnfs timeout command
63630d4: pool: Replace -si option with -storage in rep ls
0eda17d: gplazma: Mark gplazma.cell.limits.threads as deprecated
7b5ff39: spacemanager: Fix pool monitor fetch race during startup
60945bf: doors: Fix contention point and race in login broker publishing
c324557: Revert “webapi: Rest embedded server using Jersey 2.”
24d4999: [maven-release-plugin] prepare branch 2.15
4b2cd8c: ftp: fix directory listing on ‘/’
ea71188: pool: Avoid lock contention on DNS lookup in p2p component
48326f8: srm: Check for broken files during srmPutDone
12a1c8b: pool: send connection header when closing a connection.
7284a34: Revert “pool: send connection header when closing a connection.”
1c85062: pnfsmanager: Check file size and upload completion when committing temporary upload paths
6d8b93d: PnfsManager: fix restriction to support clients trying to create /
d8b899d: pool: use “same thread strategy” for nfs movers
63917eb: dcache: introduce the Restriction login attribute
ee46d36: pool: add glob filtering of storage-info in rep ls command
06b48bc: Revert “Add precondition checks to FsPath to disallow relative paths”
eae298a: pool: move RepositoryChannel creation into ReplicaDescriptor
7091dbd: libs: update dependency to 4.1.0
948be6f: pool: Move repository indexing to loading phase
3c9dd15: pool: Fix logging regression during initialization
b2d0803: pool: Refactor MetaDataCache
619bdca: pool: Add options argument to MetaDataStore::index to allow non-modifying list
c6d4001: cells: Fix envelope encoding on no-route-to-cell error
268a345: pool: Initialize meta data store in cli tools
501c334: xrootd: Add kXR_dstat support
e5f3b9a: chimera: Alter statistics target for t_tags(itagid)
15b2a10: pnfsmanager: Protect against erroneous upload paths
8d2056e: srm: Add safe-guard against invalid file ID in put requests
1002b5d: Add precondition checks to FsPath to disallow relative paths
a5862cf: pool: Don’t consider failure to find migration job a bug
ddba5d7: libs: Update third party dependencies
afc5486: gplazma2-xacml: Remove duplicate dependency
0bd49ee: nfs: try to re-use transfer class on client retry
f937305: Revert “resilience v2 (1/45): Add support for resilience attributes to pool selection unit and related classes”
2f555f1: chimera: add fileid size when converting FsInore into nfs file handle
d49fe68: cells: Don’t use CellExceptionMessage
9f807cf: pool: send connection header when closing a connection.
31ac9e5: ftp: fix return codes for the AUTH command
87a7c9d: LocationManager: move thread starting outside of constructor
5e7fd36: statistics: encode ‘/’ in filenames
0aebf12: pom: Make dependencies explicit
d1aa11d: resilience v2 (1/45): Add support for resilience attributes to pool selection unit and related classes
72ddf8f: chimera: Fix type in inumber2path stored procedure
3ba337f: xacml: Update to XACML client library version 3 with CANL support
53656ea: Introduce brace-lists for globs
c731679: chimera: Fix regression in stored procedure
51aeb29: chimera: fix broken pgsql function
0879960: chimera: fix postgres function
1ffd7c3: pool: handle duplicated start mover requests
be0b14a: pool: Fix NPE in pool yaml tool
d7a890c: chimera: Add inumber identity field to t_inodes
ffc1e2c: chimera: Fix Chimera inconsistencies
8594312: dcache: Make header in showUpdateSQL output an SQL comment
3548987: webdav: Decorate the PathMapper in order to provide the desired WebDAV door mapping for specific(OwnCloud Sync client) User-agents.
05e8f46: alarms: change executor to have bounded queue and discard events on overrun
3eab402: nfs: use noitify instead of blocking sendAndWait when sending pin/unpin messages via touch “.(get)(<file_name>)(pin)” command
7b69b49: pool: Refactor ConsistentStore
96245f9: pool: Refactor locking of the repository
32a6467: pool: Minor cleanup of repository class
9c80f9d: pool: Remove dependency from repository handles to repository
7940551: pool: Refactor repository fault handling
b8dcc2a: pool: Refactor how files are destroyed
d87a664: pool: Refactor CacheRepositoryV5#destroyWhenRemovedAndUnused
a36d800: info-provider: publish door root path
67e0487: gplazma: update output of VOMS error reporting
f538201: pool: fix HTTPS third-party transfers without X.509 credential
4197b19: pool: Refactor CacheRepositoryV5#setState
3225a8a: pool: Fix synchronization regression in pp commands
7cf5738: pool: Refactor pool initialization to reduce exclusive locking
5a34e31: ssh2 admin: Add history size
30c924d: webdav: mapping of sudden owncloud-related paths to specific hardcoded json dictionaries
715f69b: Revert “spacemanager, srm: Do not allow update to full spaces”
f8c8ccc: webdav: add support for 3rd-party HTTP pull
7c867bb: webdav: factor out the mapping between request path and dCache path
3fb5067: see comments to https://rb.dcache.org/r/8940
f6f1524: spacemanager, srm: Do not allow update to full spaces
cdf640c: spacemanager: Randomize backoff in case of transient errors
9744486: webdav: fix root directory permission check and logging
3ee4192: webdav: minor refactor
a63977c: nfs-proxy: include file’s pnfsid into debug context
1803aa2: gplazma-grid: authdb session step must fail in no mapping found
44cc9bd: nfs: convert FILE_IN_CACHE into NFS_EIO
39c65f8: webdav: log client-chain based on X-Forwarded-For header
4106cb3: replica manager (old): fix countable logic when rescanning pool repository
e44543c: pool: reduce lock contention when killing a mover
337082f: pool: Fix lock contention during heavy p2p activity
42fa54b: pool: Fix buffer leak in HTTP mover
401befd: dcache-webdav: fixed bug in TLS handshake for webdav with https introduced in the commit a8edc2642417aee14ae7944956d5fdedc2cfee1b
ac00962: xrootd: Fix classification of uploads
a046ff5: pool: Expose Berkeley DB configuration as dCache properties
1baaf2f: xrootd: Roll back asynchronous reply on open
1a1f3a5: gplamza: update LoginResultPrinter to show more AC information
1c4b9ea: voms: add diagnostic information if validation fails
9cc9afe: pool: Fix lock starvation in migration module
13225ec: pool: Fix accounting error in repository statistics
8abec4c: utils: refactor NetworkUtils.LocalAddressSupplier
0d9b3b1: pool: Avoid lock contention when opening files and setting sticky flags
0f18c8b: pool: Simplify synchronization during repository setup
a8edc26: dcache-webdav: Moved handling of authentication from Milton Security Filter to it separate handler
5c35fcd: pool: Upgrade xrootd4j to 3.0.2
1d45452: Split Glob into two classes
31cfa3f: pnfsmanager: Account for more chimera calls
5b557ec: chimera: Prepulate stat cache on inodeOf call
9a7f9f8: pool: Redirect xrootd client to door on failure to open file
1d00d93: pool: Do not fail read if xrootd client doesn’t close file
d5d6175: cleaner: Fix black listing in case of disabled pools
a4a1b4e: cells: Don’t log AsynchronousCloseException when tunnel closes
08dbe65: pool: Show progress during repository initialization
62fe5cc: pool: Drop sorting of files on initialization
e78e4e3: pool: Fix health check of file store
6470bc5: pool: Use disk ordered cursor for listing meta data on pool startup
1f9349e: pool: Log information on runtime for listing the repository
86fdfcf: common-security,ftp-client: Allow compilation with Java 7
63f088b: common: Move Checksums to fix compilation with Java 7
4eed1d5: common: Move NetLoggerBuilder out of common
b77d532: srm: Lock job while saving to create consistent persistent state
cadec14: srm: Fix saving of transient states to database
b6ede26: srm: Fix legacy close (again)
1425a74: Upgrade dependencies
56cae8c: http–3rd-party: ensure IOException logged with toString
e548d1f: x509: fix regression that prevented authentication with EEC
79b74b2: webapi: Rest embedded server using Jersey 2.
40c1c2b: README: add howo-to-contribute section
bc96edd: srm-client: Fix GSI delegation for old servers
3a841d5: srm: Do not enable TCP NO_DELAY on SRM connections
40a56e9: srm: Fix race in state reporting
4c04d49: srm: Add some useful cache statistics to info output
61eaf48: gplazma: Improve logging in voms plugin
81dbddd: info: fix test to be less critical on timing
a3c02d0: pool: Do not output expired sticky flags
8bcb522: pool: Only save sticky bits if not already set
3d67fa6: dcache-webdav: fixed missing log statement
f8416b5: poolmanager: Fix missing access latency and retention policy on pool to pool copy
2d9ea11: admin: Fix endless loop in non-interactive mode
2bbcef4: pool: Make bulk sticky bit operation robust against repository changes
889ea42: pool: Throw IllegalArgumentException on rep set sticky errors
712a1fb: spacemanager: Fix listing by pnfs id
482028c: Add caching layer for TLS validation
857aea9: pool: Extend migration module with -meta-only option
cac98f2: correct workdir
a102764: change Dockerfile dependency from master to latest
ca9e77c: docker: Add Dockerfile
c69bdc9: pool: Add option to migration module to filter by cache class
67c2876: pool: Add bulk mode for rep set sticky command
f168fd8: pool: Add migration option to filter by absense of sticky flags
08d2569: pool: Fix persistence of sticky bits
050f8fb: srm: Allow transition from InProgress to Queued
3b70e4e: dcache-webdav: moved logging functionality from Milton filter to its own handler
1fe0c4a: srm: Map obsolete states in request history tables
70396e4: cells: Tab complete on distant downstream domains too
5d1a043: cells: Fix route propagation trigger on non-default topologies
7ec11fd: gplazma: Make GlobusPrincipal compatibly with JGlobus
aecc399: ftp,dcap: Fix LoginBrokerRequestTopic subscription
11f6b0e: chimera: fix regressin introduced by fa9a749c4
668cde7: transfers: fail request if pool/space manager reject request
2f11d01: chimera: Prevent filling of stat cache of root inode
a6f9302: chimera: When updating stat cache, take the file type into account
fa9a749: chimera: Read AL and RP on directory listing
f014a49: poolmanager: little bit of java8
4f81769: Motivation: enstore based installations need a special trigger to keep t_inodes table in sync.
f9fbe60: x509: add source/client IP checking
dc0934f: doors: send Origin when logging in user
b51dab9: namespace: implementing annotated command syntax for admin commands (PnfsManagerV3) - 3
a2d66c1: dcap: accept ACLs on create/mkdir
1ac17d8: mover: implementing annotated command syntax for admin commands (NfsTransferService) - 1
dc87cbf: security: drop AuthType from Origin
84ab36f: nfs: ingore file_not_found on close
2a5ca2d: chimera: Make FsSqlDriver#addInodeLocation PostgreSQL compatible
4f22278: chimera: Do not ignore DuplicateKeyExceptions
c2a95c4: Revert “cells: Don’t use CellExceptionMessage”
19f87b7: chimera: Fix deadlock
bf22ce9: common: possible performance improvements over inefficient Map Iterators
efee30f: xrootd: Update alice token plugin to fix IPv6 compatibility
8282eca: cells: Don’t use CellExceptionMessage
017447e: xrootd: Update to xrootd4j 3.0.1
b7bbbf3: xrootd: Fix logging of Netty exceptions
4261e59: ftp-client: fixed using default encoding
f2fbfa2: webdav: update to latest milton
13bedf0: srm: remove redundant null check
00557c5: PnfsManager: remove copy-n-paste error in error message
0f14006: srm: decouple VOMSValidator and SRMUserManager from Configuration
68c6b5a: pool manager: add pool name to alarm
aeadaa4: pool: Make context factory non-lazy for remote gsiftp transfers
f1a8237: system-test: Update ctlcluster command to configure CRL settings
5222a50: system-test: Fix grid-security settings
0afdf17: cells: Start LoginManager and LocationManagerConnector after creation
dfc5582: nfs: set EOF flag when channel indicated EOF
d5714c0: poolmanager: implementing annotated command syntax for admin commands (Rebalancer)
a380094: cleaner: documentation and implementation of command annotation (ChimeraCleaner–1)
c2a8e25: poolmanager: documentation and implementation of command annotation (PartitionManager)
e222667: dcache, acl: returning null when return type if Boolean
14fbb65: rpm: enforce SL5 compatibility when building RPM packages
b08a3dc: srm: move getPath method to Configuration
bc9c557: dcache, dcache-spacemanager: fixed passing null values to non-null parameters
e1fc107: cells: Add fail fast behaviour for busy pools
46ff75b: cells: Minor refactoring to avoid casting
078e22d: Remove dead code from AbstractCell
60ddcf9: Move cell startup out of cell constructor
9cc0150: cells: Use life cycle notifications for tunnel creation
5e2f86c: Make use of new cell life cycle calls
290efe9: cells: Introduce startup callbacks
bcc51dd: cells: Delay starting nucleus until cell start
60de044: pool: Remove non-existing command from default pool setup
f750e65: cells: Provide better error message when command is not found
f93442b: cells: Make cells message executor configurable
9e066bb: cells: Drop BootstrapStore
49b6b20: cells: Drop obsolete ‘load cellprinter’ command
95a2161: cells: Refactor cell instantiation
0d16288: cells: Drop loadable interpreter
2c6424d: cells: Drop custom class loader
d8d335c: Clean up use of modifiers
9153671: srm-client, dcache: fixed passing incompatible arguments to functions
3782500: dcache, srm-common: The value of the conditional variable is always true at this point.
61eb10c: rpm: build SNAPSHOT RPM with filenames using only commit id
f4f3c11: dcache-nfs, dcache: removed unecessary use of non-short-circuit logic
c6147b0: alarms: modify log4j encoding to show actual alarm type
d2fe52f: chimera: Fix cut’n’paste bug in path2inodes stored procedure
a6376f5: pool: Allow migration task to skip location query
6e42dce: pool: Fix NPE when restoring file
0bef0f0: scripts: remove check for pkcs8 format private keys
3f77482: scripts: do not check for PKCS#8 formatted hostkey.pem on shutdown
f64d919: scripts: remove check for non-migrated dCache
24c1cda: dcache, dcache-ftp, dcache-webadmin, cells: Boxing/unboxing to parse a primitive
7f12619: pom: fix dirty version tag for deb and rpm
5f34c0b: chimera: remove deaqd code
52e4ddb: namespace: implementing annotated command syntax for admin commands (PnfsManagerV3) - 2
14eb461: namespace: implementing annotated command syntax for admin commands (PnfsManagerV3) - 4
2959f84: chimera: Creating boxed primitive value, just to extract un-boxed primitive value in ChimeraCleaner.java
49f654b: gplazma2: Add lifecycle methods to gplazma plugins
18ab73d: Fix shutdown of bounded executor
bec5378: chimera: Null value passed to non-null parameter in org.dcache.chimera.cli.Shell$WriteCommand.call()
70d080d: gplazma2: fixed findbugs warning for org.dcache.gplazma.loader.XmlResourcePluginRepositoryFactory
e027d46: pools: implementing annotated command syntax for admin commands
2426128: dirLookupPool: implementing annotated command syntax for admin interface commands (DirectoryLookUpPool)
27a524a: pool: clean-up and command conversion in PoolV4
49689dc: Update third party dependencies
cd6df73: chimera: Propagate failures to read or write in-db data
e7a899c: srm: Rename count column
614b64c: common-security: Do not log stack traces for CANL notifications
802a833: srm: Use correct logging context when saving jobs
2dc758f: common-security: Lower log messages for CANL notifications
702c1da: srm-client: Add initialization of Axis handler in SRM 1 client
675d416: srm: Fix NPE when saving requests with an unknown user
4f86664: srm: Fix race in user persistence
8520709: gplazma: fix erroneous logging in x509 plugin
c1d2fe1: pom: generate rpm friendly dirty versions number
7f42daa: gplazma: remove support for plugin caching
126c0a3: gplazma: make x509 generation of LoA principals optional
19578eb: Revert “gplazma: remove support for plugin caching”
2b33fa3: gplazma: remove support for plugin caching
0cdf9d9: pools: documentation and implementation of command annotation (2)
06c9f78: pool: jtm set timeout command fix
a07f1e4: pools: implementing annotated command syntax for admin commands (P2PClient)
26327fd: [maven-release-plugin] prepare for next development iteration

What’s new in dCache 2.15 The release notes

Executive summary

Incompatibilities

Release 2.15.40

admin

Changelog 2.15.39..2.15.40

Release 2.15.39

httpd

Changelog 2.15.38..2.15.39

Release 2.15.38

pool

Changelog 2.15.37..2.15.38

Release 2.15.37

webdav

Changelog 2.15.36..2.15.37

Release 2.15.36

webdav

Changelog 2.15.35..2.15.36

Release 2.15.35

Changes affecting multiple services

system-test

Changelog 2.15.34..2.15.35

Release 2.15.34

chimera

Changelog 2.15.33..2.15.34

Release 2.15.33

chimera

ftp

Changelog 2.15.32..2.15.33

Release 2.15.32

ftp

Changelog 2.15.31..2.15.32

Release 2.15.31

ftp

srm

Changelog 2.15.30..2.15.31

Release 2.15.30

xrootd

Changelog 2.15.29..2.15.30

Release 2.15.29

srm

systemtest

Changelog 2.15.28..2.15.29

Release 2.15.28

chimera-enstore

Changelog 2.15.27..2.15.28

Release 2.15.27

cleaner

pool

Changelog 2.15.26..2.15.27

Release 2.15.26

ftpclient

Changelog 2.15.25..2.15.26

Release 2.15.25

dcap

poolmanager

Changelog 2.15.24..2.15.25

Release 2.15.24

Changes affecting multiple services

billing

dcap

doors

pool

Changelog 2.15.23..2.15.24

Release 2.15.23

doors

Changelog 2.15.22..2.15.23

Release 2.15.22

billing

dcap

gplazma

httpd

scripts

Changelog 2.15.21..2.15.22

Release 2.15.21

commons

gplazma2

srm

Changelog 2.15.20..2.15.21

Release 2.15.20

What’s new in dCache 2.15
The release notes