dCache 5.1 Release Notes

Highlights

The 5.1 series of dCache focuses on new data transfer features.

It introduces 3rd-party copy capabilities to the xrootd protocol.

FTP doors can now support anonymous-FTP logins, and explicit TLS encrypted control channel (FTPS).

gPlazma has gained support for SciToken authentication as well as improved OpenID Connect support.

Pools can now send the full set of inotify events through SSE.

Pool manager can create dynamic pool groups based on defined tags.

Incompatibilities

The Kafka configuration was moved from dcache.properties to a dedicated kafka.properties file.

Acknowledgments

Once again, we are grateful for code and documentation contributions from contributors outside of our core team.

Onno Zweers provided many helpful corrections to the documentation.

Lea Morschel also provided documentation and code refinements.

Other contributions originated from a lecture at HTW Berlin; many thanks to Jonas Grabber, “tabea”, “Co” and “xpinkyx” for their contributions.

Release 5.1.26

dcache-xrootd

The version is upgrated to xrootd4j 3.5.7 with lossen username validation, so that

usernames like foo.1234:56 are no longer rejected.

The current release fixed java.lang.IllegalStateException: ChecksumChannel must not be written to after getChecksums.

TPC client first will be shut down when the pool netty channel goes inactive.

Changelog 5.1.25..5.1.26

2a79d33: [maven-release-plugin] prepare release 5.1.26
a107444: dcache-xrootd: bump to xrootd4j 3.4.7
abe0c02: dcache-xrootd: cancel TPC transfer when client disconnects unexpectedly from pool
1fd0d60: [maven-release-plugin] prepare for next development iteration

Release 5.1.25

pool

The unix xrootd tpc security plugin was included in order to enable the dCache TPC client to use a dCache pool as source when signed hash verification is on. However, this is now fixed and no special configuration necessary for organizations (like Tier 1) needing to communicate with EOS.

Changelog 5.1.24..5.1.25

6825773: [maven-release-plugin] prepare release 5.1.25
49cf902: pool (xrootd): make tpc security plugin default unix
131559a: [maven-release-plugin] prepare for next development iteration

Release 5.1.24

frontend

When frontend is run in a core domain and there is no history service reachable, the retry on no route to cell ends up spamming the message queues. This is now fixed.

xrootd

The current release fixed compatible level security for sigver.

The xrootd client has a command-line option, --path, which tells the server to create missing directories. This option is included in two-party copy, but setting it for TPC has no effect. TPCs which wish to write to dCache (as destination) to a non-existent subdirectory fail. This is now fixed and dCache no longer fails in these cases.

Changelog 5.1.23..5.1.24

e0eba6e: [maven-release-plugin] prepare release 5.1.24
bfd8792: dcache-xrootd: always create missing directories on write
1eb17c8: dcache-xrootd: fix compatible level security for sigver
7bb61a4: dcache-frontend: remove retry flag on sendAndWait to history service
11ef455: [maven-release-plugin] prepare for next development iteration

Release 5.1.23

dcache

QOS migration policy engine was raising JVM error when no tape pool found. This is now fixed and the normal behavior is that HTTP error code reported back with No HSM pool found.

frontend

The current release improved error responses and they are more specific now.

The current release fixed the NPE stack trace arised because the pool data could be sent with a default sweeper data object.

Changelog 5.1.22..5.1.23

c4b6e8c: [maven-release-plugin] prepare release 5.1.23
224fec2: dcache-frontend: make ErrorResponseProvider return the more specific error message
0f086a2: dcache: qos migration policy engine should not raise JVM error when no tape pool foundMotivation:
369645c: dcache-history,dcache-frontend: guard against unconfigured sweeper histogram
3d372db: [maven-release-plugin] prepare for next development iteration

Release 5.1.22

frontend

From RESTful admin API, the POST to pools/{name}/usage/mode -d {"rdonly":true} failed but {"rdonly": "true"} succeeded.

This is fixed now and both boolean and string work.

The current release fixed stack trace in logs.

Changelog 5.1.21..5.1.22

b3747a1: [maven-release-plugin] prepare release 5.1.22
3626313: dcache-frontend: allow pool enable/disable to use boolean JSON value
c7759c4: dcache-frontend,history: protect against missing highest bin in histogram data
c6d1545: [maven-release-plugin] prepare for next development iteration

Release 5.1.22

frontend

From RESTful admin API, the POST to pools/{name}/usage/mode -d {"rdonly":true} failed but {"rdonly": "true"} succeeded.

This is fixed now and both boolean and string work.

The current release fixed stack trace in logs.

Changelog 5.1.21..5.1.22

b3747a1: [maven-release-plugin] prepare release 5.1.22
3626313: dcache-frontend: allow pool enable/disable to use boolean JSON value
c7759c4: dcache-frontend,history: protect against missing highest bin in histogram data
c6d1545: [maven-release-plugin] prepare for next development iteration

Release 5.1.21

frontend

The current release added support for OIDC names and Client-IDs with spaces.

Changelog 5.1.20..5.1.21

e53b47a: [maven-release-plugin] prepare release 5.1.21
1b56e2b: dcache, frontend: release dcache-view version 1.5.7
737332b: [maven-release-plugin] prepare for next development iteration

Release 5.1.20

frontend

The current release removed unnecessary login requirement on restores and transfers.

The current release fixed a bug in the frontend if the inotify events are used.

skel

The current release repaired erroneous batch directives before cell creation.

Now it is fixed and domain is not left in zombie state after a fatal error, but restarts, as it should.

srm

Now host IP is used for comparison when determining if SURL is local.

Changelog 5.1.19..5.1.20

462bd7c: [maven-release-plugin] prepare release 5.1.20
e049838: skel: repair erroneous batch directives before cell creation
eb3ec8e: dcache-frontend: remove unnecessary login requirement on restores and transfers
2483270: srm: use host IP for comparison when determining if SURL is local
ba8781b: frontend: events inotify fix deadlock
156099d: dcache,frontend: release dcache-view version 1.5.6
87278e8: [maven-release-plugin] prepare for next development iteration

Release 5.1.19

cell

Curator client was not able to restore the connection to ZK server after network partitioning. The is now fixed.

skel

The current relase fixed tape-reserved size calculation.

webdav

The current release fixed, where the WebDAV door failed to follow RFC 4918. This make some clients reject dCache WebDAV door as a valid WebDAV endpoint.

Changelog 5.1.18..5.1.19

44d7c94: [maven-release-plugin] prepare release 5.1.19
06467af: Fix tape-reserved size calculation
d940db9: webdav: include DAV header in OPTIONS requests.
34b2beb: cells: do not re-define zookeeper watcher
5f83ba7: [maven-release-plugin] prepare for next development iteration

Release 5.1.18

canl

The current release updated lib version to 2.5.1.

gplazma

The current release fixed URL-prefix SciToken parsing and error handling if JWT contains malformed SciToken scopes.

Changelog 5.1.17..5.1.18

ae9f749: [maven-release-plugin] prepare release 5.1.18
761528a: gplazma: scitoken add unit tests and fix SciTokenScope
0a1b9aa: canl: update to version 2.5.1
448265c: [maven-release-plugin] prepare for next development iteration

Release 5.1.17

config

A typo in the dcap config file was fixed correcting dcacp.enable.kafka to dcap.enable.kafka.

dcache

Pool compatibility with Xrootd–2 and Xrootd–4 versions is now allowed.

gplazma

The SciToken plugin will now reject any JWT where there is none of the expected scopes defined. This allows dCache to support both OpenID-Connect and SciTokens.

webdav

The current release fixed an issue of transfers through dCacheView when the webdav door is configured with empty webdav.allowed.client.origins value, which is the default value.

Changelog 5.1.16..5.1.17

f5e71b8: [maven-release-plugin] prepare release 5.1.17
d204071: dcache: add null check to pool info collector util
1a9be8f: config: fix typo in property name
f27d03d: gplazma: scitoken fix two issues with SciToken plugin
dc948d0: webdav: fix CORS when all clients are allowed to connect
99e2106: dcache: allow pool compatibility with Xrootd–2 and Xrootd–4 versions
0b06dfd: [maven-release-plugin] prepare for next development iteration

Release 5.1.16

srm

The current release fixed a problem resulting in high CPU use in SrmManager if clients are attempting to pin a file and PinManager is unavailable.

A regression fixed where SrmManager will reject all QUEUED jobs and INPROGRESS BringOnline requests on restart, if there are no SRM doors running when SrmManager starts.

Changelog 5.1.15..5.1.16

521fe2e: [maven-release-plugin] prepare release 5.1.16
73cc5c4: SrmManager: fix handling of saved requests on start-up
ca2bd3d: SrmManager: avoid spamming if PinManager is down
a761455: [maven-release-plugin] prepare for next development iteration

Release 5.1.15

doors

The current release fixed a bug where running the lb set tags admin command without any arguments triggers a NullPointerException.

pool

The current release improved error messages about jobs cancellation.

scripts

The dcache-storage-descriptor command no longer requires a URL argument.

Changelog 5.1.14..5.1.15

06f58c2: [maven-release-plugin] prepare release 5.1.15
78a2168: doors: fix “lb set tags” command with no arguments
8bbfcb0: pool: improve messages when migration job is cancelled.
77f4323: scripts: fix variable ordering in dcache-storage-descriptor
9944724: docs: TheBook add chapter on SRR
4432077: [maven-release-plugin] prepare for next development iteration

Release 5.1.14

gplazma

The SciToken gplazma plugin now supports the audience (aud) claim where the claim’s value is an array. This allows dCache to support SciTokens with multiple audience values.

pool

Pool health-check log messages now include the pool’s name.

webdav

On an unsuccessful HTTP-TPC pull request, dCache will delete the file. If this deletion did not work then an error was logged. This is fixed now and failures to delete the incomplete file from a failed HTTP-TPC pull request, where the incomplete file has been deleted by some other means are now logged at DEBUG level, rather than WARN level.

xrootd

The current release refited checksum handling after xrootd4j bug fix.

Changelog 5.1.13..5.1.14

09f44ba: [maven-release-plugin] prepare release 5.1.14
bbe0dc2: dcache-xrootd: refit checksum handling after xrootd4j bug fix
81af14e: webdav: avoid logging non-error as an error
f30b95b: pool: include pool name in health-check reports
a68785d: gplazma: scitoken add support for multiple audience claims
25a4d2d: [maven-release-plugin] prepare for next development iteration

Release 5.1.13

frontend

The current release fixed QoS pin semantics.

A bug is fixed in frontend that results in a NullPointerException for billing queries where no limit is specified.

Changelog 5.1.12..5.1.13

1eb7e32: [maven-release-plugin] prepare release 5.1.13
c83c585: frontend: fix NPE if limit is not specified
9ed3404: dcache-frontend: fix QoS pin semantics
a931d26: [maven-release-plugin] prepare for next development iteration

Release 5.1.12

Changes affecting multiple services

The Apache Commons Compress library used in dCache was updated to version 1.19.

A rare deadlock situation in the Chimera database was eliminated. In cases where, within the same directory, concurrent mkdir and rmdir events happened, transactions within the database could deadlock. This would be indicated by the message

ERROR: deadlock detected

in the logs.

pool

There were reports of extraordinarily high CPU usage on pool nodes with a large number of cached files. Through an optimization of the sweeper, CPU usage was reduced significantly.

xrootd

This release fixes a vulnerability in dCache’s XRootD protocol implementation. We recommend that all sites update their XRootD doors. Details will be made available through EGI Security and, in a week’s time, through an update to these release notes.

Changelog 5.1.11..5.1.12

fe857de571: [maven-release-plugin] prepare release 5.1.12
b9895148f9: dcache-xrootd: honor read paths when listing directories
5022e47257: resilience: don’t compare Integer objects by refference
f0d9e124dd: sweeper: use in-memory map instead of repository for histogram data
7de9af5be9: dcache-xrootd: replace constants for version number
00b2df8fbd: dcache-xrootd: update protocol version numbers
522a84d482: libs: update apache.commons:commons-compress to 1.19
99f1932666: chimera: fix ABBA db deadlock when mkdir and rmdir run concurrently
21dbec4a6b: [maven-release-plugin] prepare for next development iteration

Release 5.1.11

dcap

dcap door could not handle out-of-date errors. This is now fixed.

gplazma

The current release fixed thread leak by explicitly close NamingEnumeration

httpd

The current release fixed escape status field in HttpPoolMgrEngineV3.

Changelog 5.1.10..5.1.11

da80e6a: [maven-release-plugin] prepare release 5.1.11
5626e54: dcap: restart pool selection on OUT-OF-DATE error
508a34f: gplazma-ldap: avoid thread leak by explicitly close NamingEnumeration
93de29b: httpd: escape status field in HttpPoolMgrEngineV3
e8c663f: [maven-release-plugin] prepare for next development iteration

Release 5.1.10

srm

A new user community requires the srm tools to be able to handle very large file listings. During preliminary tests, OutOfMemory errors from the srmls tool were observed. This is now fixed and srm can now support operations on very large file lists without running out of memory.

webdav

The current release added allow header to list of response headers for OPTION method request.

Changelog 5.1.9..5.1.10

2b5ef20: [maven-release-plugin] prepare release 5.1.10
51bbef8: webdav: add allow header to OPTION method request
281da6d: srm: Remove JVM memory limits
cf14b7f: [maven-release-plugin] prepare for next development iteration

Release 5.1.9

common

The current release fixed formatting of error message in Checksum.

frontend

Admins may now configure frontend to specify in which country (or countries) data may be stored. This information is visible through dCacheView.

gplazma

The current release fixe remote reading of JSON with UTF–8 CharSet and dCache can now work with OPs that use utf-8 charset.

Changelog 5.1.8..5.1.9

8e8b2b9: [maven-release-plugin] prepare release 5.1.9
6948f87: frontend: make geographic placement configurable
b5e245d: common: fix formatting of error message in Checksum
1662c2c: scitoken: fix remote reading of JSON with UTF–8 CharSet
176f4eb: [maven-release-plugin] prepare for next development iteration

Release 5.1.8

nfs

NPE on “show transfers” command is now fixed.

webdav

The current release fixed CORS for WebDAV doors that do not allow anonymous access; in particular, to support dCacheView uploading and downloading files with such authentication-required WebDAV doors.

Changelog 5.1.7..5.1.8

29688cf: [maven-release-plugin] prepare release 5.1.8
2bfc913: nfs: fix NPE on “show transfers” command
c07272b: docs: Update UserGuide to use guide-specific navigation header
173b4ae: webdav: fix cross origin resources sharing issue
11c5ce0: [maven-release-plugin] prepare for next development iteration

Release 5.1.7

chimera

The shell infrastructure supports commands being given interactively, on the commandline (e.g., ‘chimera mkdir /path/to/dir’) and from stdin (e.g., ‘echo “mkdir /path/to/dir” | chimera’). chimera now supports the latter case and properly shows command output when invoked in that fashion.

frontend

This release updates dCache View to 1.5.5.

nfs

The NFS door now correctly handles situations where newly created read-only files could occasionally not be written into:

f = os.open('test.txt', os.O_WRONLY|os.O_CREAT, 0400)
os.write(f,"Hello pNFS!")
os.fsync(f)
os.close(f)

will now succeed.

webdav

A client may issue a PUT request that targets an existing collection resource; i.e., attempt to write a file as a path that is a directory. dCache, until now, responded with an incorrect status code of 500. This release changes the status code for this operation to 405 (Method not allowed), thus keeping closer to RFC 4918.

xrootd

This release improves compatibility with the xrdcp client in versions >4.9 by responding correctly to query strings requesting a specific checksum type.

Changelog 5.1.6..5.1.7

06521c8fb8: [maven-release-plugin] prepare release 5.1.7
c994d6a39f: dcache, frontend: release dcache-view version 1.5.5
7db20d5f7f: nfs: introduce workaround ‘permission deny’ on layout commit
65d8c86ac1: chimera: chimera shell should show output when commands come from stdin.
9bf1cbc775: webdav: return 405 status code for PUT requests targeting collections
fb4576d6f6: dcache-xrootd: add checksum cgi handling to door query
7850889dac: [maven-release-plugin] prepare for next development iteration

Release 5.1.6

many

The dcache pool ls command now provides correct output even if the pool is defined with a single-digit number of bytes.

Changelog 5.1.5..5.1.6

8f7d147636: [maven-release-plugin] prepare release 5.1.6
72aad72efd: scripts: avoid copy-n-paste error when calculating pool size
f2fb97af41: [maven-release-plugin] prepare for next development iteration

Release 5.1.5

frontend

This release updates the dCache View web GUI to version 1.5.4.

ftp

HAProxy can probe endpoints to discover if they are still alive.

The FTP door has an optimisation that detects such probes and does not create the FTP command interpreter, since the FTP client (the HA-Proxy instance) is calling on behalf of itself, and will not issue any FTP requests.

This release fixes a regression that would cause erroneous NullPointerExceptions when FTP doors were probed by HAProxy.

pool

The default value for the xrootd Third-Party Copying server response timeout, pool.mover.xrootd.tpc-server-response-timeout, was increased from 2 to 30 seconds to provide more robust behaviour in the face of high loads and network congestion.

transfermanager

Error messages like the WebDAV door’s

Failed to fetch information for progress marker: failed to query pool: (0) Job not found : Job-1

where the TransferManager is unable to discover the current status of the pool mover now include the pool’s name, which should make debugging easier.

Changelog 5.1.4..5.1.5

9efd766039: [maven-release-plugin] prepare release 5.1.5
60ea0aa2d3: dcache, frontend: release dcache-view version 1.5.4
8e420a6883: pools: make the xrootd tpc response timeout less aggressive
22148b8a3d: transfermanager: include pool name in error for ‘mover ls’ failures
0d0093540a: ftp: avoid NPE on HA-Proxy probes
8eb19681b6: core: fix pool selection in killAll command of TransferManager
ca26b686e7: [maven-release-plugin] prepare for next development iteration

Release 5.1.4

Changes affecting multiple services

This release includes an updated Jetty library, with the update adressing CVE–2019–10247.

dcap

The Kafka messaging implementation in the dcap service has been made more robust, fixing issue [#4831](https://github.com/dCache/dcache/issues/4831).

frontend

Periodic activity associated with the frontend door is now logged with the door’s cell name. Such messages will also appear in the door’s pinboard.

nfs

Periodic activity associated with the NFS door is now logged with the door’s cell name. Such messages will also appear in the door’s pinboard.

pool

Attempting to start a full checksum scan (with csm check *) while an existing scan is still running is no longer reported as a bug.

Pool start-up logging now includes the corresponding pool cell name.

An internal timing check was updated, which should result in more robust pool behaviour. There should be no user-visible impact.

webdav

Periodic activity associated with the WebDAV door is now logged with the door’s cell name. Such messages will also appear in the door’s pinboard.

xrootd

A new configuration property, pool.mover.xrootd.tpc-server-response-timeout, allows setting a timeout for xrootd 3rd party copy operations. This can also be changed through the new admin command xrootd set server response timeout.

Changelog 5.1.3..5.1.4

8bc74e3f13: [maven-release-plugin] prepare release 5.1.4
c9ed0d87e3: pool: avoid IllegalStateException in ‘csm check *’ command
8cdf34d790: dcap: fix premature close of kafka sender
d88ba9bef4: sweeper: compute now after the values have been fetched
8aaabb6316: libs: use jetty 9.4.18.v20190429
03ad080bed: dcache-xrootd: add ability to override default timeout for server response (TPC)
906ab5ac03: [maven-release-plugin] prepare for next development iteration
d8f0ee3954: frontend: include CDC in scheduled activity
6007c39a82: nfs: include CDC in scheduled activity
8ba362e928: webdav: include CDC in scheduled activity
a5337a705a: pool: ensure initialisation thread has correct CDC information
0e7fcabfd8: jetty: make CanlContextFactory subclass of jetty.ssl.SslContextFactory.Server
6e566d3d9f: pom: use jetty 9.4.17.v20190418

Release 5.1.2

alarms

To ease troubleshooting, the POOL_DEAD alarm message now includes the pool name.

pinmanager

A bug was fixed where PinManager’s bulk ls admin command yielded a NullPointerException if the optional argument was omitted.

A typo prevented the error message “Remote connection failure while unpinning…” from appearing completely and correctly in the logs. The error message string now contains the message string of the underlying Exception, hopefully providing helpful details for troubleshooting.

pool

A regression that prevented a replica’s last access time from being updated was fixed.

A regression that prevented a replica’s position in the LRU queue for garbage collection from being updated was fixed.

inotify’s IN_CLOSE_WRITE event was sent at a time when it could not be guaranteed that the file triggering the event could immediately be opened. This race condition was fixed, and consumers can start using a file immediately after receiving this event.

webdav

Users asserting the “admin” role would occasionally receive NullPointerExceptions when trying to transfer files through WebDAV. This release fixes that issue.

Changelog 5.1.1..5.1.2

d8b2275279: [maven-release-plugin] prepare release 5.1.2
6f23091cff: UnpinProcessor: fix assumed typo {)
438a96a1d9: webdav: allow transfers as user with role ‘admin’
f914ddcf08: pinmanager: avoid NPE if no argument given for ‘bulk ls’ command
71cb623a8d: alarms: add pool name to POOL_DEAD alarm
033bd83c8c: pool: fix reordering of removable replicas on access
84ed9d20c6: pool: fix storage of replica last access time
a98a1ce5ad: [maven-release-plugin] prepare for next development iteration
ad9852c91a: pool: inotify avoid race in IN_CLOSE_WRITE event

Release 5.1.1

Changes affecting multiple services

Stage request from unknown locations resulted in NPE in dcap and pinmanager services. this is now fixed and using dccp to stage a file should work even if the location is unknown. ‘–’

frontend

A client that disconnects and quickly reconnects could had triggered the following NPE, this is now fixed.

resilience

The current release fixed race condition on replica state and no inaccessible file errors occures for a newly written file.

Changelog 5.1.0..5.1.1

adba768: [maven-release-plugin] prepare release 5.1.1
7d7c4c5: dcap/pinmanager: stage request for unknown location results in NPE
ef3a977: dcap batch : fix handling of dcap.kafka.topic variable
fc88f10: dcache-resilience (stable branches): fix race condition on replica state
0e979ce: frontend: fix race on client reconnecting
40307e9: [maven-release-plugin] prepare for next development iteration

Release 5.1.0

Alarms

So that they can be given different priority levels, a new alarm, POOL_DEAD, was created to distinguish a pool which is completely unreachable, from POOL_DOWN or POOL_DISABLED.

Alarms was also modified in order to be able to support a more modern version of the logback library.

Billing

The database connection was wrapped so as to catch fatal exceptions and report them as alarms (this is already done for other databases such as chimera).

When using the Kafka messaging system, the “billing” topic was hardcoded until now. This release introduces a new configuration variable, dcache.kafka.topic=billing to configure this.

When a pool reports a replica was removed, it includes information on why the replica was deleted. By default, this information is logged at the end of the line in double-quotes, where previously an empty string was logged.

dCacheView

Support macaroons for sharing files and directories. There is now an easy way for a user to generate a pre-authorised URL by requesting a macaroon from dCache.

Gravatar is an externally run service that maps an email address to an icon. The goal is that many independent services may use the same icon for the same user. At sites’ request, this behaviour is now optional.

Frontend

A bug preventing the proper filtering of active transfers according to uid was fixed. A reminder as to how this works: (a) admins always see everything; (b) authenticated users always see their own transfers; (c) if frontend.authz.unlimited-operation-visibility is set to true, authenticated users can see all transfers. Non-authenticated users can only see anonymous transfers.

The inotify support now includes sending IN_OPEN, IN_ACCESS, IN_MODIFY, IN_CLOSE_WRITE, IN_CLOSE_NOWRITE events on file activity. Uploading or writing a file will send IN_OPEN, IN_MODIFY, IN_CLOSE_WRITE. Downloading or reading a file will send the IN_OPEN, IN_ACCESS, IN_CLOSE_NOWRITE events. Support in the pool for sending these events is configurable pool.inotify-generation.enable (enabled by default). Some fine-tuning is possible using pool.inotify-generation.backlog.initial, pool.inotify-generation.backlog.per-door and pool.inotify-generation.message-batch-size where these have the same meaning as the similarly named pnfsmanager. properties.

Sending IN_MODIFY and IN_ACCESS events is rate-limited by the pool, so that multiple IO operations of the same type generate at most one event every pool.inotify-generation.io-suppression pool.inotify-generation.io-suppression.unit (by default 10 SECONDS).

FTP

Add support for anonymous FTP. Anonymous FTP is when the client authenticates with a place-holder username (e.g., anonymous or ftp) and something (perhaps their email address) as the password. When enabled, dCache will grant limited (read-only) access to that user. Many aspects may be configured, including the username (anonymous by default) and whether the password must look like an email address (by default not enforced). A specific root directory may be configured for anonymous users (by default, dCache’s root), further limiting what data is exposed. The Globus transfer service uses anonymous access for “archival” access.

Add support for FTPS doors. This is explicit FTPS sometimes written as FTP(E)S. Implicit FTPS is not widely used and is not supported in dCache. This new FTP door is represented by a new possible value for ftp.authn.protocol. Both FTPS and gsiftp encrypt the control channel and neither support encrypted data transfers. FTPS doors are advertised (in info-provider, frontend, etc) with URLs that start ftps://. Currently, only username and password authentication is supported.

gplazma

The ban plugin can now ban users by their uid or gid.

The restrictions infrastructure now supports a mult-targeted restriction: MultiTargetedRestriction. This allows a plugin to allow users to have different permissions for different parts of the namespace.

OpenID-Connect

The oidc plugin will introspect JWT to learn issuer. The access token, issued by the OpenID-Connect server is opaque; however, a JWT is a common choice for this token. If the token is a JWT then the oidc plugin can discover which OIDC server issued the token. If multiple OIDC servers are configured, knowing which one issued the token avoids having to contact each OIDC server, resulting in a faster response.

The oidc plugin now accepts the gplazma.oidc.http.slow-threshold and gplazma.oidc.http.slow-threshold.unit configuration properties, with 2 SECONDS as default values. If looking up a user takes longer than this threshold then the plugin will log the slow query.

The oidc plugin will cache the OIDC discover document. This document contains slowly changing information, so it makes sense to cache the information. The cache lifetime is controlled by the configuration properties gplazma.oidc.discovery-cache and gplazma.oidc.discovery-cache.unit, with default values of 1 HOUR. Fetching an up-to-date version of the discovery document when the cache has expired is done asynchronously: login attempts will use the old data until the fresh document is received.

The oidc plugin will now query OIDC servers in parallel, if more than one OIDC server is configured.

The oidc plugin will try to keep the TCP network connections with OIDC services (used to query a user’s identity) alive between subsequent requests.

The oidc plugin caches a user’s identity, as this lookup operation may be slow for some OIDC servers. The maximum number of identities cached is gplazma.oidc.access-token-cache.size (by default 1000). Cached information is refreshed, controlled by gplazma.oidc.access-token-cache.refresh and gplazma.oidc.access-token-cache.refresh.unit (by default 100 SECONDS). This refresh happens asynchronously: existing login activity uses the existing data until new information is returned. Users are automatically removed from the cache if they are idle. The expiry time is controlled by gplazma.oidc.access-token-cache.expire and gplazma.oidc.access-token-cache.expire.unit (by default 120 SECONDS).

SciToken

Add scitoken plugin to support SciTokens. A SciToken is a bearer token from some OAuth2 server that uses the JWT format. The SciToken includes a description of which activities the bearer to entitled. The scitoken plugin maps all tokens from the same OAuth2 server to a single account. Paths within the SciToken are mapped to dCache paths by adding a prefix. The plugin optionally supports t

The jti claim is unique for each SciToken. dCache can prevent replay attacks by remembering the previous gplazma.scitoken.token-history values and rejecting the token if already seen. If the configuration value is zero (the default) then this feature is disabled.

The aud claim describes which service the SciToken was issued. The gplazma.scitoken.audience-targets configuration property is a space-separated list of targets that dCache should consider for itself. If the configuration property is non-empty then dCache will reject any SciToken with an aud claim that does not match one of the listed claims.

NFS

Improve performance of file system locks.

Pool

For HTTP transfers, the pool now honours the pool.mover.http.chunk-size property. Some storage backends (in particular, CEPH) suffer from very poor performance if IO operations are small. Setting this property to a larger value will result in the pool caching incoming data (in memory) so that writes are the configured size. When sending data, the pool will try to read this amount of data before sending it to the client.

When accepting data, the pool may know the file’s ultimate size before the transfer has started; for example, if an HTTP client specifies the ‘Content-Length’ HTTP request header in a PUT or an HTTP TPC request, or if an FTP upload is orchestrated using SRM. Under these circumstances, the the pool will now allocate all of that capacity in the storage backend before initiating the transfer. This avoids allocating more capacity with each client write. Some storage backends (CEPH in particular) suffer from poor performance if capacity is allocated incrementally.

Improve error reporting for HTTP Third-party copy. In particular, the error message describes any redirection that took place while removing redundant information.

When making an HTTP third-party PUSH copy, the pool makes an HTTP PUT request and then verifies that the uploaded file was transferred correctly by requesting the uploaded file’s checksum. Previously, the pool always asked for the same set of checksum algorithms, irrespective of which algorithm values dCache knows for this file. dCache now builds the list of desired checksum algorithms based on the known checksums for the file.

When making an HTTP third-party PUSH copy, the pool checks the HTTP response status code from the server. The server may response 204 and 205 to indicate a successful upload where the content is identical, a status code that some systems (including OneData) use. dCache now considers 204 and 205 as successful HTTP response codes.

A successful HTTP PUT request for an HTTP third-party PUSH copy does not guarantee that a subsequent HTTP HEAD request will succeed. In particular, a pre-authorised URL (from an S3 endpoint) may allow the PUT request but not the HEAD request. Previously, dCache always considered this to be a failure and failed the request. Now, if the transfer is not required to verify checksums then the HEAD request failure is ignored.

The NFS mover now tells the client that if it wishes to update file attributes then such requests must go to the door (not the pool). A client calling fsync on an open file will now update the file’s mtime and size.

Support pools with multiple tags. The pool.tags configuration property was meant as a comma-separated list of key-value pairs; e.g., pool.tags = tag1=value1, tag2=value2, .... Due to a bug, a pool had only one tag, with the remainder tags included in that tag’s value (e.g., tag1 value1,tag2=value2). Now pools will accept multiple tags.

Pool Manager

Added option -dynamic to psu create pool command to create a pool group that will dynamically add all pools that have a specific attribute. Currently, only pool tag, defined by pool.tags property is supported.

Example: > psu create pgroup -dynamic -tags=zone=A zone-A-pools

Will create pool group zone-A-pools and any existing pool as well as any new pool that configured with tag zone=A will be automatically included into that pool group.

Add the ability to disable anonymous staging. Without enabling explicit stage protection, dCache allows anyone who can read a file to stage it. Various doors allow anonymous access, where any world-readable files may be read. If such a file is stored on tape then an anonymous user could trigger staging of those files. This version of dCache has the ability to disallow staging if the user is not authenticated.

The psu admin commands are inconsistent. Several commands that support modifying associations have the form psu addto and psu removefrom to add and remove an association (respectively). However, the commands for managing the associations between pools or poolgroups and links do not follow this form; instead, the commands are psu add link and psu unlink. This version of dCache fixes this inconsistency by also supporting the psu addto link and psu removefrom link commands.

The rc failed admin command documentation is updated, so it no longer wrongly described the argument as a PNFS-ID.

Resilience

An important bug (from a race condition) which made resilience susceptible to causing data loss during removal of excess replicas was fixed.

Several smaller bug fixes were also applied:

1) The restart command

\s Resilience pool ctrl start

now properly reinitializes the pool state on the pool operations.

2) The retry command

\s Resilience retry errors

no longer throws a string index exception.

After excluding a pool, it cannot be scanned under any circumstances; it must be explicitly included again before scanning can be activated on it; i.e.,

\s Resilience pool include [pool expression]

Logging of messages has also been added to resilience in order to be able to trace its communication with other dCache components. These records appear at INFO level in the resilience.resilience log.

For clarity, the final state of a file operation which ends up being unnecessary is no longer reported as ‘VOID’, but as ‘NOOP’, when you do:

\s Resilience file ls

WebDAV

The HTTP status code returned after successfully initiating an HTTP third-party-copy is now 202 (Accepted), rather than 200 (OK). The HTTP response also includes a Content-Type header describing the response as text/perf-marker-stream.

HTTP third-party copy now supports RFC–3230 ‘Want-Digest’ headers on COPY requests. For HTTP PUSH requests, this just acts like an HTTP HEAD request with the ‘Want-Digest’ header: the response includes the file’s checksum. For HTTP PULL requests, dCache will ensure that the most appropriate checksum algorithm is calculated. If the client advertises it suppers HTTP trailers then the checksum value is returned after the transfer has completed.

XRootD

The dCache xrootd door/server now supports security level configuration and signed hash verification. See dcache.properties under “Xrootd: support for verifying signed hashes (by server/door)” for information.

We have also added support for mapping clients with valid credentials to NOBODY; this also enables the current version of third-party-copy (which lacks the full proxy delegation that will be available in the next Golden Release [5.2]) to allow destination servers to log in anonymously to dCache as source during third-party-copy; in this latter case (only), dCache also allows the server to see read-protected files.

A bug involving access logging when the door is configured with HAproxy was fixed, and trace-level logging in general was converted to debug-level, so that the former may be reserved for handshake packet dumps only.

Changes affecting multiple services

Kafka topics can now be set independently for several services using the variables

dcache.kafka.topic = billing

(where “billing” is the default value) and the more specific

dcap.kafka.topic = ${dcache.kafka.topic}
ftp.kafka.topic = ${dcache.kafka.topic}
nfs.kafka.topic = ${dcache.kafka.topic}
pool.kafka.topic = ${dcache.kafka.topic}
webdav.kafka.topic = ${dcache.kafka.topic}
xrootd.kafka.topic = ${dcache.kafka.topic}

Macaroons now include an Issue ID (iid) caveat, which dCache ensures is the first caveat in each new macaroon. The iid caveat ensures that the ID is always unique, even when generated with the same secret, but that derived macaroons (ones with additional caveats) may still be identified.

By default, the WebDAV and frontend doors now reject any macaroon sent on an unencrypted connection. Although these doors used to accept macaroons on unencrypted connections and this behaviour is configurable, we recommend sites accept this change and have dCache reject unencrypted macaroons.

This version of dCache configures systemd limits in a single place, avoiding that ineffective configuration is included.

Changelog from 5.0.0 to 5.1.0

40307e91f2: [maven-release-plugin] prepare for next development iteration
5234c07f2a: [maven-release-plugin] prepare release 5.1.0
30b6209ad8: Revert “docker: Add a way to create docker image”
92975e7a54: resilience: adjust synchronization of file operation removal from map
7c212b24d6: pool: avoid throwing a RuntimeException for non-bugs
0863f49066: docs: TheBook UserGuide fix location of edit-me-on-github ribbon
4b2d9381de: docs: TheBook update heading importance to be more consistent
f8365ec8a7: transfermanager: do not retry starting mover if transfer is not supported
aac0eb6e52: pool: avoid log-and-throw anti-pattern
ec453280d2: frontend: ensure client is disconnected when shutting down channel
a56116e565: frontend: avoid race on cancelling channel garbage-collect task
bc68734a3c: transfermanager: avoid NPE on shutdown
886f36acb3: pool: throw exception with meaningful error message
0fc21c0217: [maven-release-plugin] prepare branch 5.1
68acbbf689: docs: add rest API active transfers description
99f049d6ee: docs: UserGuide REST API add description of space reservations
3f3ed31715: docs/UserGuide add initial frontend QoS description
4a5c1836bf: Remove copy-n-paste error
eddd8b0670: docs: add inital information about frontend REST API
82c0bbc4a3: pool: register size of successful zero-length uploads
136cbe1cde: Core: make dCache buildable on non-English locales
8ac85b095f: docs: add UserGuide
0ffdd34f46: libs: use slf4j–1.7.26
ef39e437f2: Core: make dCache buildable on non-English locales
3313ca8b8a: docs: TheBook remove reference to non-existing service
e3c4223424: gplazma: oidc better handling if JWT is unknown or invalid
f0f1379ecc: resilience: do not allow pool scans of excluded pools, period
5733f5e60c: alarms: move to logback 1.1.4 and fix handling of MDC
bab35350a6: src: remove dead code (artifacts of ReplicaManager)
f894624ee0: dcache-resilience: fix indexing on retry errors command
b2dc1245d0: pools: JSON mover info timeInSeconds should be timeInMilliseconds
0d9963becd: pool: update StateChangeListener classes with change explanation
b51ca78b8a: pool: include change explanation in replica monitoring events
84f3e15e35: pool: add explanation field when updating a replica
3cf5967975: pool: describe what triggered replica state changes
fb97274611: ftp: add support for (explicit) FTPS doors
ec34f0c332: resilience: update state on pool operations when restarted from admin command
07772db56f: libs: use nfs4j–0.18.2
aaecade7e4: chimera-shell: fix class cast of extractor in constructor
eddda93d9e: libs: update jetty version to 9.4.12.v20180830
00d89b7244: nfs: fix missing CDC initialization
18dc14c270: dcache: adding more configurations to KafkaProducer
7e8993fa45: gplazma: add SciToken support
d688b16846: common: add multi-targeted restriction
f7ac12a41a: gplazma: ban add ability to select based on uid and gid
8161f08b81: resilience: do simple existence check of replica on pool to avoid dark removes
7069ed2183: systemtest: remove ancient replica pools
4651736dd5: pool: add inotify support for open/read/write/close events
1e5e355150: docs: mongodb as pool metadata backend
b66a891fd0: docs: Fixed lists in config-PoolManager
7f3fbf0261: docs: RT#9533 and 9494: IPv6
e5a7041156: docs: RT#9534: Ceph pool backend config
cd826a7d25: gplazma: oidc cache user-info lookup result
08656e96b5: TheBook: some hints on using a banfile (#4740)
311c7182a1: Fixing example of ban.conf that breaks dCache.
424f968388: gplazma: oidc fix broken commit
4a1da8a569: gplazma: oidc optimise network usage
09d3757069: resilience: add ability to log resilience activity
18a79e7768: pool: grow file prior HTTP TPC
884dcc9e8d: pom: use rados4j–0.0.4
549b22cd8b: resilience: add ability to log resilience activity (incoming)
87aa817764: resilience: remove the ‘VOID’ operation state
cad57b6561: Add documentation to accompany patch 806de4076b3ea7c0ccad3c28abf0bf8430ffacf0
806de4076b: kafka: add ability to specify Kafka topic name
83c8b1bcd6: pool-ceph: Failure when initiating a macaroon authz HTTP PUSH, authn with X.509/macaroon to target.
1ac0ac77fd: Changes to the dCache documentation (TheBook): broken links, typos, formatting, wording, …
4c124a11d7: gplazma: oidc add discovery cache configuration properties
2abff00b2e: gplazma: oidc warn slow user-info queries
69669e6e3f: webdav: fix resource name for door root
53b95fbdb2: Updated dependency list for building on Debian
5cafe39aad: ftp: add support for anonymous FTP access
f87e4a070c: util:add new setter to ConfigurationMapFactoryBean
ddf9009b52: dcache-xrootd: use debug level instead of trace
2a2455f11c: xrootd: upgrade to xrootd4j v3.4.1
ac5419d132: gplazma: oidc introspect JWT to learn issuer
0f2a60e276: transfermanagers: recover from non-fatal error starting mover
277f773dd0: pnfsmanager: network issues triggered a NPE
65b8e499d1: Changed .gitignore to include .reviewboardrc file
9566a165bb: cells: Improving readability of cells communication code
223992f630: pool: fix CDC for repository listener notification
186a407d7f: ftp: store calculated checksum using root privileges
fc8de4575a: docs: add initial information about the ftp door
915d66e454: util:add unit test to ConfigurationMapFactoryBean
094f99fc01: dcache-xrootd: implement support for security level and signed hashes
77b0b33708: pnfsmanager: fix regression when querying attributes of the root directory
9853191a15: systemtest: fix OpenSSL DN format change
099d7e2ffc: core: describe missing FileAttribute in PoolManager message
2e3c02acc2: systemtest: remove reference to access-log plugin
968926535c: pnfsmanager: include link information in storageinfo
949936a21e: poolmanager: introduce dynamic pool groups based on pool tags
33d957a899: webdav: fix path-to-caveat for macaroon minting endpoint
59145459e2: cells: changed readme (#4634)
a4acb0ec79: gplazma gridmap plugin: compare DNs ignoring letter case for attribute names
2b64c115c5: pool: fix Guava to Java-API migration introduced in 3de5f34179
478049fbd4: poolmanager: update ‘rc failed’ command
cbd3ceba9f: pool: fix tag parsing regression introduces in #7f9a32beec
dbe93282ce: dcache: Expand star import in Indexer
70e1b12d6a: docs: describe poolmanager configuration location in zookeeper
7f9a32beec: pool: fix parsing of pool tags
6c33d339a2: webdav: fix NPE when Kafka notified file deletion
3de5f34179: various: replace guava Files usages JDK Files
aa2883f25a: doc: fix typo
0d8df30064: pool: report correct replica creation time to sweeper
b5e076fd68: psu: remove dead code
c26df9f1d4: libs: update to hazelcast–3.11
137d2ed565: xrootd: fix access logging when xrootd door is configured with HAproxy
bca2869abf: webdav/frontend: disabling basic authn should not disable macaroons
0dc483bdcc: srm: do not log a stack-trace on expected Exception errors
ea3cdd48cb: webdav/frontend: update default to reject macaroons on unencrypted channels
3bb180b849: transfermanager: fail third-party copy if the file is still being uploaded
9cb63c2142: webdav: fail COPY early if file is currently being uploaded
800c979d74: transfermanager: abort transfer if there is a bug
8550267c56: Fix typo
224574aced: chimera: README to README.md, clean contents
690d7f9145: webdav/frontend: add switch to reject macaroons sent unencrypted
b3d0f026f0: pool: introduce clone method for FileAttributes
9fde33d6d8: pom: update nfs4j to version 0.18.1
a95763ecce: gplazma: JAAS plugin logs a stack-trace on misconfiguration
01c9a760b1: httpd: write requests on admin webpage to .access-log
7991e8ed61: pool: nfs mover always returns DATA_SYNC4 as stable write indication
f400fcd0a3: packaging: remove redundant systemd configuration, add note to limits file
eee88ce105: transfermanager: include pool address in the mover start failure message
6946e79390: pool: update error messages to make them distinct
b0839feca7: poolmanager: update some psu commands to be consistent
a16ef71546: pool: avoid using the same error message in multiple places
131d0ecb6d: admin: update dependency on mina-sshd library
1483ee5d12: admin: update dependency on mina-sshd library
c48f8ef30a: libs: use nfs4j–0.18.0
7cc4963ad6: alarms: add pool dead alarm
b65acf64af: webdav.properties: note about redirects to HTTPS
a1cadfe017: pool: fix lookup for canonical hostname for IPv6 addresses
273821b44e: doc: add paragraph in direct command execution in admin section
22270ff344: admin : add direct command execution capability
17f01abd5e: pool: grow file prior FTP upload
4bb48bed23: pool: don’t update atime on flush
e4e841630f: pnfsmanager: do not emit IN_ATTRIB event on atime updates
3e6e7cf782: scripts: fix ‘dcache pool yaml’ command
63b864039e: webdav: 401 for unauthenticated requests; message in status line
b2d7393ba3: dcache, frontend: release dcache-view version 1.5.3
7a9ac72b11: door: fix issue 4551 (wring storage)
63438af62d: Update config-chimera.md
b508529b83: Update config-write-token.md
99ddbf5050: ftp: fix MLSC command for non-small directories
37c3a825c1: pom: update maven-git-commit-id-plugin to version 2.2.6
d936ee8bf4: dcache-xrootd: remove mv request hack
31e4f9c818: dcache-frontend: add documentation concerning restores
71367f26e0: dcache-xrootd: Allow ROOT read access by destination server when dCache is the source in a third-party transfer.
4e2e589eee: pom.xml: update xrootd4j dependency to 3.3.4
8a7ced4cc5: dcache-frontend: undefined suid parameter on transfers should be NULL not “null”
5f82d2710e: dcache: update kafka-client lib version to 2.1.0
cecd828583: macaroon: add an issue ID caveat
1e9353e8de: webdav/pool: add support for RFC3230 Want-Digest on COPY command
24bf16cbf6: pool: http-TPC do not fail if HEAD fails for non-verified transfer
dd66bf0ea3: pool: http-TPC consider more PUT response codes as successful
ad17e63ac0: pool: remove desired checksum from ChecksumMover
606c8236b7: nfs: do not filter device’s IP addresses based on site locality
f29ddeab04: dcache: wrap billing data source with AlarmEnabledDataSource
1f8a3c8322: pool: HTTP TPC create ‘Want-Digest’ value based on known checksums
6c17311370: common: fix random data generation in TimeseriesHistogram unit test
1c17e70bf8: docs: clarify requirement gplazma.ldap.try-uid-mapping option
a7bc5ef250: webdav: work-around Milton racy API for creating collections
3f5d8f8255: poolmanager: add ability to disable anonymous staging
a42022b6c1: webdav: fix name of root
4560d3f4ba: webdav: add Content-Type and fix status
4f29cba8a0: pool: move ChecksumChannel creation out of AbstractMover
32e1cbb7ee: pom: use rados4j–0.0.3 with ARM64 support
220e2f19ef: Removing version reference
57f1e6b460: Remove version reference in new paragraph
73ffa3dff3: Update cookbook-transport-security.md
563094fb82: pom: use nfs4j–0.17.10
73a107892f: pool: improve error logging for failed HTTP third-party copies
b7d3b663ac: gplazma voms plugin: add trust anchor refresh paramater
9c526832cf: srm: include TLS/SSL port in ‘dcache ports’ command
41c32a8c14: dcache, frontend: release dcache-view version 1.5.1
653bb697bb: pool: fix HTTP chunked upload
806e8bc3d1: xrootd: support mapping clients with valid credentials to NOBODY
b36842c800: libs: use nfs4j–0.17.9
4fd18c5dbe: pool: grow file prior HTTP upload
4d91d68ca4: Update BUILDING.md
a0a53eef1f: pool: let http mover respect pool.mover.http.chunk-size
169e6579ea: [maven-release-plugin] prepare for next development iteration

What’s new in dCache 5.1 Release notes

Highlights

Incompatibilities

Acknowledgments

Release 5.1.26

dcache-xrootd

Changelog 5.1.25..5.1.26

Release 5.1.25

pool

Changelog 5.1.24..5.1.25

Release 5.1.24

frontend

xrootd

Changelog 5.1.23..5.1.24

Release 5.1.23

dcache

frontend

Changelog 5.1.22..5.1.23

Release 5.1.22

frontend

Changelog 5.1.21..5.1.22

Release 5.1.22

frontend

Changelog 5.1.21..5.1.22

Release 5.1.21

frontend

Changelog 5.1.20..5.1.21

Release 5.1.20

frontend

skel

srm

Changelog 5.1.19..5.1.20

Release 5.1.19

cell

skel

webdav

Changelog 5.1.18..5.1.19

Release 5.1.18

canl

gplazma

Changelog 5.1.17..5.1.18

Release 5.1.17

config

dcache

gplazma

webdav

Changelog 5.1.16..5.1.17

Release 5.1.16

srm

Changelog 5.1.15..5.1.16

Release 5.1.15

doors

pool

scripts

Changelog 5.1.14..5.1.15

Release 5.1.14

gplazma

pool

webdav

xrootd

Changelog 5.1.13..5.1.14

Release 5.1.13

frontend

Changelog 5.1.12..5.1.13

Release 5.1.12

Changes affecting multiple services

pool

xrootd

Changelog 5.1.11..5.1.12

Release 5.1.11

dcap

gplazma

httpd

Changelog 5.1.10..5.1.11

Release 5.1.10

srm

webdav

Changelog 5.1.9..5.1.10

Release 5.1.9

common

What’s new in dCache 5.1
Release notes