dCache 6.2 Release Notes

Highlights

Improved pool startup time with BerkeleyDB medadata repository
Basis support for extended attributes
Java 11 runtime required
TLS support for xrootd
SciToken support with XRootD
New telemetry service
Running dcache domains as systemd services for RPM based distros
The Cleaner cell may be run in high availability mode

Incompatibilities

Minimal PostgreSQL version is 9.5
From now on dCache requires Java 11 for runtime
dCache now requires ZooKeeper version 3.5 > NOTE: the ZooKeeper server configuration format has changed, the old one will lead to logged warnings in System cell on startup: Invalid config event received:
The systemd dcache.service replaced by dcache.target
Dropped compatibiity with EGEE rpms

Note:

By increasing the required Java runtime version to 11, a new set of warnings is logged on startup, such as:

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by [...]
WARNING: Please consider reporting this to the maintainers of [...]
WARNING: Use --illegal-access=warn to enable warnings of further illegal
reflective access operations
WARNING: All illegal access operations will be denied in a future release

They are generated by third party libraries which use a type of reflection that will no longer be supported in future Java releases. These libraries are expected to be updated in the future. This does not affect functionality.

Acknowledgments

(tbd, this section uses markdown formatting)

Release 6.2.49

gplazma2-grid

An empty requires field in IGTF policy is now accepted. Files with invalid or unexpected policies are ignored.

Changelog 6.2.48..6.2.49

84cc8c49aa: [maven-release-plugin] prepare release 6.2.49
88d11eed37: gplazma2-grid: invalid IGTF policy file should fail with ParserException
035d8fc6d5: [maven-release-plugin] prepare for next development iteration

Release 6.2.48

xrootd4j

A potential for DOS in xrootd write was fixed.

Changelog 6.2.47..6.2.48

0f9134f28f: [maven-release-plugin] prepare release 6.2.48
d54ec67766: pom.xml: update to latest xrootd4j bugfix
f2e1c22e67: [maven-release-plugin] prepare for next development iteration

Release 6.2.47

frontend

Anonymous access receives a 401 error and doesn’t trigger a stack-trace when unlimitedVisibility is set to false.

pool

A bug is fixed where the checksum calculation would fail for empty files.

Changelog 6.2.46..6.2.47

c7eb0eb98f: [maven-release-plugin] prepare release 6.2.47
ae1901aaa6: dcache-frontend: protect against RuntimeError in case of denied anonymous access
433c089118: pool: don’t treat an empty file as a sparse file
543750af21: [maven-release-plugin] prepare for next development iteration

Release 6.2.46

debian

Added explicit dependency on rsyslog for debian packages so there are no errors during installation.

info

The delay between messages sent by info was increased to reduce log messages of “DGA {…} triggering too quickly”.

pool

If the scrubber state file is empty, scrubber will now log a more helpful error message and proceed scrubbing the pool.

Changelog 6.2.45..6.2.46

b2e85ba189: [maven-release-plugin] prepare release 6.2.46
8bc07b2d64: pool: improve handling empty saved state by scrubber
3d291c54d1: dcache-xroot,pom.xml: bump xrootd4j to 4.3.1
335c2b8d01: dcache-xroot: flesh out channel inactive and exception caught
7a2a773261: info: increase delay between messages
43fc1560a8: deb: add explicit dependency on rsyslog package
e2001056ca: [maven-release-plugin] prepare for next development iteration

Release 6.2.45

dcache

Improved documentation for kafka Producer properties.

Improved error message for inexistent statistics path.

dcache-xroot

The current relase Updated master xrootd4j to 4.3.0.

frontend

NPE is fixed when SRR when space information is not available yet.

xrootd

Rendezvous TPC without requiring a JWT token to be passed by the third-party client is possible (again).

Changelog 6.2.44..6.2.45

ce13ea9f9c: [maven-release-plugin] prepare release 6.2.45
5356036597: dcache:improve documentation for kafka properties
b7816f31bc: added missing import statement to XrootdRedirectHandler
cc9fe9c458: dcache-xrootd: fix TPC rendezvous to work with token authorization
9074653311: frontend: fix NPE in SRR when space information is not available yet.
4d87218ff8: dcache-xroot: update master xrootd4j to 4.3.0, and stable branches to next minor version
552471dd34: dcache-core: improve error message for inexistent statistics path
298769c978: [maven-release-plugin] prepare for next development iteration

Release 6.2.44

Technical release

Release 6.2.43

pnfsmanager

Doors no longer allow a user to discover whether or not a file or directory exists within directories they cannot access.

pom

Updated spring libs to version 5.2.20, CVE–2022–22965

Changelog 6.2.42..6.2.43

ceb303f5e6: [maven-release-plugin] prepare release 6.2.43
f45fce2e15: pnfsmanager: avoid leaking whether or not file exists
1294a04939: pom: update spring libs to 5.2.20
dd3d12ba71: [maven-release-plugin] prepare for next development iteration

Release 6.2.42

dcache-chimera

Cleaner batch delete exception has been fixed.

frontend

A bug is fixed that inverted the property frontend.srr.public=true. True means now allow public access and false does not.

Changelog 6.2.41..6.2.42

856047ab99: [maven-release-plugin] prepare release 6.2.42
e6029f78a4: nearline-storage: fix request count on error path in AbstractRequest#failed
c486249071: nearline-storage: fix queued count when request initialization falied
15916a5494: pool: add bunch of unit tests for NearlineStorageHandlerTest
b341519c19: frontend: fix problem with making SRR resources public
3b9988ac97: dcache-chimera: fix cleaner batch delete exception
3035152b24: [maven-release-plugin] prepare for next development iteration

Release 6.2.42

dcache-chimera

Cleaner batch delete exception has been fixed.

frontend

A bug is fixed that inverted the property frontend.srr.public=true. True means now allow public access and false does not.

Changelog 6.2.41..6.2.42

856047ab99: [maven-release-plugin] prepare release 6.2.42
e6029f78a4: nearline-storage: fix request count on error path in AbstractRequest#failed
c486249071: nearline-storage: fix queued count when request initialization falied
15916a5494: pool: add bunch of unit tests for NearlineStorageHandlerTest
b341519c19: frontend: fix problem with making SRR resources public
3b9988ac97: dcache-chimera: fix cleaner batch delete exception
3035152b24: [maven-release-plugin] prepare for next development iteration

Release 6.2.41

frontend

The property frontend.srr.public=true|false is added to frontend to control SRR resource restriction. The default value is false.

ftp

The FTP cell is killed asynchronously now after the session terminates so the door will remain responsive when many clients terminate their FTP session concurrently.

pool

HTTP-TPC transfers now prefer IPv6 address, if both endpoints support it.

Changelog 6.2.40..6.2.41

3d046adaea: [maven-release-plugin] prepare release 6.2.41
e17c43ee26: frontend: make srr resources public
9a18970585: ftp: kill cell asynchronously
53a22e79ec: pool: http-tpc prefer IPv6 address
cc2b7dc69a: [maven-release-plugin] prepare for next development iteration

Release 6.2.40

common

A bug was fixed where dCache will no longer accept certificates issued by a trusted CA after that CA updates their CA certificate while keeping the public/private key-pair the same. This is typically done to change something in CA’s certificate.

There are places where dCache configuration requires an absolute path. In some places attempting to use a relative path will yield an unhepful null in the log file. This is now fixed, so a meaningful error message is logged instead.

pool

An NPE on attempt by client to read from a broken replica is fixed now.

webdav

The current release fixed handling of webdav.authz.allowed-paths so it no longer triggers a NullPointerException.

Changelog 6.2.39..6.2.40

d0005970a3: [maven-release-plugin] prepare release 6.2.40
693b91ff3b: pool: fix NPE on attempt by client to read from a broken replica
0e635deadc: webdav: fix NPE if ‘webdav.authz.allowed-paths’ disallows a request
d7a6ce1c65: libraries: upgrade version of canl
86e2494cf2: common: provide reasonable error message if path is not absolute
a8cc3908f1: [maven-release-plugin] prepare for next development iteration

Release 6.2.39

Pool

The current relase fixed a problem where a file’s tape location that used to exist in the namespace and was subsequently removed can reappear if the file made precious and flushed to tape.

TransferManager

TransferManager now logs bugs with the corresponding stack-trace, making fixes any such bug easier.

Changelog 6.2.38..6.2.39

dda1e54814: [maven-release-plugin] prepare release 6.2.39
89add8b061: pool: only send new URLs when flushing file
2d55238b5d: TransferManager: log bugs with corresponding stack-trace
10cc03a78b: book: Add chapter about telemetry cell
415450e545: [maven-release-plugin] prepare for next development iteration

Release 6.2.38

gplazma

dCache will now reject WLCG-AuthZ-JWT profile tokens with a ‘wlcg.ver’ claim that it does not support. Other tokens (e.g., SciTokens) are unaffected by this change.

loginbroker

This patch fixes SRM based upload or download where the client requests an xroot-based transfer. The SRM door will now also consider any xroot door with xrootd.security.tls.mode configured to OPTIONAL when building a TURL targeting either xroot or xroots protocols.

pnfsmanager

PnfsManager now has two commands, ‘reset chimera stats’ and ‘reset stats’, to support resetting the gauge and counter statistics available through the ‘info’ command.

dCache no longer leaks information about whether or not files exist when using macaroons or Scitokens / WLCG AuthZ JWT profile tokens.

webdav

The WebDAV door cannot send the HTTP response to some HTTP request if the client has already disconnected. dCache no longer logs an error that it cannot send the HTTP response.

A client that disconnects during a proxied HTTP transfer (GET or PUT) is no longer logged in the WebDAV door’s log file; instead, it is logged in the cell’s pinboard. The billing message is updated to make it clearer what went wrong.

Changelog 6.2.37..6.2.38

7ce49ad6a9: [maven-release-plugin] prepare release 6.2.38
af03ea4434: skel: bump logback to 1.2.10
af37119d08: pnfsmanager: add support for resetting gauge and counter statistics
42621fcea0: loginbroker: add support for multiple protocol families
7787b87148: webdav: better logging if client disconnects during proxied transfer
7effd2874c: webdav: don’t log an error if client disconnects before response sent
5b80b3a77f: PnfsManager: check restrictions before resolving path to PNFS-ID
b9e86fb9ca: gplazma: scitoken validate ‘wlcg.ver’ claim
a36f469d86: [maven-release-plugin] prepare for next development iteration

Release 6.2.37

common-security

A better error message is logged when attempting to use a password-protected credential: java.io.IOException: Error decrypting private key: the password is incorrect or the PEM data is corrupted.

webdav

The HTTP-TPC should have fewer failed transfers when faced with many short-lived transfers.

It’s now possible to access a summary of HTTP-TPC transfers and configuration information via the WebDAV door’s info admin command.

Changelog 6.2.36..6.2.37

6fdd3a905c: [maven-release-plugin] prepare release 6.2.37
ada24fcd35: common-security: prevent NPE on password protected cert
d625f40a4e: webdav: http-tpc update heuristics for failing transfer
f06d3fa1af: webdav: http-tpc provide status information in ‘info’ admin command
27da72f0ed: [maven-release-plugin] prepare for next development iteration

Release 6.2.36

dcache

The RemoteTransferManager now better describes the current state of a transfer.

gplazma

The scitoken gplazma plugin now supports extracting the sub and wlcg.groups claims in the same fashion as the oidc plugin.

pool

The current release fixed open queue flag when template defined by queue define class.

The current release fixed open queue flag when template defined by queue define class issue.

webdav

The WebDAV door’s http-tpc ls command now has the possibility to show the current state of the transfer. This may be useful diagnosing transfers spending a lot of time deciding on which pool to transfer should take place.

Changelog 6.2.35..6.2.36

0328238eec: [maven-release-plugin] prepare release 6.2.36
ec93df914d: RemoteTransferManager: update state description
abba94dfe7: webdav: http-tpc add “prep” duration to ‘http-tpc ls’ command
68a2218f06: webdav: http-tpc show whether transfer is queued on pool
d6ed8385da: webdav: http-tpc update ‘http-tpc ls’ command to include state
841af52850: glazma: scitoken add support for additional principals
8fcac9689f: srr: limit publisched endpoints to GLOBAL scope
599f1a341d: Revert “frontend: fix handling of SRR requests over IPv6”
e7a663a145: pool: adjust StorageClassContainerTest for dcache pre- 7.1
90206858a7: pool: fix open queue flag when template defined by queue define class
ae3916b98d: [maven-release-plugin] prepare for next development iteration

Release 6.2.35

dcache-core

The current release fixed NPE in TransferManager when no pool was selected before transfer is cancelled.

frontend

The current release fixed handling of SRR requests over IPv6.

gplazma

The scitoken plugin now supports OPs that publish their public keys without any corresponding ID (i.e., no kid value).

pinmanager

A more detailed pinmanager log entry is provided for pin requests that are caught in a retry loop, and eventually time out.

resilience

A runtime exception triggered by side effect of logging has been fixed.

storagedescriptor

The script for generating SRR records (from the info service output) has been updated. The field lastupdated has been adjusted to the correct name latestupdate.

webdav

A bug is fixed where HTTP-TPC PULL request can fail (under heavy load) with the downloaded file being deleted, but dCache reports the transfer as successful.

dCache now provides a faster respones to the HTTP-TPC client (typically FTS) should the door decide to fail a transfer.

The WebDAV door should be now faster at accepting new transfers and faster at handling transfer completions when handling many small transfers.

The HTTP-TPC support in the WebDAV door is now e more robust to high number of transfers finishing concurrently.

xroot

Now it is possible to support kXR_delete as a write request on the pool.

xrootd4j

Library updates for xrootd4j which contains fix for Unix Protocol.

Changelog 6.2.34..6.2.35

cd05a920c8: [maven-release-plugin] prepare release 6.2.35
f48b03a893: webdav: http-tpc improve throughput with short transfers
d4b48a7161: frontend: fix handling of SRR requests over IPv6
94d1af20cb: webdav: http-tpc don’t wait if door fails transfer
d158aaf301: webdav: http-tpc avoid resetting transfer state
11ed3886a8: dcache-core: fix NPE in TransferManager when no pool was selected before transfer is cancelled
9388fa02cb: resilience: fix runtime exception triggered by side effect of logging
84deca3d27: gplazma: scitoken add support for OPs that advertise keys without kid
5941661529: webdav: http-tpc move transfer finalisation off of message queue
435261c4ba: pom.xml: update xrootd4j dependencies to 4.2.5/4.1.6/4.0.11
04f2272903: CanlContextFactory: throw FileNotFound for missing ca certs dir
0097c289b1: pinmanager: provide more details information on timeout
f429b91125: storagedescriptor: fix element name to ‘latestupdate’
16f1aaf0fd: dcache-xroot: support kXR_delete as a write request on the pool
da1e8dabc3: [maven-release-plugin] prepare for next development iteration

Release 6.2.34

ftp

The FTP door now provides more succinct information on pinboard, should use less CPU and take better advantage of the available cores.

pnfsmanager

Attempts to create a file where the parent directory is either missing or not a directory now provide clearer error messages.

pool

The pool now provides more information for bugs reported as java.lang.IllegalStateException: Handle is closed.

webdav

The HTTP-TPC response is improved if a client attempts to pull a file into a non-existing directory, or attempts to use an existing file as an ancestor directory.

xrootd

Improved exception handling for Xrootd.

Changelog 6.2.33..6.2.34

dbf2413c8e: [maven-release-plugin] prepare release 6.2.34
df638f6c0b: pool: provide more information when handle is double-closed
b3fc7e18cd: dcache-xrootd: Alternate fix for client write to closed checksum channel
4e182448f8: pnfsmanager: fix confusing error message.
9d7dda31e7: webdav: fix error handling for bad paths
13cfb3a620: ftp: switch to more reasonable logging of Subject
6ef7512ab7: [maven-release-plugin] prepare for next development iteration

Release 6.2.33

universal-spring-cell

This patch fixes a problem where a call to api/PoolManager returned 404 not found. The JSON serialized object is returned now as before.

xrootd

The xrootd4j version was updated to 4.0.10

Changelog 6.2.32..6.2.33

f2e6b97ff2: [maven-release-plugin] prepare release 6.2.33
e992fa127b: pom.xml: bump to xrootd4j 4.0.10
56a103196e: universal-spring-cell: allow for serialization of Optional
83c1c32f4f: [maven-release-plugin] prepare for next development iteration

Release 6.2.32

billing

Billing will no longer throw a NullPointerException with certain information from an NFS door serving an NFSv3 client.

ftp

Now file sizes for LIST output in bytes is displayed.

The FTP door should be more robust against a (currently unknown) bug that results in the ftp session attempting to send progress queries to the pool after that is no longer possible.

gplazma

The banfile and scitoken plugins, and the two admin commands test login and explain login are updated to accept the username: prefix (e.g., username:paul). The prefix user: continues to work but is now deprecated.

pool

The current release improved error messaging if a setup file badly configures an HSM instance.

A bug is fixed where removing a create hsm statement in a pool’s setup file and running the reload command kills the pool.

xroot

The documentation concerning TLS properties has been fixed.

Changelog 6.2.31..6.2.32

0cbce3329c: [maven-release-plugin] prepare release 6.2.32
07b0d2977e: dcache-xroot, book, skel: fix documentation concerning TLS properties
5f2ebb2fe9: bootstrap: fix two problems with how bugs are reported on startup
a399cadf23: ftp: try harder to ensure any timer tasks are cancelled
31c80a16fb: pool: removing hsm with reload command kills pool
e49357d577: gplazma: switch from user: to username:
504eeda1ba: ftp: show file sizes for LIST output in bytes
99f40a20d2: util/frontend: change transfer rate to double
5fac1c3c5c: bootstrap: don’t hide bugs in ‘create’ command
06eed198e2: ftp: ensure cell is killed even if shutdown triggers a bug
f7f6f04770: billing: fix NPE in billing from NFSv3 message
2f683966e4: pool: improve error message on badly configured hsm instance
5e54ee71b5: [maven-release-plugin] prepare for next development iteration

Release 6.2.31

acl

The group ACEs now apply to desired group instead of to a users with the same numeric id.

gplazma

The WLCG Common JWT Profiles (v1.0) describes the wlcg.groups claim for expressing group-membership. We are increasingly seeing this being used as the OIDC equivalent to the group-membership expressed through the VOMS extension.

During the auth phase of the login process, the group membership information contained in a wlcg.groups claim from the OP is now available as OpenIdGroupPrincipal principals. Subsequent plugins (e.g., multimap) may be used to convert these principals to more directly useful principals.

webdav

dCache now responds with a 507 (Insufficient Storage) on GET request if dCache must do a pool-to-pool internal-transfer or stage the file but no pools have sufficient free capacity to support this.

Changelog 6.2.30..6.2.31

7b81d48510: [maven-release-plugin] prepare release 6.2.31
1732b4a2aa: all: reformat code using Google Style Sheet
7045c15fce: acl: set IDENTIFIER_GROUP flag if WHO is GROUP or OWNER_GROUP
17ef1d8a3a: webdav: return 507 if insufficient space on GET request
84e995f98e: gplazma: scitoken make ExemptFromNamespaceChecks principal optional
4f3fa2bd4c: gplazma/pnfsmanager: update namespace so scitoken ‘scope’ takes priority
669f19df79: [maven-release-plugin] prepare for next development iteration

Release 6.2.30

gplazma

During the auth phase of the login process, the group membership information contained in a ‘wlcg.groups’ claim from the OP is now available as OpenIdGroupPrincipal principals. Subsequent plugins (e.g., multimap) may be used to convert these principals to more directly useful principals.

pool

A rare NullPointerException for HTTP-TPC transfers is fixed.

Changelog 6.2.29..6.2.30

d372de5fa1: [maven-release-plugin] prepare release 6.2.30
8481729ace: gplazma: oidc add support for the ‘wlcg.groups’ claim
e664995cdd: pool: http-tpc fix NPE when monitoring network traffic
3d87507028: [maven-release-plugin] prepare for next development iteration

Release 6.2.29

frontend

SRR returns data now with official json schema

webdav

Fix how sym-links are shown in the static HTML (web-browser) view from the WebDAV door.

Changelog 6.2.28..6.2.29

41c3d128d1: [maven-release-plugin] prepare release 6.2.29
24d79fdd76: frontend: add service to provide Storage resource reporting
dc6442a9ae: webdav: update representation of symbolic links in HTML page
f5c36a56ea: [maven-release-plugin] prepare for next development iteration

Release 6.2.28

dcache-xroot

The next release of xrootd4j is used now, which improves handling TPC read requests.

namespace-chimera

There are less operations on the database now as the filesize doesn’t get updated after a file was flushed to tape.

pool, webdav

The webdav admin interface has been extended to include monitoring information about on-going transfers. The command can optionally show pool information if transfer-manager is update; updating transfer-manager is not required by this change. Percent transfer progress for PULL requests is optionally available if pools are updated; updating pools is not required by this change.

Changelog 6.2.27..6.2.28

6be706def4: [maven-release-plugin] prepare release 6.2.28
c6e80f9c4a: namespace-chimera: ignore file size update on file flush
b86f7dfcf7: webdav: add ‘http-tpc ls’ admin command
be2289ff80: pool/webdav: http-tpc add RemoteConnections perfmarker metadata
f1b2c262b0: dcache-xroot: bump dependency to next xrootd4j release
e57b3578cd: [maven-release-plugin] prepare for next development iteration

Release 6.2.27

pnfsmanager

Fixed a problem where dCache accepts a HSM script that, when called to flush a file, returns successfully but without providing any URIs on stdout. Such requests are considered failed, with the flush request blocked until retried explicitly with the queue activate PNFSID admin command.

pool

The Kafka message emitted when a file is flushed to tape or staged back from tape now includes the locations field: a JSON array of JSON Strings, where each JSON String is a tape-location URL. For flush operations, this is the (potentially empty, potentially multiple) list of URLs returned by the HSM instance. For stage operations, this is the (potentially empty, potentially multiple) URLs that match the selected HSM instance. Both messages now also describe a hsm JSON Object that contain the HSM type, HSM instance name, and HSM provider.

srmmanager

A race condition is fixed that, if triggered, results in a memory leak. This leak can also affect the TURLs returned by SrmManager, where out-of-date information about doors is used.

Changelog 6.2.26..6.2.27

ef08d9263e: [maven-release-plugin] prepare release 6.2.27
2ed4135f2a: srmmanager: fix race condition in LoginBrokerSubscriber
616fea3eba: pool/pnfsmanager: don’t accept a non-Enstore flush without locations
6528e45d8d: pool: update Kafka tape events to include additional information
9863ed9322: [maven-release-plugin] prepare for next development iteration

Release 6.2.26

alarms

Clearer documentation for billing/alarms services regarding HA requirements.

gplazma

dCache has a new session gPlazma plugin that is designed to be a viable alternative to the StorageAuthzDB and KPWD plugins but with shorter (hopefully less confusing) configuration, while also being more flexible.

To support the removal of authzdb plugin, we should provide an easy way for sites to migrate their existing configuration from using authzdb (as a session plugin) to omnisession.

Sites now have a utility to help them migrating from authzdb to omnisession.

Note: it is also possible to use the authzdb plugin as a mapping plugin. This utility does not help in migrating that use to some alternative solution.

To support the removal of authzdb plugin, we should provide an easy way for sites to migrate their existing configuration from using authzdb (as a session plugin) to omnisession.

Sites now have a utility to help them migrating from authzdb to omnisession.

Note: it is also possible to use the authzdb plugin as a mapping plugin. This utility does not help in migrating that use to some alternative solution.

skel

Whether a cell/service supports HA or not can now be derived from the properties file (except for doors).

Please see <name>.cell.replicable.

srm

The access log file for SRM requests now includes the client-supplied list of protocols, if any were provided.

Changelog 6.2.25..6.2.26

e01c39a7ba: [maven-release-plugin] prepare release 6.2.26
7d560e0015: rpm: add dcache-convert-authzdb-to-omnisession into list of packaged files
7252087e5c: gplazma: add util to convert authzdb file to omnisession
a78fc2594d: alarms, billing: specify shared rdbms in the case of replicated service
ed589449e9: skel: add explicit replicable property even when value is false
82815edb7b: gplazma: add omnisession plugin
fd9179fab4: srm: log transfer protocols in access log
02a6d29785: [maven-release-plugin] prepare for next development iteration

Release 6.2.25

dcache-core

The old web pages claim the transfer size and speed are reported in KB and KB/s, although the actually used units are KiB and KiB/s. Reporting the correct unit is important for monitoring. This is now fixed.

frontend

The current release fixed regression in generating swagger JSON for frontend’s REST API.

gplazma

dCache may now be configured so that the multimap oidc predicates match the sub claim value from a specific OAuth2 Provider using the format SUBVALUE@OP, where OP is the dCache-internal alias for the OP. Existing multimap configuration continues to work, but admin is warned to update the multimap configuration.

Changelog 6.2.24..6.2.25

63a0dcafe2: [maven-release-plugin] prepare release 6.2.25
fc0932484a: gplazma: multimap now supports OP in ‘oidc’ predicate
ec7329cb83: frontend: bump swagger version to 1.6.2
6458e8bb3c: dcache-core: correct reported units of transfer size and speed
f901019cda: [maven-release-plugin] prepare for next development iteration

Release 6.2.24

checksum

The policies map is now added to the JSON info object that is returned for checksum module information.

Changelog 6.2.23..6.2.24

4237fda233: [maven-release-plugin] prepare release 6.2.24
dfcb4a8221: checksum-module: add missing map to JSON info
25b46fa5d0: [maven-release-plugin] prepare for next development iteration

Release 6.2.23

xroot

The current release fixed the parsing to account for host/error pairs.

Upload transfer checksum failure is fixed and the gfal scenario now works.

xrootd

xrootd4j is updated to new version which fixed problem with netty LogHandler slowing down TPC client (cutting bandwidth by a factor of 10).

Changelog 6.2.22..6.2.23

30ea020a04: [maven-release-plugin] prepare release 6.2.23
1291969a99: dcache-xrootd: upgrade xrootd4j to 4.0.8
203624c23e: dcache-xroot: parsing of ‘triedrc’ CGI lacks separation by optional comma
cdd6404bea: dcache-xroot: fix upload transfer checksum failure (stable branches)
abe8a2b8e7: [maven-release-plugin] prepare for next development iteration

Release 6.2.22

ftp

Globus transfer agent is now able to create symbolic links.

nfs

The current release introduced nfs.clientdb.config property prefix to populate the BerkeleyDB configuration and now the nfs door is not picky about free disk space in /var/lib/dcache/nfs and the BerkeleyDB used as client store by nfs door can be configured with properties.

pool

dCacheView now works for redirected transfers for non-anonymous data access, provided the WebDAV door is using TLS encryption and is configured with ‘webdav.redirect.allow-https’ set to ‘true’.

poolmanager

A very rare race-condition is fixed that could be triggered if a pool completes a stage or pool-to-pool request (to satisfy a client’s read request) immediately before dying.

webdav

dCacheView is able to view and download files when authenticated and when the WebDAV door proxies the transfer.

Changelog 6.2.21..6.2.22

2b3c319d6d: [maven-release-plugin] prepare release 6.2.22
70b1a317d8: nfs: add possibility to configure BerkeleyDB client store
c8027ca0c0: webdav: use Jetty’s built-in support for CORS
f63ba858c8: poolmanager: fix _waitingFor access outside of synchronized block
2d7f00158c: ftp: add support for the SITE SYMLINK command
129d87a583: pool: http allow client to send credentials when TLS is used
5d3b1c0de5: dcache-chimera: use RemotePoolMonitor to discover file locality
7de3af9df4: [maven-release-plugin] prepare for next development iteration

Release 6.2.21

frontend

frontend/api readers and writers always zero issue is fixed now

gplazma

The LDAP plugin behaviuor now more closely follows that of other mapping plugins. This allows deployments where LDAP is tried first and, if that fails to identify the user, fall-back strategies are used.

pool

The pool size reporting when static/runtime config is not defined now is improved.

Changelog 6.2.20..6.2.21

f88fd1627e: [maven-release-plugin] prepare release 6.2.21
7ada2fef7e: gplazma: ldap throw exception if no principal is added
91e75edacd: poolV4: fix reader/writer counts
e29f378820: pool: remove invalid test case of configured space
8ffdfb69f7: pool: fix pool size reporting when static/runtime config is not defined.
e0bcc3f4e6: [maven-release-plugin] prepare for next development iteration

Release 6.2.20

dcache-xroot

The cirrent release fixed memory leak, race and other small issues with reconnect to pool.

frontend

The dcache-view.endpoints.webdav property now overrides any auto-discovered WebDAV endpoint, making explicit configuration easier.

pnfsmanager

A regression is fixed the prevents creating a symbolic link.

systemd

dcache-generator now successfully runs after restart.

xrootd

xrootd4j is updated to 4.0.6.

The current relase has updated to xrootd4j library to 4.0.6.

The correct subject is used to access the path for both GSI and SciToken protocols.

Changelog 6.2.19..6.2.20

489a117972: [maven-release-plugin] prepare release 6.2.20
2046a18e14: pool: make sure that RepositoryChannel when createChannel detects error
b4a55acb9b: update to xrootd 4.0.7
22f919abd0: dcache-xrootd: update to xrootd4j 4.0.6
fd9c6640ff: frontend: bump version of dCacheView
2a1a2c8bc0: pnfsmanager: fix NPE regression from 3dfed7e8b0
2fdc1d9894: dcache-xroot: fix memory leak, race and other small issues with reconnect to pool
e0b500e7b3: dcache-xroot: store most recent login subject in door
9d354f1fa1: frontend: bump dCacheView to v1.6.2
c7fa0f6870: systemd: fix generator run after reboot
3a41bdc6eb: [maven-release-plugin] prepare for next development iteration

Release 6.2.19

nearline-storage

The current release fixed store cancel regression.

pool

As in distributed system packet loss (due to timeout or re-connects) can’t be avoid, dCache retries many of requests. However, this is not the case for mover redirects. If a door did not receive reqirect information, then transfer will never happen. This is now fixed and re-send redirect if client is not connected within 5sec.

poolmanager

When poolmanager requests that a pool stage a file, the pool may return the special error code HSM_DELAY_ERROR (10013). On receiving this error, poolmanager suspends the request.

This is now fixed.

Changelog 6.2.18..6.2.19

780e7dcaad: [maven-release-plugin] prepare release 6.2.19
a53f00cca9: nearline-storage: fix store cancel regression
cb0388a2d2: pool: re-send redirect if client is not connected within 5sec
a1ad7f7e7a: poolmanager: fix NPE if pool delays stage without giving a reason
79ab2fd0f4: nearline-storage: introduce FlushRequest#getReplicaCreationTime
c1e5a4aa6b: [maven-release-plugin] prepare for next development iteration

Release 6.2.18

pool

There are several cases when pool scans the various hsm queus: on mover start, to check load assumptions, on heartbeat, to populate pool manager with load information, on info command, to report load numbers to admin interface of dcache-view collectors

As this is O(n) operation, in case of a large queues those scans might take seconds, which is unacceptable, especially wen starting a new mover.

This is now fixed and getting hsm start is O(1) and doesn’t impact pools operation

xrootd

Now the door will ignore xrootd.enable.tried-hosts property if property false instead of returning errror.

Changelog 6.2.17..6.2.18

e428eb796f: [maven-release-plugin] prepare release 6.2.18
d98a894414: pool: don’t scan flush/restore/remove queue to get stats
dddbab46f7: dcache-xroot: (door) ignore tried if property false instead of returning error
2a333645c0: [maven-release-plugin] prepare for next development iteration

Release 6.2.18

pool

As this is O(n) operation, in case of a large queues those scans might take seconds, which is unacceptable, especially wen starting a new mover.

This is now fixed and getting hsm start is O(1) and doesn’t impact pools operation

xrootd

Now the door will ignore xrootd.enable.tried-hosts property if property false instead of returning errror.

Changelog 6.2.17..6.2.18

e428eb796f: [maven-release-plugin] prepare release 6.2.18
d98a894414: pool: don’t scan flush/restore/remove queue to get stats
dddbab46f7: dcache-xroot: (door) ignore tried if property false instead of returning error
2a333645c0: [maven-release-plugin] prepare for next development iteration

Release 6.2.17

nfs

Fixes regression introduced in 6.2.15.

Changelog 6.2.16..6.2.17

598ec78fb7: [maven-release-plugin] prepare release 6.2.17
929340be36: nfs: fix access to path before namespace info available
243b3d4daa: [maven-release-plugin] prepare for next development iteration

Release 6.2.16

Changes affecting multiple services

Sometimes file checksums are missing in namespace. Transfer requiring checksum verification fails then with an error saying that “No checksums found”, but doesn’t log a stack trace server side anymore.

dcache-xroot

During periods of heavy usage on a pool node, if I/O stalls during read, the xroot client may try to reconnect and continue reading. This release fixes a bug which fails the transfer on the attempted reconnect with a “uuid no longer valid” error (because the file mover has already been closed and removed). Clients should now be able to resume reading the file on retry.

Changelog 6.2.15..6.2.16

ac73913b6b: [maven-release-plugin] prepare release 6.2.16
d4883ca2c3: dcache-xroot: Allow client to reattempt open on pool when I/O stalls
b4349ced2f: check if checksums are present and throw CacheException if they are not
41fc04fd04: [maven-release-plugin] prepare for next development iteration

Release 6.2.15

admin

The current release fixed doors login dump cache admin command to show the information sent by the door along with the login result.

common

Printing of bearer tokens is improved to provide better assurance that bearer tokens are not leaked.

gplazma

All login failures are now recoreded in pinboard as a single-line summary, explaining why the login attempt failed.

namespace

Now Path information available for all transfers.

webdav

The current release fixed a regression where dCache would complain about badly formatted IPv6 addresses, preventing those addresses from being included in the access log files.

xroot

The current relase updated xrootd4j to 4.0.5 which fixes improper use of destination token when contacting source server during third-pary transfer.

Changelog 6.2.14..6.2.15

5b5da688ad: [maven-release-plugin] prepare release 6.2.15
c32494919a: dcache-xroot: update to xrootd4j–4.0.5
8c6a6a96b1: webdav/frontend: fix IPv6 address logging
34ee7f95f4: common: update how BearerToken is shown in Subjects#toString
19cfc9d636: authn: add missing hashCode and equals methods for PasswordCredential
735fbe2210: authn: fix “login dump cache” admin command output
08b9944719: gplazma: provide single-line summary in pinboard for each login failure
827c6195e1: authn: log failures in UnionLoginStrategy
a18373b6be: authn: add missing hashCode and equals methods for BearerTokenCredential
881ea70cbb: common: Update BearerToken representation to include some information
3264f7b4e2: namespace: ensure that file’s path always set on GetAttributes
0d3261d6df: [maven-release-plugin] prepare for next development iteration

Release 6.2.14

gplazma

dCache now supports OPs that assert group-membership with eduPersonEntitlement claims (as described by AARC guideline G002). The multimap plugin may be used to map these to corresponding group/gid values

webdav

Restarting transfermanager while there are ongoing HTTP-TPC transfers no longer results in an endless loop, with transfermanager logging a message every ~5 seconds per ongoing HTTP-TPC transfer.

Changelog 6.2.13..6.2.14

315ad704aa: [maven-release-plugin] prepare release 6.2.14
8f289b6d0c: Revert “pool: do not update file size on flush”
96bcb14900: gplazma: add support for eduPersonEntitlement assertions
e3d68154eb: build(deps): bump version.jackson from 2.10.1 to 2.12.1
d430a76b9b: webdav-transfermanager: fail gracefully if transfermanager restarted
544829d5ce: [maven-release-plugin] prepare for next development iteration

Release 6.2.13

biiling

The current relase fixed unit info for bandwidth measurements in billing.properties (bytes/sec, not MiB/sec).

frontend

Frontend now logs any time it returns 500 Internal Server Error to the client.

dcache-view Previously displayed MB/sec for transfer rate, but the value is the old KB/sec. This is now improved and pages display the values advertised.

pool

Less memory consumption by pool, especially on startup.

The current release assures now that checksum scanner wiill mark files broken and sends alarm.

webdav

An HTTP-TPC transfer will abort automatically if transfer-manager is restarted, without waiting for the client (FTS) to time-out.

Changelog 6.2.12..6.2.13

c159dc6d54: [maven-release-plugin] prepare release 6.2.13
6c4faecd00: dcache-pool: Assure checksum scanner marks files broken and sends alarm
e4e0bae712: pool: do not re-read storage info when creating CacheRepositoryEntry
f17bfa3d4a: webdav/transfermanager: fail gracefully on restart, for perf marker
e644f58438: default: fix unit info for bandwidth measurements in billing.properties
daa8b11173: frontend: create a log entry when we return 500 Internal Server Error
98719cd6a3: pool: do not update file size on flush
7da96d813a: dcache-frontend: change transfer rate to compute what it advertised
660d88600b: pool: use soft reference to storage info object
27370e09e4: [maven-release-plugin] prepare for next development iteration

Release 6.2.12

book

A new section is added to the xrootd chapter explaining the need for SAN extensions on the hostcert in the case of hostname aliasing.

ftp

Globus transfer service can now list directories that contain incomplete files, those that are still being uploaded.

The FTP no longer leaking sockets when dynamically calculating file checksums (e.g., MD5). In practise, this issue only affects transfers via the Globus transfer service.

The current release fixed Globus transfer service directory listing for medium size directories.

The current release fixed Globus transfers hanging when verify file integrity after transfer is enabled and the pool is not configured to calculate an MD5 checksum.

Honour the ftp.net.internal configuration property for dynamic checksum calculation.

Changelog 6.2.11..6.2.12

3581765d10: [maven-release-plugin] prepare release 6.2.12
241581617b: ftp: do not leak sockets when calculating checksums dynamically
a1a0a167e6: ftp: add work-around to calculate MD5 for Globus
16afc1af6b: ftp: add place-holder file size work-around for Globus directory listing
5d46fbe63f: rpm take build directory path into quotes
c692f6da11: ftp: fix MLSC support for medium-length directories
825f517834: ftp: fix on-the-fly checksum calculation
3d592c7ea7: book: document host alias issue with xroot TPC
f66725dbd4: [maven-release-plugin] prepare for next development iteration

Release 6.2.11

nfs

The new version of nfs4j–0.21.1 fixed readdir reply size miscalculation for NFSv3.

pool

The current release fix a problem where, under heavy load, redirected https transfers result in garbled SSL information, preventing the HTTP client from working.

services

Recovery processe in the case where a pool’s message queue is overloaded, as old (and now useless) requests may be simply discarded is improved.

vehicles

The current release improved memory consumption on pools.

webdav

There were reports of the access log file missing entries. This problem seems to be triggered if the client disconnects before dCache has finished processing the request, which is certainly true for cancelled HTTP-TPC transfers. This is now fixed.

xrootd

xrootd4j-authz-plugin-alice is updated to 1.2.0.

The xroot protocol access log entries for the xrootd door and pool have been enhanced to include information about the kXR_query requests.

Changelog 6.2.10..6.2.11

ec8b1e67a8: [maven-release-plugin] prepare release 6.2.11
78040db522: prepare for next development iteration
3689629eb1: [maven-release-plugin] prepare release 6.2.10
01f71f5f59: TransferManager: choose better pool query message TTL
4597553c31: webdav/frontend: ensure cancelled HTTP-TPC transfers are logged
15fede73da: xrootd/pool: improve access log to record more details of kXR_query
0fff2d36db: libs: use nfs4j–0.21.1
42e840560b: vehicles: internalize common strings in StorageInfo
25bbf82fba: pool: sort flush queue only if we going to use the result
932055e31b: pool: https avoid sharing SSLEngine between TCP connections
04c96a851f: removing commit-msg file
395acb44d6: updated alice-plugin version to 1.2.0
851452f57d: pom.xml:update xrootd4j-authz-plugin-alice version to 1.2.0

Release 6.2.10

The current release fixes EGI-SVG–2020–16939 vulnerabilities.

Release 6.2.9

dcache

The log files now can be enforced even with systemd deployments.

dcache.log.destination property has been introduced which controls the log destination. The default value console preserves the current behaviour: systemd services log to journald, system-V services log to stdout (redirected to /var/log/dcache/.log).

The option file instruct logback to send message to the location defined by dcache.log.file property. The plain file FileAppender is used as dCache packages already comes with pre-defined logrotate configuration.

xrootd

The xrootd4j version was updated to 4.0.4.

Changelog 6.2.8..6.2.9

f8ffbdacd4: [maven-release-plugin] prepare release 6.2.9
88ef6a9fc5: logging: add an option to control logger destination
ec72bc0f61: pom: update to xrootd4j 4.0.4
64961d79a9: [maven-release-plugin] prepare for next development iteration

Release 6.2.8

common-auth

Fix regression introduced by limiting UidPrincipal to be only an output principal.

pool

The mover thread is not interrupted outside of IO activity to avoid disabling of repository.

HTTP-TPC transfers where dCache pushes data to an Apache server (for example as used by Dynafed) will now work for zero-length files.

webdav

HTTPS-PUT requests are now possible.

Changelog 6.2.7..6.2.8

464e3e0330: [maven-release-plugin] prepare release 6.2.8
452bc43016: pool: http-tpc do not loop if HEAD ‘Content-Length’ response is missing
7054e2791a: webdav:HTTPS PUT req issue
bcbb79f1ca: pool: preserve CDC on p2p transfer
1b76cac1b6: pool: http-tpc drop ‘Accept-Encoding’ on HEAD requests
f3dc5f36e7: pool: detach IO thread when mover finished
6cf85465f7: common-auth: mark UidPrincipal as In/Out principal
eb65fa3679: [maven-release-plugin] prepare for next development iteration

Release 6.2.7

pool

dCache pools now support GSI-based xrootd-TPC by default.

Changelog 6.2.6..6.2.7

fe2631db39: [maven-release-plugin] prepare release 6.2.7
1ded2cdb9a: pool: update xrootd-tpc authenication default
e932921c28: [maven-release-plugin] prepare for next development iteration

Release 6.2.6

nfs

If door failed to start a mover due to timeout or no route to cell, then pool selection or mover start is retried. However, as the retry procedure in the transfer class was not idempotent, the nfs door has it’s own workaround to avoid extra pool selection. This partial solution plays not well with retry logic of transfer class and produced inconsistent results. This is fixed now and lost messages to start mover successfully delivered after number of retries.

pool

The HTTP-TPC PUSH requests (which use the HTTP PUT request to send data) are now more robust against slow remote servers that redirect the transfer.

The current release fixed NullPointerException bug in MongoDB-backed pool metadata storage and SpotBugs NORMAL-level warning.

With the chenges introduced in current release, when using HTTP-TPC, the timeouts dCache pool uses when requesting information from the remote server have been adjusted, based on operational experience. In general, dCache will now wait longer for the remote server to complete any post-processing before giving up and failing the transfer.

Changelog 6.2.5..6.2.6

3c8b6b0e56: [maven-release-plugin] prepare release 6.2.6
1bcc96fbbd: chimera: update EnstoreStorageInfoExtractor to use FileState
f29ccdb158: chimera: use UTF–8 constant instead of string in ChimeraEnstoreStorageInfoExtractor
a24c79412b: pool: http-tpc adjust GET and HEAD request timeouts
c45529130d: pool: http-tpc PUT request are repeatable
7b710e4a16: nfs: drop internal mover re-start loop
31d869d32f: pool: support removing MongoDB storage-info entry
7dfbe630ce: [maven-release-plugin] prepare for next development iteration

Release 6.2.5

dcap

Sometimes file restores on dCap door were hanging. This is now fixed.

Changelog 6.2.4..6.2.5

8629039c76: [maven-release-plugin] prepare release 6.2.5
d94a5f5a06: [maven-release-plugin] prepare for next development iteration
efeb443cbe: dcap: fix automatic door retry on transient errors

Release 6.2.4

gplazma

X.509 certificates that use the “SHA–384 with RSA” algorithm are now logged with this name instead of the raw OID value.

httpd

The current release fix pool usage color scheme css used for UsageInto and PoolInfo pages.

pool

dCache pools that are undertaking HTTP-TPC transfers will now wait longer for the remote servers to complete any post-transfer activity. Transfers that involve heavily loaded remote servers are now more likely to succeed.

Changelog 6.2.3..6.2.4

1e55f3b3ff: [maven-release-plugin] prepare release 6.2.4
317ce07c20: fix error, tag 6.2 instead 6.2.3
f485ddc295: srm-common: make X509 client authn optional, support bearer tokens
7f6978aae7: gplazma: include additional algorithm in LoginResultPrinter
20a7493253: pool: http-tpc increase timeout waiting for remote server post-processing
819d18a334: httpd: fix pool usage color scheme css
65c01798d0: correct snapshot version for logback-test-config
2352b87474: correct snapshot version

Release 6.2.3

nfs

The current release fixed previously observed infinite WRITE+COMMIT loop.

Changelog 6.2.2..6.2.3

2bd19d2080: [maven-release-plugin] prepare release 6.2.3
0709f64e67: nfs-proxy: op WRITE must use the same verifier as COMMIT
295ed840a9: [maven-release-plugin] prepare for next development iteration

Release 6.2.2

dcache

There have been several occasions where in a dCache instance using an embedded ZooKeeper server such HA services stopped working because an ephemeral node was elected leader that has no associated service anymore; it should have been deleted. This state might go undetected for a longer time, and removing such stale nodes in order for the services to continue functioning as intended requires manual intervention.

This is fixed now and the functioning of HA services using ZooKeeper-based leader elections is more reliable.

resilience

A more robust handling of PSU changes which remove pool, unit or group mappings was introduced.

Recently introduced improvements changed the way resilience handles file “removal” (no longer setting the repository entry to ‘removed’ but simply by caching the replica). This change, however, did not take into account the handling of broken files. Encountering a broken file, it would indiscriminately attempt to remove it, whether it was cached or not; this was leading to an infinite loop, with the file operation continuously iterating without doing any further work. This situation could potentially hang the pool scans (if there are as many broken files as there are scan threads), and even the file operation queue. The current release fixed this issue and there is no any potential for stalled operations when encountering broken replicas.

Changelog 6.2.1..6.2.2

d91ba83234: [maven-release-plugin] prepare release 6.2.2
44fabfc33e: dcache-core: fix embedded ZooKeeper persisting stale ephemeral nodes
859be7aca0: dcache-resilience: ignore broken cached files
d7ddc2e486: dcache-resilience: protect access of pool info map against NoSuchElementException
12287c4beb: [maven-release-plugin] prepare for next development iteration

Release 6.2.1

macaroons

Fixed macaroons validation where in case of multiple combined caveats for some of them only the digital signature was checked.

resilience

If a FileOperation is canceled while its subtask is running a NullPointerException was thrown. This is now fixed.

rpm

dCache rpm-package requires now Java 11.

systemd

dCache forces now the use of systemctl on systemd-capable systems.

Changelog 6.2.0..6.2.1

130a5c708a: [maven-release-plugin] prepare release 6.2.1
61d44927ef: dcache-resilience: avoid NPE in file operation cancel
916d8c4e04: pool: fix issue #5491 NoSuchFileException
9a2dea5446: macaroons: verifyCaveat should return false for unprocessed caveats
b970c780e8: rpm: make dcache package depend on java–11
c47ffcf8b9: systemd: add option to enforce systemd
fa4f6851fc: [maven-release-plugin] prepare for next development iteration

Release 6.2.0

An optional telememetry service is added to collect dCache statistics by dCache.org. The telemetry cell has to be explicitly defined in a layout file as well as enabled:

[telemetryDomain]
[telemetryDomain/telemetry]
telemetry.cell.enable=true
telemetry.instance.site-name=dCache instance on \${host.name}
telemetry.instance.location.latitude=53.5772
telemetry.instance.location.longitude=9.8772

The dcache version, online capacity, site name and optional location will be sent to the stats.dcache.org collector.

Starting from version 3.2.0 dcache domain processes can be managed as systemd services. However, this functionality was available for debian-based systems only. With version 6.2.0 systemd integration is added into RPM packages as well. The Debian based installations must stop dcache.service manually before installing new package. Please check the documentation for further information. To support RHEL–6 based systems, the legacy sysV-like daemon still available.

Billing

The data from the mover info message and door request message has been combined into a single message when reporting to the kafka server. Kafka receives a request message type with mover info integrated, if it exists. A failed door request or a remove request (which also uses DoorRequestInfoMessage) will simply have a null mover message field.

Chimera

The minimal PosgreSQL version is changed to 9.5. A new table t_xattr (inumber, key, value) is introduced to store user defined extended attributes.

NFS

The embedded NFS server publiches itself as 4.2 protocol compatible.

Each NFS door gets an unique identifiert that becomes a part of nfs state ids. This prevents unlikely event of nfs mover conflicts. The unique id for each door is stored in zookeeper as:

/dcache/nfs/doors/<cell>@<domain>/state-handler-id

The NFS transfer service was returning a multipath address list to nfs clients that contains duplicate entries. This was triggering kernel warnings like:

[Di Apr 23 13:50:01 2019] RPC:   addr 131.169.a.b already in xprt switch
[Di Apr 23 13:50:01 2019] RPC:   rpc_clnt_test_xprt failed: -98 addr 131.169.a.b not added
[Di Apr 23 13:50:02 2019] RPC:   addr 131.169.a.c already in xprt switch
[Di Apr 23 13:50:02 2019] RPC:   rpc_clnt_test_xprt failed: -98 addr 131.169.a.c not added
[Di Apr 23 13:55:00 2019] RPC:   addr 131.169.a.d already in xprt switch
[Di Apr 23 13:55:00 2019] RPC:   rpc_clnt_test_xprt failed: -98 addr 131.169.a.d not added

This is fixed now.

The discovery of an NFS4 domain, which is used by idmapper, is made more robust against invalid configurations of the dns resolver. Improved throughput of parallel opens.

Pool

Pools that use BerkeleyDBReplicaStore have impoved startup time by reducing the number of stat requests to the underlying filesystem.

SRM / SRM Manager

Removed hidden possibility to generate perfomance HTML reports and plots.

Telemetry

An optional telememetry service is added to collect dCache statistics by dCache.org.

Transfer Manager

WebDAV

A Webdav door can pass xattr. prefixed url params as extended attributes to a pool. For example:

    $ curl --upload-file file.txt \
        https://localhost:2881/tape/file.txt?xattr.key1=value1&xattr.key2=value2"

This information will be available to an HSM as part of the storage info.

XRootD

TLS can now be enabled for connections to both door and pool. Please read the documentation in The Book for details (https://dcache.org/old/manuals/Book–6.2/config-xrootd.shtml#tls). There is also further information in the xrootd.properties file.

SciToken support has been added to the xrootd door. Please read the documentation in The Book for details (https://dcache.org/old/manuals/Book–6.2/config-xrootd.shtml, under “SciTokens”). There is also further information in the xrootd.properties file.

When using TPC with GSI, the option to fall back to generating a proxy from the host certificate or to use a proxy provided at a given path instead of a delegated one, can cause issues if third-party source endpoints do not have a mapping for the proxy DN. Before 6.2, this fallback was automatic, but with 6.2 it can be controlled via the property

    xrootd.gsi.tpc.delegation-only=true

Turn on fallback (set this property to false) only if you are certain all potential endpoints will recognize the alternate DN.

As a reminder: we have also changed the dCache door to behave like the xrootd server and create missing directories (see [229210209c]); the default pool plugin has also been changed to unix (rather than undefined), as this is needed for TPC with dCache and EOS sources (but is otherwise ignored): see [a5482ad5f2]).

Zookeeper

The ZooKeeper ensemble which is used by dCache is required to be of version 3.5. Version 3.4 will no longer be supported.

Cleaner

The Cleaner cell may be run in high availability mode. Several Cleaners negotiate leadership via ZooKeeper and only the leader is active at any point in time. When the leader disappears, a new leader is elected automatically.

Changelog from 6.1.0 to 6.2.0

fa4f6851fc: [maven-release-plugin] prepare for next development iteration
a732fcb967: [maven-release-plugin] prepare release 6.2.0
70caee8202: archetype: configure maven-compiler-plugin to use java11
2621b05455: pool: do not fail transfer if RFC3230 is badly formed.
5b3888fb87: nfs: don’t attempt to select online-only pools when file on HSM
948217aebb: cleaner: add support for db password file
70bff6adb5: util: fix pgpass support for jdbc urls with options
8eff87bec9: pool: rework interrupt processing on p2p
eaada9757f: systemd: convert dcache.service into dcache.target
a5cb11ad1b: [maven-release-plugin] prepare branch 6.2
9df0bd2330: change <tag>
e5f3fe470c: [maven-release-plugin] prepare branch 6.2
ae91987797: libs: use nfs4j–0.21.x
c52d53cd73: Revert “libs: use nfs4j–0.21.x”
fcb4a168f0: libs: use nfs4j–0.21.x
a1d8b9abab: gplazma: automate filtering of principals
b4fe9166f4: common: annotate principals based on their role during login.
6a79ecb4ce: pnfsmanager: add the “show list activity” command
05a910fae8: common: add support for sections in ColumnWriter
985a43c2cb: dcache-bulk: fix several issues with request storage
92fca35216: dcache-chimera: Show that Cleaner service is replicable
a79fd7c12b: systemd: print deprecation warning only on systemd capable systems
f22003dbe9: pool: add Account wrapper that checks file capacity usage
4af94c84cb: docs: UserGuide describe extended attributes in frontend
87a6fbe54d: dcache-core: Move LeadershipListenerGroup for reusability
86ce323540: systemd: drop dependency on zookeeper service
9cde8bf80b: docs: UserGuide fix .html -> .shtml extension in links
eab0b80276: dcache-core: Move HAServiceLeadershipManager for reusability
17654d4318: docs: Add common chapter on extended attributes
065c6d3042: dcache-bulk: restructure test jobs
4105665d8d: dcache-bulk: add ping service for testing waiting state
4d273857b0: dcache-bulk: modifications to bulk jobs
c45da26e27: common: do not log bearer token values.
16f56cba36: frontend: add support for extended attributes
b751c39b14: pnfsmanager: avoid NPE if file is deleted during directory listing
4ace8e1e16: dcache-chimera: high available cleaner
5dfef491ce: rpm: stop dcache service before update and disable on uninstall
5fb54f364f: systemd: export PATH variable in the dcache-generator
8fcde432fb: rpm: remove pre-historic rpm support
a5e4f7adbe: pool: rework interrupt processing in AbstractMoverProtocolTransferService
ca257e79f5: rpm: don’t use RPM’s native systemd macros to stop dcache
d7de1f0030: rpm: add systemd target and generator
8e2a926545: rpm: drop pre-historic package compatibility
96101cdc9f: docs: UserGuide update WebDAV to describe xattr on upload
d205d6ba19: src: don’t use deprecated Throwables#propagate(Throwable throwable)
a5688b1745: srm-server: use URLDecoder#decode(String s, Charset cs)
300fd1c8f1: docs: fix issues with WebDAV chapter in the user-guide.
e06f4ef029: cells: fix stacktrace dump if cell takes too long to shutdown
ee4de45062: docs: describe how WebDAV properties may be used with extended attributes
61cdbebfc8: webdav: add support for extended attributes
de40f748db: Revert “chimera: use hazelcast cache for pnfs id to inumber mapping”
7765e99f6f: poolmanager: fix tab usage in RendezvousPoolManagerHandlerTest.java
2243d6d696: common: fix LoAPrincipal deserialisation
f0dbff3d1c: frontend: events make backlog configuration
834b5702bd: webdav/pnfsmanager: fix problems with xattr assignment on file creation
8dd7ea6fa0: pnfsmanager: support storing and querying FileAttribute.XATTR
71f957c4fa: core: update PnfsHandler to support extended attributes
ff1b5e74ac: pool: shutdown NettyTransferService implementations asynchronously
b3b421ae31: dcache-xrootd: bump to xrootd4j 4.0.3
be5de06f6b: pool: clean interrupted state after mover complete (again)
9caf6d7e9c: chimera: allow default durations for ‘pin’, ‘stage’ and ‘bringonline’ dot commands; add ‘unpin’
0ef6d91fe2: Update srmmanager.properties
905e39e281: Update gplazma.properties
ac6dbc3bdf: Update pnfsmanager.properties
c96eee8e50: Update spacemanager.properties
74cfffa038: Update pinmanager.properties
f1857d3c91: Update topo.properties
db42b3f9f1: pnfsmanager.properties: fix typo in comment
465f6cb4f1: pinmanager: use PoolManagerStub when talking to pool manager
3f48856942: cells: make ThreadGroup dump a single log message
a3de7c09f6: dependencies: exclude spring’s implementation of Jakata Common Logging
8697edac15: gplazma2-argus avoid shipping dCache with commons-logging
17360de1a1: pool: avoid instantiating http server twice
d60801bfee: pool: shutdown xrootd TPC event-loop group asynchronously
af7878c447: build(deps): bump guava from 24.1-jre to 24.1.1-jre
e3680d2874: build(deps): bump xercesImpl from 2.11.0 to 2.12.0
6b9e505616: build(deps): bump version.netty from 4.1.45.Final to 4.1.50.Final
b5f12700a4: dcache (pool manager): override suspend when “excluded” (hosts) is not empty’
0132c7aca4: pool: clear mover thread’s interrupted state before returning to pool
784b9b27a8: nfs: fix door startup crash on DNS error
9ee78c3fae: Doc bug: rm ls shows remove queue, not store queue
17d542576a: pool: avoid duplicate IP addresses in nfs multipath array
1ebad872c2: pool: use Long#compare to sort cache entries
b2317d35c1: nfs: remove redundant methods from PoolDeviceMap
7832755ce9: The Book: fix typo in config-nfs
81f4d682d0: dcache-xrootd: cancel TPC transfer when client disconnects unexpectedly from pool
ae8354b532: core: support extended attributes in RemoteNamespaceProvider
65286ee10b: dcache-xrootd: fix ssl handler context loading when TLS is not enabled
ebeb9076e9: pool: check compare fileSize from namespace and fileSize on disk before starting Mover
b7712bc23e: telemetry: new cell to collect usage data
504a729931: PnfsManager: implement support for extended attributes
b37b9850d8: srm: remove rrd based stats counter and gauges
d322e8800f: pool: repository account capacity correctly if file channel closed
2231b12e45: dcache-xrootd: make ‘tried’/excluded hosts an optional feature
16a42a8a78: cells: fix logger configuration before SystemCell is created
9891b06108: src: suppress VA_FORMAT_STRING_USES_NEWLINE warning
1c4e0e9fd9: logback: remove special zookeeper–3.4 workaround
0abb24a601: namespace: add support for extended attributes
a73d71d4cb: chimera: update FileSystemProvider so listing xattr returns a Set
559746c786: pool:add property je.freeDisk
fa52f1b15d: pool: remove historic VspDataOutputStream
6adab4c292: pool: add file’s PNFS-ID to capacity accounting
86e7b84f3a: pool: avoid unnecessary pool capacity account modification
9212eb3116: pool: deleting empty files need not update pool capacity account
7c3fa201c7: docs: document maven 3.5.0 requirement
db6c1cd3e5: pom: require maven version 3.5.0 in maven-enforcer-plugin
d4d16e5147: pool: fix space miscalculation for partially uploaded files
4da5e01bf4: Fix Strings#toThreeSigFig for negative numbers
20be71cbce: pool: remove boolean ‘force’ argument from sendPoolManagerMessage
bfa76eca68: system-test: updated required maven version in README
79fe6a15a7: docs: removed typos in system-test README
97d64c8c58: docs: added embedded db connection info
4618e5c8ff: docs: TheBook update index to include the chapter on Kafka
122c15ec33: pool: don’t use RequestExecutionTimeGuage to measure startup time
49f17c89f6: dcache: rework QoS Transition Engine synchronous mode
f34c0700d6: dcache-core: embedded ZooKeeper may start without existent snapshots
c04aa4336a: shell: update chimera shell to support extended attributes
1f8abae6dd: poolmanager: fix output of ‘rc ls’ command
45e4221b12: pool: pool restart optimization
58dd557411: pool: don’t use hard-coded time constant to query nfs door about stall clients
9c35cacf27: Revert “dcache-core: Fixed exception thrown in admin interface”
f509dd6b67: cells: prettify outputs of zk admin commands
a5482ad5f2: pool (xrootd): make tpc security plugin default unix
dac76a51c9: dcache-core: Fixed exception thrown in admin interface
6a34873492: docs: formatting for better readability
60e3de4dbf: util: update PnfsId#compareTo to use Arrays#compare
0023f22347: pool: don’t wrap with optional a nonnull annotated objects
60d56c6af5: docs: remove duplicate system requirements
858a313e7b: libs: update to zookeeper 3.5
651c8e7567: Updated system requirements
8d46aa7c3b: src: end of col.toArray(col.size()) vs col.toArray(new T[0])
9980df241f: docs: add missing architecture diagram
721776a80c: src: simplify Collection->stream->toArray(T[]::new)
5bbb74d3c0: libs: update to pg-jdbc 4.2.12
48049b525d: srm: remove historic ant build script
dc2dd74062: docs: UserGuide fix missing back-tick
a4d62f40ea: frontend: honour door and user root for inotify subscriptions
89dc2d3800: frontend: refactor inotify client-path to dCache-path mapping
98169977ff: frontend: add SelectionContext for event subscription
e309156938: nfs: don’t use CDC in try-with-resource block
3045e80cbb: nfs: restore transfers debug context on mover shutdown
a899884dd7: common: annotate in which dCache version Principals were added
f011a98209: pool: drop Authorization HTTP header on redirected
8e3ecb3ab1: util: don’t call String#toString in Transfer class
179d5e7bd0: cells: fix links in javadoc of CDC
80a905a01a: login-broker: fix ‘lb set update’ command
4ec1f835d3: nfs: close directory stream after use
f1d61f2a2e: chimera: remove unused DirectoryStreamHelper#listOf
0030d66fc7: nfs: use TreeSet when directory listing is created
ed48aa21ba: chimera: make DirectoryStream java8 friendly
2930138d85: nfs: fix missing place holder in log a message
229210209c: dcache-xrootd: always create missing directories on write
c9a685a184: dcache-xrootd: enforce failure when clients without TLS attempt TPC where it is required
d348c76d2c: dcache-frontend: remove retry flag on sendAndWait to history service
8e156be0db: src: don’t use guava’s Charsets in favor of StandardCharsets
0c155981dd: dcache-bulk (05): add request storage (in-memory/disk)
086bcc2c40: dcache-bulk: don’t overwrite BulkJob state
39d63ed78b: nfs: add client store for server reboot recovery
48b0c2d19d: enstore: populate the correct storage info information on set
b511d7b453: dcache-bulk: copy paste error in BulkRequestArgumentDescriptor constructor*
5bef4b8301: dcache-parent: Use Objects#requireNonNull instead of guava’s alternative
cc5ee9b3d1: pom: fixed typo in maven-compiler-plugin java version
4e414e6dcb: dcache-bulk: add BulkJobArgumentDescriptor
dfcd446cfe: dcache-bulk: fix two bugs in the TargetExpansionJob
bbc04fa94f: src: revisit ‘java 9+’ marked todos
8531c4e2e5: dcache: restore synchronized on two methods in QoSTransitionEngine
1072059639: pom: enable modernizer plugin to help java version migration
99f2fe49bb: dcache: fix QoSTransitioEngine logic
56f9c75d3c: pom: dont use deprecated options for maven-compiler-plugin
d163c1a33a: dcache-resilience,dcache: move utility class to pool manager package
6aa4553e0a: docs: describe dependency on java–11
e25a93613b: docs: specify dependency on java11
11799e7271: libs; use nfs4j–0.20.2 with minor bugfixes
da9e15a353: nearline-storage: remove disk copy if restore from HSM failed
00b40161c8: pom: use property agsLine
051dd05ba8: pom: require Java–11 at compile and run time
986abe48a7: hsm: fix missing closing bracket in error message
6f81aff617: tag: unify snapshot package name with deb and rpm
f8f9979586: system-test: use CopyNearlineStorage as configure HSM
e137380001: chimera: make PostgreSQL 9.5 minimal supported version
311e45ad01: dcache-bulk (04): add job store
fba82bb73b: dcache-bulk: fix possible NPE in completion handler
311a70e39f: dcache: give QoSTransitionEngine a synchronous option
550860147e: chimera: fix broken commit b44fa4e
890e156311: chimera: fix broken commit b44fa4e
babcbad1a2: chimera: fix broken commit b44fa4e
7764dd82b3: chimera: fix broken commit b44fa4e
b44fa4e85e: chimera: introduce postgresql 9.5 friendly f_create_inode95 procedure
f9c4cfb148: dcache-bulk (03): define single target job providers and factory
71d74bc230: dcache-bulk (02): define multiple and single target jobs
e14905061e: dcache-bulk: (1) add module, base job, completion handler
a4b8ba78c8: nfs4: add xattr support
782fcead53: doc: add README.md to system test with configuration description
1a480ecde4: chimera: optimize FsSqlDriver#setXattr for postgres
56620b06a7: chimera: add extended attribute support
658233b128: libs: use nfs4j–0.20.1
0b57f11a33: chimera: fix postgres version discovery
639cc34212: dcache-xrootd: add SciToken plugin
4e8c33b382: dcache-xrootd: add TLS support to door and pool
2354a29eda: dcache-xrootd: add delegation-only GSI property
dbdb7f6e75: dcache-xrootd: eliminate try-catch block (InterruptedException no longer declared)
dec851bce5: chimera: fsstat should handle big file systems
1ba70abfc2: dcache-frontend: make ErrorResponseProvider return the more specific error message
b5f9e5d2e4: dcache: qos migration policy engine should not raise JVM error when no tape pool found
0d1cdd7964: dcache-frontend: request file attributes explicitly
a7a940bc0c: dcache: send merged mover and door info to kafka as request
dba949e0ac: sweeper: make sure the histogram object is properly configured
01b7061ba4: namespace: accept XATTR as supported attribute
933cedb4f3: The Book: Update documentation about how to use maven archetypes
1a13e26772: dcache-history,dcache-frontend: guard against unconfigured sweeper histogram
68e44b8b24: pom.xml: sleepycat version update
a1080dd2cc: wendav: add url param based xattr support
5fc1d8a813: vehicles: add FileAttribute#XATTR to represent extended attributes
28cb37a59d: common: add Xattrs to handle extended attributes
f1a55a3a9b: message: cells call of message toString() as part of CellMessge.toString()
bf147085e3: dcache-frontend,history: protect against missing highest bin in histogram data
f0aba8f424: dcache-frontend: allow pool enable/disable to use boolean JSON value
c7bd1a5942: Documentation: Update webdav write and read url
8a2d5c991e: ftp-client: Remove whitespaces in blank lines and line endings
b864ff1d33: nfs: give each door a unique state handler id
08298eeafc: frontend: events inotify decouple sending events from message queue
a77a06bd14: frontend: fix memory leak in long-running SSE connections
20e3c76dce: libs: use nfs4j–0.20.0
70987bc53e: [maven-release-plugin] prepare for next development iteration

What’s new in dCache 6.2 Release notes

Highlights

Incompatibilities

Note:

Acknowledgments

Release 6.2.49

gplazma2-grid

Changelog 6.2.48..6.2.49

Release 6.2.48

xrootd4j

Changelog 6.2.47..6.2.48

Release 6.2.47

frontend

pool

Changelog 6.2.46..6.2.47

Release 6.2.46

debian

info

pool

Changelog 6.2.45..6.2.46

Release 6.2.45

dcache

dcache-xroot

frontend

xrootd

Changelog 6.2.44..6.2.45

Release 6.2.44

Release 6.2.43

pnfsmanager

pom

Changelog 6.2.42..6.2.43

Release 6.2.42

dcache-chimera

frontend

Changelog 6.2.41..6.2.42

Release 6.2.42

dcache-chimera

frontend

Changelog 6.2.41..6.2.42

Release 6.2.41

frontend

ftp

pool

Changelog 6.2.40..6.2.41

Release 6.2.40

common

pool

webdav

Changelog 6.2.39..6.2.40

Release 6.2.39

Pool

TransferManager

Changelog 6.2.38..6.2.39

Release 6.2.38

gplazma

loginbroker

pnfsmanager

webdav

Changelog 6.2.37..6.2.38

Release 6.2.37

common-security

webdav

Changelog 6.2.36..6.2.37

Release 6.2.36

dcache

gplazma

pool

webdav

Changelog 6.2.35..6.2.36

Release 6.2.35

dcache-core

frontend

gplazma

pinmanager

resilience

storagedescriptor

webdav

xroot

xrootd4j

Changelog 6.2.34..6.2.35

What’s new in dCache 6.2
Release notes