Highlights

  • Improved pool startup time with BerkeleyDB medadata repository
  • Basis support for extended attributes
  • Java 11 runtime required
  • TLS support for xrootd
  • SciToken support with XRootD
  • New telemetry service
  • Running dcache domains as systemd services for RPM based distros
  • The Cleaner cell may be run in high availability mode

Incompatibilities

  • Minimal PostgreSQL version is 9.5
  • From now on dCache requires Java 11 for runtime
  • dCache now requires ZooKeeper version 3.5 > NOTE: the ZooKeeper server configuration format has changed, the old one will lead to logged warnings in System cell on startup: Invalid config event received:
  • The systemd dcache.service replaced by dcache.target
  • Dropped compatibiity with EGEE rpms

Note:

By increasing the required Java runtime version to 11, a new set of warnings is logged on startup, such as:

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by [...]
WARNING: Please consider reporting this to the maintainers of [...]
WARNING: Use --illegal-access=warn to enable warnings of further illegal
reflective access operations
WARNING: All illegal access operations will be denied in a future release

They are generated by third party libraries which use a type of reflection that will no longer be supported in future Java releases. These libraries are expected to be updated in the future. This does not affect functionality.

Acknowledgments

(tbd, this section uses markdown formatting)

Release 6.2.47

frontend

Anonymous access receives a 401 error and doesn’t trigger a stack-trace when unlimitedVisibility is set to false.

pool

A bug is fixed where the checksum calculation would fail for empty files.

Changelog 6.2.46..6.2.47

c7eb0eb98f
[maven-release-plugin] prepare release 6.2.47
ae1901aaa6
dcache-frontend: protect against RuntimeError in case of denied anonymous access
433c089118
pool: don’t treat an empty file as a sparse file
543750af21
[maven-release-plugin] prepare for next development iteration

Release 6.2.46

debian

Added explicit dependency on rsyslog for debian packages so there are no errors during installation.

info

The delay between messages sent by info was increased to reduce log messages of “DGA {…} triggering too quickly”.

pool

If the scrubber state file is empty, scrubber will now log a more helpful error message and proceed scrubbing the pool.

Changelog 6.2.45..6.2.46

b2e85ba189
[maven-release-plugin] prepare release 6.2.46
8bc07b2d64
pool: improve handling empty saved state by scrubber
3d291c54d1
dcache-xroot,pom.xml: bump xrootd4j to 4.3.1
335c2b8d01
dcache-xroot: flesh out channel inactive and exception caught
7a2a773261
info: increase delay between messages
43fc1560a8
deb: add explicit dependency on rsyslog package
e2001056ca
[maven-release-plugin] prepare for next development iteration

Release 6.2.45

dcache

Improved documentation for kafka Producer properties.

Improved error message for inexistent statistics path.

dcache-xroot

The current relase Updated master xrootd4j to 4.3.0.

frontend

NPE is fixed when SRR when space information is not available yet.

xrootd

Rendezvous TPC without requiring a JWT token to be passed by the third-party client is possible (again).

Changelog 6.2.44..6.2.45

ce13ea9f9c
[maven-release-plugin] prepare release 6.2.45
5356036597
dcache:improve documentation for kafka properties
b7816f31bc
added missing import statement to XrootdRedirectHandler
cc9fe9c458
dcache-xrootd: fix TPC rendezvous to work with token authorization
9074653311
frontend: fix NPE in SRR when space information is not available yet.
4d87218ff8
dcache-xroot: update master xrootd4j to 4.3.0, and stable branches to next minor version
552471dd34
dcache-core: improve error message for inexistent statistics path
298769c978
[maven-release-plugin] prepare for next development iteration

Release 6.2.44

Technical release

Release 6.2.43

pnfsmanager

Doors no longer allow a user to discover whether or not a file or directory exists within directories they cannot access.

pom

Updated spring libs to version 5.2.20, CVE–2022–22965

Changelog 6.2.42..6.2.43

ceb303f5e6
[maven-release-plugin] prepare release 6.2.43
f45fce2e15
pnfsmanager: avoid leaking whether or not file exists
1294a04939
pom: update spring libs to 5.2.20
dd3d12ba71
[maven-release-plugin] prepare for next development iteration

Release 6.2.42

dcache-chimera

Cleaner batch delete exception has been fixed.

frontend

A bug is fixed that inverted the property frontend.srr.public=true. True means now allow public access and false does not.

Changelog 6.2.41..6.2.42

856047ab99
[maven-release-plugin] prepare release 6.2.42
e6029f78a4
nearline-storage: fix request count on error path in AbstractRequest#failed
c486249071
nearline-storage: fix queued count when request initialization falied
15916a5494
pool: add bunch of unit tests for NearlineStorageHandlerTest
b341519c19
frontend: fix problem with making SRR resources public
3b9988ac97
dcache-chimera: fix cleaner batch delete exception
3035152b24
[maven-release-plugin] prepare for next development iteration

Release 6.2.42

dcache-chimera

Cleaner batch delete exception has been fixed.

frontend

A bug is fixed that inverted the property frontend.srr.public=true. True means now allow public access and false does not.

Changelog 6.2.41..6.2.42

856047ab99
[maven-release-plugin] prepare release 6.2.42
e6029f78a4
nearline-storage: fix request count on error path in AbstractRequest#failed
c486249071
nearline-storage: fix queued count when request initialization falied
15916a5494
pool: add bunch of unit tests for NearlineStorageHandlerTest
b341519c19
frontend: fix problem with making SRR resources public
3b9988ac97
dcache-chimera: fix cleaner batch delete exception
3035152b24
[maven-release-plugin] prepare for next development iteration

Release 6.2.41

frontend

The property frontend.srr.public=true|false is added to frontend to control SRR resource restriction. The default value is false.

ftp

The FTP cell is killed asynchronously now after the session terminates so the door will remain responsive when many clients terminate their FTP session concurrently.

pool

HTTP-TPC transfers now prefer IPv6 address, if both endpoints support it.

Changelog 6.2.40..6.2.41

3d046adaea
[maven-release-plugin] prepare release 6.2.41
e17c43ee26
frontend: make srr resources public
9a18970585
ftp: kill cell asynchronously
53a22e79ec
pool: http-tpc prefer IPv6 address
cc2b7dc69a
[maven-release-plugin] prepare for next development iteration

Release 6.2.40

common

A bug was fixed where dCache will no longer accept certificates issued by a trusted CA after that CA updates their CA certificate while keeping the public/private key-pair the same. This is typically done to change something in CA’s certificate.

There are places where dCache configuration requires an absolute path. In some places attempting to use a relative path will yield an unhepful null in the log file. This is now fixed, so a meaningful error message is logged instead.

pool

An NPE on attempt by client to read from a broken replica is fixed now.

webdav

The current release fixed handling of webdav.authz.allowed-paths so it no longer triggers a NullPointerException.

Changelog 6.2.39..6.2.40

d0005970a3
[maven-release-plugin] prepare release 6.2.40
693b91ff3b
pool: fix NPE on attempt by client to read from a broken replica
0e635deadc
webdav: fix NPE if ‘webdav.authz.allowed-paths’ disallows a request
d7a6ce1c65
libraries: upgrade version of canl
86e2494cf2
common: provide reasonable error message if path is not absolute
a8cc3908f1
[maven-release-plugin] prepare for next development iteration

Release 6.2.39

Pool

The current relase fixed a problem where a file’s tape location that used to exist in the namespace and was subsequently removed can reappear if the file made precious and flushed to tape.

TransferManager

TransferManager now logs bugs with the corresponding stack-trace, making fixes any such bug easier.

Changelog 6.2.38..6.2.39

dda1e54814
[maven-release-plugin] prepare release 6.2.39
89add8b061
pool: only send new URLs when flushing file
2d55238b5d
TransferManager: log bugs with corresponding stack-trace
10cc03a78b
book: Add chapter about telemetry cell
415450e545
[maven-release-plugin] prepare for next development iteration

Release 6.2.38

gplazma

dCache will now reject WLCG-AuthZ-JWT profile tokens with a ‘wlcg.ver’ claim that it does not support. Other tokens (e.g., SciTokens) are unaffected by this change.

loginbroker

This patch fixes SRM based upload or download where the client requests an xroot-based transfer. The SRM door will now also consider any xroot door with xrootd.security.tls.mode configured to OPTIONAL when building a TURL targeting either xroot or xroots protocols.

pnfsmanager

PnfsManager now has two commands, ‘reset chimera stats’ and ‘reset stats’, to support resetting the gauge and counter statistics available through the ‘info’ command.

dCache no longer leaks information about whether or not files exist when using macaroons or Scitokens / WLCG AuthZ JWT profile tokens.

webdav

The WebDAV door cannot send the HTTP response to some HTTP request if the client has already disconnected. dCache no longer logs an error that it cannot send the HTTP response.

A client that disconnects during a proxied HTTP transfer (GET or PUT) is no longer logged in the WebDAV door’s log file; instead, it is logged in the cell’s pinboard. The billing message is updated to make it clearer what went wrong.

Changelog 6.2.37..6.2.38

7ce49ad6a9
[maven-release-plugin] prepare release 6.2.38
af03ea4434
skel: bump logback to 1.2.10
af37119d08
pnfsmanager: add support for resetting gauge and counter statistics
42621fcea0
loginbroker: add support for multiple protocol families
7787b87148
webdav: better logging if client disconnects during proxied transfer
7effd2874c
webdav: don’t log an error if client disconnects before response sent
5b80b3a77f
PnfsManager: check restrictions before resolving path to PNFS-ID
b9e86fb9ca
gplazma: scitoken validate ‘wlcg.ver’ claim
a36f469d86
[maven-release-plugin] prepare for next development iteration

Release 6.2.37

common-security

A better error message is logged when attempting to use a password-protected credential: java.io.IOException: Error decrypting private key: the password is incorrect or the PEM data is corrupted.

webdav

The HTTP-TPC should have fewer failed transfers when faced with many short-lived transfers.

It’s now possible to access a summary of HTTP-TPC transfers and configuration information via the WebDAV door’s info admin command.

Changelog 6.2.36..6.2.37

6fdd3a905c
[maven-release-plugin] prepare release 6.2.37
ada24fcd35
common-security: prevent NPE on password protected cert
d625f40a4e
webdav: http-tpc update heuristics for failing transfer
f06d3fa1af
webdav: http-tpc provide status information in ‘info’ admin command
27da72f0ed
[maven-release-plugin] prepare for next development iteration

Release 6.2.36

dcache

The RemoteTransferManager now better describes the current state of a transfer.

gplazma

The scitoken gplazma plugin now supports extracting the sub and wlcg.groups claims in the same fashion as the oidc plugin.

pool

The current release fixed open queue flag when template defined by queue define class.

The current release fixed open queue flag when template defined by queue define class issue.

webdav

The WebDAV door’s http-tpc ls command now has the possibility to show the current state of the transfer. This may be useful diagnosing transfers spending a lot of time deciding on which pool to transfer should take place.

Changelog 6.2.35..6.2.36

0328238eec
[maven-release-plugin] prepare release 6.2.36
ec93df914d
RemoteTransferManager: update state description
abba94dfe7
webdav: http-tpc add “prep” duration to ‘http-tpc ls’ command
68a2218f06
webdav: http-tpc show whether transfer is queued on pool
d6ed8385da
webdav: http-tpc update ‘http-tpc ls’ command to include state
841af52850
glazma: scitoken add support for additional principals
8fcac9689f
srr: limit publisched endpoints to GLOBAL scope
599f1a341d
Revert “frontend: fix handling of SRR requests over IPv6”
e7a663a145
pool: adjust StorageClassContainerTest for dcache pre- 7.1
90206858a7
pool: fix open queue flag when template defined by queue define class
ae3916b98d
[maven-release-plugin] prepare for next development iteration

Release 6.2.35

dcache-core

The current release fixed NPE in TransferManager when no pool was selected before transfer is cancelled.

frontend

The current release fixed handling of SRR requests over IPv6.

gplazma

The scitoken plugin now supports OPs that publish their public keys without any corresponding ID (i.e., no kid value).

pinmanager

A more detailed pinmanager log entry is provided for pin requests that are caught in a retry loop, and eventually time out.

resilience

A runtime exception triggered by side effect of logging has been fixed.

storagedescriptor

The script for generating SRR records (from the info service output) has been updated. The field lastupdated has been adjusted to the correct name latestupdate.

webdav

A bug is fixed where HTTP-TPC PULL request can fail (under heavy load) with the downloaded file being deleted, but dCache reports the transfer as successful.

dCache now provides a faster respones to the HTTP-TPC client (typically FTS) should the door decide to fail a transfer.

The WebDAV door should be now faster at accepting new transfers and faster at handling transfer completions when handling many small transfers.

The HTTP-TPC support in the WebDAV door is now e more robust to high number of transfers finishing concurrently.

xroot

Now it is possible to support kXR_delete as a write request on the pool.

xrootd4j

Library updates for xrootd4j which contains fix for Unix Protocol.

Changelog 6.2.34..6.2.35

cd05a920c8
[maven-release-plugin] prepare release 6.2.35
f48b03a893
webdav: http-tpc improve throughput with short transfers
d4b48a7161
frontend: fix handling of SRR requests over IPv6
94d1af20cb
webdav: http-tpc don’t wait if door fails transfer
d158aaf301
webdav: http-tpc avoid resetting transfer state
11ed3886a8
dcache-core: fix NPE in TransferManager when no pool was selected before transfer is cancelled
9388fa02cb
resilience: fix runtime exception triggered by side effect of logging
84deca3d27
gplazma: scitoken add support for OPs that advertise keys without kid
5941661529
webdav: http-tpc move transfer finalisation off of message queue
435261c4ba
pom.xml: update xrootd4j dependencies to 4.2.5/4.1.6/4.0.11
04f2272903
CanlContextFactory: throw FileNotFound for missing ca certs dir
0097c289b1
pinmanager: provide more details information on timeout
f429b91125
storagedescriptor: fix element name to ‘latestupdate’
16f1aaf0fd
dcache-xroot: support kXR_delete as a write request on the pool
da1e8dabc3
[maven-release-plugin] prepare for next development iteration

Release 6.2.34

ftp

The FTP door now provides more succinct information on pinboard, should use less CPU and take better advantage of the available cores.

pnfsmanager

Attempts to create a file where the parent directory is either missing or not a directory now provide clearer error messages.

pool

The pool now provides more information for bugs reported as java.lang.IllegalStateException: Handle is closed.

webdav

The HTTP-TPC response is improved if a client attempts to pull a file into a non-existing directory, or attempts to use an existing file as an ancestor directory.

xrootd

Improved exception handling for Xrootd.

Changelog 6.2.33..6.2.34

dbf2413c8e
[maven-release-plugin] prepare release 6.2.34
df638f6c0b
pool: provide more information when handle is double-closed
b3fc7e18cd
dcache-xrootd: Alternate fix for client write to closed checksum channel
4e182448f8
pnfsmanager: fix confusing error message.
9d7dda31e7
webdav: fix error handling for bad paths
13cfb3a620
ftp: switch to more reasonable logging of Subject
6ef7512ab7
[maven-release-plugin] prepare for next development iteration

Release 6.2.33

universal-spring-cell

This patch fixes a problem where a call to api/PoolManager returned 404 not found. The JSON serialized object is returned now as before.

xrootd

The xrootd4j version was updated to 4.0.10

Changelog 6.2.32..6.2.33

f2e6b97ff2
[maven-release-plugin] prepare release 6.2.33
e992fa127b
pom.xml: bump to xrootd4j 4.0.10
56a103196e
universal-spring-cell: allow for serialization of Optional
83c1c32f4f
[maven-release-plugin] prepare for next development iteration

Release 6.2.32

billing

Billing will no longer throw a NullPointerException with certain information from an NFS door serving an NFSv3 client.

ftp

Now file sizes for LIST output in bytes is displayed.

The FTP door should be more robust against a (currently unknown) bug that results in the ftp session attempting to send progress queries to the pool after that is no longer possible.

gplazma

The banfile and scitoken plugins, and the two admin commands test login and explain login are updated to accept the username: prefix (e.g., username:paul). The prefix user: continues to work but is now deprecated.

pool

The current release improved error messaging if a setup file badly configures an HSM instance.

A bug is fixed where removing a create hsm statement in a pool’s setup file and running the reload command kills the pool.

xroot

The documentation concerning TLS properties has been fixed.

Changelog 6.2.31..6.2.32

0cbce3329c
[maven-release-plugin] prepare release 6.2.32
07b0d2977e
dcache-xroot, book, skel: fix documentation concerning TLS properties
5f2ebb2fe9
bootstrap: fix two problems with how bugs are reported on startup
a399cadf23
ftp: try harder to ensure any timer tasks are cancelled
31c80a16fb
pool: removing hsm with reload command kills pool
e49357d577
gplazma: switch from user: to username:
504eeda1ba
ftp: show file sizes for LIST output in bytes
99f40a20d2
util/frontend: change transfer rate to double
5fac1c3c5c
bootstrap: don’t hide bugs in ‘create’ command
06eed198e2
ftp: ensure cell is killed even if shutdown triggers a bug
f7f6f04770
billing: fix NPE in billing from NFSv3 message
2f683966e4
pool: improve error message on badly configured hsm instance
5e54ee71b5
[maven-release-plugin] prepare for next development iteration

Release 6.2.31

acl

The group ACEs now apply to desired group instead of to a users with the same numeric id.

gplazma

The WLCG Common JWT Profiles (v1.0) describes the wlcg.groups claim for expressing group-membership. We are increasingly seeing this being used as the OIDC equivalent to the group-membership expressed through the VOMS extension.

During the auth phase of the login process, the group membership information contained in a wlcg.groups claim from the OP is now available as OpenIdGroupPrincipal principals. Subsequent plugins (e.g., multimap) may be used to convert these principals to more directly useful principals.

webdav

dCache now responds with a 507 (Insufficient Storage) on GET request if dCache must do a pool-to-pool internal-transfer or stage the file but no pools have sufficient free capacity to support this.

Changelog 6.2.30..6.2.31

7b81d48510
[maven-release-plugin] prepare release 6.2.31
1732b4a2aa
all: reformat code using Google Style Sheet
7045c15fce
acl: set IDENTIFIER_GROUP flag if WHO is GROUP or OWNER_GROUP
17ef1d8a3a
webdav: return 507 if insufficient space on GET request
84e995f98e
gplazma: scitoken make ExemptFromNamespaceChecks principal optional
4f3fa2bd4c
gplazma/pnfsmanager: update namespace so scitoken ‘scope’ takes priority
669f19df79
[maven-release-plugin] prepare for next development iteration

Release 6.2.30

gplazma

During the auth phase of the login process, the group membership information contained in a ‘wlcg.groups’ claim from the OP is now available as OpenIdGroupPrincipal principals. Subsequent plugins (e.g., multimap) may be used to convert these principals to more directly useful principals.

pool

A rare NullPointerException for HTTP-TPC transfers is fixed.

Changelog 6.2.29..6.2.30

d372de5fa1
[maven-release-plugin] prepare release 6.2.30
8481729ace
gplazma: oidc add support for the ‘wlcg.groups’ claim
e664995cdd
pool: http-tpc fix NPE when monitoring network traffic
3d87507028
[maven-release-plugin] prepare for next development iteration

Release 6.2.29

frontend

SRR returns data now with official json schema

webdav

Fix how sym-links are shown in the static HTML (web-browser) view from the WebDAV door.

Changelog 6.2.28..6.2.29

41c3d128d1
[maven-release-plugin] prepare release 6.2.29
24d79fdd76
frontend: add service to provide Storage resource reporting
dc6442a9ae
webdav: update representation of symbolic links in HTML page
f5c36a56ea
[maven-release-plugin] prepare for next development iteration

Release 6.2.28

dcache-xroot

The next release of xrootd4j is used now, which improves handling TPC read requests.

namespace-chimera

There are less operations on the database now as the filesize doesn’t get updated after a file was flushed to tape.

pool, webdav

The webdav admin interface has been extended to include monitoring information about on-going transfers. The command can optionally show pool information if transfer-manager is update; updating transfer-manager is not required by this change. Percent transfer progress for PULL requests is optionally available if pools are updated; updating pools is not required by this change.

Changelog 6.2.27..6.2.28

6be706def4
[maven-release-plugin] prepare release 6.2.28
c6e80f9c4a
namespace-chimera: ignore file size update on file flush
b86f7dfcf7
webdav: add ‘http-tpc ls’ admin command
be2289ff80
pool/webdav: http-tpc add RemoteConnections perfmarker metadata
f1b2c262b0
dcache-xroot: bump dependency to next xrootd4j release
e57b3578cd
[maven-release-plugin] prepare for next development iteration

Release 6.2.27

pnfsmanager

Fixed a problem where dCache accepts a HSM script that, when called to flush a file, returns successfully but without providing any URIs on stdout. Such requests are considered failed, with the flush request blocked until retried explicitly with the queue activate PNFSID admin command.

pool

The Kafka message emitted when a file is flushed to tape or staged back from tape now includes the locations field: a JSON array of JSON Strings, where each JSON String is a tape-location URL. For flush operations, this is the (potentially empty, potentially multiple) list of URLs returned by the HSM instance. For stage operations, this is the (potentially empty, potentially multiple) URLs that match the selected HSM instance. Both messages now also describe a hsm JSON Object that contain the HSM type, HSM instance name, and HSM provider.

srmmanager

A race condition is fixed that, if triggered, results in a memory leak. This leak can also affect the TURLs returned by SrmManager, where out-of-date information about doors is used.

Changelog 6.2.26..6.2.27

ef08d9263e
[maven-release-plugin] prepare release 6.2.27
2ed4135f2a
srmmanager: fix race condition in LoginBrokerSubscriber
616fea3eba
pool/pnfsmanager: don’t accept a non-Enstore flush without locations
6528e45d8d
pool: update Kafka tape events to include additional information
9863ed9322
[maven-release-plugin] prepare for next development iteration

Release 6.2.26

alarms

Clearer documentation for billing/alarms services regarding HA requirements.

gplazma

dCache has a new session gPlazma plugin that is designed to be a viable alternative to the StorageAuthzDB and KPWD plugins but with shorter (hopefully less confusing) configuration, while also being more flexible.

To support the removal of authzdb plugin, we should provide an easy way for sites to migrate their existing configuration from using authzdb (as a session plugin) to omnisession.

Sites now have a utility to help them migrating from authzdb to omnisession.

Note: it is also possible to use the authzdb plugin as a mapping plugin. This utility does not help in migrating that use to some alternative solution.

To support the removal of authzdb plugin, we should provide an easy way for sites to migrate their existing configuration from using authzdb (as a session plugin) to omnisession.

Sites now have a utility to help them migrating from authzdb to omnisession.

Note: it is also possible to use the authzdb plugin as a mapping plugin. This utility does not help in migrating that use to some alternative solution.

skel

Whether a cell/service supports HA or not can now be derived from the properties file (except for doors).

Please see <name>.cell.replicable.

srm

The access log file for SRM requests now includes the client-supplied list of protocols, if any were provided.

Changelog 6.2.25..6.2.26

e01c39a7ba
[maven-release-plugin] prepare release 6.2.26
7d560e0015
rpm: add dcache-convert-authzdb-to-omnisession into list of packaged files
7252087e5c
gplazma: add util to convert authzdb file to omnisession
a78fc2594d
alarms, billing: specify shared rdbms in the case of replicated service
ed589449e9
skel: add explicit replicable property even when value is false
82815edb7b
gplazma: add omnisession plugin
fd9179fab4
srm: log transfer protocols in access log
02a6d29785
[maven-release-plugin] prepare for next development iteration

Release 6.2.25

dcache-core

The old web pages claim the transfer size and speed are reported in KB and KB/s, although the actually used units are KiB and KiB/s. Reporting the correct unit is important for monitoring. This is now fixed.

frontend

The current release fixed regression in generating swagger JSON for frontend’s REST API.

gplazma

dCache may now be configured so that the multimap oidc predicates match the sub claim value from a specific OAuth2 Provider using the format SUBVALUE@OP, where OP is the dCache-internal alias for the OP. Existing multimap configuration continues to work, but admin is warned to update the multimap configuration.

Changelog 6.2.24..6.2.25

63a0dcafe2
[maven-release-plugin] prepare release 6.2.25
fc0932484a
gplazma: multimap now supports OP in ‘oidc’ predicate
ec7329cb83
frontend: bump swagger version to 1.6.2
6458e8bb3c
dcache-core: correct reported units of transfer size and speed
f901019cda
[maven-release-plugin] prepare for next development iteration

Release 6.2.24

checksum

The policies map is now added to the JSON info object that is returned for checksum module information.

Changelog 6.2.23..6.2.24

4237fda233
[maven-release-plugin] prepare release 6.2.24
dfcb4a8221
checksum-module: add missing map to JSON info
25b46fa5d0
[maven-release-plugin] prepare for next development iteration

Release 6.2.23

xroot

The current release fixed the parsing to account for host/error pairs.

Upload transfer checksum failure is fixed and the gfal scenario now works.

xrootd

xrootd4j is updated to new version which fixed problem with netty LogHandler slowing down TPC client (cutting bandwidth by a factor of 10).

Changelog 6.2.22..6.2.23

30ea020a04
[maven-release-plugin] prepare release 6.2.23
1291969a99
dcache-xrootd: upgrade xrootd4j to 4.0.8
203624c23e
dcache-xroot: parsing of ‘triedrc’ CGI lacks separation by optional comma
cdd6404bea
dcache-xroot: fix upload transfer checksum failure (stable branches)
abe8a2b8e7
[maven-release-plugin] prepare for next development iteration

Release 6.2.22

ftp

Globus transfer agent is now able to create symbolic links.

nfs

The current release introduced nfs.clientdb.config property prefix to populate the BerkeleyDB configuration and now the nfs door is not picky about free disk space in /var/lib/dcache/nfs and the BerkeleyDB used as client store by nfs door can be configured with properties.

pool

dCacheView now works for redirected transfers for non-anonymous data access, provided the WebDAV door is using TLS encryption and is configured with ‘webdav.redirect.allow-https’ set to ‘true’.

poolmanager

A very rare race-condition is fixed that could be triggered if a pool completes a stage or pool-to-pool request (to satisfy a client’s read request) immediately before dying.

webdav

dCacheView is able to view and download files when authenticated and when the WebDAV door proxies the transfer.

Changelog 6.2.21..6.2.22

2b3c319d6d
[maven-release-plugin] prepare release 6.2.22
70b1a317d8
nfs: add possibility to configure BerkeleyDB client store
c8027ca0c0
webdav: use Jetty’s built-in support for CORS
f63ba858c8
poolmanager: fix _waitingFor access outside of synchronized block
2d7f00158c
ftp: add support for the SITE SYMLINK command
129d87a583
pool: http allow client to send credentials when TLS is used
5d3b1c0de5
dcache-chimera: use RemotePoolMonitor to discover file locality
7de3af9df4
[maven-release-plugin] prepare for next development iteration

Release 6.2.21

frontend

frontend/api readers and writers always zero issue is fixed now

gplazma

The LDAP plugin behaviuor now more closely follows that of other mapping plugins. This allows deployments where LDAP is tried first and, if that fails to identify the user, fall-back strategies are used.

pool

The pool size reporting when static/runtime config is not defined now is improved.

Changelog 6.2.20..6.2.21

f88fd1627e
[maven-release-plugin] prepare release 6.2.21
7ada2fef7e
gplazma: ldap throw exception if no principal is added
91e75edacd
poolV4: fix reader/writer counts
e29f378820
pool: remove invalid test case of configured space
8ffdfb69f7
pool: fix pool size reporting when static/runtime config is not defined.
e0bcc3f4e6
[maven-release-plugin] prepare for next development iteration

Release 6.2.20

dcache-xroot

The cirrent release fixed memory leak, race and other small issues with reconnect to pool.

frontend

The dcache-view.endpoints.webdav property now overrides any auto-discovered WebDAV endpoint, making explicit configuration easier.

pnfsmanager

A regression is fixed the prevents creating a symbolic link.

systemd

dcache-generator now successfully runs after restart.

xrootd

xrootd4j is updated to 4.0.6.

The current relase has updated to xrootd4j library to 4.0.6.

The correct subject is used to access the path for both GSI and SciToken protocols.

Changelog 6.2.19..6.2.20

489a117972
[maven-release-plugin] prepare release 6.2.20
2046a18e14
pool: make sure that RepositoryChannel when createChannel detects error
b4a55acb9b
update to xrootd 4.0.7
22f919abd0
dcache-xrootd: update to xrootd4j 4.0.6
fd9c6640ff
frontend: bump version of dCacheView
2a1a2c8bc0
pnfsmanager: fix NPE regression from 3dfed7e8b0
2fdc1d9894
dcache-xroot: fix memory leak, race and other small issues with reconnect to pool
e0b500e7b3
dcache-xroot: store most recent login subject in door
9d354f1fa1
frontend: bump dCacheView to v1.6.2
c7fa0f6870
systemd: fix generator run after reboot
3a41bdc6eb
[maven-release-plugin] prepare for next development iteration

Release 6.2.19

nearline-storage

The current release fixed store cancel regression.

pool

As in distributed system packet loss (due to timeout or re-connects) can’t be avoid, dCache retries many of requests. However, this is not the case for mover redirects. If a door did not receive reqirect information, then transfer will never happen. This is now fixed and re-send redirect if client is not connected within 5sec.

poolmanager

When poolmanager requests that a pool stage a file, the pool may return the special error code HSM_DELAY_ERROR (10013). On receiving this error, poolmanager suspends the request.

This is now fixed.

Changelog 6.2.18..6.2.19

780e7dcaad
[maven-release-plugin] prepare release 6.2.19
a53f00cca9
nearline-storage: fix store cancel regression
cb0388a2d2
pool: re-send redirect if client is not connected within 5sec
a1ad7f7e7a
poolmanager: fix NPE if pool delays stage without giving a reason
79ab2fd0f4
nearline-storage: introduce FlushRequest#getReplicaCreationTime
c1e5a4aa6b
[maven-release-plugin] prepare for next development iteration

Release 6.2.18

pool

There are several cases when pool scans the various hsm queus: on mover start, to check load assumptions, on heartbeat, to populate pool manager with load information, on info command, to report load numbers to admin interface of dcache-view collectors

As this is O(n) operation, in case of a large queues those scans might take seconds, which is unacceptable, especially wen starting a new mover.

This is now fixed and getting hsm start is O(1) and doesn’t impact pools operation

xrootd

Now the door will ignore xrootd.enable.tried-hosts property if property false instead of returning errror.

Changelog 6.2.17..6.2.18

e428eb796f
[maven-release-plugin] prepare release 6.2.18
d98a894414
pool: don’t scan flush/restore/remove queue to get stats
dddbab46f7
dcache-xroot: (door) ignore tried if property false instead of returning error
2a333645c0
[maven-release-plugin] prepare for next development iteration

Release 6.2.18

pool

There are several cases when pool scans the various hsm queus: on mover start, to check load assumptions, on heartbeat, to populate pool manager with load information, on info command, to report load numbers to admin interface of dcache-view collectors

As this is O(n) operation, in case of a large queues those scans might take seconds, which is unacceptable, especially wen starting a new mover.

This is now fixed and getting hsm start is O(1) and doesn’t impact pools operation

xrootd

Now the door will ignore xrootd.enable.tried-hosts property if property false instead of returning errror.

Changelog 6.2.17..6.2.18

e428eb796f
[maven-release-plugin] prepare release 6.2.18
d98a894414
pool: don’t scan flush/restore/remove queue to get stats
dddbab46f7
dcache-xroot: (door) ignore tried if property false instead of returning error
2a333645c0
[maven-release-plugin] prepare for next development iteration

Release 6.2.17

nfs

Fixes regression introduced in 6.2.15.

Changelog 6.2.16..6.2.17

598ec78fb7
[maven-release-plugin] prepare release 6.2.17
929340be36
nfs: fix access to path before namespace info available
243b3d4daa
[maven-release-plugin] prepare for next development iteration

Release 6.2.16

Changes affecting multiple services

Sometimes file checksums are missing in namespace. Transfer requiring checksum verification fails then with an error saying that “No checksums found”, but doesn’t log a stack trace server side anymore.

dcache-xroot

During periods of heavy usage on a pool node, if I/O stalls during read, the xroot client may try to reconnect and continue reading. This release fixes a bug which fails the transfer on the attempted reconnect with a “uuid no longer valid” error (because the file mover has already been closed and removed). Clients should now be able to resume reading the file on retry.

Changelog 6.2.15..6.2.16

ac73913b6b
[maven-release-plugin] prepare release 6.2.16
d4883ca2c3
dcache-xroot: Allow client to reattempt open on pool when I/O stalls
b4349ced2f
check if checksums are present and throw CacheException if they are not
41fc04fd04
[maven-release-plugin] prepare for next development iteration

Release 6.2.15

admin

The current release fixed doors login dump cache admin command to show the information sent by the door along with the login result.

common

Printing of bearer tokens is improved to provide better assurance that bearer tokens are not leaked.

gplazma

All login failures are now recoreded in pinboard as a single-line summary, explaining why the login attempt failed.

namespace

Now Path information available for all transfers.

webdav

The current release fixed a regression where dCache would complain about badly formatted IPv6 addresses, preventing those addresses from being included in the access log files.

The current release fixed a regression where dCache would complain about badly formatted IPv6 addresses, preventing those addresses from being included in the access log files.

xroot

The current relase updated xrootd4j to 4.0.5 which fixes improper use of destination token when contacting source server during third-pary transfer.

Changelog 6.2.14..6.2.15

5b5da688ad
[maven-release-plugin] prepare release 6.2.15
c32494919a
dcache-xroot: update to xrootd4j–4.0.5
8c6a6a96b1
webdav/frontend: fix IPv6 address logging
34ee7f95f4
common: update how BearerToken is shown in Subjects#toString
19cfc9d636
authn: add missing hashCode and equals methods for PasswordCredential
735fbe2210
authn: fix “login dump cache” admin command output
08b9944719
gplazma: provide single-line summary in pinboard for each login failure
827c6195e1
authn: log failures in UnionLoginStrategy
a18373b6be
authn: add missing hashCode and equals methods for BearerTokenCredential
881ea70cbb
common: Update BearerToken representation to include some information
3264f7b4e2
namespace: ensure that file’s path always set on GetAttributes
0d3261d6df
[maven-release-plugin] prepare for next development iteration

Release 6.2.14

gplazma

dCache now supports OPs that assert group-membership with eduPersonEntitlement claims (as described by AARC guideline G002). The multimap plugin may be used to map these to corresponding group/gid values

webdav

Restarting transfermanager while there are ongoing HTTP-TPC transfers no longer results in an endless loop, with transfermanager logging a message every ~5 seconds per ongoing HTTP-TPC transfer.

Changelog 6.2.13..6.2.14

315ad704aa
[maven-release-plugin] prepare release 6.2.14
8f289b6d0c
Revert “pool: do not update file size on flush”
96bcb14900
gplazma: add support for eduPersonEntitlement assertions
e3d68154eb
build(deps): bump version.jackson from 2.10.1 to 2.12.1
d430a76b9b
webdav-transfermanager: fail gracefully if transfermanager restarted
544829d5ce
[maven-release-plugin] prepare for next development iteration

Release 6.2.13

biiling

The current relase fixed unit info for bandwidth measurements in billing.properties (bytes/sec, not MiB/sec).

frontend

Frontend now logs any time it returns 500 Internal Server Error to the client.

dcache-view Previously displayed MB/sec for transfer rate, but the value is the old KB/sec. This is now improved and pages display the values advertised.

pool

Less memory consumption by pool, especially on startup.

The current release assures now that checksum scanner wiill mark files broken and sends alarm.

webdav

An HTTP-TPC transfer will abort automatically if transfer-manager is restarted, without waiting for the client (FTS) to time-out.

Changelog 6.2.12..6.2.13

c159dc6d54
[maven-release-plugin] prepare release 6.2.13
6c4faecd00
dcache-pool: Assure checksum scanner marks files broken and sends alarm
e4e0bae712
pool: do not re-read storage info when creating CacheRepositoryEntry
f17bfa3d4a
webdav/transfermanager: fail gracefully on restart, for perf marker
e644f58438
default: fix unit info for bandwidth measurements in billing.properties
daa8b11173
frontend: create a log entry when we return 500 Internal Server Error
98719cd6a3
pool: do not update file size on flush
7da96d813a
dcache-frontend: change transfer rate to compute what it advertised
660d88600b
pool: use soft reference to storage info object
27370e09e4
[maven-release-plugin] prepare for next development iteration

Release 6.2.12

book

A new section is added to the xrootd chapter explaining the need for SAN extensions on the hostcert in the case of hostname aliasing.

ftp

Globus transfer service can now list directories that contain incomplete files, those that are still being uploaded.

The FTP no longer leaking sockets when dynamically calculating file checksums (e.g., MD5). In practise, this issue only affects transfers via the Globus transfer service.

The current release fixed Globus transfer service directory listing for medium size directories.

The current release fixed Globus transfers hanging when verify file integrity after transfer is enabled and the pool is not configured to calculate an MD5 checksum.

Honour the ftp.net.internal configuration property for dynamic checksum calculation.

Changelog 6.2.11..6.2.12

3581765d10
[maven-release-plugin] prepare release 6.2.12
241581617b
ftp: do not leak sockets when calculating checksums dynamically
a1a0a167e6
ftp: add work-around to calculate MD5 for Globus
16afc1af6b
ftp: add place-holder file size work-around for Globus directory listing
5d46fbe63f
rpm take build directory path into quotes
c692f6da11
ftp: fix MLSC support for medium-length directories
825f517834
ftp: fix on-the-fly checksum calculation
3d592c7ea7
book: document host alias issue with xroot TPC
f66725dbd4
[maven-release-plugin] prepare for next development iteration

Release 6.2.11

nfs

The new version of nfs4j–0.21.1 fixed readdir reply size miscalculation for NFSv3.

pool

The current release fix a problem where, under heavy load, redirected https transfers result in garbled SSL information, preventing the HTTP client from working.

services

Recovery processe in the case where a pool’s message queue is overloaded, as old (and now useless) requests may be simply discarded is improved.

vehicles

The current release improved memory consumption on pools.

webdav

There were reports of the access log file missing entries. This problem seems to be triggered if the client disconnects before dCache has finished processing the request, which is certainly true for cancelled HTTP-TPC transfers. This is now fixed.

xrootd

xrootd4j-authz-plugin-alice is updated to 1.2.0.

The xroot protocol access log entries for the xrootd door and pool have been enhanced to include information about the kXR_query requests.

Changelog 6.2.10..6.2.11

ec8b1e67a8
[maven-release-plugin] prepare release 6.2.11
78040db522
prepare for next development iteration
3689629eb1
[maven-release-plugin] prepare release 6.2.10
01f71f5f59
TransferManager: choose better pool query message TTL
4597553c31
webdav/frontend: ensure cancelled HTTP-TPC transfers are logged
15fede73da
xrootd/pool: improve access log to record more details of kXR_query
0fff2d36db
libs: use nfs4j–0.21.1
42e840560b
vehicles: internalize common strings in StorageInfo
25bbf82fba
pool: sort flush queue only if we going to use the result
932055e31b
pool: https avoid sharing SSLEngine between TCP connections
04c96a851f
removing commit-msg file
395acb44d6
updated alice-plugin version to 1.2.0
851452f57d
pom.xml:update xrootd4j-authz-plugin-alice version to 1.2.0

Release 6.2.10

The current release fixes EGI-SVG–2020–16939 vulnerabilities.

Release 6.2.9

dcache

The log files now can be enforced even with systemd deployments.

dcache.log.destination property has been introduced which controls the log destination. The default value console preserves the current behaviour: systemd services log to journald, system-V services log to stdout (redirected to /var/log/dcache/.log).

The option file instruct logback to send message to the location defined by dcache.log.file property. The plain file FileAppender is used as dCache packages already comes with pre-defined logrotate configuration.

xrootd

The xrootd4j version was updated to 4.0.4.

Changelog 6.2.8..6.2.9

f8ffbdacd4
[maven-release-plugin] prepare release 6.2.9
88ef6a9fc5
logging: add an option to control logger destination
ec72bc0f61
pom: update to xrootd4j 4.0.4
64961d79a9
[maven-release-plugin] prepare for next development iteration

Release 6.2.8

common-auth

Fix regression introduced by limiting UidPrincipal to be only an output principal.

pool

The mover thread is not interrupted outside of IO activity to avoid disabling of repository.

HTTP-TPC transfers where dCache pushes data to an Apache server (for example as used by Dynafed) will now work for zero-length files.

webdav

HTTPS-PUT requests are now possible.

Changelog 6.2.7..6.2.8

464e3e0330
[maven-release-plugin] prepare release 6.2.8
452bc43016
pool: http-tpc do not loop if HEAD ‘Content-Length’ response is missing
7054e2791a
webdav:HTTPS PUT req issue
bcbb79f1ca
pool: preserve CDC on p2p transfer
1b76cac1b6
pool: http-tpc drop ‘Accept-Encoding’ on HEAD requests
f3dc5f36e7
pool: detach IO thread when mover finished
6cf85465f7
common-auth: mark UidPrincipal as In/Out principal
eb65fa3679
[maven-release-plugin] prepare for next development iteration

Release 6.2.7

pool

dCache pools now support GSI-based xrootd-TPC by default.

Changelog 6.2.6..6.2.7

fe2631db39
[maven-release-plugin] prepare release 6.2.7
1ded2cdb9a
pool: update xrootd-tpc authenication default
e932921c28
[maven-release-plugin] prepare for next development iteration

Release 6.2.6

nfs

If door failed to start a mover due to timeout or no route to cell, then pool selection or mover start is retried. However, as the retry procedure in the transfer class was not idempotent, the nfs door has it’s own workaround to avoid extra pool selection. This partial solution plays not well with retry logic of transfer class and produced inconsistent results. This is fixed now and lost messages to start mover successfully delivered after number of retries.

pool

The HTTP-TPC PUSH requests (which use the HTTP PUT request to send data) are now more robust against slow remote servers that redirect the transfer.

The current release fixed NullPointerException bug in MongoDB-backed pool metadata storage and SpotBugs NORMAL-level warning.

With the chenges introduced in current release, when using HTTP-TPC, the timeouts dCache pool uses when requesting information from the remote server have been adjusted, based on operational experience. In general, dCache will now wait longer for the remote server to complete any post-processing before giving up and failing the transfer.

Changelog 6.2.5..6.2.6

3c8b6b0e56
[maven-release-plugin] prepare release 6.2.6
1bcc96fbbd
chimera: update EnstoreStorageInfoExtractor to use FileState
f29ccdb158
chimera: use UTF–8 constant instead of string in ChimeraEnstoreStorageInfoExtractor
a24c79412b
pool: http-tpc adjust GET and HEAD request timeouts
c45529130d
pool: http-tpc PUT request are repeatable
7b710e4a16
nfs: drop internal mover re-start loop
31d869d32f
pool: support removing MongoDB storage-info entry
7dfbe630ce
[maven-release-plugin] prepare for next development iteration

Release 6.2.5

dcap

Sometimes file restores on dCap door were hanging. This is now fixed.

Changelog 6.2.4..6.2.5

8629039c76
[maven-release-plugin] prepare release 6.2.5
d94a5f5a06
[maven-release-plugin] prepare for next development iteration
efeb443cbe
dcap: fix automatic door retry on transient errors

Release 6.2.4

gplazma

X.509 certificates that use the “SHA–384 with RSA” algorithm are now logged with this name instead of the raw OID value.

httpd

The current release fix pool usage color scheme css used for UsageInto and PoolInfo pages.

pool

dCache pools that are undertaking HTTP-TPC transfers will now wait longer for the remote servers to complete any post-transfer activity. Transfers that involve heavily loaded remote servers are now more likely to succeed.

Changelog 6.2.3..6.2.4

1e55f3b3ff
[maven-release-plugin] prepare release 6.2.4
317ce07c20
fix error, tag 6.2 instead 6.2.3
f485ddc295
srm-common: make X509 client authn optional, support bearer tokens
7f6978aae7
gplazma: include additional algorithm in LoginResultPrinter
20a7493253
pool: http-tpc increase timeout waiting for remote server post-processing
819d18a334
httpd: fix pool usage color scheme css
65c01798d0
correct snapshot version for logback-test-config
2352b87474
correct snapshot version

Release 6.2.3

nfs

The current release fixed previously observed infinite WRITE+COMMIT loop.

Changelog 6.2.2..6.2.3

2bd19d2080
[maven-release-plugin] prepare release 6.2.3
0709f64e67
nfs-proxy: op WRITE must use the same verifier as COMMIT
295ed840a9
[maven-release-plugin] prepare for next development iteration

Release 6.2.2

dcache

There have been several occasions where in a dCache instance using an embedded ZooKeeper server such HA services stopped working because an ephemeral node was elected leader that has no associated service anymore; it should have been deleted. This state might go undetected for a longer time, and removing such stale nodes in order for the services to continue functioning as intended requires manual intervention.

This is fixed now and the functioning of HA services using ZooKeeper-based leader elections is more reliable.

resilience

A more robust handling of PSU changes which remove pool, unit or group mappings was introduced.

Recently introduced improvements changed the way resilience handles file “removal” (no longer setting the repository entry to ‘removed’ but simply by caching the replica). This change, however, did not take into account the handling of broken files. Encountering a broken file, it would indiscriminately attempt to remove it, whether it was cached or not; this was leading to an infinite loop, with the file operation continuously iterating without doing any further work. This situation could potentially hang the pool scans (if there are as many broken files as there are scan threads), and even the file operation queue. The current release fixed this issue and there is no any potential for stalled operations when encountering broken replicas.

Changelog 6.2.1..6.2.2

d91ba83234
[maven-release-plugin] prepare release 6.2.2
44fabfc33e
dcache-core: fix embedded ZooKeeper persisting stale ephemeral nodes
859be7aca0
dcache-resilience: ignore broken cached files
d7ddc2e486
dcache-resilience: protect access of pool info map against NoSuchElementException
12287c4beb
[maven-release-plugin] prepare for next development iteration

Release 6.2.1

macaroons

Fixed macaroons validation where in case of multiple combined caveats for some of them only the digital signature was checked.

resilience

If a FileOperation is canceled while its subtask is running a NullPointerException was thrown. This is now fixed.

rpm

dCache rpm-package requires now Java 11.

systemd

dCache forces now the use of systemctl on systemd-capable systems.

Changelog 6.2.0..6.2.1

130a5c708a
[maven-release-plugin] prepare release 6.2.1
61d44927ef
dcache-resilience: avoid NPE in file operation cancel
916d8c4e04
pool: fix issue #5491 NoSuchFileException
9a2dea5446
macaroons: verifyCaveat should return false for unprocessed caveats
b970c780e8
rpm: make dcache package depend on java–11
c47ffcf8b9
systemd: add option to enforce systemd
fa4f6851fc
[maven-release-plugin] prepare for next development iteration

Release 6.2.0

An optional telememetry service is added to collect dCache statistics by dCache.org. The telemetry cell has to be explicitly defined in a layout file as well as enabled:

[telemetryDomain]
[telemetryDomain/telemetry]
telemetry.cell.enable=true
telemetry.instance.site-name=dCache instance on \${host.name}
telemetry.instance.location.latitude=53.5772
telemetry.instance.location.longitude=9.8772

The dcache version, online capacity, site name and optional location will be sent to the stats.dcache.org collector.

Starting from version 3.2.0 dcache domain processes can be managed as systemd services. However, this functionality was available for debian-based systems only. With version 6.2.0 systemd integration is added into RPM packages as well. The Debian based installations must stop dcache.service manually before installing new package. Please check the documentation for further information. To support RHEL–6 based systems, the legacy sysV-like daemon still available.

Billing

The data from the mover info message and door request message has been combined into a single message when reporting to the kafka server. Kafka receives a request message type with mover info integrated, if it exists. A failed door request or a remove request (which also uses DoorRequestInfoMessage) will simply have a null mover message field.

Chimera

The minimal PosgreSQL version is changed to 9.5. A new table t_xattr (inumber, key, value) is introduced to store user defined extended attributes.

NFS

The embedded NFS server publiches itself as 4.2 protocol compatible.

Each NFS door gets an unique identifiert that becomes a part of nfs state ids. This prevents unlikely event of nfs mover conflicts. The unique id for each door is stored in zookeeper as:

/dcache/nfs/doors/<cell>@<domain>/state-handler-id

The NFS transfer service was returning a multipath address list to nfs clients that contains duplicate entries. This was triggering kernel warnings like:

[Di Apr 23 13:50:01 2019] RPC:   addr 131.169.a.b already in xprt switch
[Di Apr 23 13:50:01 2019] RPC:   rpc_clnt_test_xprt failed: -98 addr 131.169.a.b not added
[Di Apr 23 13:50:02 2019] RPC:   addr 131.169.a.c already in xprt switch
[Di Apr 23 13:50:02 2019] RPC:   rpc_clnt_test_xprt failed: -98 addr 131.169.a.c not added
[Di Apr 23 13:55:00 2019] RPC:   addr 131.169.a.d already in xprt switch
[Di Apr 23 13:55:00 2019] RPC:   rpc_clnt_test_xprt failed: -98 addr 131.169.a.d not added

This is fixed now.

The discovery of an NFS4 domain, which is used by idmapper, is made more robust against invalid configurations of the dns resolver. Improved throughput of parallel opens.

Pool

Pools that use BerkeleyDBReplicaStore have impoved startup time by reducing the number of stat requests to the underlying filesystem.

SRM / SRM Manager

Removed hidden possibility to generate perfomance HTML reports and plots.

Telemetry

An optional telememetry service is added to collect dCache statistics by dCache.org.

Transfer Manager

WebDAV

A Webdav door can pass xattr. prefixed url params as extended attributes to a pool. For example:

    $ curl --upload-file file.txt \
        https://localhost:2881/tape/file.txt?xattr.key1=value1&xattr.key2=value2"

This information will be available to an HSM as part of the storage info.

XRootD

TLS can now be enabled for connections to both door and pool. Please read the documentation in The Book for details (https://dcache.org/old/manuals/Book–6.2/config-xrootd.shtml#tls). There is also further information in the xrootd.properties file.

SciToken support has been added to the xrootd door. Please read the documentation in The Book for details (https://dcache.org/old/manuals/Book–6.2/config-xrootd.shtml, under “SciTokens”). There is also further information in the xrootd.properties file.

When using TPC with GSI, the option to fall back to generating a proxy from the host certificate or to use a proxy provided at a given path instead of a delegated one, can cause issues if third-party source endpoints do not have a mapping for the proxy DN. Before 6.2, this fallback was automatic, but with 6.2 it can be controlled via the property

    xrootd.gsi.tpc.delegation-only=true

Turn on fallback (set this property to false) only if you are certain all potential endpoints will recognize the alternate DN.

As a reminder: we have also changed the dCache door to behave like the xrootd server and create missing directories (see [229210209c]); the default pool plugin has also been changed to unix (rather than undefined), as this is needed for TPC with dCache and EOS sources (but is otherwise ignored): see [a5482ad5f2]).

Zookeeper

The ZooKeeper ensemble which is used by dCache is required to be of version 3.5. Version 3.4 will no longer be supported.

Cleaner

The Cleaner cell may be run in high availability mode. Several Cleaners negotiate leadership via ZooKeeper and only the leader is active at any point in time. When the leader disappears, a new leader is elected automatically.

Changelog from 6.1.0 to 6.2.0

fa4f6851fc
[maven-release-plugin] prepare for next development iteration
a732fcb967
[maven-release-plugin] prepare release 6.2.0
70caee8202
archetype: configure maven-compiler-plugin to use java11
2621b05455
pool: do not fail transfer if RFC3230 is badly formed.
5b3888fb87
nfs: don’t attempt to select online-only pools when file on HSM
948217aebb
cleaner: add support for db password file
70bff6adb5
util: fix pgpass support for jdbc urls with options
8eff87bec9
pool: rework interrupt processing on p2p
eaada9757f
systemd: convert dcache.service into dcache.target
a5cb11ad1b
[maven-release-plugin] prepare branch 6.2
9df0bd2330
change <tag>
e5f3fe470c
[maven-release-plugin] prepare branch 6.2
ae91987797
libs: use nfs4j–0.21.x
c52d53cd73
Revert “libs: use nfs4j–0.21.x”
fcb4a168f0
libs: use nfs4j–0.21.x
a1d8b9abab
gplazma: automate filtering of principals
b4fe9166f4
common: annotate principals based on their role during login.
6a79ecb4ce
pnfsmanager: add the “show list activity” command
05a910fae8
common: add support for sections in ColumnWriter
985a43c2cb
dcache-bulk: fix several issues with request storage
92fca35216
dcache-chimera: Show that Cleaner service is replicable
a79fd7c12b
systemd: print deprecation warning only on systemd capable systems
f22003dbe9
pool: add Account wrapper that checks file capacity usage
4af94c84cb
docs: UserGuide describe extended attributes in frontend
87a6fbe54d
dcache-core: Move LeadershipListenerGroup for reusability
86ce323540
systemd: drop dependency on zookeeper service
9cde8bf80b
docs: UserGuide fix .html -> .shtml extension in links
eab0b80276
dcache-core: Move HAServiceLeadershipManager for reusability
17654d4318
docs: Add common chapter on extended attributes
065c6d3042
dcache-bulk: restructure test jobs
4105665d8d
dcache-bulk: add ping service for testing waiting state
4d273857b0
dcache-bulk: modifications to bulk jobs
c45da26e27
common: do not log bearer token values.
16f56cba36
frontend: add support for extended attributes
b751c39b14
pnfsmanager: avoid NPE if file is deleted during directory listing
4ace8e1e16
dcache-chimera: high available cleaner
5dfef491ce
rpm: stop dcache service before update and disable on uninstall
5fb54f364f
systemd: export PATH variable in the dcache-generator
8fcde432fb
rpm: remove pre-historic rpm support
a5e4f7adbe
pool: rework interrupt processing in AbstractMoverProtocolTransferService
ca257e79f5
rpm: don’t use RPM’s native systemd macros to stop dcache
d7de1f0030
rpm: add systemd target and generator
8e2a926545
rpm: drop pre-historic package compatibility
96101cdc9f
docs: UserGuide update WebDAV to describe xattr on upload
d205d6ba19
src: don’t use deprecated Throwables#propagate(Throwable throwable)
a5688b1745
srm-server: use URLDecoder#decode(String s, Charset cs)
300fd1c8f1
docs: fix issues with WebDAV chapter in the user-guide.
e06f4ef029
cells: fix stacktrace dump if cell takes too long to shutdown
ee4de45062
docs: describe how WebDAV properties may be used with extended attributes
61cdbebfc8
webdav: add support for extended attributes
de40f748db
Revert “chimera: use hazelcast cache for pnfs id to inumber mapping”
7765e99f6f
poolmanager: fix tab usage in RendezvousPoolManagerHandlerTest.java
2243d6d696
common: fix LoAPrincipal deserialisation
f0dbff3d1c
frontend: events make backlog configuration
834b5702bd
webdav/pnfsmanager: fix problems with xattr assignment on file creation
8dd7ea6fa0
pnfsmanager: support storing and querying FileAttribute.XATTR
71f957c4fa
core: update PnfsHandler to support extended attributes
ff1b5e74ac
pool: shutdown NettyTransferService implementations asynchronously
b3b421ae31
dcache-xrootd: bump to xrootd4j 4.0.3
be5de06f6b
pool: clean interrupted state after mover complete (again)
9caf6d7e9c
chimera: allow default durations for ‘pin’, ‘stage’ and ‘bringonline’ dot commands; add ‘unpin’
0ef6d91fe2
Update srmmanager.properties
905e39e281
Update gplazma.properties
ac6dbc3bdf
Update pnfsmanager.properties
c96eee8e50
Update spacemanager.properties
74cfffa038
Update pinmanager.properties
f1857d3c91
Update topo.properties
db42b3f9f1
pnfsmanager.properties: fix typo in comment
465f6cb4f1
pinmanager: use PoolManagerStub when talking to pool manager
3f48856942
cells: make ThreadGroup dump a single log message
a3de7c09f6
dependencies: exclude spring’s implementation of Jakata Common Logging
8697edac15
gplazma2-argus avoid shipping dCache with commons-logging
17360de1a1
pool: avoid instantiating http server twice
d60801bfee
pool: shutdown xrootd TPC event-loop group asynchronously
af7878c447
build(deps): bump guava from 24.1-jre to 24.1.1-jre
e3680d2874
build(deps): bump xercesImpl from 2.11.0 to 2.12.0
6b9e505616
build(deps): bump version.netty from 4.1.45.Final to 4.1.50.Final
b5f12700a4
dcache (pool manager): override suspend when “excluded” (hosts) is not empty’
0132c7aca4
pool: clear mover thread’s interrupted state before returning to pool
784b9b27a8
nfs: fix door startup crash on DNS error
9ee78c3fae
Doc bug: rm ls shows remove queue, not store queue
17d542576a
pool: avoid duplicate IP addresses in nfs multipath array
1ebad872c2
pool: use Long#compare to sort cache entries
b2317d35c1
nfs: remove redundant methods from PoolDeviceMap
7832755ce9
The Book: fix typo in config-nfs
81f4d682d0
dcache-xrootd: cancel TPC transfer when client disconnects unexpectedly from pool
ae8354b532
core: support extended attributes in RemoteNamespaceProvider
65286ee10b
dcache-xrootd: fix ssl handler context loading when TLS is not enabled
ebeb9076e9
pool: check compare fileSize from namespace and fileSize on disk before starting Mover
b7712bc23e
telemetry: new cell to collect usage data
504a729931
PnfsManager: implement support for extended attributes
b37b9850d8
srm: remove rrd based stats counter and gauges
d322e8800f
pool: repository account capacity correctly if file channel closed
2231b12e45
dcache-xrootd: make ‘tried’/excluded hosts an optional feature
16a42a8a78
cells: fix logger configuration before SystemCell is created
9891b06108
src: suppress VA_FORMAT_STRING_USES_NEWLINE warning
1c4e0e9fd9
logback: remove special zookeeper–3.4 workaround
0abb24a601
namespace: add support for extended attributes
a73d71d4cb
chimera: update FileSystemProvider so listing xattr returns a Set
559746c786
pool:add property je.freeDisk
fa52f1b15d
pool: remove historic VspDataOutputStream
6adab4c292
pool: add file’s PNFS-ID to capacity accounting
86e7b84f3a
pool: avoid unnecessary pool capacity account modification
9212eb3116
pool: deleting empty files need not update pool capacity account
7c3fa201c7
docs: document maven 3.5.0 requirement
db6c1cd3e5
pom: require maven version 3.5.0 in maven-enforcer-plugin
d4d16e5147
pool: fix space miscalculation for partially uploaded files
4da5e01bf4
Fix Strings#toThreeSigFig for negative numbers
20be71cbce
pool: remove boolean ‘force’ argument from sendPoolManagerMessage
bfa76eca68
system-test: updated required maven version in README
79fe6a15a7
docs: removed typos in system-test README
97d64c8c58
docs: added embedded db connection info
4618e5c8ff
docs: TheBook update index to include the chapter on Kafka
122c15ec33
pool: don’t use RequestExecutionTimeGuage to measure startup time
49f17c89f6
dcache: rework QoS Transition Engine synchronous mode
f34c0700d6
dcache-core: embedded ZooKeeper may start without existent snapshots
c04aa4336a
shell: update chimera shell to support extended attributes
1f8abae6dd
poolmanager: fix output of ‘rc ls’ command
45e4221b12
pool: pool restart optimization
58dd557411
pool: don’t use hard-coded time constant to query nfs door about stall clients
9c35cacf27
Revert “dcache-core: Fixed exception thrown in admin interface”
f509dd6b67
cells: prettify outputs of zk admin commands
a5482ad5f2
pool (xrootd): make tpc security plugin default unix
dac76a51c9
dcache-core: Fixed exception thrown in admin interface
6a34873492
docs: formatting for better readability
60e3de4dbf
util: update PnfsId#compareTo to use Arrays#compare
0023f22347
pool: don’t wrap with optional a nonnull annotated objects
60d56c6af5
docs: remove duplicate system requirements
858a313e7b
libs: update to zookeeper 3.5
651c8e7567
Updated system requirements
8d46aa7c3b
src: end of col.toArray(col.size()) vs col.toArray(new T[0])
9980df241f
docs: add missing architecture diagram
721776a80c
src: simplify Collection->stream->toArray(T[]::new)
5bbb74d3c0
libs: update to pg-jdbc 4.2.12
48049b525d
srm: remove historic ant build script
dc2dd74062
docs: UserGuide fix missing back-tick
a4d62f40ea
frontend: honour door and user root for inotify subscriptions
89dc2d3800
frontend: refactor inotify client-path to dCache-path mapping
98169977ff
frontend: add SelectionContext for event subscription
e309156938
nfs: don’t use CDC in try-with-resource block
3045e80cbb
nfs: restore transfers debug context on mover shutdown
a899884dd7
common: annotate in which dCache version Principals were added
f011a98209
pool: drop Authorization HTTP header on redirected
8e3ecb3ab1
util: don’t call String#toString in Transfer class
179d5e7bd0
cells: fix links in javadoc of CDC
80a905a01a
login-broker: fix ‘lb set update’ command
4ec1f835d3
nfs: close directory stream after use
f1d61f2a2e
chimera: remove unused DirectoryStreamHelper#listOf
0030d66fc7
nfs: use TreeSet when directory listing is created
ed48aa21ba
chimera: make DirectoryStream java8 friendly
2930138d85
nfs: fix missing place holder in log a message
229210209c
dcache-xrootd: always create missing directories on write
c9a685a184
dcache-xrootd: enforce failure when clients without TLS attempt TPC where it is required
d348c76d2c
dcache-frontend: remove retry flag on sendAndWait to history service
8e156be0db
src: don’t use guava’s Charsets in favor of StandardCharsets
0c155981dd
dcache-bulk (05): add request storage (in-memory/disk)
086bcc2c40
dcache-bulk: don’t overwrite BulkJob state
39d63ed78b
nfs: add client store for server reboot recovery
48b0c2d19d
enstore: populate the correct storage info information on set
b511d7b453
dcache-bulk: copy paste error in BulkRequestArgumentDescriptor constructor*
5bef4b8301
dcache-parent: Use Objects#requireNonNull instead of guava’s alternative
cc5ee9b3d1
pom: fixed typo in maven-compiler-plugin java version
4e414e6dcb
dcache-bulk: add BulkJobArgumentDescriptor
dfcd446cfe
dcache-bulk: fix two bugs in the TargetExpansionJob
bbc04fa94f
src: revisit ‘java 9+’ marked todos
8531c4e2e5
dcache: restore synchronized on two methods in QoSTransitionEngine
1072059639
pom: enable modernizer plugin to help java version migration
99f2fe49bb
dcache: fix QoSTransitioEngine logic
56f9c75d3c
pom: dont use deprecated options for maven-compiler-plugin
d163c1a33a
dcache-resilience,dcache: move utility class to pool manager package
6aa4553e0a
docs: describe dependency on java–11
e25a93613b
docs: specify dependency on java11
11799e7271
libs; use nfs4j–0.20.2 with minor bugfixes
da9e15a353
nearline-storage: remove disk copy if restore from HSM failed
00b40161c8
pom: use property agsLine
051dd05ba8
pom: require Java–11 at compile and run time
986abe48a7
hsm: fix missing closing bracket in error message
6f81aff617
tag: unify snapshot package name with deb and rpm
f8f9979586
system-test: use CopyNearlineStorage as configure HSM
e137380001
chimera: make PostgreSQL 9.5 minimal supported version
311e45ad01
dcache-bulk (04): add job store
fba82bb73b
dcache-bulk: fix possible NPE in completion handler
311a70e39f
dcache: give QoSTransitionEngine a synchronous option
550860147e
chimera: fix broken commit b44fa4e
890e156311
chimera: fix broken commit b44fa4e
babcbad1a2
chimera: fix broken commit b44fa4e
7764dd82b3
chimera: fix broken commit b44fa4e
b44fa4e85e
chimera: introduce postgresql 9.5 friendly f_create_inode95 procedure
f9c4cfb148
dcache-bulk (03): define single target job providers and factory
71d74bc230
dcache-bulk (02): define multiple and single target jobs
e14905061e
dcache-bulk: (1) add module, base job, completion handler
a4b8ba78c8
nfs4: add xattr support
782fcead53
doc: add README.md to system test with configuration description
1a480ecde4
chimera: optimize FsSqlDriver#setXattr for postgres
56620b06a7
chimera: add extended attribute support
658233b128
libs: use nfs4j–0.20.1
0b57f11a33
chimera: fix postgres version discovery
639cc34212
dcache-xrootd: add SciToken plugin
4e8c33b382
dcache-xrootd: add TLS support to door and pool
2354a29eda
dcache-xrootd: add delegation-only GSI property
dbdb7f6e75
dcache-xrootd: eliminate try-catch block (InterruptedException no longer declared)
dec851bce5
chimera: fsstat should handle big file systems
1ba70abfc2
dcache-frontend: make ErrorResponseProvider return the more specific error message
b5f9e5d2e4
dcache: qos migration policy engine should not raise JVM error when no tape pool found
0d1cdd7964
dcache-frontend: request file attributes explicitly
a7a940bc0c
dcache: send merged mover and door info to kafka as request
dba949e0ac
sweeper: make sure the histogram object is properly configured
01b7061ba4
namespace: accept XATTR as supported attribute
933cedb4f3
The Book: Update documentation about how to use maven archetypes
1a13e26772
dcache-history,dcache-frontend: guard against unconfigured sweeper histogram
68e44b8b24
pom.xml: sleepycat version update
a1080dd2cc
wendav: add url param based xattr support
5fc1d8a813
vehicles: add FileAttribute#XATTR to represent extended attributes
28cb37a59d
common: add Xattrs to handle extended attributes
f1a55a3a9b
message: cells call of message toString() as part of CellMessge.toString()
bf147085e3
dcache-frontend,history: protect against missing highest bin in histogram data
f0aba8f424
dcache-frontend: allow pool enable/disable to use boolean JSON value
c7bd1a5942
Documentation: Update webdav write and read url
8a2d5c991e
ftp-client: Remove whitespaces in blank lines and line endings
b864ff1d33
nfs: give each door a unique state handler id
08298eeafc
frontend: events inotify decouple sending events from message queue
a77a06bd14
frontend: fix memory leak in long-running SSE connections
20e3c76dce
libs: use nfs4j–0.20.0
70987bc53e
[maven-release-plugin] prepare for next development iteration