Highlights

  • better nfs v4.x performance of state validation in highly concurrent environment

Incompatibilities

  • the mountd service is started only if NFS v3 is enabled
  • removed historic code of HSM flush manager

Acknowledgments

(tbd, this section uses markdown formatting)

Release 8.0.11

dcache

A clarification on authorized keys format has been added.

Changelog 8.0.10..8.0.11

d7dba3390a
[maven-release-plugin] prepare release 8.0.11
f03030ce4d
Add clarification on authorized keys format
843aa5336c
[maven-release-plugin] prepare for next development iteration

Release 8.0.10

dcache-bulk

Runtime exceptions do not kill the queue thread now and do not leave the request in a stalled state.

webdav

Existing monitoring information does not show HTTP-TPC transfers, which may result in admins being unaware of dCache activity.

This is now fixed and The Active Transfers page (from httpd) and the transfers resource (from frontend) now include HTTP-TPC transfers.

WebDAV HTTP-TPC now supports multiple RemoteTransferManager services, which allows for high-availability deployments. The WebDAV door and all RemoteTransferManager services must be upgraded to support this new feature.

Changelog 8.0.9..8.0.10

1e11319d30
[maven-release-plugin] prepare release 8.0.10
f8020c82cf
dcache-bulk: (version 1) handle unexpected exceptions in queue
1fca124200
webdav/httpd/frontend: add HTTP-TPC transfers to transfer list
b35d3e847a
webdav/transfermanager: support multiple RemoteTransferManager
c02d6ece3c
[maven-release-plugin] prepare for next development iteration

Release 8.0.10

dcache-bulk

Runtime exceptions do not kill the queue thread now and do not leave the request in a stalled state.

webdav

Existing monitoring information does not show HTTP-TPC transfers, which may result in admins being unaware of dCache activity.

This is now fixed and The Active Transfers page (from httpd) and the transfers resource (from frontend) now include HTTP-TPC transfers.

WebDAV HTTP-TPC now supports multiple RemoteTransferManager services, which allows for high-availability deployments. The WebDAV door and all RemoteTransferManager services must be upgraded to support this new feature.

Changelog 8.0.9..8.0.10

1e11319d30
[maven-release-plugin] prepare release 8.0.10
f8020c82cf
dcache-bulk: (version 1) handle unexpected exceptions in queue
1fca124200
webdav/httpd/frontend: add HTTP-TPC transfers to transfer list
b35d3e847a
webdav/transfermanager: support multiple RemoteTransferManager
c02d6ece3c
[maven-release-plugin] prepare for next development iteration

Release 8.0.9

frontend, bulk

Both numeric and string values for PIN lifetime are valid now. Also, skipDirs for DELETE can be both boolean and string. All other current arguments are required to be strings, as before.

Changelog 8.0.8..8.0.9

7887ef8a2d
[maven-release-plugin] prepare release 8.0.9
30ec4454d5
dcache-frontend,bulk: fix typing of argument values
baf0d890f8
[maven-release-plugin] prepare for next development iteration

Release 8.0.8

frontend

Anonymous access receives a 401 error and doesn’t trigger a stack-trace when unlimitedVisibility is set to false.

clearOnSuccess, clearOnFailure and cancelOnFailure can have boolean and string values now.

pool

A bug is fixed where the checksum calculation would fail for empty files.

Changelog 8.0.7..8.0.8

1a412d30c9
[maven-release-plugin] prepare release 8.0.8
9fae9e8a75
dcache-frontend: protect against RuntimeError in case of denied anonymous access
551fc779da
restore inadvertently removed ‘target’ extraction in toBulkRequest()
faffec45bf
dcache-frontend: bulk-requests POST, allow either string or boolean for boolean values
e449c33482
pool: don’t treat an empty file as a sparse file
ea3e74ef2a
[maven-release-plugin] prepare for next development iteration

Release 8.0.7

ChimeraVfs

This release restores the ability to modify file levels.

info

The delay between messages sent by info was increased to reduce log messages of “DGA {…} triggering too quickly”.

pool

If the scrubber state file is empty, scrubber will now log a more helpful error message and proceed scrubbing the pool.

Changelog 8.0.6..8.0.7

5781884976
[maven-release-plugin] prepare release 8.0.7
4a2eb74fab
pool: improve handling empty saved state by scrubber
b90c4fbec2
dcache-xroot,pom.xml: bump xrootd4j to 4.3.1
bf11ceecbb
dcache-xroot: flesh out channel inactive and exception caught
42f94b841e
info: increase delay between messages
f0d69b8c0f
ChimeraVfs: Check file level in addition to inode type when disallowing file size change
ce556c9015
[maven-release-plugin] prepare for next development iteration

Release 8.0.6

dcache

Improved documentation for kafka Producer properties.

Improved error message for inexistent statistics path.

dcache-xroot

The current relase Updated master xrootd4j to 4.3.0.

frontend

NPE is fixed when SRR when space information is not available yet.

nfs

Closing on open-for-read file now does not trigger extra layout recall.

xrootd

Rendezvous TPC without requiring a JWT token to be passed by the third-party client is possible (again).

Changelog 8.0.5..8.0.6

a92fa0cf2d
[maven-release-plugin] prepare release 8.0.6
d488867ee8
dcache:improve documentation for kafka properties
4389adec33
nfs: re-send kill only for write movers
cc04c4f5dd
dcache-xrootd: fix TPC rendezvous to work with token authorization
675302e989
dcache-core: improve error message for inexistent statistics path
2fe8b1446f
frontend: fix NPE in SRR when space information is not available yet.
b01c233ed6
dcache-xroot: update master xrootd4j to 4.3.0, and stable branches to next minor version
c5b07458ac
[maven-release-plugin] prepare for next development iteration

Release 8.0.5

script

Direct deprecation warnings are now redirected to stderr.

Changelog 8.0.4..8.0.5

2901d045e5
[maven-release-plugin] prepare release 8.0.5
b9e835c028
info: fix broken unit-test, remove redundant one
1eb775efa2
dcache script: redirect deprecation warning to stderr
bf93e96c83
[maven-release-plugin] prepare for next development iteration

Release 8.0.4

pnfsmanager

Doors no longer allow a user to discover whether or not a file or directory exists within directories they cannot access.

pom

Updated spring libs to version 5.2.20, CVE–2022–22965

pool

The pool now provides more information when a migration job was cancelled.

Changelog 8.0.3..8.0.4

5d7043b9e6
[maven-release-plugin] prepare release 8.0.4
6654c88269
pom: update spring libs to 5.2.20
3506bcd74f
pool: describe why migration job was cancelled
9f0d88d370
pnfsmanager: avoid leaking whether or not file exists
476efc5c11
[maven-release-plugin] prepare for next development iteration

Release 8.0.3

dcache-xroot

A ZTN token can now be given without further downstream tokens expressed, and full authorization of the subject will take place.

Changelog 8.0.2..8.0.3

e7fc0dc8e9
[maven-release-plugin] prepare release 8.0.3
40681c2702
dcache-xroot: Allow ZTN authentication to function as fallback authorization
fa567e2bc2
[maven-release-plugin] prepare for next development iteration

Release 8.0.2

dcache-chimera

Cleaner batch delete exception has been fixed.

Chimera shell support has been added for getting, adding and removing sha1, sha156 and sha512 valued checksums.

User’s guid has been updated.

dcache-core

Sha–1 checksums can and must now correctly be requested as sha, as defined by RFC 3230.

frontend

A bug is fixed that inverted the property frontend.srr.public=true. True means now allow public access and false does not.

nfs

user subject always available in the processing thread.

srr

LinkLocal address will not be published.

util

nfs door has been updated to map broken file to IO error.

Changelog 8.0.1..8.0.2

f5ff9dcb2a
[maven-release-plugin] prepare release 8.0.2
ee5c3329de
srr: don’t publish link local addresses (v2)
1e386a1be7
util: treat broken file as unrecoverable error in Trnasfer class
bd8fefa54a
nearline-storage: fix request count on error path in AbstractRequest#failed
4ababb6902
nearline-storage: fix queued count when request initialization falied
3e492aef9f
pool: add bunch of unit tests for NearlineStorageHandlerTest
ee191ff429
UderGuide: describe supported checksums, add example including sha
c052e902da
frontend: fix problem with making SRR resources public
80a277d033
dcache-chimera: add sha checksum commands to chimera shell
fdbd2cbe2c
dcache-core,common: fix user having to request sha-1, not sha
7e189ce96e
nfs: enforce subject propagation on RPC level
0a0b7f6c14
dcache-core: prevent ocasional message ttl test failure
f05ff3b670
Fix typo
c2b795c275
dcache-chimera: fix cleaner batch delete exception
8280d097a4
[maven-release-plugin] prepare for next development iteration

Release 8.0.2

dcache-chimera

Cleaner batch delete exception has been fixed.

Chimera shell support has been added for getting, adding and removing sha1, sha156 and sha512 valued checksums.

User’s guid has been updated.

dcache-core

Sha–1 checksums can and must now correctly be requested as sha, as defined by RFC 3230.

frontend

A bug is fixed that inverted the property frontend.srr.public=true. True means now allow public access and false does not.

nfs

user subject always available in the processing thread.

srr

LinkLocal address will not be published.

util

nfs door has been updated to map broken file to IO error.

Changelog 8.0.1..8.0.2

f5ff9dcb2a
[maven-release-plugin] prepare release 8.0.2
ee5c3329de
srr: don’t publish link local addresses (v2)
1e386a1be7
util: treat broken file as unrecoverable error in Trnasfer class
bd8fefa54a
nearline-storage: fix request count on error path in AbstractRequest#failed
4ababb6902
nearline-storage: fix queued count when request initialization falied
3e492aef9f
pool: add bunch of unit tests for NearlineStorageHandlerTest
ee191ff429
UderGuide: describe supported checksums, add example including sha
c052e902da
frontend: fix problem with making SRR resources public
80a277d033
dcache-chimera: add sha checksum commands to chimera shell
fdbd2cbe2c
dcache-core,common: fix user having to request sha-1, not sha
7e189ce96e
nfs: enforce subject propagation on RPC level
0a0b7f6c14
dcache-core: prevent ocasional message ttl test failure
f05ff3b670
Fix typo
c2b795c275
dcache-chimera: fix cleaner batch delete exception
8280d097a4
[maven-release-plugin] prepare for next development iteration

Release 8.0.1

Changes affecting multiple services

The OIDC ‘sub’ (subject) and ‘jti’ (JWT ID) claims are logged to access log file for WebDav, frontend and SRM doors if OIDC is used.

admin-shell

Fix the line termination printed to console. It’s \r\n now.

bulk

Request-specific pinning is enabled now, but generic unpinning is still possible.

dCache now returns 404 instead of 403 if a submitted id in GET or PATCH bulk-request is not recognized by the service.

Jobs which originally got stuck in STARTED state now complete; their failure information contains the reason for premature completion.

frontend

The property frontend.srr.public=true|false is added to frontend to control SRR resource restriction. The default value is false.

ftp

The FTP cell is killed asynchronously now after the session terminates so the door will remain responsive when many clients terminate their FTP session concurrently.

pool

HTTP-TPC transfers now prefer IPv6 address, if both endpoints support it.

If a HSM script survives SIGTERM it will now be stopped with SIGKILL and a warning is logged. This reduces expired or canceled requests staying in CANCELED state.

If a NFS mover is killed twice it’s now logged without a stack trace. This provides more stable NFS error recovery, too.

webdav

Fix handling of ‘webdav.authz.allowed-paths’ so it no longer triggers a NullPointerException if a request targets a path that is not in allowed-paths.

xroot

Change default TLS to OPTIONAL on pools so defaults for xroot TLS are uniform on pools and doors.

Changelog 8.0.0..8.0.1

df6fcc1f9a
[maven-release-plugin] prepare release 8.0.1
fc1d423b3d
frontend: make srr resources public
8fee90cce9
webdav/frontend/srm/gplazma: log OIDC ‘sub’ and ‘jti’ claims
ca3dbb82f4
ftp: kill cell asynchronously
57bceb1b2f
pool: http-tpc prefer IPv6 address
d8fe9f420c
pool: handle multiple shutdowns of a nfs mover
3ef40e0f64
dcache-bulk: aborted request gets stuck in the STARTED state
bb223954b7
dcache-bulk: pin by request id and unpin optionally by request
2b4c8a148c
pool: try to forcefully kill HSM process if needed
a4b8babd1a
admin shell: fix line termination in SshOutputStream
cdbe5ad0f2
dcache-xroot: set default TLS to OPTIONAL for pools
edf2a83165
bulk, frontend-rest: return 404, not 403, when request id does not exist
b241a4c150
webdav: fix NPE if ‘webdav.authz.allowed-paths’ disallows a request
e0b3b623d3
chimera: fix error when adding new label to a file
a8cb617f44
[maven-release-plugin] prepare for next development iteration

Release 8.0.0

Bulk

The handling of pin requests was fixed to be fully asynchronous and not time out. There was also a regression fix for aborted requests which would leave the request in the STARTED state instead of completing.

Chimera-shell

The chimera shell now supports creation symbolic and hard links.

Frontend

Transfer rate was changed to a double value so that very fast transfers do not get rounded down to 0. ‘pin’ and ‘unpin’ were added to the namespace resource POST method for individual files. In both REST and NFS (dot command), pin and unpin was modified to use the uid as request id. Bulk requests were fixed to support any combination of attribute styles (camel-, snake- and kebab-case) for a given request.

Clearer error message which does not denote a bug when CA certs directory is missing and HTTPS is configured.

The “key” field of alarm log entries is now build as a colon-separated list of keywords.

FTP

GSIFTP was reverted to use the Java SSL library instead of OpenSSL (API incompatibilities).

gplazma

It is now possible to configure a group name using the simple group prefix in the banfile and scitokens gPlazma plugins. This prefix will also work for the “test login” and “explain login” admin commands.

The oidc plugin is now able to accept a JWT access token without querying the user-info endpoint. If the token is not a JWT then the oidc plugin will call the user-info endpoint to discover the necessary information about the user.

The oidc gPlazma plugin now supports the ‘aud’ claim. The list of allowed audience values is configured via the ‘gplazma.oidc.audience-targets’ configuration property.

NFS

The mountd server is started only if NFSv3 is enabled.

Exposed files PNFSID, checksum and locality as read-only extended attribyutes. $ attr -l file1.txt Attribute "dcache.id" has a 36 byte value for file1.txt Attribute "dcache.locality" has a 8 byte value for file1.txt Attribute "dcache.checksum" has a 10 byte value for file1.txt

NOTE: this functionality is in BETA state and should be used with cation and client side caching might produce wrong results

Update NFS4J library to 0.23.x version with various fixes and improvements.

In version 8.0.0 we have introduced a workaround for linux NFSv4.1/pNFS client race condition that under high write IO workload the client might close a file, despite the fact that not all VM dirty pages are flushed and putting the client into unresponsive state.

Pool

Removed support of unused centralized flush manager.

When transferring files with HTTP-TPC, the pool will try to reuse TCP connections. Idle connections are closed only if they are idle for too long or if a maximum number of connections is reached. The exact behaviour is configurable. This is helpful when transferring many files with the same remote endpoint, particularly if the files are small.

Previously, when a client downloads a file via HTTP, the pool considers the transfer complete only after the client disconnects from the pool (or the pool disconnects the client due to inactivity). Now, if the client requests the entire file’s contents then the pool will consider the transfer complete once all the data has been sent. A request for partial file contents will continue to follow the existing behaviour.

The NearlineStorage interface is updated to support stateful plugins that can fail during initialisation.

Resilience

There was a fix to a runtime exception triggered as a side effect of logging.

Telemetry

Added a command, print data, that prints data that is send to the collector.

WebDAV

Clearer error message which does not denote a bug when CA certs directory is missing and HTTPS is configured.

XRootD

Since 7.2 there have been a number of upgrades and fixes for important regressions. These have been noted in the time-releases. The most significant were a fix to handle clients writing to a closed checksum channel, to support delete requests on the pool, and to provide a workaround for older xroot clients using prime values in the GSI handshake shorter than the recommended length.

Changelog from 7.2.0 to 8.0.0

a8cb617f44
[maven-release-plugin] prepare for next development iteration
a8b9c63d61
[maven-release-plugin] prepare release 8.0.0
5b44b8e61e
common: provide reasonable error message if path is not absolute
b0812a7572
pool: fix NPE on attempt by client to read from a broken replica
8323f0172d
common-security: re-read server credentials on context re-creation
d2ed77eabd
rest api: add checksum paramater to namespace resource
8b28862cda
frontend: update dCacheView to v2.0.2
0a0a095821
TransferManager: log bugs with corresponding stack-trace
ff4077fa10
CopyManager: reply success when the target transfer completes.
d4c5943b94
pool: only send new URLs when flushing file
33f2937057
[maven-release-plugin] prepare branch 8.0
ca1e3c7ce8
nfs4.1: limit layout-recall workaround to flex_files layout only
c0c77a2828
skel: minor improvements to example configuration for needrestart
a6230571ed
common: add support for “group” principals in various places
143802520a
qos-verifier: (2 of 8) pool selection utilities
4f561f9914
dcache-xroot: set default TLS to OPTIONAL
6f415a50e9
dcache-core: fix admin command to query pins by state
b2cdfac20a
util: drop Transfer#get/getOnlineFilesOnly
1746a22e62
gplazma: oidc rewrite OidcAuthPluginTest
7582bf9434
gplazma: oidc add support for profiles
40603217d9
gplazma: oidc fix minor/typo errors in IdentityProviderTests
2671c9ffae
srm-server: log trs tape queue states when tape requests are added
86cbd0cddc
chimera: remove unused constructor in ChimeraDirectoryEntry
5a001af333
nfs4.1: recall write layout on close
d232f34c85
libs: use nfs4j–0.23.0
a85e84ba8a
book: Add chapter about telemetry cell to index
0b15b16b53
book: Add chapter about telemetry cell
ece6360996
gplazma: oidc add support for offline JWT validation
6b99107559
nfs4: expose pnfsid, checksum, locality as extended attribytes
da1fcdc883
telemetry: Add command to display data
8baabae881
libs: update ZooKeeper version
b8639d1f80
nfs: use local pool monitor for reads
d38d5e940e
dcache-chimera: avoid conversion id->path->id on getLocality
0191f0b40e
common: move Jdbc base classes from qos
c7711c7ab0
skel: bump logback to 1.2.10
ac93764537
book: fix typo
3223a9faad
qos-verifier: (1 of 8) data store interfaces and implementation (jdbc)
8a4a02cd02
gplazma: oidc support chaining of TokenProcessor implementations
1f63fbbeab
pnfsmanager: add support for resetting gauge and counter statistics
1947b9386f
libs: use jline2–2.14.6 with small bugfix
30828d4930
loginbroker: add support for multiple protocol families