Highlights

This version introduces a quota system and support for file labels. Support for a dedicated scheduling strategy for bring-online requests was added to SRM-Manager.

Incompatibilities

This release doesn’t have any incompatibilities.

Release 7.2.19

frontend, bulk

Both numeric and string values for PIN lifetime are valid now. Also, skipDirs for DELETE can be both boolean and string. All other current arguments are required to be strings, as before.

Changelog 7.2.18..7.2.19

176420e31e
[maven-release-plugin] prepare release 7.2.19
cd28af3b31
dcache-frontend,bulk: fix typing of argument values
81a032acaf
[maven-release-plugin] prepare for next development iteration

Release 7.2.18

frontend

Anonymous access receives a 401 error and doesn’t trigger a stack-trace when unlimitedVisibility is set to false.

clearOnSuccess, clearOnFailure and cancelOnFailure can have boolean and string values now.

pool

A bug is fixed where the checksum calculation would fail for empty files.

Changelog 7.2.17..7.2.18

d822ccf0a8
[maven-release-plugin] prepare release 7.2.18
67055873da
dcache-frontend: protect against RuntimeError in case of denied anonymous access
cd88da4c98
restore inadvertently removed ‘target’ extraction in toBulkRequest()
51291bfc85
dcache-frontend: bulk-requests POST, allow either string or boolean for boolean values
56e2b54a5f
pool: don’t treat an empty file as a sparse file
d92924806c
[maven-release-plugin] prepare for next development iteration

Release 7.2.17

ChimeraVfs

This release restores the ability to modify file levels.

info

The delay between messages sent by info was increased to reduce log messages of “DGA {…} triggering too quickly”.

pool

If the scrubber state file is empty, scrubber will now log a more helpful error message and proceed scrubbing the pool.

Changelog 7.2.16..7.2.17

617359eb22
[maven-release-plugin] prepare release 7.2.17
0fb0b3bdbc
pool: improve handling empty saved state by scrubber
d9b3dff4ef
dcache-xroot,pom.xml: bump xrootd4j to 4.3.1
ed50d1efee
dcache-xroot: flesh out channel inactive and exception caught
6a2daf6b18
info: increase delay between messages
44c356058a
ChimeraVfs: Check file level in addition to inode type when disallowing file size change
cfcd34c4e7
[maven-release-plugin] prepare for next development iteration

Release 7.2.16

dcache

Improved documentation for kafka Producer properties.

Improved error message for inexistent statistics path.

dcache-xroot

The current relase Updated master xrootd4j to 4.3.0.

frontend

NPE is fixed when SRR when space information is not available yet.

xrootd

Rendezvous TPC without requiring a JWT token to be passed by the third-party client is possible (again).

Changelog 7.2.15..7.2.16

ca1438defe
[maven-release-plugin] prepare release 7.2.16
3fec613542
dcache:improve documentation for kafka properties
93d7afab92
dcache-xrootd: fix TPC rendezvous to work with token authorization
b7193551e7
frontend: fix NPE in SRR when space information is not available yet.
149ea52085
dcache-xroot: update master xrootd4j to 4.3.0, and stable branches to next minor version
4201b7274b
dcache-core: improve error message for inexistent statistics path
ee4183d2c1
[maven-release-plugin] prepare for next development iteration

Release 7.2.15

script

Direct deprecation warnings are now redirected to stderr.

Changelog 7.2.14..7.2.15

79444b8bc1
[maven-release-plugin] prepare release 7.2.15
2bcb871750
info: fix broken unit-test, remove redundant one
1636a3ea95
dcache script: redirect deprecation warning to stderr
39302d27c4
[maven-release-plugin] prepare for next development iteration

Release 7.2.14

pnfsmanager

Doors no longer allow a user to discover whether or not a file or directory exists within directories they cannot access.

pom

Updated spring libs to version 5.2.20, CVE–2022–22965

pool

The pool now provides more information when a migration job was cancelled.

Changelog 7.2.13..7.2.14

a18f2adb33
[maven-release-plugin] prepare release 7.2.14
b95737de1e
pom: update spring libs to 5.2.20
f4c804724c
pool: describe why migration job was cancelled
e9ee3233ae
pnfsmanager: avoid leaking whether or not file exists
e263b9ace3
[maven-release-plugin] prepare for next development iteration

Release 7.2.13

dcache-xroot

A ZTN token can now be given without further downstream tokens expressed, and full authorization of the subject will take place.

Changelog 7.2.12..7.2.13

c13043a222
[maven-release-plugin] prepare release 7.2.13
5997a85b01
dcache-xroot: Allow ZTN authentication to function as fallback authorization
32d04ceb3d
[maven-release-plugin] prepare for next development iteration

Release 7.2.12

dcache-chimera

Cleaner batch delete exception has been fixed.

Chimera shell support has been added for getting, adding and removing sha1, sha156 and sha512 valued checksums.

User’s guid has been updated.

dcache-core

Sha–1 checksums can and must now correctly be requested as sha, as defined by RFC 3230.

frontend

A bug is fixed that inverted the property frontend.srr.public=true. True means now allow public access and false does not.

nfs

user subject always available in the processing thread.

srr

LinkLocal address will not be published.

util

nfs door has been updated to map broken file to IO error.

Changelog 7.2.11..7.2.12

f833efb11a
[maven-release-plugin] prepare release 7.2.12
2c2c48baf2
srr: don’t publish link local addresses (v2)
ee246b02a1
util: treat broken file as unrecoverable error in Trnasfer class
f498803529
nearline-storage: fix request count on error path in AbstractRequest#failed
e00ac992b4
nearline-storage: fix queued count when request initialization falied
da08b097eb
pool: add bunch of unit tests for NearlineStorageHandlerTest
8fede904e3
UserGuide: describe supported checksums, add example including sha
62757d2630
frontend: fix problem with making SRR resources public
3a1cb7e1ff
dcache-chimera: add sha checksum commands to chimera shell
9f04c91f51
dcache-core,common: fix user having to request sha-1, not sha
6a3c3e30a1
nfs: enforce subject propagation on RPC level
f6e1075015
dcache-chimera: fix cleaner batch delete exception
0e7f23bc05
[maven-release-plugin] prepare for next development iteration

Release 7.2.12

dcache-chimera

Cleaner batch delete exception has been fixed.

Chimera shell support has been added for getting, adding and removing sha1, sha156 and sha512 valued checksums.

User’s guid has been updated.

dcache-core

Sha–1 checksums can and must now correctly be requested as sha, as defined by RFC 3230.

frontend

A bug is fixed that inverted the property frontend.srr.public=true. True means now allow public access and false does not.

nfs

user subject always available in the processing thread.

srr

LinkLocal address will not be published.

util

nfs door has been updated to map broken file to IO error.

Changelog 7.2.11..7.2.12

f833efb11a
[maven-release-plugin] prepare release 7.2.12
2c2c48baf2
srr: don’t publish link local addresses (v2)
ee246b02a1
util: treat broken file as unrecoverable error in Trnasfer class
f498803529
nearline-storage: fix request count on error path in AbstractRequest#failed
e00ac992b4
nearline-storage: fix queued count when request initialization falied
da08b097eb
pool: add bunch of unit tests for NearlineStorageHandlerTest
8fede904e3
UserGuide: describe supported checksums, add example including sha
62757d2630
frontend: fix problem with making SRR resources public
3a1cb7e1ff
dcache-chimera: add sha checksum commands to chimera shell
9f04c91f51
dcache-core,common: fix user having to request sha-1, not sha
6a3c3e30a1
nfs: enforce subject propagation on RPC level
f6e1075015
dcache-chimera: fix cleaner batch delete exception
0e7f23bc05
[maven-release-plugin] prepare for next development iteration

Release 7.2.11

Changes affecting multiple services

The OIDC ‘sub’ (subject) and ‘jti’ (JWT ID) claims are logged to access log file for WebDav, frontend and SRM doors if OIDC is used.

admin-shell

Fix the line termination printed to console. It’s \r\n now.

bulk

dCache now returns 404 instead of 403 if a submitted id in GET or PATCH bulk-request is not recognized by the service.

Jobs which originally got stuck in STARTED state now complete; their failure information contains the reason for premature completion.

frontend

The property frontend.srr.public=true|false is added to frontend to control SRR resource restriction. The default value is false.

ftp

The FTP cell is killed asynchronously now after the session terminates so the door will remain responsive when many clients terminate their FTP session concurrently.

pool

HTTP-TPC transfers now prefer IPv6 address, if both endpoints support it.

If a HSM script survives SIGTERM it will now be stopped with SIGKILL and a warning is logged. This reduces expired or canceled requests staying in CANCELED state.

If a NFS mover is killed twice it’s now logged without a stack trace. This provides more stable NFS error recovery, too.

Changelog 7.2.10..7.2.11

abafc5e050
[maven-release-plugin] prepare release 7.2.11
2328d630e4
frontend: make srr resources public
f3f48d6e65
webdav/frontend/srm/gplazma: log OIDC ‘sub’ and ‘jti’ claims
7653bc78df
ftp: kill cell asynchronously
b2de5dcdcd
pool: http-tpc prefer IPv6 address
bd807f0da4
pool: handle multiple shutdowns of a nfs mover
3607a08821
dcache-bulk: aborted request gets stuck in the STARTED state
dc2a01fea6
pool: try to forcefully kill HSM process if needed
8fbab1884e
admin shell: fix line termination in SshOutputStream
e858de6448
[maven-release-plugin] prepare for next development iteration
0970afbe2d
bulk, frontend-rest: return 404, not 403, when request id does not exist

Release 7.2.10

common

The current release fixed certificate reload when updated.

There are places where dCache configuration requires an absolute path. In some places attempting to use a relative path will yield an unhepful null in the log file. This is now fixed, so a meaningful error message is logged instead.

pool

An NPE on attempt by client to read from a broken replica is fixed now.

rest

webdav

The current release fixed handling of webdav.authz.allowed-paths so it no longer triggers a NullPointerException.

Changelog 7.2.9..7.2.10

034deb4042
[maven-release-plugin] prepare release 7.2.10
bfff8d424f
webdav: fix NPE if ‘webdav.authz.allowed-paths’ disallows a request
16f7723398
common: provide reasonable error message if path is not absolute
55cfd70b30
pool: fix NPE on attempt by client to read from a broken replica
1de476bd75
common-security: re-read server credentials on context re-creation
9afb28cf3e
rest api: add checksum parameter to namespace resource
65359b538f
[maven-release-plugin] prepare for next development iteration

Release 7.2.9

Pool

The current relase fixed a problem where a file’s tape location that used to exist in the namespace and was subsequently removed can reappear if the file made precious and flushed to tape.

TransferManager

TransferManager now logs bugs with the corresponding stack-trace, making fixes any such bug easier.

dcahe-core

When asking for count pins unpin-failed, only report the number of pins in state FAILED_TO_UNPIN and not PINNING.

srm-server

Now log trs tape queue states when tape requests are added.

Changelog 7.2.8..7.2.9

3f78e6f5b7
[maven-release-plugin] prepare release 7.2.9
3eb7542a49
CopyManager: reply success when the target transfer completes.
e41bd289b0
frontend: update dCacheView to v2.0.2
9b38e4fba2
pool: only send new URLs when flushing file
dea284457d
TransferManager: log bugs with corresponding stack-trace
255ba86249
dcache-core: fix admin command to query pins by state
2fc6ff0406
srm-server: log trs tape queue states when tape requests are added
3268e0ace8
readme: Remove newline at the end
8356c837d4
Update config-xrootd.md
6458b47484
Update config-xrootd.md
2e124bb413
book: Add chapter about telemetry cell
68ade2a7e0
[maven-release-plugin] prepare for next development iteration

Release 7.2.8

dcacheview

This patch introduces some improved icons and other minor visual tweaks.

gplazma

The ‘scitoken’ gplazma plugin will now accept WLCG AuthZ tokens without direct authorisation statements. When accepting such a token, the plugin refrains from adding any of the additional principals from the configuration property; only the principals directly from the token are added.

dCache will now reject WLCG-AuthZ-JWT profile tokens with a ‘wlcg.ver’ claim that it does not support. Other tokens (e.g., SciTokens) are unaffected by this change.

The token-based authentication gPlazma plugins (scitoken, oidc) now include the identity of the OP that created the token. The multimap plugin is updated to support matching on this principal.

loginbroker

This patch fixes SRM based upload or download where the client requests an xroot-based transfer. The SRM door will now also consider any xroot door with xrootd.security.tls.mode configured to OPTIONAL when building a TURL targeting either xroot or xroots protocols.

pnfsmanager

PnfsManager now has two commands, ‘reset chimera stats’ and ‘reset stats’, to support resetting the gauge and counter statistics available through the ‘info’ command.

dCache no longer leaks information about whether or not files exist when using macaroons or Scitokens / WLCG AuthZ JWT profile tokens.

webdav

The WebDAV door cannot send the HTTP response to some HTTP request if the client has already disconnected. dCache no longer logs an error that it cannot send the HTTP response.

A client that disconnects during a proxied HTTP transfer (GET or PUT) is no longer logged in the WebDAV door’s log file; instead, it is logged in the cell’s pinboard. The billing message is updated to make it clearer what went wrong.

webdav, transfermanager

dCache provides marginally better performance for HTTP-TPC, which starts to become significant when transferring many small files.

Changelog 7.2.7..7.2.8

6351804eb5
[maven-release-plugin] prepare release 7.2.8
63e035cfd6
gplazma: scitoken fix/improve support for group-based AuthN in WLCG profile
304bc36936
gplazma: scitoken/oidc/multimap add support for OP principal
3b7d463834
skel: bump logback to 1.2.10
8534813023
pnfsmanager: add support for resetting gauge and counter statistics
8229fc9eb8
loginbroker: add support for multiple protocol families
61e47caa89
dcacheview: upgrade to dCacheView 2.0.1
55f999ccb6
webdav/transfermanager: skip PnfsManager lookup if possible
05bede5d29
webdav: better logging if client disconnects during proxied transfer
ebbc63ddfb
webdav: don’t log an error if client disconnects before response sent
dc1dc553c0
PnfsManager: check restrictions before resolving path to PNFS-ID
bf0ef165a3
gplazma: scitoken validate ‘wlcg.ver’ claim
a2945ee543
[maven-release-plugin] prepare for next development iteration

Release 7.2.7

common-security

A better error message is logged when attempting to use a password-protected credential: java.io.IOException: Error decrypting private key: the password is incorrect or the PEM data is corrupted.

nfs

With this change, pool doesn’t enter an infinite retry loop if staging is not allowed on this pool.

pnfsmanager

Files written directly by encp (Enstore) can be read by dCache again. The revert cost 5% in performance on write. We will try to address the performance hit asap.

pool

This patch introduces the flag -noheader for use with rep ls -s that produces output similiar to what we used to see in previous dCache releases. In addition, a bug is fixed that prevented printing of precious and sticky file counts.

Pool statistic files are now copied without throwing an exception.

webdav

The HTTP-TPC should have fewer failed transfers when faced with many short-lived transfers.

It’s now possible to access a summary of HTTP-TPC transfers and configuration information via the WebDAV door’s info admin command.

Changelog 7.2.6..7.2.7

21d84be630
[maven-release-plugin] prepare release 7.2.7
a1a9a31503
common-security: prevent NPE on password protected cert
ee04084970
webdav: http-tpc update heuristics for failing transfer
5d53d308e7
Remove double whitespaces
9c202de000
chimera/pnfsmanager: restore compatibility with Enstore - restore ability to read files written directly by encp (Enstore client).
57bc507924
nfs: propagate permission denied on stage requests
699da511b4
webdav: http-tpc provide status information in ‘info’ admin command
1c2771773b
pool: fix rep ls formating
1f6b78d7b5
pool: fix pool statistic file overwrite option
6cec90347f
[maven-release-plugin] prepare for next development iteration

Release 7.2.6

core

Macaroons created with both a “path” caveat and “root” caveat work as expected.

dcache

The RemoteTransferManager now better describes the current state of a transfer.

gplazma

The scitoken gplazma plugin now supports extracting the sub and wlcg.groups claims in the same fashion as the oidc plugin.

pool

The current release fixed open queue flag when template defined by queue define class issue.

webdav

The WebDAV door’s http-tpc ls command now has the possibility to show the current state of the transfer. This may be useful diagnosing transfers spending a lot of time deciding on which pool to transfer should take place.

Macaroon requests with a path in the request URL and with a non-default door root now generate correct macaroons. WebDAV doors with ‘/’ root are unaffected by this problem.

Changelog 7.2.5..7.2.6

6e3816cb12
[maven-release-plugin] prepare release 7.2.6
5a0e179a94
RemoteTransferManager: update state description
32da6ef8ef
webdav: http-tpc add “prep” duration to ‘http-tpc ls’ command
04933fa957
webdav: http-tpc show whether transfer is queued on pool
1ffdd14c44
webdav: http-tpc update ‘http-tpc ls’ command to include state
e4ac684b4c
glazma: scitoken add support for additional principals
7333b9c923
srr: limit publisched endpoints to GLOBAL scope
59e00cd64b
core: fix macaroon behaviour with root and path
03011c2abe
webdav: ensure URL-path request macaroon have relative “path” caveat
e777580ad2
pool: fix open queue flag when template defined by queue define class
b28ba03f8e
srm-server: trs don’t cache tape names forever
bda7d82b0a
[maven-release-plugin] prepare for next development iteration

Release 7.2.5

dcache-core

The current release fixed NPE in TransferManager when no pool was selected before transfer is cancelled.

frontend

The current release fixed handling of SRR requests over IPv6.

gplazma

The scitoken plugin now supports OPs that publish their public keys without any corresponding ID (i.e., no kid value).

nfs

nfs can write into space reservations now.

pinmanager

A more detailed pinmanager log entry is provided for pin requests that are caught in a retry loop, and eventually time out.

resilience

A runtime exception triggered by side effect of logging has been fixed.

storagedescriptor

The script for generating SRR records (from the info service output) has been updated. The field lastupdated has been adjusted to the correct name latestupdate.

webdav

A bug is fixed where HTTP-TPC PULL request can fail (under heavy load) with the downloaded file being deleted, but dCache reports the transfer as successful.

dCache now provides a faster respones to the HTTP-TPC client (typically FTS) should the door decide to fail a transfer.

The WebDAV door should be now faster at accepting new transfers and faster at handling transfer completions when handling many small transfers.

The HTTP-TPC support in the WebDAV door is now e more robust to high number of transfers finishing concurrently.

xroot

Now it is possible to support kXR_delete as a write request on the pool.

xrootd4j

Library updates for xrootd4j which contains fix for Unix Protocol.

Packaging

Starting from dcache–7.2.5 the rpm package includes rsyslog configuration to produce classic log files in /var/log/dcache directory in addition to journal entries. To enable the logging into files the /etc/systemd/journald.conf should be configured as:

ForwardToSyslog=yes

Changelog 7.2.4..7.2.5

90ec3b60fe
[maven-release-plugin] prepare release 7.2.5
999fee1296
webdav: http-tpc improve throughput with short transfers
071e6914eb
frontend: fix handling of SRR requests over IPv6
b9ff1db9e6
rpm: include default rsyslog configuration
36c108d0de
webdav: http-tpc don’t wait if door fails transfer
5a49c6a506
webdav: http-tpc avoid resetting transfer state
7794d6bdb2
dcache-core: fix NPE in TransferManager when no pool was selected before transfer is cancelled
02244c83dc
resilience: fix runtime exception triggered by side effect of logging
59d5c6fa99
gplazma: scitoken add support for OPs that advertise keys without kid
417d0d9616
nfsv: convert UnixNumericXxxPrincipal into dCache analogs
f5b0e71449
webdav: http-tpc move transfer finalisation off of message queue
183a51bed6
pom.xml: update xrootd4j dependencies to 4.2.5/4.1.6/4.0.11
8e382cea4f
CanlContextFactory: throw FileNotFound for missing ca certs dir
a5ce12b212
pinmanager: provide more details information on timeout
6e5981526b
storagedescriptor: fix element name to ‘latestupdate’
d874e60106
dcache-xroot: support kXR_delete as a write request on the pool
184152bba0
[maven-release-plugin] prepare for next development iteration

Release 7.2.4

bulk

The way Bulk handles pin requests was changed to provide greater throughput and reliability.

ftp

The FTP door now provides more succinct information on pinboard, should use less CPU and take better advantage of the available cores.

libraries

The canl version was updated to version 2.6.0 to fix a bug that made dCache no longer accept certificates issued by a trusted CA after that CA updated their CA certificate while keeping the public/private key-pair the same.

pnfsmanager

Attempts to create a file where the parent directory is either missing or not a directory now provide clearer error messages.

pool

The pool is updated to support stateful NearlineStorage plugins that can fail during initialisation.

The pool now provides more information for bugs reported as java.lang.IllegalStateException: Handle is closed.

webdav

The HTTP-TPC response is improved if a client attempts to pull a file into a non-existing directory, or attempts to use an existing file as an ancestor directory.

xrootd

Improved exception handling for Xrootd.

The xrootd4j version was updated to 4.2.4

Changelog 7.2.3..7.2.4

7e8bd48424
[maven-release-plugin] prepare release 7.2.4
9313fad943
pom.xml: update to xrootd4j 4.2.4/4.1.5
bc5a8c3146
libraries: upgrade version of canl
45c8c8ae7a
dcache-bulk: fix handling of asynchronous pin requests
354c514596
dcache-frontend: support different attribute styles for bulk request
ff1ca76ef4
pool: update NearlineStorage to allow the start method to fail.
dbd4ca9c7e
pool: provide more information when handle is double-closed
caa67359e8
dcache-xrootd: Alternate fix for client write to closed checksum channel
5291aa0316
pnfsmanager: fix confusing error message.
331b056cbc
webdav: fix error handling for bad paths
f7ebd146b3
ftp: switch to more reasonable logging of Subject
84467de11c
[maven-release-plugin] prepare for next development iteration

Release 7.2.3

chimera

Added information to the cleaner admin info output, showing which pools are in the process of being cleaned.

A bug is fixed that triggered downstream errors when getUid() is called.

gftp door

The GFTP door was running into out-of-memory errors. This is fixed now.

frontend

api/v1/namespace/path is extended by adding ‘pin’ and ‘unpin’ activities to the POST method.

frontend, chimera

The behaviour of pin/unpin to api/v1/namespace/path POST was fixed to not remove the qos pin.

universal-spring-cell

This patch fixes a problem where a call to api/PoolManager returned 404 not found. The JSON serialized object is returned now as before.

xrootd

The xroot4j version was updated to 4.2.3

Changelog 7.2.2..7.2.3

90f94d25f3
[maven-release-plugin] prepare release 7.2.3
d7eda62d77
common-security: revert GSI/FTP to Java SSL
fbf0c0810e
dcache-frontend,dcache-chimera: fix pin/unpin in REST and NFS to use UID as request id
7d0c535992
srm-server: trs fix logging
eede36fa09
srm-server: trs – improve tape info fetching
8b6dcf34f9
dcache-chimera: add Unix principal conversion to AccessControlContext subject
5b250e549e
dcache-chimera: add which pools are being cleaned to cleaner admin info
4dc9f2b94b
dcache-frontend: add ‘pin’ and ‘unpin’ activities to namespace resource POST method
a2cf3d8ff2
srm-server: add trs logging
7624bbccde
srm-server: revert unnecessary type change
61f12b59d8
pom.xml: bump to xrootd4j 4.2.3
3364f25836
universal-spring-cell: allow for serialization of Optional
d09ef7f063
srm-server: trs – use Optional types, add unit tests
b5ca9b0b1d
[maven-release-plugin] prepare for next development iteration
324d79bfde
Update install.md

Release 7.2.2

billing

Billing will no longer throw a NullPointerException with certain information from an NFS door serving an NFSv3 client.

dcache

Running TPC HTTP(s) transfers with gridsite delegation against hosts on dual stack noticed this exception: java.lang.IllegalArgumentException: ‘[xxxx:xxxxx:xxx:xxx:fe03:7377]’ is not an IP string literal. this is now fixed

ftp

Now file sizes for LIST output in bytes is displayed.

The FTP door should be more robust against a (currently unknown) bug that results in the ftp session attempting to send progress queries to the pool after that is no longer possible.

gplazma

The banfile and scitoken plugins, and the two admin commands test login and explain login are updated to accept the username: prefix (e.g., username:paul). The prefix user: continues to work but is now deprecated.

nfs

An exception for access check in ChimeraVfs has been fixed.

Tag update regression was fixed.

pool

The current release improved error messaging if a setup file badly configures an HSM instance.

A bug is fixed where removing a create hsm statement in a pool’s setup file and running the reload command kills the pool.

srm-server

NullPointerException when a tape with expired requests does not have an oldest request age value has been fixed.

srmclient

The RPM packaging system and related software (e.g,. yum) now understands that the srm-client package requires java.

xroot

xrootd4j has been upgraded to 4.2.2 with a fixed regression introduced into the unix protocol plugin.

The documentation concerning TLS properties has been fixed.

Changelog 7.2.1..7.2.2

a4abef2012
[maven-release-plugin] prepare release 7.2.2
8006da747a
srm-server: Fix NullPointerException in SRM TRS
a8700de057
nfs: fix tag update regression
9866499238
srmclient: add Java–11 dependency
df4bc057b3
pool: add support for stateful NearlineStorage
65df64150f
core: update config testing to honour CellLifeCycleAware mocks
19e6d6c8a2
bootstrap: startup can hide bugs
e19425b263
dcache-xroot, book, skel: fix documentation concerning TLS properties
556106b7b0
bootstrap: fix two problems with how bugs are reported on startup
c3173860f3
ftp: try harder to ensure any timer tasks are cancelled
3b65a4fae7
pool: removing hsm with reload command kills pool
8e18034673
Handle IPv6 address when running HTTP(s) TPC with gridsite delegation
d7b600c972
Auto create table for TransferManager persistency
e0c8816ffd
gplazma: switch from user: to username:
38eac1c7e7
ftp: show file sizes for LIST output in bytes
f427ba7044
dcache-xroot: upgrade xrootd4j to 4.2.2
b6c8502e3e
util/frontend: change transfer rate to double
e7129fbcf3
bootstrap: don’t hide bugs in ‘create’ command
67825f2bcf
ftp: ensure cell is killed even if shutdown triggers a bug
7a55a57b65
nfs-vfs: fix access check in ChimeraVfs (fixes 19fa7cf2425)
9e65e4a96e
billing: fix NPE in billing from NFSv3 message
be19093a68
pool: improve error message on badly configured hsm instance
57f35a9242
[maven-release-plugin] prepare for next development iteration

Release 7.2.1

acl

The group ACEs now apply to desired group instead of to a users with the same numeric id.

gplazma

The WLCG Common JWT Profiles (v1.0) describes the wlcg.groups claim for expressing group-membership. We are increasingly seeing this being used as the OIDC equivalent to the group-membership expressed through the VOMS extension.

During the auth phase of the login process, the group membership information contained in a wlcg.groups claim from the OP is now available as OpenIdGroupPrincipal principals. Subsequent plugins (e.g., multimap) may be used to convert these principals to more directly useful principals.

srm-server

The current release fixed a potential NPE by not deleting tape objects and checking for as well as handling null values appropriately.

webdav

dCache now responds with a 507 (Insufficient Storage) on GET request if dCache must do a pool-to-pool internal-transfer or stage the file but no pools have sufficient free capacity to support this.

Changelog 7.2.0..7.2.1

09aac3a496
[maven-release-plugin] prepare release 7.2.1
672dbc04d2
all: reformat code using Google Style Sheet
6a36cbf12c
acl: set IDENTIFIER_GROUP flag if WHO is GROUP or OWNER_GROUP
a314c63346
Fix code format
ee5dcef4e9
webdav: return 507 if insufficient space on GET request
30c76de0c7
gplazma: scitoken make ExemptFromNamespaceChecks principal optional
c60c9f0841
gplazma/pnfsmanager: update namespace so scitoken ‘scope’ takes priority
6cbecb1993
srm-server: fix potential NPE in trs
462dd9bdf4
[maven-release-plugin] prepare for next development iteration

Release 7.2.0

Runtime

Java flight recorder

When debugging an issue on a running system often we need to collect jvm performance stats with ‘Java flight recorder’. Starting from release 7.2 the Java flight recorder attach listener is enabled by default. Site admins can collect and provide developers with additional information when high CPU load or memory consuption is observed as:

jcmd <pid> JFR.start duration=60s filename=/tmp/dcache.jfr,

Please note, that jcmd command is a part of java-11-openjdk-devel package (on RHEL and clones)

Handling of OutOfMemoryError

Depending which thread have received OutOfMemoryError the JVM might or might not exit. In a later cache, dCache might remain in an unpredictable state, where a component might be exposed as functional when its not.

With 7.2 we have update the java options to include ExitOnOutOfMemoryError, which forces JVM to exit when an OOM is detected.

NOTE: There are several situations when jvm generates an OutOfMemoryError. The ExitOnOutOfMemoryError option works ONLY when allocation in heap space fails.

Admin

Updated mina-sshd library to version 2.7.0 with various security enhancements.

Alarms

Documentation has been added specifying that the alarms service is not replicable.

Billing

Documentation has been added specifying that billing is replicable only if it shares an underlying store (i.e., an RDBMS instance).

Bulk

In preparation for future integration of tape recall optimization scheduling, some of the bulk processing has been adjusted to ensure the highest possible throughput to the scheduling endpoint. These changes are largely invisible to the user.

Cleaner

Previously, pools were cleaned synchronously one after another by the cleaner component. Doing so in parallel is expected to provide performance benefits.

The property cleaner.limits.threads now also controls the number of pools processed in parallel.

Checksums

dCache now supports the stronger hash functions SHA–1, SHA–256 and SHA–512.

Chimera (Shell)

It is now possible to add a label to a file.

The following admin commands for adding, removing and querying labels of a file object are available:

labels get path lists the labels of a file.

label rm path labelname removes the given label from a file.

label add path labelname adds a new label to a file.

ls_virtualdir labelname lists all the files having the given label.

Frontend

File labels support

If the file object has labels, these can now be queried, set and removed through a RESTful API.

To query labels, the curl -v http://localhost:3880/api/v1/namespace/path/filename\?label=true should be used.

To add a new label:

curl -X POST -H "Accept: application/json" -H "Content-Type: application/json" http://localhost:3880/api/v1/namespace/path/filename -d '{"action" : "set-label", "label" : "label" }'

And to remove an existing label:

curl -X POST -H "Accept: application/json" -H "Content-Type: application/json" http://localhost:3880/api/v1/namespace/path/filename -d '{"action" : "rm-label", "label" : "label" }'

Quota support

A RESTful API for the quota system (see further below) has been implemented; it allows for creating (POST), modifying (PATCH), listing (GET) and removing (DELETE) both user and group quotas.

All commands except GET require admin privileges. For GET, anonymous users are blocked. Otherwise, a full list of quotas can be obtained using:

curl -X GET -H "Accept: application/json" http://localhost:3880/api/v1/quota/user

curl -X GET -H "Accept: application/json" http://localhost:3880/api/v1/quota/group

The list can be limited to the quota for the user’s uid or primary gid by appending the query ‘\?user=true’, or, if known, a GET can be issued with the id as final path element:

curl -X GET -H "Accept: application/json" http://localhost:3880/api/v1/quota/user/4215

etc.

A full description of the API is available through the Swagger interface (https://localhost:3880/api/v1).

Bug fixes

In reporting movers (api/v1/pools/{pool}/movers), a problem with counts has been fixed so that both active and queued movers are included, and so that these are correctly broken down by their I/O activity (read/write).

In reporting pool usage (api/v1/pools/{pool}/usage), the csm (checksum) module object was missing profile information; this is now included.

PNFS Manager

User and group quota

Release 7.2 introduces a user and group quota implementation. The quota system operates on the chimera namespace. It periodically counts space usage broken down by Retention Policy, UID and GID, and stores these counts in two chimera DB tables –– t_user_quota and t_group_quota. Besides space counts these tables hold quota limits for each space count category by UID and by GID. A null for a limit means “no quota”. No entries in these tables for a UID or GIDs means “no quota” for that UID or GID. CUSTODIAL Retention Policy corresponds to files going to tape, REPLICA corresponds to disk-only files, and OUTPUT currently is not used in dCache. The quota system is controlled by the master switch:

        dcache.enable.quota

which by default is false. To enable the quota system, this variable has to be set to true on hosts running the PnfsManager and NFS services.

Queries executed by periodic updates may run for a significant amount of time on the chimera back-end. Therefore default update frequency is once a day:

    pnfsmanager.quota.update.interval=1
    (one-of?MILLISECONDS|SECONDS|MINUTES|HOURS|DAYS)pnfsmanager.quota.update.interval.time.unit = DAYS

The default has been chosen to match an installation having close to 1B chimera file entries. On smaller installations it is advisable to increase the frequency by decreasing the above interval value.

This means that quota enforcement is eventual. If users run out of allocated quota they still will be able to write data over quota until the next update happens. Likewise, removing over quota data will not be noticed by the quota system until the next update.

Internally the quota check involves holding two maps <uid, Quota> (user quota map) and <gid, Quota> (group quota map) in memory at the level of JdbcFs. When create entry function is called the uid, gid and Retention Policy (based on RetentionPolicy tag of the parent directory or default retention policy if there is no tag) are used to check if maps have necessary entries and exception is thrown if space counts exceed limits. Both maps are refreshed in memory from DB back-end every minute to pick possible new entries and new limits as well as to eventually catch the updated space usage counts.

From above follows that quota system is not going to work well with explicit space reservations (or space reservations expressed as WriteToken tag) on dCache installations having both disk-only and tape-backed pools. It requires RetentionPolicy tag. Although, since a general practice is to not to mix files with different retention policies under the same directories, this could work if RetentionPolicy tag is added to the containing directories.

On tape-only or disk-only installations, where Retention Policy of files is unambiguous, the quota can rely on default, system-wide Retention policy value. On tape-only system, the default value supplied with dCache distribution is good: (one-of?CUSTODIAL|REPLICA|OUTPUT)dcache.default-retention-policy=CUSTODIAL On disk-only systems, it needs to be set to : dcache.default-retention-policy=REPLICA

On a mixed system, a dcache.default-retention-policy=CUSTODIAL can be utilized for files bound to tape and RetentionPolicy tag can be set in the root of the directrory tree where disk-only files are stored.

An admin user can interact with the quota system via the PnfsManager admin interface. For this, the following commands are provided:

    set group quota [OPTIONS] <gid>  # Set group quota
    set user quota [OPTIONS] <uid>  # Set user quota

    remove group quota <gid>  # remove group quota
    remove user quota <uid>  # remove user quota

    show group quota [-gid=<string>] [-h]  # Print group quota
    show user quota [-h] [-uid=<string>]  # Print user quota

Use help <command> to learn how to use the command.

A user-facing quota interface is implemented through RESTFul frontend (see above).

File system statistics

Added functionality to cache total files and total space used on DB backend. Added two properties that control how often heavy queries that calculates the total number of files and total used space in the namespace us ru

    pnfsmanager.fs-stat-cache.time = 3600
    pnfsmanager.fs-stat-cache.time.unit = SECONDS

The resulting numbers are typically seen when running df command.

This patch has shown to speed up NFS availablity (among other things) after startup, especially on large installations.

Pool

Before dCache version 7.2, the scrubber process was starting with the pool startup. As a result, as long as a pool inventory is not complete, the scrubber will fail with Operation not allowed while repository is in state LOADING. With this version, the scrubber now will start only after the inventory is complete.

Security (common)

A general change to the internal SSL handlers to enable, where possible, a pass-through to native SSL on the node (usually OpenSSL) rather than using the Java implementation has been made. This change affects xrootd, https (webdav) and gsi (GridFTP). It should give improved performance (up to about 20%) when data is being encrypted.

SRM Manager

Optimally recalling data from tape is achieved by reducing the number of tape mounts and on-tape seeks by recalling as much volume as possible per mount. To that end a dedicated scheduling strategy exclusively for bring-online requests is introduced into the SrmManager. It is capable of clustering requests by tape according to a set of configurable criteria before passing them on to the rest of the system. In its current state it requires two files with information on targeted tapes, their capacity and occupancy as well as the mapping of tape-resident files to tape name. The file formats are described in the book.

The scheduler can be activated by adding the following line to the srmmanager section of the properties file:

srmmanager.plugins.enable-bring-online-clustering = true

and configured as described in the book.

It is important to note that the scheduler can only be effective when a dCache instance contains exactly one SrmManager.

WebUI (dCacheView)

The admin view now includes a separate page for plots of file lifetimes by pool group. All file lifetime plots have been simplified to a single data series (average lifetime). The viewport on the pool plots page has also been adjusted for easier scrolling on smaller (laptop) screens. The “print” button on the magnified plots dialog has been removed because it is not supported in all browsers.

For the new quota system, a quota view consisting of a read-only table has been added to the admin view. A similar table (limited to the user’s quotas) also appears in the UserProfile view.

XRootD

Several important fixes have gone in since 7.1.

First, the parsing of the ‘triedrc’ CGI was incomplete; it now properly accounts for error code/message pairings.

Second, a race condition which was causing certain clients to report that a file is not found when requesting the checksum for an upload (persist on successful close) transfer was previously fixed in the door. This fix has now been migrated to the pool (where it really belongs) and made more robust.

Finally, the third-party-copy client has been changed to conform with the behavior of the SLAC vanilla client so that it requests the exact number of remaining bytes on the final read chunk. This should eliminate an issue previously encountered with a particular CEPH plugin. The default chunk size has also been increased to 8 MiB to match the SLAC client.

Changelog from 7.1.0 to 7.2.0

12d1e46
[maven-release-plugin] prepare branch 7.2
ff88aac
nfs: fix typo pricipal -> principal
adec8b1
libs: use zookeeper 3.5.9
e3990d6
srm-server: make trs admin commands use rwlock
3438812
frontend: add service to provide Storage resource reporting
49d8ba6
dcache: Modernize PoolStatisticsV0
7effd50
webdav: update representation of symbolic links in HTML page
5e77f79
book: described trs admin commands
9071fa1
srm-server: add trs configuration admin commands
97c54ae
dcache: remove redundant file
c756308
src: don’t use com.google.common.io.ByteStreams
4b6338f
pool: remove dead code in org.dcache.pool.p2p.Companion
2fc1f8d
restful: provide functionality to remove or add labels of a file
84b45e6
namespace-chimera: ignore file size update on file flush
1a5b28d
book: added clarification to trs description
9602167
pool: scrubber should ignore the files if checksum is missing
862093f
billing: reduce unhelpful log spamming
278f712
pool: add Repository#waitForLoad method
05ee28c
webdav: add ‘http-tpc ls’ admin command
bf7610a
book: describe srm tape recall scheduler and file formats
e975365
build(deps): bump jetty-webapp from 9.4.41.v20210516 to 9.4.43.v20210629
a24b2c8
dcache-xroot: bump dependency to next xrootd4j release
cd8bf7b
src: unify all instances of logger to a same style
fae6d73
srm-server: add tape info provider SPI and JSON support
61decad
libs: use sshd-core 2.7.0
33dcbda
src: unify all instances of logger to a same style
c4f755b
ssh: remote unnecessary String#valueOf call in logging
4a1ffa1
webdav: fix regression from commit a2b252c07f
6f69d12
testing: consolidate builder classes
184c826
srmmanager: fix race condition in LoginBrokerSubscriber
53c4ba1
dcache-frontend: add RESTFUL quota resource
9212aa4
Revert “libs: use sshd-core 2.7.0”
f27c206
libs: use sshd-core 2.7.0
5b07905
pool/pnfsmanager: don’t accept a non-Enstore flush without locations
5e6d306
pool: update Kafka tape events to include additional information
7daf803
github: add citation file
eb588fd
dcache-core: add quota messaging and support in PnfsManager
6e9fefe
namespace: add delete quota to driver and PnfsManager admin commands
f338304
common: support requirements on parsed value in ByteSizeParser
6c34d24
common: Update ByteSizeParser to support multiple representations
5bf281a
common: update ByteSizeParser to follow builder pattern
6809690
pnfsmanager: fix regression introduced with commit 61a4e8dc4e
c53315f
common: provide more diagnostic information
a1f1aff
pool: improve flush information
bacafac
Motivation:
302db70
Revert “dcache-xroot: prevent attempt to write to channel for which checksum has been computed”
489af68
srm: add srm bring-online request scheduler
3b89d2e
dcache-xroot: prevent attempt to write to channel for which checksum has been computed
0a1d34d
restfulapi: add restful Api call to query the lable of a givin file object
e54e3c4
skel, pool.xml: fix regressions from 13117–13118
4697c7a
dcache-xroot: increase server-side frame size to 8 MiB
8567435
dcache-xroot: increase tpc client block/chunk size to 8 MiB
d0bae1f
Revert “libs: use sshd-core 2.7.0”
81e676d
libs: use sshd-core 2.7.0
d84c908
PnfsManager: improve wording of quota command
13215ee
rpm: add dcache-convert-authzdb-to-omnisession into list of packaged files
ae11f68
skel: add explicit replicable property even when value is false
03ca593
alarms, billing: specify shared rdbms in the case of replicated service
061ce56
build(deps): bump commons-compress from 1.19 to 1.21
a733b3f
gplazma: add util to convert authzdb file to omnisession
e160a3c
gplazma: add omnisession plugin
f33acca
srm: log transfer protocols in access log
7f1f63a
gplazma: multimap now supports OP in ‘oidc’ predicate
33868f5
dcache-frontend: remove unreachable catch clause
69f9ece
dcache: use charset constant instead of String-based dynamic discovery
a6a096d
core: fix NPE in TryCatchTemplate
0c19f64
dcache-core: add checksums sha–1, sha–256 and sha–512
37ae4b8
dcache-core: correct reported units of transfer size and speed
2cf3ce5
nearline: add simulated stage delay options to FileSystemNearlineStorage
a95a832
dcache-bulk: make SignalAware reusable
8218698
starting with a few macaroon hands-on examples
56154bc
qos-engine: use batching to notify concerning completed actions
554f4e2
dcache-xroot: move upload commit for persist-on-successful-close to the pool
61a4e8d
Implementation of UID/GID quota system in dCache
695b7c8
skel: remove obsolete python scripts
0067f72
pool: simplify inotify support in pool.
9a2c99a
gplazma: oidc check token validity if JWT
0ed6b58
chimerashell: add recursive option to ‘chown’ command
17e7c30
Added a link to the user guide to the main README
4d5f054
SRM : Fix IPV6 logging for SRM
75b395b
dcache-chimera: introducing parallel pool cleaning
ab900b8
dcache-qos: cast non-Object arrays to varargs Object[]
a5f691f
qos-adjuster: add waiting queue and state; polling
e5bbab7
dcache-chimera:add admin commands to support add/remove file lables
dc4e6db
frontend: fix wrong success on not defined action
ac3a3e4
pool: http-tpc add support for multiple trust stores
ab72c86
pool: update configuration to split http-tpc and remote-gsiftp CA settings
4154dc1
build(deps): bump jetty-servlets
ae40d43
checksum-module: add missing map to JSON info
e22b598
pool: update ReplicaDescriptor to implement AutoCloseable
b7d7ee0
pool: checksum scanner: don’t create EnumSet of OpenOption in a loop
d9ac931
chimera:adding labels metadata to file objects
5393593
dcache-qos (10.6): qos scanner admin interface, local client and remote receiver
3e29dd3
bulk: change PIN, UNPIN and UPDATE-QOS to breadth-first
0c555c8
build(deps): bump httpclient from 4.5.3 to 4.5.13
a705b0a
ssh: update DirectCommand to use SshOutputStream
ddd6c88
common-security, doors: switch to openssl where possible
f64a880
FsSqlDriver: use more conservative SQL query as default, so that system tests passs. Move more advanced SQL query to psql specific driver.
a96ca9e
User guide: bulk-request URLs contain /api/v1
1848354
srm: remove redundant Pgpass class
8b0bcba
ssh: drop custom AuthorizedKeyParser
cb7282c
dcache-qos (10.4/5): qos scanner handlers
f76991c
dcache-xrootd: upgrade xrootd4j to 4.1.1
a63e603
Namespace: add FS stat cache table
b19bb24
Fix typo
2133f6e
dcache-xroot: parsing of ‘triedrc’ CGI lacks separation by optional comma
77eadb3
nfs: add possibility to configure BerkeleyDB client store
2c09996
poolmanager: fix _waitingFor access outside of synchronized block
6fa08f5
ftp: add support for the SITE SYMLINK command
bf463de
pool: http allow client to send credentials when TLS is used
0a22bac
gplamza: oidc support fetching username and groups direct from OP
275df29
build: disable stack-trace trimming in surefire
e780ace
build(deps): bump jetty-io from 9.4.35.v20201120 to 9.4.40.v20210413
206185d
frontend: bump swagger version to 1.6.2
838d422
runtime: force JVM exit on OutOfMemoryError
c2bc2ae
runtime: enable Java flight recorder attach listener
939673d
dcache-qos (10.3): pool operation map and counters
84d65bc
dcache (request container): extract reusable deliverables and endpoint from unit test suite
6c7b4da
nearline: don’t use guava to sort the output of various ‘ls’ commands
9f69e5c
dcache-chimera: use RemotePoolMonitor to discover file locality
a31b429
poolmanager: remove redundant test in WatchdogThread
959cac7
ftp:clean dead code
6d7b6a4
book: add cleaner chapter
4a1b405
webdav: use Jetty’s built-in support for CORS
f26be37
pool: build list metadata entries only when we are going to use it
99b4d18
pool: use Set#of to construct O_READ and O_RW constants
e78b504
pool: remove invalid test case of configured space
b4f957b
pool: fix pool size reporting when static/runtime config is not defined.
f40adce
poolV4: fix reader/writer counts
14a5fc5
[maven-release-plugin] prepare for next development iteration