dCache 7.2 Release Notes

Highlights

This version introduces a quota system and support for file labels. Support for a dedicated scheduling strategy for bring-online requests was added to SRM-Manager.

Incompatibilities

This release doesn’t have any incompatibilities.

Release 7.2.43

xrootd4j

Update xrootd4j to a new version, that fixes a byte buffer leak in ZTNCredentialUtils.

Changelog 7.2.42..7.2.43

8f5efe75ac: [maven-release-plugin] prepare release 7.2.43
40d33b1883: ci: add gitlab-ci.yml
3d2ab50137: pom.xml: bump xrootd4j to next version (4.5.7, 4.4.8, 4.3.9, 4.2.13)
2d1f8f1167: [maven-release-plugin] prepare for next development iteration

Release 7.2.42

metadata

No user- or admin observable changes.

Changelog 7.2.41..7.2.42

563eaaaf0e: [maven-release-plugin] prepare release 7.2.42
e6c00c8a47: metadata: add (some) ORCIDs to CITATION.cff
eadea4cfeb: [maven-release-plugin] prepare for next development iteration

Release 7.2.42

### #

Changelog 7.2.41..7.2.42

563eaaaf0e: [maven-release-plugin] prepare release 7.2.42
e6c00c8a47: metadata: add (some) ORCIDs to CITATION.cff
eadea4cfeb: [maven-release-plugin] prepare for next development iteration

Release 7.2.41

gplazma

POSC works with tokens when user root is ‘/’.

Changelog 7.2.40..7.2.41

b6c5970027: [maven-release-plugin] prepare release 7.2.41
fc88dffcd5: Motivation:
0802e01f69: [maven-release-plugin] prepare for next development iteration

Release 7.2.40

Technical Release

Release 7.2.39

xrootd4j

update to next xrootd4j version (4.5.6, 4.4.7, 4.3.8, 4.2.12)

Changelog 7.2.38..7.2.39

403b7409c1: [maven-release-plugin] prepare release 7.2.39
dc1b4aae41: pom.xml: update to next xrootd4j version (4.5.6, 4.4.7, 4.3.8, 4.2.12)
62e2c4cf10: [maven-release-plugin] prepare for next development iteration

Release 7.2.38

pom

Update stable branches to include bug fix to the TPC client behavior so that it establishes a secure connection to its source when the source TLS setting is STRICT.

Changelog 7.2.37..7.2.38

1e8d1dc26f: [maven-release-plugin] prepare release 7.2.38
51de4d9ee7: pom.xml: update to next xrood4j version (4.5.5, 4.3.7, 4.2.11)
74a9e38251: [maven-release-plugin] prepare for next development iteration

Release 7.2.37

frontend

Use of relative path when issuing mv through the RESTful /api/v1/namespace was broken. This is fixed now.

Changelog 7.2.36..7.2.37

8276addb3e: [maven-release-plugin] prepare release 7.2.37
9e14e92945: dcache-frontend: resolve on parent of source when renaming
5c82be367c: [maven-release-plugin] prepare for next development iteration

Release 7.2.37

frontend

Use of relative path when issuing mv through the RESTful /api/v1/namespace was broken. This is fixed now.

Changelog 7.2.36..7.2.37

8276addb3e: [maven-release-plugin] prepare release 7.2.37
9e14e92945: dcache-frontend: resolve on parent of source when renaming
5c82be367c: [maven-release-plugin] prepare for next development iteration

Release 7.2.37

frontend

Use of relative path when issuing mv through the RESTful /api/v1/namespace was broken. This is fixed now.

Changelog 7.2.36..7.2.37

8276addb3e: [maven-release-plugin] prepare release 7.2.37
9e14e92945: dcache-frontend: resolve on parent of source when renaming
5c82be367c: [maven-release-plugin] prepare for next development iteration

Release 7.2.36

gplazma

dCache now handles malformed .info policies correctly: logging the problem but continuing to allow logins.

Changelog 7.2.35..7.2.36

c8f9332591: [maven-release-plugin] prepare release 7.2.36
8645aa0ddb: gplazma: x509 do not report bad .info files as a dCache bug
84f375a6d4: [maven-release-plugin] prepare for next development iteration

Release 7.2.35

frontend

Request for pin without lifetime unit specified works now.

telemetry

Caught exceptions are now logged without a stacktrace.

xroot

Clients should be blocked from pre-emptive removal if a file/directory already exists when doing copy.

Changelog 7.2.34..7.2.35

b45130a36d: [maven-release-plugin] prepare release 7.2.35
583873a5dc: dcache-frontend: convert empty to null when processing TimeUnit attribute
d4160fc5e1: dcache-xroot: return kXR_ItExists error code when file exists
314c9b0c3c: telemetry: Don’t log stacktrace for caught expceptions
1d6dc12d41: [maven-release-plugin] prepare for next development iteration

Release 7.2.35

frontend

Request for pin without lifetime unit specified works now.

telemetry

Caught exceptions are now logged without a stacktrace.

xroot

Clients should be blocked from pre-emptive removal if a file/directory already exists when doing copy.

Changelog 7.2.34..7.2.35

b45130a36d: [maven-release-plugin] prepare release 7.2.35
583873a5dc: dcache-frontend: convert empty to null when processing TimeUnit attribute
d4160fc5e1: dcache-xroot: return kXR_ItExists error code when file exists
314c9b0c3c: telemetry: Don’t log stacktrace for caught expceptions
1d6dc12d41: [maven-release-plugin] prepare for next development iteration

Release 7.2.34

common

Prevent two potential NPEs in common utility methods.

IndexOutOfBoundsException should not occur during timeseries histogram creation; stack trace is replaced by a logging statement.

gplazma

For xrootd “persist on successful close” (POSC), authorization is successful for both x509 and OIDC/Scitokens.

history

The history service should no longer generate the reported NaN stack trace in the histogram pipeline.

Changelog 7.2.33..7.2.34

bcf9acf151: [maven-release-plugin] prepare release 7.2.34
64333ff688: common: provisional fix for out of bounds error in timeseries histogram
997597b095: gplazma: allow upload permissions with MultiTargetRestriction
fbf9c310b6: common: prevent NPE by explict cast of (Double)double in ternary condition
8472767744: commons: add null check to nanToZero method
f0d2ac0003: dcache-history: avoid NaN stack trace in histogram conversion
ffb3a73581: [maven-release-plugin] prepare for next development iteration

Release 7.2.33

chimera

There should be less load on the trash table database of the cleaner and thus higher performance.

dcache-xroot

With xrootd.authz.anonymous-operations=READONLY, anonymous login allows world-readable files to be read; with xrootd.authz.anonymous-operations=NONE, it does not allow anonymous operations. This is for two-party interactions. Third-party copy with OIDC tokens will work regardless of the value given to this property.

Changelog 7.2.32..7.2.33

53ec9d5337: [maven-release-plugin] prepare release 7.2.33
cdf7f38be0: dcache-xroot: make handling of anonymous restrictions consistent
2470c0e636: pom.xml: update to next xrood4j version (4.5.4, 4.4.5, 4.3.6, 4.2.10)
89375abc52: dcache-chimera: reduce frequency of expensive cleaner inode deletion
3bebca89a7: [maven-release-plugin] prepare for next development iteration

Release 7.2.32

pool

Updated host certificate is available to https mover. Fixed p2p transfers over HTTPS.

Changelog 7.2.31..7.2.32

c4f50ad271: [maven-release-plugin] prepare release 7.2.32
92276f44ff: pool: reload host certificate if changed
4b16717151: [maven-release-plugin] prepare for next development iteration

Release 7.2.31

chimera

When receiving a reply from a pool it is no longer waiting for, cleaner-hsm will not send a new delete request or forget the currently-waited-on pool anymore. Logging is added to inform when cleaner-hsm receives a reply from a pool it is no longer waiting for, indicating that the timeout might be too small.

xroot

There should be no longer exceptions reporting no exception handler to handle the exception.

Changelog 7.2.30..7.2.31

f422670ddb: [maven-release-plugin] prepare release 7.2.31
06e13ef3ab: dcache-xroot: handle outbound errors on channel promise
26d65f2b95: dcache-chimera: cleaner better handles stale delete reply
0da9c6fbfa: book: add NOTE on domain naming, uniqueness
7c366f22eb: [maven-release-plugin] prepare for next development iteration

Release 7.2.30

Pool, Chimera

Prevent NullPointerExceptions in the cleaner and repeating pool restarted messages in PoolManager when pools report null as their connected hsm instances.

Changelog 7.2.29..7.2.30

489ab7aa58: [maven-release-plugin] prepare release 7.2.30
3a69017bbc: pool,chimera: fix pool pings having a null-valued hsm field
05652c9c5b: [maven-release-plugin] prepare for next development iteration

Release 7.2.29

chimera

A bug was fixed that threw a NullPointerException when the associated hsms for a pool are stored as null.

xroot

Update xrootd4j to support kXR_prefname in kXR_locate request.

Changelog 7.2.28..7.2.29

53002fff14: [maven-release-plugin] prepare release 7.2.29
7a626cd6ba: dcache-chimera: prevent PoolInformationBase NPE
4b945a05e2: pom.xml: update xrootd4j (4.5.3, etc.)
853430caa0: [maven-release-plugin] prepare for next development iteration

Release 7.2.28

chimera

Cleaner cells that do not have leadership will now more strictly enforce not triggering delete runs.

nfs-proxy

Proxy-IO with posix locks now works as expected.

Changelog 7.2.27..7.2.28

d5ff60f719: [maven-release-plugin] prepare release 7.2.28
571b1daebc: nfs-proxy: use open-stateid when proxying IO request to a pool
73616ac932: dcache-chimera: make sure non-leader does not execute clean run
7034c53d18: [maven-release-plugin] prepare for next development iteration

Release 7.2.27

nearline-storage

A NPE on error path has been fixed.

The race condition when flushing removed file is fixed now.

Changelog 7.2.26..7.2.27

5cdf29cc8f: [maven-release-plugin] prepare release 7.2.27
c0cfa5cec8: nearline-storage: fix race in flushing removed file
4ce83bc9b6: nearline-storage: fix NPE on error path
da729555d0: [maven-release-plugin] prepare for next development iteration

Release 7.2.26

frontend

The current release fixed type default of RESTful pool/nearline/queues resource

Changelog 7.2.25..7.2.26

b71c62f412: [maven-release-plugin] prepare release 7.2.26
038f3300cf: dcache-frontend: fix type default of RESTful pool/nearline/queues resource
d382f39280: [maven-release-plugin] prepare for next development iteration

Release 7.2.25

Release 7.2.24 was skipped due to technical reasons.

chimera

Pools that are no longer actively being cleaned were previously falsely still listed as active in the admin interface when the cleaning attempt ended due to an exception. This has been fixed.

core

If a pin is unpinned while a pin request for that same pin is still ongoing in pinmanager, pinning will not be retried.

Changelog 7.2.23..7.2.25

93db528d03: [maven-release-plugin] prepare release 7.2.25
dd6c0ec23e: pom.xml: fix file
4f07c497d5: [maven-release-plugin] prepare for next development iteration
6633988fcb: [maven-release-plugin] prepare release 7.2.24
7a4f9cc6c2: dcache-core: let pinmanager stop retrying pinning upon unpin state
bdcf04769b: dcache-chimera: disk-cleaner, error handling in parallel pool deletes

Release 7.2.23

gplazma2-grid

An empty requires field in IGTF policy is now accepted. Files with invalid or unexpected policies are ignored.

webdav

A bug was fixed that caused a NullPointerException while sending Kafka messages.

jquery was updated to version 3.6.1 and bootstrap was updated to version 5.2.1

Changelog 7.2.22..7.2.23

2f4956c120: [maven-release-plugin] prepare release 7.2.23
8b39f49a8f: dcache: NPE on removal via WebDAV and token
daeb0e3fb5: webdav: update 3rd party js libraries
62fbc57f9c: doc: add recipe to cookbook for migration/draining of resilient pools
7978ac1a94: gplazma2-grid: invalid IGTF policy file should fail with ParserException
2255486e27: [maven-release-plugin] prepare for next development iteration

Release 7.2.22

common

dCache now reports RemoteTransferManager as being replicable.

dcache-xroot

A possible thread starvation and inability to allow more clients to connect hase been fixed.

webdav

Fix a bug where the WebDAV door will continue attempting to kill a cancelled HTTP-TPC transfer indefinitely if the corresponding RemoteTransferManager cell has been switched off.

The current release fixed a NullPointerException when reporting on active transfers if the client has disconnected but the door has not yet cancelled the transfer.

xroot

Documentation now offers some explicit guidance as to pool configuration for xrootd reqarding memory.

Changelog 7.2.21..7.2.22

616282d9d1: [maven-release-plugin] prepare release 7.2.22
7194d3caef: dcache-xroot: interrupt doOnOpen thread in the event of channel inactive
a08ec82ca1: webdav: http-tpc do not keep retrying to cancel transfer
0b0a79c5ad: webdav: fix NPE if client aborts transfer
dd504beb38: dcache-xroot: add Book section on direct memory requirements for pools*
5c49935a70: pom.xml: update to latest xrootd4j bugfix
76725e6b46: [maven-release-plugin] prepare for next development iteration
408fc0fab2: properties: report that remote transfer manager is replicable

Release 7.2.21

dcache

A clarification on authorized keys format has been added.

Changelog 7.2.20..7.2.21

9983551668: [maven-release-plugin] prepare release 7.2.21
468e401ea6: Add clarification on authorized keys format
d8f4a594db: [maven-release-plugin] prepare for next development iteration

Release 7.2.20

dcache-bulk

Runtime exceptions do not kill the queue thread now and do not leave the request in a stalled state.

webdav

Existing monitoring information does not show HTTP-TPC transfers, which may result in admins being unaware of dCache activity.

This is now fixed and The Active Transfers page (from httpd) and the transfers resource (from frontend) now include HTTP-TPC transfers.

WebDAV HTTP-TPC now supports multiple RemoteTransferManager services, which allows for high-availability deployments. The WebDAV door and all RemoteTransferManager services must be upgraded to support this new feature.

Changelog 7.2.19..7.2.20

068c101757: [maven-release-plugin] prepare release 7.2.20
6bfeecf2cc: dcache-bulk: (version 1) handle unexpected exceptions in queue
4f0328d84c: webdav/httpd/frontend: add HTTP-TPC transfers to transfer list
ed926ca25e: webdav/transfermanager: support multiple RemoteTransferManager
9fd267feb6: [maven-release-plugin] prepare for next development iteration

Release 7.2.20

dcache-bulk

Runtime exceptions do not kill the queue thread now and do not leave the request in a stalled state.

webdav

Existing monitoring information does not show HTTP-TPC transfers, which may result in admins being unaware of dCache activity.

This is now fixed and The Active Transfers page (from httpd) and the transfers resource (from frontend) now include HTTP-TPC transfers.

Changelog 7.2.19..7.2.20

068c101757: [maven-release-plugin] prepare release 7.2.20
6bfeecf2cc: dcache-bulk: (version 1) handle unexpected exceptions in queue
4f0328d84c: webdav/httpd/frontend: add HTTP-TPC transfers to transfer list
ed926ca25e: webdav/transfermanager: support multiple RemoteTransferManager
9fd267feb6: [maven-release-plugin] prepare for next development iteration

Release 7.2.19

frontend, bulk

Both numeric and string values for PIN lifetime are valid now. Also, skipDirs for DELETE can be both boolean and string. All other current arguments are required to be strings, as before.

Changelog 7.2.18..7.2.19

176420e31e: [maven-release-plugin] prepare release 7.2.19
cd28af3b31: dcache-frontend,bulk: fix typing of argument values
81a032acaf: [maven-release-plugin] prepare for next development iteration

Release 7.2.18

frontend

Anonymous access receives a 401 error and doesn’t trigger a stack-trace when unlimitedVisibility is set to false.

clearOnSuccess, clearOnFailure and cancelOnFailure can have boolean and string values now.

pool

A bug is fixed where the checksum calculation would fail for empty files.

Changelog 7.2.17..7.2.18

d822ccf0a8: [maven-release-plugin] prepare release 7.2.18
67055873da: dcache-frontend: protect against RuntimeError in case of denied anonymous access
cd88da4c98: restore inadvertently removed ‘target’ extraction in toBulkRequest()
51291bfc85: dcache-frontend: bulk-requests POST, allow either string or boolean for boolean values
56e2b54a5f: pool: don’t treat an empty file as a sparse file
d92924806c: [maven-release-plugin] prepare for next development iteration

Release 7.2.17

ChimeraVfs

This release restores the ability to modify file levels.

info

The delay between messages sent by info was increased to reduce log messages of “DGA {…} triggering too quickly”.

pool

If the scrubber state file is empty, scrubber will now log a more helpful error message and proceed scrubbing the pool.

Changelog 7.2.16..7.2.17

617359eb22: [maven-release-plugin] prepare release 7.2.17
0fb0b3bdbc: pool: improve handling empty saved state by scrubber
d9b3dff4ef: dcache-xroot,pom.xml: bump xrootd4j to 4.3.1
ed50d1efee: dcache-xroot: flesh out channel inactive and exception caught
6a2daf6b18: info: increase delay between messages
44c356058a: ChimeraVfs: Check file level in addition to inode type when disallowing file size change
cfcd34c4e7: [maven-release-plugin] prepare for next development iteration

Release 7.2.16

dcache

Improved documentation for kafka Producer properties.

Improved error message for inexistent statistics path.

dcache-xroot

The current relase Updated master xrootd4j to 4.3.0.

frontend

NPE is fixed when SRR when space information is not available yet.

xrootd

Rendezvous TPC without requiring a JWT token to be passed by the third-party client is possible (again).

Changelog 7.2.15..7.2.16

ca1438defe: [maven-release-plugin] prepare release 7.2.16
3fec613542: dcache:improve documentation for kafka properties
93d7afab92: dcache-xrootd: fix TPC rendezvous to work with token authorization
b7193551e7: frontend: fix NPE in SRR when space information is not available yet.
149ea52085: dcache-xroot: update master xrootd4j to 4.3.0, and stable branches to next minor version
4201b7274b: dcache-core: improve error message for inexistent statistics path
ee4183d2c1: [maven-release-plugin] prepare for next development iteration

Release 7.2.15

script

Direct deprecation warnings are now redirected to stderr.

Changelog 7.2.14..7.2.15

79444b8bc1: [maven-release-plugin] prepare release 7.2.15
2bcb871750: info: fix broken unit-test, remove redundant one
1636a3ea95: dcache script: redirect deprecation warning to stderr
39302d27c4: [maven-release-plugin] prepare for next development iteration

Release 7.2.14

pnfsmanager

Doors no longer allow a user to discover whether or not a file or directory exists within directories they cannot access.

pom

Updated spring libs to version 5.2.20, CVE–2022–22965

pool

The pool now provides more information when a migration job was cancelled.

Changelog 7.2.13..7.2.14

a18f2adb33: [maven-release-plugin] prepare release 7.2.14
b95737de1e: pom: update spring libs to 5.2.20
f4c804724c: pool: describe why migration job was cancelled
e9ee3233ae: pnfsmanager: avoid leaking whether or not file exists
e263b9ace3: [maven-release-plugin] prepare for next development iteration

Release 7.2.13

dcache-xroot

A ZTN token can now be given without further downstream tokens expressed, and full authorization of the subject will take place.

Changelog 7.2.12..7.2.13

c13043a222: [maven-release-plugin] prepare release 7.2.13
5997a85b01: dcache-xroot: Allow ZTN authentication to function as fallback authorization
32d04ceb3d: [maven-release-plugin] prepare for next development iteration

Release 7.2.12

dcache-chimera

Cleaner batch delete exception has been fixed.

Chimera shell support has been added for getting, adding and removing sha1, sha156 and sha512 valued checksums.

User’s guid has been updated.

dcache-core

Sha–1 checksums can and must now correctly be requested as sha, as defined by RFC 3230.

frontend

A bug is fixed that inverted the property frontend.srr.public=true. True means now allow public access and false does not.

nfs

user subject always available in the processing thread.

srr

LinkLocal address will not be published.

util

nfs door has been updated to map broken file to IO error.

Changelog 7.2.11..7.2.12

f833efb11a: [maven-release-plugin] prepare release 7.2.12
2c2c48baf2: srr: don’t publish link local addresses (v2)
ee246b02a1: util: treat broken file as unrecoverable error in Trnasfer class
f498803529: nearline-storage: fix request count on error path in AbstractRequest#failed
e00ac992b4: nearline-storage: fix queued count when request initialization falied
da08b097eb: pool: add bunch of unit tests for NearlineStorageHandlerTest
8fede904e3: UserGuide: describe supported checksums, add example including sha
62757d2630: frontend: fix problem with making SRR resources public
3a1cb7e1ff: dcache-chimera: add sha checksum commands to chimera shell
9f04c91f51: dcache-core,common: fix user having to request sha-1, not sha
6a3c3e30a1: nfs: enforce subject propagation on RPC level
f6e1075015: dcache-chimera: fix cleaner batch delete exception
0e7f23bc05: [maven-release-plugin] prepare for next development iteration

Release 7.2.12

dcache-chimera

Cleaner batch delete exception has been fixed.

Chimera shell support has been added for getting, adding and removing sha1, sha156 and sha512 valued checksums.

User’s guid has been updated.

dcache-core

Sha–1 checksums can and must now correctly be requested as sha, as defined by RFC 3230.

frontend

A bug is fixed that inverted the property frontend.srr.public=true. True means now allow public access and false does not.

nfs

user subject always available in the processing thread.

srr

LinkLocal address will not be published.

util

nfs door has been updated to map broken file to IO error.

Changelog 7.2.11..7.2.12

f833efb11a: [maven-release-plugin] prepare release 7.2.12
2c2c48baf2: srr: don’t publish link local addresses (v2)
ee246b02a1: util: treat broken file as unrecoverable error in Trnasfer class
f498803529: nearline-storage: fix request count on error path in AbstractRequest#failed
e00ac992b4: nearline-storage: fix queued count when request initialization falied
da08b097eb: pool: add bunch of unit tests for NearlineStorageHandlerTest
8fede904e3: UserGuide: describe supported checksums, add example including sha
62757d2630: frontend: fix problem with making SRR resources public
3a1cb7e1ff: dcache-chimera: add sha checksum commands to chimera shell
9f04c91f51: dcache-core,common: fix user having to request sha-1, not sha
6a3c3e30a1: nfs: enforce subject propagation on RPC level
f6e1075015: dcache-chimera: fix cleaner batch delete exception
0e7f23bc05: [maven-release-plugin] prepare for next development iteration

Release 7.2.11

Changes affecting multiple services

The OIDC ‘sub’ (subject) and ‘jti’ (JWT ID) claims are logged to access log file for WebDav, frontend and SRM doors if OIDC is used.

admin-shell

Fix the line termination printed to console. It’s \r\n now.

bulk

dCache now returns 404 instead of 403 if a submitted id in GET or PATCH bulk-request is not recognized by the service.

Jobs which originally got stuck in STARTED state now complete; their failure information contains the reason for premature completion.

frontend

The property frontend.srr.public=true|false is added to frontend to control SRR resource restriction. The default value is false.

ftp

The FTP cell is killed asynchronously now after the session terminates so the door will remain responsive when many clients terminate their FTP session concurrently.

pool

HTTP-TPC transfers now prefer IPv6 address, if both endpoints support it.

If a HSM script survives SIGTERM it will now be stopped with SIGKILL and a warning is logged. This reduces expired or canceled requests staying in CANCELED state.

If a NFS mover is killed twice it’s now logged without a stack trace. This provides more stable NFS error recovery, too.

Changelog 7.2.10..7.2.11

abafc5e050: [maven-release-plugin] prepare release 7.2.11
2328d630e4: frontend: make srr resources public
f3f48d6e65: webdav/frontend/srm/gplazma: log OIDC ‘sub’ and ‘jti’ claims
7653bc78df: ftp: kill cell asynchronously
b2de5dcdcd: pool: http-tpc prefer IPv6 address
bd807f0da4: pool: handle multiple shutdowns of a nfs mover
3607a08821: dcache-bulk: aborted request gets stuck in the STARTED state
dc2a01fea6: pool: try to forcefully kill HSM process if needed
8fbab1884e: admin shell: fix line termination in SshOutputStream
e858de6448: [maven-release-plugin] prepare for next development iteration
0970afbe2d: bulk, frontend-rest: return 404, not 403, when request id does not exist

Release 7.2.10

common

The current release fixed certificate reload when updated.

There are places where dCache configuration requires an absolute path. In some places attempting to use a relative path will yield an unhepful null in the log file. This is now fixed, so a meaningful error message is logged instead.

pool

An NPE on attempt by client to read from a broken replica is fixed now.

rest

webdav

The current release fixed handling of webdav.authz.allowed-paths so it no longer triggers a NullPointerException.

Changelog 7.2.9..7.2.10

034deb4042: [maven-release-plugin] prepare release 7.2.10
bfff8d424f: webdav: fix NPE if ‘webdav.authz.allowed-paths’ disallows a request
16f7723398: common: provide reasonable error message if path is not absolute
55cfd70b30: pool: fix NPE on attempt by client to read from a broken replica
1de476bd75: common-security: re-read server credentials on context re-creation
9afb28cf3e: rest api: add checksum parameter to namespace resource
65359b538f: [maven-release-plugin] prepare for next development iteration

Release 7.2.9

Pool

The current relase fixed a problem where a file’s tape location that used to exist in the namespace and was subsequently removed can reappear if the file made precious and flushed to tape.

TransferManager

TransferManager now logs bugs with the corresponding stack-trace, making fixes any such bug easier.

dcahe-core

When asking for count pins unpin-failed, only report the number of pins in state FAILED_TO_UNPIN and not PINNING.

srm-server

Now log trs tape queue states when tape requests are added.

Changelog 7.2.8..7.2.9

3f78e6f5b7: [maven-release-plugin] prepare release 7.2.9
3eb7542a49: CopyManager: reply success when the target transfer completes.
e41bd289b0: frontend: update dCacheView to v2.0.2
9b38e4fba2: pool: only send new URLs when flushing file
dea284457d: TransferManager: log bugs with corresponding stack-trace
255ba86249: dcache-core: fix admin command to query pins by state
2fc6ff0406: srm-server: log trs tape queue states when tape requests are added
3268e0ace8: readme: Remove newline at the end
8356c837d4: Update config-xrootd.md
6458b47484: Update config-xrootd.md
2e124bb413: book: Add chapter about telemetry cell
68ade2a7e0: [maven-release-plugin] prepare for next development iteration

Release 7.2.8

dcacheview

This patch introduces some improved icons and other minor visual tweaks.

gplazma

The ‘scitoken’ gplazma plugin will now accept WLCG AuthZ tokens without direct authorisation statements. When accepting such a token, the plugin refrains from adding any of the additional principals from the configuration property; only the principals directly from the token are added.

dCache will now reject WLCG-AuthZ-JWT profile tokens with a ‘wlcg.ver’ claim that it does not support. Other tokens (e.g., SciTokens) are unaffected by this change.

The token-based authentication gPlazma plugins (scitoken, oidc) now include the identity of the OP that created the token. The multimap plugin is updated to support matching on this principal.

loginbroker

This patch fixes SRM based upload or download where the client requests an xroot-based transfer. The SRM door will now also consider any xroot door with xrootd.security.tls.mode configured to OPTIONAL when building a TURL targeting either xroot or xroots protocols.

pnfsmanager

PnfsManager now has two commands, ‘reset chimera stats’ and ‘reset stats’, to support resetting the gauge and counter statistics available through the ‘info’ command.

dCache no longer leaks information about whether or not files exist when using macaroons or Scitokens / WLCG AuthZ JWT profile tokens.

webdav

The WebDAV door cannot send the HTTP response to some HTTP request if the client has already disconnected. dCache no longer logs an error that it cannot send the HTTP response.

A client that disconnects during a proxied HTTP transfer (GET or PUT) is no longer logged in the WebDAV door’s log file; instead, it is logged in the cell’s pinboard. The billing message is updated to make it clearer what went wrong.

webdav, transfermanager

dCache provides marginally better performance for HTTP-TPC, which starts to become significant when transferring many small files.

Changelog 7.2.7..7.2.8

6351804eb5: [maven-release-plugin] prepare release 7.2.8
63e035cfd6: gplazma: scitoken fix/improve support for group-based AuthN in WLCG profile
304bc36936: gplazma: scitoken/oidc/multimap add support for OP principal
3b7d463834: skel: bump logback to 1.2.10
8534813023: pnfsmanager: add support for resetting gauge and counter statistics
8229fc9eb8: loginbroker: add support for multiple protocol families
61e47caa89: dcacheview: upgrade to dCacheView 2.0.1
55f999ccb6: webdav/transfermanager: skip PnfsManager lookup if possible
05bede5d29: webdav: better logging if client disconnects during proxied transfer
ebbc63ddfb: webdav: don’t log an error if client disconnects before response sent
dc1dc553c0: PnfsManager: check restrictions before resolving path to PNFS-ID
bf0ef165a3: gplazma: scitoken validate ‘wlcg.ver’ claim
a2945ee543: [maven-release-plugin] prepare for next development iteration

Release 7.2.7

common-security

A better error message is logged when attempting to use a password-protected credential: java.io.IOException: Error decrypting private key: the password is incorrect or the PEM data is corrupted.

nfs

With this change, pool doesn’t enter an infinite retry loop if staging is not allowed on this pool.

pnfsmanager

Files written directly by encp (Enstore) can be read by dCache again. The revert cost 5% in performance on write. We will try to address the performance hit asap.

pool

This patch introduces the flag -noheader for use with rep ls -s that produces output similiar to what we used to see in previous dCache releases. In addition, a bug is fixed that prevented printing of precious and sticky file counts.

Pool statistic files are now copied without throwing an exception.

webdav

The HTTP-TPC should have fewer failed transfers when faced with many short-lived transfers.

It’s now possible to access a summary of HTTP-TPC transfers and configuration information via the WebDAV door’s info admin command.

Changelog 7.2.6..7.2.7

21d84be630: [maven-release-plugin] prepare release 7.2.7
a1a9a31503: common-security: prevent NPE on password protected cert
ee04084970: webdav: http-tpc update heuristics for failing transfer
5d53d308e7: Remove double whitespaces
9c202de000: chimera/pnfsmanager: restore compatibility with Enstore - restore ability to read files written directly by encp (Enstore client).
57bc507924: nfs: propagate permission denied on stage requests
699da511b4: webdav: http-tpc provide status information in ‘info’ admin command
1c2771773b: pool: fix rep ls formating
1f6b78d7b5: pool: fix pool statistic file overwrite option
6cec90347f: [maven-release-plugin] prepare for next development iteration

Release 7.2.6

core

Macaroons created with both a “path” caveat and “root” caveat work as expected.

dcache

The RemoteTransferManager now better describes the current state of a transfer.

gplazma

The scitoken gplazma plugin now supports extracting the sub and wlcg.groups claims in the same fashion as the oidc plugin.

pool

The current release fixed open queue flag when template defined by queue define class issue.

webdav

The WebDAV door’s http-tpc ls command now has the possibility to show the current state of the transfer. This may be useful diagnosing transfers spending a lot of time deciding on which pool to transfer should take place.

Macaroon requests with a path in the request URL and with a non-default door root now generate correct macaroons. WebDAV doors with ‘/’ root are unaffected by this problem.

Changelog 7.2.5..7.2.6

6e3816cb12: [maven-release-plugin] prepare release 7.2.6
5a0e179a94: RemoteTransferManager: update state description
32da6ef8ef: webdav: http-tpc add “prep” duration to ‘http-tpc ls’ command
04933fa957: webdav: http-tpc show whether transfer is queued on pool
1ffdd14c44: webdav: http-tpc update ‘http-tpc ls’ command to include state
e4ac684b4c: glazma: scitoken add support for additional principals
7333b9c923: srr: limit publisched endpoints to GLOBAL scope
59e00cd64b: core: fix macaroon behaviour with root and path
03011c2abe: webdav: ensure URL-path request macaroon have relative “path” caveat
e777580ad2: pool: fix open queue flag when template defined by queue define class
b28ba03f8e: srm-server: trs don’t cache tape names forever
bda7d82b0a: [maven-release-plugin] prepare for next development iteration

Release 7.2.5

dcache-core

The current release fixed NPE in TransferManager when no pool was selected before transfer is cancelled.

frontend

The current release fixed handling of SRR requests over IPv6.

gplazma

The scitoken plugin now supports OPs that publish their public keys without any corresponding ID (i.e., no kid value).

nfs

nfs can write into space reservations now.

pinmanager

A more detailed pinmanager log entry is provided for pin requests that are caught in a retry loop, and eventually time out.

resilience

A runtime exception triggered by side effect of logging has been fixed.

storagedescriptor

The script for generating SRR records (from the info service output) has been updated. The field lastupdated has been adjusted to the correct name latestupdate.

webdav

A bug is fixed where HTTP-TPC PULL request can fail (under heavy load) with the downloaded file being deleted, but dCache reports the transfer as successful.

dCache now provides a faster respones to the HTTP-TPC client (typically FTS) should the door decide to fail a transfer.

The WebDAV door should be now faster at accepting new transfers and faster at handling transfer completions when handling many small transfers.

The HTTP-TPC support in the WebDAV door is now e more robust to high number of transfers finishing concurrently.

xroot

Now it is possible to support kXR_delete as a write request on the pool.

xrootd4j

Library updates for xrootd4j which contains fix for Unix Protocol.

Packaging

Starting from dcache–7.2.5 the rpm package includes rsyslog configuration to produce classic log files in /var/log/dcache directory in addition to journal entries. To enable the logging into files the /etc/systemd/journald.conf should be configured as:

ForwardToSyslog=yes

Changelog 7.2.4..7.2.5

90ec3b60fe: [maven-release-plugin] prepare release 7.2.5
999fee1296: webdav: http-tpc improve throughput with short transfers
071e6914eb: frontend: fix handling of SRR requests over IPv6
b9ff1db9e6: rpm: include default rsyslog configuration
36c108d0de: webdav: http-tpc don’t wait if door fails transfer
5a49c6a506: webdav: http-tpc avoid resetting transfer state
7794d6bdb2: dcache-core: fix NPE in TransferManager when no pool was selected before transfer is cancelled
02244c83dc: resilience: fix runtime exception triggered by side effect of logging
59d5c6fa99: gplazma: scitoken add support for OPs that advertise keys without kid
417d0d9616: nfsv: convert UnixNumericXxxPrincipal into dCache analogs
f5b0e71449: webdav: http-tpc move transfer finalisation off of message queue
183a51bed6: pom.xml: update xrootd4j dependencies to 4.2.5/4.1.6/4.0.11
8e382cea4f: CanlContextFactory: throw FileNotFound for missing ca certs dir
a5ce12b212: pinmanager: provide more details information on timeout
6e5981526b: storagedescriptor: fix element name to ‘latestupdate’
d874e60106: dcache-xroot: support kXR_delete as a write request on the pool
184152bba0: [maven-release-plugin] prepare for next development iteration

Release 7.2.4

bulk

The way Bulk handles pin requests was changed to provide greater throughput and reliability.

ftp

The FTP door now provides more succinct information on pinboard, should use less CPU and take better advantage of the available cores.

libraries

The canl version was updated to version 2.6.0 to fix a bug that made dCache no longer accept certificates issued by a trusted CA after that CA updated their CA certificate while keeping the public/private key-pair the same.

pnfsmanager

Attempts to create a file where the parent directory is either missing or not a directory now provide clearer error messages.

pool

The pool is updated to support stateful NearlineStorage plugins that can fail during initialisation.

The pool now provides more information for bugs reported as java.lang.IllegalStateException: Handle is closed.

webdav

The HTTP-TPC response is improved if a client attempts to pull a file into a non-existing directory, or attempts to use an existing file as an ancestor directory.

xrootd

Improved exception handling for Xrootd.

The xrootd4j version was updated to 4.2.4

Changelog 7.2.3..7.2.4

7e8bd48424: [maven-release-plugin] prepare release 7.2.4
9313fad943: pom.xml: update to xrootd4j 4.2.4/4.1.5
bc5a8c3146: libraries: upgrade version of canl
45c8c8ae7a: dcache-bulk: fix handling of asynchronous pin requests
354c514596: dcache-frontend: support different attribute styles for bulk request
ff1ca76ef4: pool: update NearlineStorage to allow the start method to fail.
dbd4ca9c7e: pool: provide more information when handle is double-closed
caa67359e8: dcache-xrootd: Alternate fix for client write to closed checksum channel
5291aa0316: pnfsmanager: fix confusing error message.
331b056cbc: webdav: fix error handling for bad paths
f7ebd146b3: ftp: switch to more reasonable logging of Subject
84467de11c: [maven-release-plugin] prepare for next development iteration

Release 7.2.3

chimera

Added information to the cleaner admin info output, showing which pools are in the process of being cleaned.

A bug is fixed that triggered downstream errors when getUid() is called.

gftp door

The GFTP door was running into out-of-memory errors. This is fixed now.

frontend

api/v1/namespace/path is extended by adding ‘pin’ and ‘unpin’ activities to the POST method.

frontend, chimera

The behaviour of pin/unpin to api/v1/namespace/path POST was fixed to not remove the qos pin.

universal-spring-cell

This patch fixes a problem where a call to api/PoolManager returned 404 not found. The JSON serialized object is returned now as before.

xrootd

The xroot4j version was updated to 4.2.3

Changelog 7.2.2..7.2.3

90f94d25f3: [maven-release-plugin] prepare release 7.2.3
d7eda62d77: common-security: revert GSI/FTP to Java SSL
fbf0c0810e: dcache-frontend,dcache-chimera: fix pin/unpin in REST and NFS to use UID as request id
7d0c535992: srm-server: trs fix logging
eede36fa09: srm-server: trs – improve tape info fetching
8b6dcf34f9: dcache-chimera: add Unix principal conversion to AccessControlContext subject
5b250e549e: dcache-chimera: add which pools are being cleaned to cleaner admin info
4dc9f2b94b: dcache-frontend: add ‘pin’ and ‘unpin’ activities to namespace resource POST method
a2cf3d8ff2: srm-server: add trs logging
7624bbccde: srm-server: revert unnecessary type change
61f12b59d8: pom.xml: bump to xrootd4j 4.2.3
3364f25836: universal-spring-cell: allow for serialization of Optional
d09ef7f063: srm-server: trs – use Optional types, add unit tests
b5ca9b0b1d: [maven-release-plugin] prepare for next development iteration
324d79bfde: Update install.md

Release 7.2.2

billing

Billing will no longer throw a NullPointerException with certain information from an NFS door serving an NFSv3 client.

dcache

Running TPC HTTP(s) transfers with gridsite delegation against hosts on dual stack noticed this exception: java.lang.IllegalArgumentException: ‘[xxxx:xxxxx:xxx:xxx:fe03:7377]’ is not an IP string literal. this is now fixed

ftp

Now file sizes for LIST output in bytes is displayed.

The FTP door should be more robust against a (currently unknown) bug that results in the ftp session attempting to send progress queries to the pool after that is no longer possible.

gplazma

The banfile and scitoken plugins, and the two admin commands test login and explain login are updated to accept the username: prefix (e.g., username:paul). The prefix user: continues to work but is now deprecated.

nfs

An exception for access check in ChimeraVfs has been fixed.

Tag update regression was fixed.

pool

The current release improved error messaging if a setup file badly configures an HSM instance.

A bug is fixed where removing a create hsm statement in a pool’s setup file and running the reload command kills the pool.

srm-server

NullPointerException when a tape with expired requests does not have an oldest request age value has been fixed.

srmclient

The RPM packaging system and related software (e.g,. yum) now understands that the srm-client package requires java.

xroot

xrootd4j has been upgraded to 4.2.2 with a fixed regression introduced into the unix protocol plugin.

The documentation concerning TLS properties has been fixed.

Changelog 7.2.1..7.2.2

a4abef2012: [maven-release-plugin] prepare release 7.2.2
8006da747a: srm-server: Fix NullPointerException in SRM TRS
a8700de057: nfs: fix tag update regression
9866499238: srmclient: add Java–11 dependency
df4bc057b3: pool: add support for stateful NearlineStorage
65df64150f: core: update config testing to honour CellLifeCycleAware mocks
19e6d6c8a2: bootstrap: startup can hide bugs
e19425b263: dcache-xroot, book, skel: fix documentation concerning TLS properties
556106b7b0: bootstrap: fix two problems with how bugs are reported on startup
c3173860f3: ftp: try harder to ensure any timer tasks are cancelled
3b65a4fae7: pool: removing hsm with reload command kills pool
8e18034673: Handle IPv6 address when running HTTP(s) TPC with gridsite delegation
d7b600c972: Auto create table for TransferManager persistency
e0c8816ffd: gplazma: switch from user: to username:
38eac1c7e7: ftp: show file sizes for LIST output in bytes
f427ba7044: dcache-xroot: upgrade xrootd4j to 4.2.2
b6c8502e3e: util/frontend: change transfer rate to double
e7129fbcf3: bootstrap: don’t hide bugs in ‘create’ command
67825f2bcf: ftp: ensure cell is killed even if shutdown triggers a bug
7a55a57b65: nfs-vfs: fix access check in ChimeraVfs (fixes 19fa7cf2425)
9e65e4a96e: billing: fix NPE in billing from NFSv3 message
be19093a68: pool: improve error message on badly configured hsm instance
57f35a9242: [maven-release-plugin] prepare for next development iteration

Release 7.2.1

acl

The group ACEs now apply to desired group instead of to a users with the same numeric id.

gplazma

The WLCG Common JWT Profiles (v1.0) describes the wlcg.groups claim for expressing group-membership. We are increasingly seeing this being used as the OIDC equivalent to the group-membership expressed through the VOMS extension.

During the auth phase of the login process, the group membership information contained in a wlcg.groups claim from the OP is now available as OpenIdGroupPrincipal principals. Subsequent plugins (e.g., multimap) may be used to convert these principals to more directly useful principals.

srm-server

The current release fixed a potential NPE by not deleting tape objects and checking for as well as handling null values appropriately.

webdav

dCache now responds with a 507 (Insufficient Storage) on GET request if dCache must do a pool-to-pool internal-transfer or stage the file but no pools have sufficient free capacity to support this.

Changelog 7.2.0..7.2.1

09aac3a496: [maven-release-plugin] prepare release 7.2.1
672dbc04d2: all: reformat code using Google Style Sheet
6a36cbf12c: acl: set IDENTIFIER_GROUP flag if WHO is GROUP or OWNER_GROUP
a314c63346: Fix code format
ee5dcef4e9: webdav: return 507 if insufficient space on GET request
30c76de0c7: gplazma: scitoken make ExemptFromNamespaceChecks principal optional
c60c9f0841: gplazma/pnfsmanager: update namespace so scitoken ‘scope’ takes priority
6cbecb1993: srm-server: fix potential NPE in trs
462dd9bdf4: [maven-release-plugin] prepare for next development iteration

Release 7.2.0

Runtime

Java flight recorder

When debugging an issue on a running system often we need to collect jvm performance stats with ‘Java flight recorder’. Starting from release 7.2 the Java flight recorder attach listener is enabled by default. Site admins can collect and provide developers with additional information when high CPU load or memory consuption is observed as:

jcmd <pid> JFR.start duration=60s filename=/tmp/dcache.jfr,

Please note, that jcmd command is a part of java-11-openjdk-devel package (on RHEL and clones)

Handling of OutOfMemoryError

Depending which thread have received OutOfMemoryError the JVM might or might not exit. In a later cache, dCache might remain in an unpredictable state, where a component might be exposed as functional when its not.

With 7.2 we have update the java options to include ExitOnOutOfMemoryError, which forces JVM to exit when an OOM is detected.

NOTE: There are several situations when jvm generates an OutOfMemoryError. The ExitOnOutOfMemoryError option works ONLY when allocation in heap space fails.

Admin

Updated mina-sshd library to version 2.7.0 with various security enhancements.

Alarms

Documentation has been added specifying that the alarms service is not replicable.

Billing

Documentation has been added specifying that billing is replicable only if it shares an underlying store (i.e., an RDBMS instance).

Bulk

In preparation for future integration of tape recall optimization scheduling, some of the bulk processing has been adjusted to ensure the highest possible throughput to the scheduling endpoint. These changes are largely invisible to the user.

Cleaner

Previously, pools were cleaned synchronously one after another by the cleaner component. Doing so in parallel is expected to provide performance benefits.

The property cleaner.limits.threads now also controls the number of pools processed in parallel.

Checksums

dCache now supports the stronger hash functions SHA–1, SHA–256 and SHA–512.

Chimera (Shell)

It is now possible to add a label to a file.

The following admin commands for adding, removing and querying labels of a file object are available:

labels get path lists the labels of a file.

label rm path labelname removes the given label from a file.

label add path labelname adds a new label to a file.

ls_virtualdir labelname lists all the files having the given label.

Frontend

File labels support

If the file object has labels, these can now be queried, set and removed through a RESTful API.

To query labels, the curl -v http://localhost:3880/api/v1/namespace/path/filename\?label=true should be used.

To add a new label:

curl -X POST -H "Accept: application/json" -H "Content-Type: application/json" http://localhost:3880/api/v1/namespace/path/filename -d '{"action" : "set-label", "label" : "label" }'

And to remove an existing label:

curl -X POST -H "Accept: application/json" -H "Content-Type: application/json" http://localhost:3880/api/v1/namespace/path/filename -d '{"action" : "rm-label", "label" : "label" }'

Quota support

A RESTful API for the quota system (see further below) has been implemented; it allows for creating (POST), modifying (PATCH), listing (GET) and removing (DELETE) both user and group quotas.

All commands except GET require admin privileges. For GET, anonymous users are blocked. Otherwise, a full list of quotas can be obtained using:

curl -X GET -H "Accept: application/json" http://localhost:3880/api/v1/quota/user

curl -X GET -H "Accept: application/json" http://localhost:3880/api/v1/quota/group

The list can be limited to the quota for the user’s uid or primary gid by appending the query ‘\?user=true’, or, if known, a GET can be issued with the id as final path element:

curl -X GET -H "Accept: application/json" http://localhost:3880/api/v1/quota/user/4215

etc.

A full description of the API is available through the Swagger interface (https://localhost:3880/api/v1).

Bug fixes

In reporting movers (api/v1/pools/{pool}/movers), a problem with counts has been fixed so that both active and queued movers are included, and so that these are correctly broken down by their I/O activity (read/write).

In reporting pool usage (api/v1/pools/{pool}/usage), the csm (checksum) module object was missing profile information; this is now included.

PNFS Manager

User and group quota

Release 7.2 introduces a user and group quota implementation. The quota system operates on the chimera namespace. It periodically counts space usage broken down by Retention Policy, UID and GID, and stores these counts in two chimera DB tables –– t_user_quota and t_group_quota. Besides space counts these tables hold quota limits for each space count category by UID and by GID. A null for a limit means “no quota”. No entries in these tables for a UID or GIDs means “no quota” for that UID or GID. CUSTODIAL Retention Policy corresponds to files going to tape, REPLICA corresponds to disk-only files, and OUTPUT currently is not used in dCache. The quota system is controlled by the master switch:

        dcache.enable.quota

which by default is false. To enable the quota system, this variable has to be set to true on hosts running the PnfsManager and NFS services.

Queries executed by periodic updates may run for a significant amount of time on the chimera back-end. Therefore default update frequency is once a day:

    pnfsmanager.quota.update.interval=1
    (one-of?MILLISECONDS|SECONDS|MINUTES|HOURS|DAYS)pnfsmanager.quota.update.interval.time.unit = DAYS

The default has been chosen to match an installation having close to 1B chimera file entries. On smaller installations it is advisable to increase the frequency by decreasing the above interval value.

This means that quota enforcement is eventual. If users run out of allocated quota they still will be able to write data over quota until the next update happens. Likewise, removing over quota data will not be noticed by the quota system until the next update.

Internally the quota check involves holding two maps <uid, Quota> (user quota map) and <gid, Quota> (group quota map) in memory at the level of JdbcFs. When create entry function is called the uid, gid and Retention Policy (based on RetentionPolicy tag of the parent directory or default retention policy if there is no tag) are used to check if maps have necessary entries and exception is thrown if space counts exceed limits. Both maps are refreshed in memory from DB back-end every minute to pick possible new entries and new limits as well as to eventually catch the updated space usage counts.

From above follows that quota system is not going to work well with explicit space reservations (or space reservations expressed as WriteToken tag) on dCache installations having both disk-only and tape-backed pools. It requires RetentionPolicy tag. Although, since a general practice is to not to mix files with different retention policies under the same directories, this could work if RetentionPolicy tag is added to the containing directories.

On tape-only or disk-only installations, where Retention Policy of files is unambiguous, the quota can rely on default, system-wide Retention policy value. On tape-only system, the default value supplied with dCache distribution is good: (one-of?CUSTODIAL|REPLICA|OUTPUT)dcache.default-retention-policy=CUSTODIAL On disk-only systems, it needs to be set to : dcache.default-retention-policy=REPLICA

On a mixed system, a dcache.default-retention-policy=CUSTODIAL can be utilized for files bound to tape and RetentionPolicy tag can be set in the root of the directrory tree where disk-only files are stored.

An admin user can interact with the quota system via the PnfsManager admin interface. For this, the following commands are provided:

    set group quota [OPTIONS] <gid>  # Set group quota
    set user quota [OPTIONS] <uid>  # Set user quota

    remove group quota <gid>  # remove group quota
    remove user quota <uid>  # remove user quota

    show group quota [-gid=<string>] [-h]  # Print group quota
    show user quota [-h] [-uid=<string>]  # Print user quota

Use help <command> to learn how to use the command.

A user-facing quota interface is implemented through RESTFul frontend (see above).

File system statistics

Added functionality to cache total files and total space used on DB backend. Added two properties that control how often heavy queries that calculates the total number of files and total used space in the namespace us ru

    pnfsmanager.fs-stat-cache.time = 3600
    pnfsmanager.fs-stat-cache.time.unit = SECONDS

The resulting numbers are typically seen when running df command.

This patch has shown to speed up NFS availablity (among other things) after startup, especially on large installations.

Pool

Before dCache version 7.2, the scrubber process was starting with the pool startup. As a result, as long as a pool inventory is not complete, the scrubber will fail with Operation not allowed while repository is in state LOADING. With this version, the scrubber now will start only after the inventory is complete.

Security (common)

A general change to the internal SSL handlers to enable, where possible, a pass-through to native SSL on the node (usually OpenSSL) rather than using the Java implementation has been made. This change affects xrootd, https (webdav) and gsi (GridFTP). It should give improved performance (up to about 20%) when data is being encrypted.

SRM Manager

Optimally recalling data from tape is achieved by reducing the number of tape mounts and on-tape seeks by recalling as much volume as possible per mount. To that end a dedicated scheduling strategy exclusively for bring-online requests is introduced into the SrmManager. It is capable of clustering requests by tape according to a set of configurable criteria before passing them on to the rest of the system. In its current state it requires two files with information on targeted tapes, their capacity and occupancy as well as the mapping of tape-resident files to tape name. The file formats are described in the book.

The scheduler can be activated by adding the following line to the srmmanager section of the properties file:

srmmanager.plugins.enable-bring-online-clustering = true

and configured as described in the book.

It is important to note that the scheduler can only be effective when a dCache instance contains exactly one SrmManager.

WebUI (dCacheView)

The admin view now includes a separate page for plots of file lifetimes by pool group. All file lifetime plots have been simplified to a single data series (average lifetime). The viewport on the pool plots page has also been adjusted for easier scrolling on smaller (laptop) screens. The “print” button on the magnified plots dialog has been removed because it is not supported in all browsers.

For the new quota system, a quota view consisting of a read-only table has been added to the admin view. A similar table (limited to the user’s quotas) also appears in the UserProfile view.

XRootD

Several important fixes have gone in since 7.1.

First, the parsing of the ‘triedrc’ CGI was incomplete; it now properly accounts for error code/message pairings.

Second, a race condition which was causing certain clients to report that a file is not found when requesting the checksum for an upload (persist on successful close) transfer was previously fixed in the door. This fix has now been migrated to the pool (where it really belongs) and made more robust.

Finally, the third-party-copy client has been changed to conform with the behavior of the SLAC vanilla client so that it requests the exact number of remaining bytes on the final read chunk. This should eliminate an issue previously encountered with a particular CEPH plugin. The default chunk size has also been increased to 8 MiB to match the SLAC client.

Changelog from 7.1.0 to 7.2.0

12d1e46: [maven-release-plugin] prepare branch 7.2
ff88aac: nfs: fix typo pricipal -> principal
adec8b1: libs: use zookeeper 3.5.9
e3990d6: srm-server: make trs admin commands use rwlock
3438812: frontend: add service to provide Storage resource reporting
49d8ba6: dcache: Modernize PoolStatisticsV0
7effd50: webdav: update representation of symbolic links in HTML page
5e77f79: book: described trs admin commands
9071fa1: srm-server: add trs configuration admin commands
97c54ae: dcache: remove redundant file
c756308: src: don’t use com.google.common.io.ByteStreams
4b6338f: pool: remove dead code in org.dcache.pool.p2p.Companion
2fc1f8d: restful: provide functionality to remove or add labels of a file
84b45e6: namespace-chimera: ignore file size update on file flush
1a5b28d: book: added clarification to trs description
9602167: pool: scrubber should ignore the files if checksum is missing
862093f: billing: reduce unhelpful log spamming
278f712: pool: add Repository#waitForLoad method
05ee28c: webdav: add ‘http-tpc ls’ admin command
bf7610a: book: describe srm tape recall scheduler and file formats
e975365: build(deps): bump jetty-webapp from 9.4.41.v20210516 to 9.4.43.v20210629
a24b2c8: dcache-xroot: bump dependency to next xrootd4j release
cd8bf7b: src: unify all instances of logger to a same style
fae6d73: srm-server: add tape info provider SPI and JSON support
61decad: libs: use sshd-core 2.7.0
33dcbda: src: unify all instances of logger to a same style
c4f755b: ssh: remote unnecessary String#valueOf call in logging
4a1ffa1: webdav: fix regression from commit a2b252c07f
6f69d12: testing: consolidate builder classes
184c826: srmmanager: fix race condition in LoginBrokerSubscriber
53c4ba1: dcache-frontend: add RESTFUL quota resource
9212aa4: Revert “libs: use sshd-core 2.7.0”
f27c206: libs: use sshd-core 2.7.0
5b07905: pool/pnfsmanager: don’t accept a non-Enstore flush without locations
5e6d306: pool: update Kafka tape events to include additional information
7daf803: github: add citation file
eb588fd: dcache-core: add quota messaging and support in PnfsManager
6e9fefe: namespace: add delete quota to driver and PnfsManager admin commands
f338304: common: support requirements on parsed value in ByteSizeParser
6c34d24: common: Update ByteSizeParser to support multiple representations
5bf281a: common: update ByteSizeParser to follow builder pattern
6809690: pnfsmanager: fix regression introduced with commit 61a4e8dc4e
c53315f: common: provide more diagnostic information
a1f1aff: pool: improve flush information
bacafac: Motivation:
302db70: Revert “dcache-xroot: prevent attempt to write to channel for which checksum has been computed”
489af68: srm: add srm bring-online request scheduler
3b89d2e: dcache-xroot: prevent attempt to write to channel for which checksum has been computed
0a1d34d: restfulapi: add restful Api call to query the lable of a givin file object
e54e3c4: skel, pool.xml: fix regressions from 13117–13118
4697c7a: dcache-xroot: increase server-side frame size to 8 MiB
8567435: dcache-xroot: increase tpc client block/chunk size to 8 MiB
d0bae1f: Revert “libs: use sshd-core 2.7.0”
81e676d: libs: use sshd-core 2.7.0
d84c908: PnfsManager: improve wording of quota command
13215ee: rpm: add dcache-convert-authzdb-to-omnisession into list of packaged files
ae11f68: skel: add explicit replicable property even when value is false
03ca593: alarms, billing: specify shared rdbms in the case of replicated service
061ce56: build(deps): bump commons-compress from 1.19 to 1.21
a733b3f: gplazma: add util to convert authzdb file to omnisession
e160a3c: gplazma: add omnisession plugin
f33acca: srm: log transfer protocols in access log
7f1f63a: gplazma: multimap now supports OP in ‘oidc’ predicate
33868f5: dcache-frontend: remove unreachable catch clause
69f9ece: dcache: use charset constant instead of String-based dynamic discovery
a6a096d: core: fix NPE in TryCatchTemplate
0c19f64: dcache-core: add checksums sha–1, sha–256 and sha–512
37ae4b8: dcache-core: correct reported units of transfer size and speed
2cf3ce5: nearline: add simulated stage delay options to FileSystemNearlineStorage
a95a832: dcache-bulk: make SignalAware reusable
8218698: starting with a few macaroon hands-on examples
56154bc: qos-engine: use batching to notify concerning completed actions
554f4e2: dcache-xroot: move upload commit for persist-on-successful-close to the pool
61a4e8d: Implementation of UID/GID quota system in dCache
695b7c8: skel: remove obsolete python scripts
0067f72: pool: simplify inotify support in pool.
9a2c99a: gplazma: oidc check token validity if JWT
0ed6b58: chimerashell: add recursive option to ‘chown’ command
17e7c30: Added a link to the user guide to the main README
4d5f054: SRM : Fix IPV6 logging for SRM
75b395b: dcache-chimera: introducing parallel pool cleaning
ab900b8: dcache-qos: cast non-Object arrays to varargs Object[]
a5f691f: qos-adjuster: add waiting queue and state; polling
e5bbab7: dcache-chimera:add admin commands to support add/remove file lables
dc4e6db: frontend: fix wrong success on not defined action
ac3a3e4: pool: http-tpc add support for multiple trust stores
ab72c86: pool: update configuration to split http-tpc and remote-gsiftp CA settings
4154dc1: build(deps): bump jetty-servlets
ae40d43: checksum-module: add missing map to JSON info
e22b598: pool: update ReplicaDescriptor to implement AutoCloseable
b7d7ee0: pool: checksum scanner: don’t create EnumSet of OpenOption in a loop
d9ac931: chimera:adding labels metadata to file objects
5393593: dcache-qos (10.6): qos scanner admin interface, local client and remote receiver
3e29dd3: bulk: change PIN, UNPIN and UPDATE-QOS to breadth-first
0c555c8: build(deps): bump httpclient from 4.5.3 to 4.5.13
a705b0a: ssh: update DirectCommand to use SshOutputStream
ddd6c88: common-security, doors: switch to openssl where possible
f64a880: FsSqlDriver: use more conservative SQL query as default, so that system tests passs. Move more advanced SQL query to psql specific driver.
a96ca9e: User guide: bulk-request URLs contain /api/v1
1848354: srm: remove redundant Pgpass class
8b0bcba: ssh: drop custom AuthorizedKeyParser
cb7282c: dcache-qos (10.4/5): qos scanner handlers
f76991c: dcache-xrootd: upgrade xrootd4j to 4.1.1
a63e603: Namespace: add FS stat cache table
b19bb24: Fix typo
2133f6e: dcache-xroot: parsing of ‘triedrc’ CGI lacks separation by optional comma
77eadb3: nfs: add possibility to configure BerkeleyDB client store
2c09996: poolmanager: fix _waitingFor access outside of synchronized block
6fa08f5: ftp: add support for the SITE SYMLINK command
bf463de: pool: http allow client to send credentials when TLS is used
0a22bac: gplamza: oidc support fetching username and groups direct from OP
275df29: build: disable stack-trace trimming in surefire
e780ace: build(deps): bump jetty-io from 9.4.35.v20201120 to 9.4.40.v20210413
206185d: frontend: bump swagger version to 1.6.2
838d422: runtime: force JVM exit on OutOfMemoryError
c2bc2ae: runtime: enable Java flight recorder attach listener
939673d: dcache-qos (10.3): pool operation map and counters
84d65bc: dcache (request container): extract reusable deliverables and endpoint from unit test suite
6c7b4da: nearline: don’t use guava to sort the output of various ‘ls’ commands
9f69e5c: dcache-chimera: use RemotePoolMonitor to discover file locality
a31b429: poolmanager: remove redundant test in WatchdogThread
959cac7: ftp:clean dead code
6d7b6a4: book: add cleaner chapter
4a1b405: webdav: use Jetty’s built-in support for CORS
f26be37: pool: build list metadata entries only when we are going to use it
99b4d18: pool: use Set#of to construct O_READ and O_RW constants
e78b504: pool: remove invalid test case of configured space
b4f957b: pool: fix pool size reporting when static/runtime config is not defined.
f40adce: poolV4: fix reader/writer counts
14a5fc5: [maven-release-plugin] prepare for next development iteration

What’s new in dCache 7.2 Release notes

Highlights

Incompatibilities

Release 7.2.43

xrootd4j

Changelog 7.2.42..7.2.43

Release 7.2.42

metadata

Changelog 7.2.41..7.2.42

Release 7.2.42

Changelog 7.2.41..7.2.42

Release 7.2.41

gplazma

Changelog 7.2.40..7.2.41

Release 7.2.40

Technical Release

Release 7.2.39

xrootd4j

Changelog 7.2.38..7.2.39

Release 7.2.38

pom

Changelog 7.2.37..7.2.38

Release 7.2.37

frontend

Changelog 7.2.36..7.2.37

Release 7.2.37

frontend

Changelog 7.2.36..7.2.37

Release 7.2.37

frontend

Changelog 7.2.36..7.2.37

Release 7.2.36

gplazma

Changelog 7.2.35..7.2.36

Release 7.2.35

frontend

telemetry

xroot

Changelog 7.2.34..7.2.35

Release 7.2.35

frontend

telemetry

xroot

Changelog 7.2.34..7.2.35

Release 7.2.34

common

gplazma

history

Changelog 7.2.33..7.2.34

Release 7.2.33

chimera

dcache-xroot

Changelog 7.2.32..7.2.33

Release 7.2.32

pool

Changelog 7.2.31..7.2.32

Release 7.2.31

chimera

xroot

Changelog 7.2.30..7.2.31

Release 7.2.30

Pool, Chimera

Changelog 7.2.29..7.2.30

Release 7.2.29

chimera

xroot

Changelog 7.2.28..7.2.29

Release 7.2.28

chimera

nfs-proxy

Changelog 7.2.27..7.2.28

Release 7.2.27

nearline-storage

Changelog 7.2.26..7.2.27

Release 7.2.26

frontend

Changelog 7.2.25..7.2.26

Release 7.2.25

chimera

core

What’s new in dCache 7.2
Release notes