dCache 10.2 Release Notes

Highlights

Expose user/group quotas via remote quota protocol
Java 21 is supported as runtime environment
PoC support for firefly network markers
Support of Label-based virtual read-only directories

Incompatibilities

Pool Manager partitions by default allowed to stage
Dropped gPlazma ARGUS plugin
A stage request canceled in the poolmanager will propagate the cancel to pool
Pool internal health check is disabled if an external check command is configured.
Access to files with QoS ‘HSM’-only will be denied, even if they are still available on disks.

Acknowledgments

Anton Schwarz, Heidelberg University, DESY Summer Stutent Programm

Release 10.2.2

bulk

This fix has reduced the number 0f stack traces in logs.

tape

Users reported 2 day pin lifetime on staged files (which is a default) despite specifying different values. This is now fixed.

Changelog 10.2.1..10.2.2

df5eb3b5c7: [maven-release-plugin] prepare release 10.2.2
a1b2a95b69: tape REST api: additional fix tohandling of prefixed paths
cac439eebe: bulk/qos: shutdown executor services when stopped
92c11c94d3: [maven-release-plugin] prepare for next development iteration

Release 10.2.1

xroot

Return destination address (that is, the haproxy address) if xrootd.enable.proxy-protocol=true is set, instead of the actual door address.

Changelog 10.2.0..10.2.1

81a63e36cd: [maven-release-plugin] prepare release 10.2.1
93c5074d95: xroot: handle haproxy and checksum command
2aa172f661: [maven-release-plugin] prepare for next development iteration

Release 10.2.0

Cleaner

Cleaner-hsm’s admin info command now shows information about which pools have currently been instructed to delete files from an attached HSM. These pools are still expected to send a reply concerning success or failure of the deletion. Additionally, the pool selection behaviour was changed to only select not already waited for pools for hsm-cleaning.

Frontend

Logic was updated. None existent QOS name will now longer throw a 500, it will return a 400. QOS of type “tape+disk” will no longer throw a 400, it will return the same as “disk+tape”.

Chimera

Traditional file systems organize data in directories. Directories are typically a collection of files whose grouping is based on some of the above mentioned criteria. However, each file in a directory can belong to several logical groups, such as a special event type or experiment condition.

With our previous releases releases in dCache, we had have implemented user metadata handling, so that any short string label describing raw data, calibrated data or detector telemetry can be attached to a file via the RESTful API, or via graphical dCache-view interface.

With the latest changes in dCache, it is now possible to populate virtual read-only directories with files as soon as they get the corresponding label.

These virtual directories are exposed through NFS and WebDAV protocols via the commands ls -1 /mnt/.(collection)(labelName) and http://localhost:portnumber/.(collection)(labelName), respectively.

For example, the following command returns the list of all files labeled by blue-bird:

$ ls  -1 /mnt/".(collection)(blue-bird)"/

file_large_bird_1.log-7
file_large_bird_2.log-7
file_large_bird_3.log-2
file_large_bird_3.log-7
file_large_bird_4.log-7
file_large_bird_5.log-7
file_large_bird_6.log-2
file_large_bird_6.log-7
$

To access the file a simple cat /mnt/".(collection)(yellow)"/file_large_bird_4.log-7 should be used.

gplazma

The gPlazma’ argus’ plugin was removed due to a lack of usage and the decommissioning of the corresponding EGI-managed infrastructure.

Added the possibility of prototyping auth and map plugins in Python.

httpd

Fixed PoolManager web representation to generate valid HTML. Updated PoolSelection configuration section to use common css. Update the color palette to match (more modern) dcache.org colors.

NFS

The NFS door has been updated to expose the REPLICA-ONLINE quota via the remote quota protocol, which uses UDP protocol. The nfs.net.rquota.port property can specify the UDP port number used by the service.

Pool

Fixed invalid logging format by NFS write operation. The sweeper purge command accepts the optional -storageClass=<class> option to purge only files of the specified storage class.

Added configuration option pool.mover.nfs.multipath to specify which IP addresses should be advertised to pNFS clients.

Removed obsolete/test code.

dCache provides a possibility to specify a command that the pool can use to perform repository checks. This check runs parallel to the internal repository health check. Starting from dcache–10.2, an internal check is skipped if the command is specified.

Added optional network fireflies to allow network traffic monitoring by WLHC and ES-Net. The fireflies can be enabled by pool.enable.firefly=true configuration property. To exclude local traffic from reporting, pool.firefly.excludes can be used. Example:

pool.firefly.excludes = a.b.c.d/16, aa:bb:cc:dd:ee:ff/64, v.x.w.z

Where a.b.c.d/16, aa:bb:cc:dd:ee:ff/64 and v.x.w.z are site local IP addresses or subnets.

Poolmanager

The stage-allowed option in the pool manager is now on yes by default. When a request in PoolManager that has triggered the stage is canceled, PoolManager will cancel the stage in the pool.

If a pool with the file is offline and a tape copy is available, dCache will trigger the stage and wait until the file is restored on disk. However, if the pool becomes available again, the stage request is not interrupted, and the client will wait for the tape. With 10.2 the PoolManager will use the online replica to serve the request.

NOTE: The on-going stage will not be interrupted.

QoS

The QoS service database name is now configurable.

Access to files with QoS ‘HSM’-only will be denied, even if they are still available on disks.

Zookeeper

Suppress non-fatal errors in the logs

Updated zookeeper and curator to versions 3.8.4 and 5.7.0

Changelog from 10.1.0 to 10.2.0

2aa172f661: [maven-release-plugin] prepare for next development iteration
9f6b619b6b: [maven-release-plugin] prepare release 10.2.0
e334fb04c0: http(old): modernize color palette
53173e10a8: build(deps): bump org.eclipse.jetty:jetty-server
9d3eb6e108: build(deps): bump org.eclipse.jetty:jetty-servlets
1e916a95a9: dcache: reject access to files with nearline QoS
2423686780: pool manager: retry request on pool up
ae3fd77003: [maven-release-plugin] prepare branch @{releaseLabel}
2de64addb2: webdav: adding listing for files tagged by labels
daad2ab76f: pool: fix double decrement of hsm requests
d0c4ad74df: nfs: add remote quota protocol support
4b7dbd4f28: pool: add network traffic flow markers
fb734fb6c6: pool: don’t treat InterruptedIOException as a disk IO error
f745948d53: pool: skip internal repository check if external command provided
4d7e0de505: qos: make database name configurable
84c3d0be26: ci: split container image registry and repository
4a9305c028: pool: remove fault-injection in dcap mover
17b79d521b: pool: add property to specity IP addresses exposed to NFS clients
d3dbba97bc: dcache-vehicles: remove obsolete comments
9954491560: logback: suppress zookeeper warnings
e1e086ea60: ci: use latest helm chart for postgresql
d350f45c8d: Revert “Create dependabot.yml”
a212817054: Create dependabot.yml
cfeed0cc3c: migration: provide default copy behavior per #7623
be9d1c7ff5: pool: update sweeper purge command to accept option storage class
7b2f1a34a8: cells: update CellStub#send to return CompletableFuture
2400de1d01: scripts: accept java 17–21
6885af34b1: libs: update aspectj to java–21 compatible version
6381cc6568: ci: run spotbugs only on master branch
f260a34c50: poolgroup: error code update
c2406da578: ci: tests to verify a successful migration request
b6a528ca20: poolmanager: cancel stage when restore is canceled
f1ba73bc3c: ci:workaround oidc-agent bug
92e1c67ec3: ci: print oidc agent stats
561e3bf2b0: pool: include source pool name into pp ls output
ebae424a0d: libs: use nfs4j–0.26.0
c165794462: ci: split current/old dcache deployments
d99c031355: bulk: fix broken command activities
d84682eb5a: Corrected a few typos
a0ba728fa8: ci: add keycloack and dummy token test
83297e934e: gplazma: multimap fix op regression
6fe3676e57: ci: use llm to generate release notes
d74fd8a8af: ci: add minio s3 as tape
45a6cc15ec: bulk: check targets for empty strings
345edefb8c: gitlab: trigger mirroring on pull_request
d981ebc160: nearline-storage: allow retrying of request cancelation
bb836bef2d: gplazma2-oidc-te: fix invalid initialization of plugin
12c8fef0f4: github: add action for atumatic github-release
09919b7750: ci: verify file location and migration p2p
d4c2c75d73: maven: ensure that we have unique file names to conflicting jars
60292c7687: http-old: use common CSS for pool manager configuration section
8ea0efc76e: namespace: skip files/directories from listings on bad storageinfo
25412a1c0e: docs: clarify OIDC provider ID and issue claim
494063afbc: build(deps): bump org.webjars:swagger-ui from 3.1.7 to 3.23.11
ad395dd2f5: github: pull gitlab CI pipeline status every 60s
10abac183b: gplazma alise initial version of plugin
ef2dc1e0e5: common: add issuer URI to OAuthProviderPrincipal
3cdddc0d01: ci: use installed packages dependencies to detect duplicates
7ba5b8fffd: libs: drop XACML leftover dependency saxon:saxon
d6cb3f15f2: ci: update webdaav-door path to run in kubernetes
2ad55888ef: ci: option means allow_failure: true :)
2dde22eb22: ci: add optional (for now) test to catch multiple versions of dependencies
12a86d520f: core: reformat pinmanager spring xml file
5faa6359b2: build(deps): bump org.springframework:spring-expression
8fc469086e: ci: drop CVMFS leftovers
b1c05399ca: gplazma-pyscript: add mapping functionality (#7639)
d147cbcba3: chimera: remove unused code
44efb5c1ac: WLG TAPE REST API: fix handling of frontend.root in release API
e57606532f: test: add test case for adding and querying labels via rest api
591e361d2b: Improved Pyscript Auth Module
7b2b3dfd10: gplazma: add gplazma2-jython module for auth using Python (#7627)
f7796051de: gitlab: mirror tags
61a75dcea5: frontend: change tests to use updated variables
82bf63868c: qos: add tests, rename tests and clean up
db160657c4: qos: bad request update and additional qos target
b772cd54a1: gplazma2-argus: drop obsolete plugin
83d591f0a2: github: change mirroring action
5658a7a889: ci: use shorter k8s namespace names
3ada526bf4: ci: more robust collection of billing logs
183e99fb88: ci: don’t explicitly delete kafka collector
725b3a6523: ci: don’t start kafka collector in interactive mode
94b0bb65eb: CONTRIBUTING.md: typos, formatting fixes
8eee558bf4: Add CONTRIBUTING.md
29ab1b54aa: poolmanager: enable staging by default
2cdef22064: pool manager: log when stage request disallowed by partition configuration
3fee4baf6a: dcache-chimera: add active pools to info
6b4ebad205: docs: describe min version ov postgresql
df1233d40d: ci: drop centos–7 based wn. drop cvmfs
d8d934cf52: ci: use ZCG for containers
4043dd3a35: libs: update mina-sshd to version 2.13.1
fb1d797962: libs: bump zookeeper and curator versions to 3.8.4 and 5.7.0
0d2629a3a4: http(old): fix generated html for pool manager
fa8a90b80b: pool: fix missing logging placeholder for nfs write OP
cce2da5a98: ci: split el9 wn initialization from s2 tests
aea6edec50: Reapply “Fix issue with infinite replicas when replicas > 1”
cfcc172b3d: ci: remove localhost property
7c4f2f673d: ci: remove comment out command
3619ce9b9f: ci: refactor files
0b72715fa3: ci: delete unused file
d5868b69aa: ci: frontend test classes.
8b2a8e1c0c: ci: rename test suite
7f2c24adff: ci: make gitlab happy
8fc59673bb: ci: make gitlab happy
7dd7c4358b: ci: make gitlab happy
1fcad4a132: ci: make gitlab happy
3ddca9abdd: ci: renaming and commente back in
a414f3a2bf: ci: renaming and commente back in.
d2dac63028: ci: 101
8d0ff6ca4b: ci: 100th time is the charm.. (for gitlab-ci just in case)
1806d01690: ci: add qos tests and junit report. rename endpoints.
66a8eb3234: ci: add junit report one last time again 4
1a2c562496: ci: add junit report one last time again 3
c5f774f226: ci: add junit report one last time again 2
6e2e17f494: ci: add junit report one last time again
87afb895b5: ci: add junit report one last time
52efe7db55: ci: add junit report one more time yet again
aea948c922: ci: add junit report one more time
99eba2c3b7: ci: add junit report yet again
f7f2fbbcda: ci: add junit report again
2b5916229e: ci: add junit report
dadbc54d20: ci: variable name change and update to test classes.
0d37a3ba53: ci: comment in gitlab-ci sections
2ce31ab468: ci: add flag for report.
f1004535cd: ci: add flag for report.
274b300731: ci: replacing correct variable in poolEndpoint test
9efa0424e7: ci: copy readme file to container. Add more variables. Get rid of migrationTest txt file.
2f5c1e2016: Revert “Fix issue with infinite replicas when replicas > 1”
20d9c78e59: ci: fix comments for qos-policy tests
2d7350e93d: ci: add upload file capability and check for it. Fix other migration tests.
069c9388da: ci: add qos-policy http file and variables for pool and qos-policy inside private var client.
986913c561: ci: comment out value pair tests for GET request for migration endpoint
fd9a7ef8d9: ci: fix titles and tests for migration endpoint
a974d8bfae: ci: include variables
eb6d644da7: ci: Update name yet again
4a9cc4caee: ci: Update correct name of file
ad439bfe94: ci: Add stage in ci for pool endpoint request to run in. Add variables, pool http file, migration yml file.
7ae7109f41: github: trigger sync on PR
03a27be538: nfs: add functionallity to do ls on file labels
9655c37c45: [maven-release-plugin] prepare for next development iteration
526c419366: ci: refactor files
0b68e41cb6: ci: delete unused file
c84ed9fc4c: ci: frontend test classes.
ab742aea5c: ci: rename test suite
fedf87976e: ci: make gitlab happy
d27d9e6aba: ci: make gitlab happy
e0534ea46a: ci: make gitlab happy
29f1006ab5: ci: make gitlab happy
c411d312fa: ci: renaming and commente back in
37e77d7ab6: ci: renaming and commente back in.
a5bf0749a3: ci: 101
51705c90ca: ci: 100th time is the charm.. (for gitlab-ci just in case)
61caee3182: ci: add qos tests and junit report. rename endpoints.
9b993dc0d5: ci: add junit report one last time again 4
465ab125bb: ci: add junit report one last time again 3
f824f720b8: ci: add junit report one last time again 2
e75eed331c: ci: add junit report one last time again
406467ab9e: ci: add junit report one last time
6ad655939d: ci: add junit report one more time yet again
76051acd41: ci: add junit report one more time
7ee17d93ef: ci: add junit report yet again
b59a8d6cc8: ci: add junit report again
589d1d3ef4: ci: add junit report
9d8471d6e5: ci: variable name change and update to test classes.
0733143bc5: ci: comment in gitlab-ci sections
f73544b111: ci: add flag for report.
17c1bb33ab: ci: add flag for report.
959945e1c6: ci: replacing correct variable in poolEndpoint test
6dd4058bf6: ci: copy readme file to container. Add more variables. Get rid of migrationTest txt file.
f4a5f4cc2b: Revert “Fix issue with infinite replicas when replicas > 1”
0c1d688c45: Fix issue with infinite replicas when replicas > 1
513157afab: ci: fix comments for qos-policy tests
4e5cbf9acd: ci: add upload file capability and check for it. Fix other migration tests.
42bab8322f: ci: add qos-policy http file and variables for pool and qos-policy inside private var client.
f740e3f27f: ci: comment out value pair tests for GET request for migration endpoint
a5ed3cb4a3: ci: fix titles and tests for migration endpoint
fcc8a53134: ci: include variables
30ca379910: ci: Update name yet again
8120fbb037: ci: Update correct name of file
e80ad235ee: ci: Add stage in ci for pool endpoint request to run in. Add variables, pool http file, migration yml file.

(tbd as soon as the branching is done)

What’s new in dCache 10.2 Release notes