dCache 11.0 Release Notes

Highlights

SciTag support
Retry stage requests on pool up

Incompatibilities

Database automatic schema migration can’t downgrade to the earlier schema versions

Acknowledgments

We want to thank Marian Babik and Onno Zweers for their contributions.

Release 11.0.3

frontend

dCache has two identify admin role: login attribute and RolePrincipal. As HttpServletRequests#isAdmin checks only the RolePrincipal, attribute based roles are ignored.

This is now fixed.

gplazma

Excessive caching of failed login attempts leads to system has been reported.

The current relase fexed that and now following properties could be used to control the overload:

gplazma.authz.loginFailure.cache.size=10000

When the cache reaches this size, older entries are evicted. Remove the least-recently-used (LRU) entries when that limit is exceeded. important: the cache size is an approximate size not exact.

gplazma.authz.loginFailure.cache-size-expiry=1

timeout cache entry life time. By default expire entries 1 hour after they are written. If a key is expired due to time, it is gone even if the cache is not full.

gplazma.authz.loginFailure.cache-size-expiry.unit=HOURS

timeout cache entry life time unit.

The current release fixed BEARER TOKEN empty case and the empty token should be rejected.

tape

dcache tape rest API returned ONLINE (or DISK) locality for incomplete files. Return locality ONLINE if locality is NONE for files w/ zero size.

webdav

A bug is fixed that prevented dCaches support for the X-OC-MTIME HTTP request header on PUT requests from working. Note that, for this patch to be effective, both the WebDAV doors and all pools that accept such upload requests need to be updated.

Changelog 11.0.2..11.0.3

0e7ddac355: [maven-release-plugin] prepare release 11.0.3
337595fff3: fix broken commit 48a544cb6
f3f0600c30: gplazma: fix static field for RecordFailedLogins
6833bed8d2: gmplazma: excessive caching of failed login attempts leads to system overload.
9257700847: frontend: respect admin role when requested
20f391353e: ci: frontend tests cleanup
f37b5479f9: webdav: fix X-OC-MTIME support when uploading a file
87decdf8e7: ci: publish container image with branch tag
881266efbd: dcache tape rest API: return ONLINE (or DISK) locality for incomplete files
1542fcf417: gplazma: check BEARER TOKEN empty case
96de4016d5: [maven-release-plugin] prepare for next development iteration

Release 11.0.2

nfs

NFS door associates transfers with layout- or open- state ids. On state disposal, for example, when client closes a file, transfers shutdown is called, that will remove the transfer when mover is finished. However, if no mover associated with a transfer, then such transfers will stay in the door.

This is now fixed and less orphan transfers in the nfs door.

pool

As HTTP is a stateless protocol, when Range request is used, then mover has no idea if that was a last block, or client will request yet another chunk. Thus, as long as client it connected, mover will be active, before idle connection removal is triggered. If too many clients do that, then pool might run out of movers.

This is now fixed.

rest

Now access to id resource is restricted to admin user.

Changelog 11.0.1..11.0.2

4e507ad182: [maven-release-plugin] prepare release 11.0.2
d961939c1e: Bulk: remove skipIfOnline method from PinManagerActivity
4e6b684036: nfs41: remove transfers with no movers on layout state disposal
f5d0a6cf65: pool: shutdown mover on “Connection: close” http header
d38ca8e228: REST API: restrict access to id resource to admin user
09478755db: ci: fix script execution bit
c440089a26: [maven-release-plugin] prepare for next development iteration

Release 11.0

Admin

The underlying sshd library is updated.

JFR

Java flight recorder will record and display door -> dcache interactions and message round-trip durations.

NFS

The user now has the possibility to dynamically group files with the same label into virtual directories independent of the actual file location. These virtual read-only directories are dynamically populated with files as soon as the files get the corresponding label.

$ ls -l /dcache/".(collection)"/AnnaBerthaLudwig will return the list of all files tagged by AnnaBerthaLudwig`

report.txt-9
report.txt-10

Where 9 and 10 are the parent ids.

Files can be downloaded via:

cat /dcache/".(collection)"/report.txt-9

It is also possible to list all existing labels as:

 ls -l /dcache/".(collection)"

will return

user@user-O:~/dcache$ ls -l   /mnt/".(collection)"/
total 3
drwxr-xr-x 13 root root 512 Mai  9 12:06 AnnaBerthaLudwig
drwxr-xr-x 13 root root 512 Mai  8 12:06 Xray
drwxr-xr-x 13 root root 512 Mai  8 12:06 1895

PNFS Manager

The liquibase version is updated.

Pool

Pools set max diskspace command accepts % as unit, which is treated as a percentage of the total disk size under data directory.

On http upload or TPC-pull the mover will calculate multiple checksums if requested by the client.

The migration module can now spread files to a desired number of pools. If no sufficient pools are online, two strategies are available: wait and limit, which will wait or stop.

If pool space shrinks, then the pool will automatically decrease its size.

Poolmanager

If a pool with the file is online and a tape copy is available, then dCache will trigger the stage and wait until the file is restored on disk. However, if the pool becomes available again, the stage request is not interrupted, and the client will wait for tape. With 11.0 release, the poolmanager will retry the request when the pool is up.

Changelog from 10.2.0 to 11.0.0

b8049c371a: [maven-release-plugin] prepare branch @{releaseLabel}
b60aba5bcb: frontend:make DirectoryListSource Polimorph
61b7dbfa7b: Scitags: Firefly usage and stats (#7778)
cc1b65ab5b: jetty: disable sending version number
b75b1a7605: ci: enable more pynfs NFS tests
217fed9d8f: ci: enable all pynfs tests related to setattr
5021255d1d: cells: simplify initialization of LocationManagerConnector
34658ed0d7: Scitags: fallback mechanism (#7775)
543bfc5711: cells: use core domain endpoint only if hostname resolvable
3541affb3d: Replace jdk 11 with 17, because dCache now requires it; update dCache version in example
04c71e03b3: Changing jdk to 17, because dCache now requires it; also, jcmd is in java-17-openjdk-headless
ec2aa806e5: Revert “ci: use multistage container build to reduce image size”
1c23511c62: pom: use jython-slim
c3d4db45c3: ci: use multistage container build to reduce image size
88f3d566be: ci: use multistage container build to reduce image size
4da17af73f: github: mirror not sync PR
e47acf7470: github-sync: fix repo name
44bee8c24f: github: mirror PRs to gitlab
36f54db61b: labels: fixing issue #7727
65e190f498: Update webdav.md: add STAGE to table of activities
23e49b719e: Update macaroons.md: add STAGE to list of activities
5b5fd42f66: book: update descriptions of partition types
02a3bbb66c: Update hsmcp.rb: do not escape string that is returned to the pool
f6d6e31dc6: libs: use jetty 9.4.57.v20241219
8517165f16: doors,pools: extend ProtocolInfo to propagate Scitags to FlowMarkers
db878cc947: Update gplazma.properties: explain gplazma.oidc.provider options
0315f6fbe0: book: fix typo
5c7c0a6eee: nfs: add improve ChimeraVfs javadoc
09fb7b1f56: build(deps): bump io.netty:netty-handler
ed2bcad2a7: libs: update sshd-core to version 2.15.0
9c97b20034: BUILDING.md: specify the location of system-test logs files
10c49db78c: srm: fix liquibase checksum mismatch
0a094590b6: CONTRIBUTING: link to style guide
9f4a464829: utils: drop extra null check in ImmutableAttributesMap
5578721ee4: pool:fix outofspace error sending pool into diasbled mode
466c97e390: qos: Uncaught NullPointerException due to QoS being set to null
208bfbe64a: pool: add workaround space mismanagementd by XFS (and others?)
705a6dd636: billing: database fixes necessitated by Liquibase update
2440b22277: cells: fix race condition between thread start and flag check
153a018b06: dcache-billing-indexer: add explicit dependency on commons-io
68f7ffee9c: libs: update Liquibase to 4.29.2
2b5f8bc6b0: Bulk: PinManagerActivity do not set state of files that are pinned indefinitely to SKIPPED
76fe850773: chimera: on shutdown propagate close to DB specific driver
3b12aa7233: utils: use JFR events to track door interaction with other cells
693fffa3fc: poolselection: fix typo
f41cd3e586: migration: new MigrationModule option and copy completion message
bfbf240c1d: pool: separate cases for disk error and no space available
abe4bd9327: bulk: update message returned when trying to stage a file present on disk
484e967e1f: bulk: handle absolute/relative paths in uniform fashion
8b9dfb3997: qos: QOS fails with ’Attribute is not defined: QOS_POLICY'
600ed1ff98: cells: always try to re-establish dead tunnel, unless stopped
289e9f1542: ci: skip maven-release-plugin generated commit messages by release notes
8c71927df6: pool: allow max disksize to be specified as percent of filesystem
103bfebf20: AI release notes: ignore commits by maven-release-plugin
59ea69ff01: jvm: drop UseCompressedOops JVM option
b63677e21b: cells: remove backward compatibility 7.0
69f5c771b5: poolmanager: fix typo
9023ceaee4: vehicles: support door requesting multiple checksums
4b922a1d6d: docs: diagram on successful write workflow
30829c96aa: cells: ignore empty core domain uris propagated by zk
6a3e89952d: docs: update to the minimal dCache installation
d92fdf159f: test releases xml file creation
47ef9a7735: docs: correcting typos in chapter 2
404dc6dbdb: container: add plugin classpath if defined
fff8b79e79: tape REST api: additional fix tohandling of prefixed paths
5f67bddf57: bulk/qos: shutdown executor services when stopped
e98ab94264: xroot: handle haproxy and checksum command
8d83ed59fe: ci: use java-21 for runtime
cbff12d45d: chimera: correct a typo
7e2040ecb8: http(old): modernize color palette
8254b6727b: build(deps): bump org.eclipse.jetty:jetty-server
c18934e22d: build(deps): bump org.eclipse.jetty:jetty-servlets
43a6082e70: github: drop branch name manipulations
703e5e00eb: github: keep branch name for regualr push events
e697949a06: github: use GITHUB_REF_NAME to for pull requests [skip ci]
cd9c90d230: github: mirror pull requests
f945c3db98: pool manager: retry request on pool up
1a3c12e8b6: dcache: reject access to files with nearline QoS
046511bb60: [maven-release-plugin] prepare for next development iteration

What's new in dCache 11.0 Release notes