dCache 11.1 Release Notes

Highlights

NFSv4.1 Read-delegation support

Incompatibilities

Drop support for nfsv4_1_files layout type.
Drop admin commands to set/get/list flags associated with files.
Obsolete pool.enable.hsm-flag.
Stop populating of t_storageinfo chimera table.
Removed Legacy admin shell

Acknowledgments

We thank Tino Reichardt for his contribution.

Release 11.1.0

Admin

dCache allowed admins to associate key-value (flags) pairs with files, which were stored in pnfs level-2 (t_lavel_2 table). Those flags did not influence the dcache behavior and were invisible to users. This functionality is removed now.

The ssh admin interface could start the legacy shell with an alternative way to address dCache cells (cd vs. \c). This functionality is removed now. Any custom scripts that might depend on such functionality must be updated.

Chimera

The dCache namespace stored so-called flags, key-value pairs associated with files, in the t_level_2 table. This functionality is removed as it is not being used. The t_level_2 is not touched and stays as-is. The table should be empty for most sites or contain no new records since 2018. Admins are free to truncate the table to reclaim the space, if desired.

The storage info of a file is defined by directory tags, such as OSMTemplate and sGroup or storage_group and file_family. Additionally, after file is flushed in HSM, the storage info was stored in t_storageinfo table. This was required to be able to restore the file with the original storageinfo, even if directory tags are changed. However, the functionality was never used by sites. With release 11.1, dCache stops populating and reading t_storageinfo table.

Cleaner

Previously, cleaner-hsm would send one delete request per HSM at a time to a single pool each. However, this level of parallelization is not very efficient, especially when the delete times out. Now, for each HSM, it can delete files in parallel via multiple pools. Instead of waiting for one delete request batch per HSM to finish or time out, it will send further delete requests for that HSM to other pools connected to it on successive cleaner-hsm runs. By decreasing the time between successive runs, the number of parallel cleaning pools is increased (as long as there are available pools).

DCAP

Removed the capability of associating a checksum as a key-value pair on upload. The checksum had never been used by dCache or exposed to users.

gplazma

Added multimap plugin section in the dCache book.

httpd

The pool and pool group usage pages have been updated to display data volumes in a human-readable form.

Monitoring

With release 11.1, dCache comes with an integrated prometheus-exporter that can be enabled by dcache.enable.prometheus.exporter=true property. The property dcache.enable.prometheus.exporter.endpoint controls on which interface and port number the exporter is exposed. The default value is localhost:9876. There are not too many metrics available righ now, however, the exporter can be used to monitor JVM memory, number of running threads, or number of open files.

NFS

NFS door supports read-delegation. Read delegations allow the server to delegate OPEN-CLOSE operations to clients if a file is accessed several times. This functionality aims to improve file access latency in workloads like open->read-close; loop, which some HPC applications use.

In order to provide delegation-based read access on dCache pools, the nfsv4_1_files layout type is no longer supported. Site admins must remove the lt=nfsv4_1_files option from nfs door export files.

NOTE: As the client will not return delegations right after transfer, the NFS movers on pools might stay active even if no applications have opened the file. Moreover, the transferTime and transferSize correspond to a file delegation period and do not reflect a single open->close cycle.

Pool

Berkeley DB, which is used to store metadata on pools, by default keeps only 100 files open. On some installations, this causes a slow start, as the number of db files is much larger. Now, the default is increased to 512.

On data upload, dCache pool can tell PnfsManager whatever pool’s lfs mode is hsm or precious. This functionality is controlled by pool.enable.hsm-flag option, which defaults to false since 2017. This option is now obsolete.

Changelog from 11.0.0 to 11.1.0

7cec96cf27: build(deps): bump io.netty:netty-codec-http
c23fd18b50: qos: don’t shadow fields from super class in ReplicaStateAdjuster
2e12f1dca2: qos-adjuster: use pnfsid for internal bookkeeping
ef27b1f9c7: qos: disable dynamic replica reduction
a6b7237387: chimera: reduce number of internal location copies
acf6cc0762: Human readable sizes for Webadmin Interface
38ac3d0e13: WLCG tape API: do not report empty fields in json responses
74b01b98ab: poolmanager: fix wrandom patrition loop
a778f9763c: poolmanager: update wrandom partition to respect gap
5f0e7be8e8: bulk: remove residual code having to do with delay_clear
6b5438f4b5: build(deps): bump org.bouncycastle:bcpkix-jdk18on from 1.78 to 1.78.1 (#7861)
5f922dd51e: Revert “libs: update jackson to version 2.18.3”
9b19f5fca2: omnisession: add support for role attribute
c735ee9258: ci: consume billing kafka messages from the beginning
8e052af8b7: ci: use minio helm chart from charts.min.io
d15becab64: ci: don’t use bitnami helm charts for kafka and zookeeper
b4db67e462: ci: don’t use bitnami helm chart for postgresql
80b45777ec: ci: don’t use bitnami images for kubectl commands
fa1987338b: pool: move QoSMessageHandler#SYSTEM_OWNER into StickyRecord
1239271d25: namespace: raise alarm if storage class misconfigured
c01a82fb05: pool: define replocas' lastAccessTime and creationTime on start up when repository is not present
dfc4f69574: ci: add gpg to almalinux 10 installation
5ef4e2bb6a: ci: use almalinux:10-minimal as a standard image
2008cde212: ssh: drop legacy shell
e5b8a66450: ci: ignore netty-tcnative-boringssl-static duplicates
2a0716d7be: libs: update netty version to 4.2.3
ef4d8b8e08: vehicles: remove legacy PoolModifyPersistencyMessage
08ca0d0f7e: pool: expose receiving from pool state in rep ls
d96d0c132e: runtime: enable prometheus exporter
9f5d209816: ci: adding variable admin_role to environment
aeabefe0c6: frontend: organization of swagger
8dcf074205: cells: stop using deprecated ThreadGroup methods
5a4adf3d42: auth: add DesiredRole#toString
cab11e7486: qos: make db name configurable
36d6a5ef8a: dcache rest api: do not filter out fields with default and null values when building json reply object. Enable get/set of OUTPUT quota
8de56cb839: voms: accept only FQANs that match VO name
8353e67690: ci: adding migration endpoint tests to the gitlab pipeline
4d22cb12aa: quota: implement protocols specific return codes
b68a2d6ae7: ci: adding qos endpoint tests to the gitlab pipeline
e09d70a568: chimera: stop populating and querying t_storageinfo table
47cc2bc55a: nfs: add admin command to list current delegations
f5f92781c0: fix broken commit 48a544cb6
2842808071: pnfsmanager: drop support for file flags and handling of pnfs level-2
75617d45e0: gplazma: fix static field for RecordFailedLogins
a406b1b82d: gplazma: update link to IGTFInfo file format
ce93ab1799: pool: drop pool.enable.hsm-flag
dee4ee71f4: pnfsmanager: drop flags set/get/delete commands
7cbd21ae26: dcap: drop checksum string as flags
bcde31a09f: admin: remove set/get flags command and derivates
1a5b83d78b: ci: publish container image with branch tag
48a544cb69: gmplazma: excessive caching of failed login attempts leads to system overload.
71b18ee76e: frontend: refactor to camel case to keep naming convention consistent between tests
d6e7dc671a: ci: tests to check staging functionality
28b76ef158: container: use almalinux-10 minimal as it produces smaller images
2a77534942: gplazma: check BEARER TOKEN empty case
7dc561d58f: frontend: respect admin role when requested
bec50c40cf: chimera: fix string representation of Stat
196f8c9cce: webdav: fix X-OC-MTIME support when uploading a file
ee57075247: Removal of random README file
5332c10f19: dcache tape rest API: return ONLINE (or DISK) locality for incomplete files
9db7e752c5: pool: change default number of open files by Berkeley DB
af20384b1a: libs: drop unused dependency
bafadec421: Revert “chimera: fix string representation of Stat”
d8269db15e: Reapply “gplazma2: drop unsupported org.reflections library”
ca8a492385: container: avoid epmty entry in the classpath
2595222ca0: chimera: fix string representation of Stat
e731648a3a: chimera: add ‘stat’ command to shell
00e680d4bf: ci: frontend tests cleanup
1d27fbb9e4: libs: switch t nfs4j-0.27.1
893734557e: docs: add multimap plugin section
d0cc39c403: nfs: update proxy to use flex_files layout driver
7266edd31a: pnfs: drop nfsv4_1_files layout type support
7284974e6d: nfs4: libs update to nfs4j-0.27.0
5b57c61b98: book: describe hsm-cleaner parallelization
0eff1a062c: REST API: restrict access to id resource to admin user
8be70fb7b8: Revert “gplazma2: drop unsupported org.reflections library”
cd6ea4fae8: cleaner: allow multiple pools per hsm to be cleaned in parallel
41a07b64fb: libs: update jackson to version 2.18.3
42144b3544: gplazma2: drop unsupported org.reflections library
40f72e4508: nfs41: unconditionally remote transfers associated with dead clients
046b83e912: pool: shutdown mover on “Connection: close” http header
fd4b899110: nfs41: remove transfers with no movers on layout state disposal
7fefd91353: pom: update exec-maven-plugin to 3.5.0
eee28545f2: libs: update aspectj version to 1.9.24
3f3bdce14a: ci: fix script execution bit
4391b28fec: frontend: add listing all existing labels
5696faa7d0: ci: fix igtf rpm signing key
40bb21acaa: ci: collect k8s events before logs
111f60ca52: ci: collect k8s events
6814424adb: ci: disable unsupported pynfs tests
f18d3cd646: ci: don’t use egi software repo for CA certs
ef40ce10c5: Bulk: remove skipIfOnline method from PinManagerActivity
0b30436285: [maven-release-plugin] prepare for next development iteration

What's new in dCache 11.1 Release notes