dCache 10.0 Release Notes

Highlights

Added pool metadata directory configuration option
Use environment variables as configuration properties

Incompatibilities

dropped native CEPH support. Sites must migrate their pools before updating dCache
dropped idle timeout handler in netty-based movers (xroot, http)

Acknowledgments

Many Thanks to Petter A. Urkedal and Lukas Mansour for their contributions.

Release 10.0.1

rpm

Java 11 will be installed as dependency when Java 17 is not already present.

webdav

When running multiple Remote Transfermanager, two transfer that were started simultaneously had the same ID which led to the second transfer becoming orphan after the first one completed. This is fixed now.

Changelog 10.0.0..10.0.1

1bbfbbf8b9: [maven-release-plugin] prepare release 10.0.1
afd5dd5f6e: ci: use almalinux9 for rpm install test
d9f3706793: docs: update oidc chapter to explain trust anchors
3c15025ab8: rpm: don’t force java–11 it java–17 is installed
61e29f8397: ci: eplocitly specify kubernetes namespace
9762947f8a: ci: fix typo
b6b8386a19: webdav: use transfermanager+id to identify TPC transfer
da81c86739: [maven-release-plugin] prepare for next development iteration

Release 10.0.0

Billing

The property billing.format.json was added to write the logs in JSON format instead of plain text if it’s set to true.

Configuration

In the container world, it’s common to use environment variables to pass the configuration parameters. With release 10.0 dCache support injects of environment variables into a configuration with special values prefixed with env.. For example, to control wan port numbers with environment variables DCACHE_WAN_PORT_MIN and DCACHE_WAN_PORT_MAX the layout file might have the following configuration:

dcache.net.wan.port.min=${env.DCACHE_WAN_PORT_MIN}
dcache.net.wan.port.max=${env.DCACHE_WAN_PORT_MAX}

> NOTE: the nested variables are not allowed. E.g. ${env.$VAR1-$VAR2} will not be resolved.

DCAP

Added support for additional Kafka parameters.

Frontend

FTP

Added support for additional Kafka parameters.

gplazma

Update OIDC-related documentation.

Pinmanager

A pin request via admin interface no longer waits for the pin to be established before returning.

Pool

Sites that have tried to run dCache with built-in CEPH decided to go for CephFS (mounted as regular FS) as it provides better performance, scalability and erasure encoding. Thus, built-in CEPH is not recommended for deployments and is removed in version 10.0. Sites that have such a setup should migrate pools before updating the dCache.

Admin command csm info will display enabled digests for checksum calculation.

Added property pool.path.meta to specify the location of pools metadata directory, defaults to pool.path.

Added pool command csm use directio on|off to enable/disable O_DIRECT to bypass the filesystem cache during checksum calculation.

Prior version 10.0 for xroot and http movers dcache hat two ways to detect and cancel idle connections: Job Timeout Manager (jtm)and idleHandler. Now, the latter one is removed and only jtm-baes one is used. The following pool properties are obsolete:

pool.mover.xrootd.timeout.idle
pool.mover.xrootd.timeout.idle.unit
pool.mover.http.timeout.idle
pool.mover.http.timeout.idle.unit

Starting version 3.0 has dropped the limit on the maximal number of concurrent p2p transfers for a single pool. This is typically not a problem, however, in some situations, namely multiple transfers from many pools to a single one, during rebalancing, for example, a pool might become unresponsive due to IO starvation. With version 10.0 the maximal number of concurrent p2p transfers was re-introduced and pp set max active command is undeprecated. The default value is unlimited.

Dependencies

Updated external dependencies in preparation for Java17 migration.

Changelog from 9.2.0 to 10.0.0

51631eb409: qos: add flag to enable/disable role based authorization for transitions
f3ad57bfe2: common-security: add custom version of ProxyCSRGenerator#generate
3dafdddcb5: [maven-release-plugin] prepare branch @{releaseLabel}
4bc0305ae7: config: group logging configs in system-test
83ced04067: billing: Add option for json format
8f3b984fab: nfs: calculate desired memory fraction for correct memory allocation
dce4087df6: info: use DGA refresh rate as message timeout
bdb8da5cd3: Revert “chimera: make list method to support virtual directory listing”
824275ebe0: pool: make pool_read only if fs is ro
1fbd7ae388: chimera: make list method to support virtual directory listing
030d5ce33b: xroot: switch to xrootd4j–4.6.0
c357ac60d0: libs: update apache-curator version to 5.6.0
d78a4805ed: tape-api: map request supplied paths based on frontend.root
6b9e1c6b9f: srr: fix “empty path annotation” warning
4bace63646: notifications: Add BillingMessageSerializerVisitor
00ba8c7bff: bulk: fix divergent command params format strings
b5fc763f43: core: remove unintended changes to listPoolGroupXml
48e5cafb68: cells: fix error reporting on tunnel disconnect
a244b9f10a: Remove dependency on GNU patch.
55f5091d0f: ci: fix job dependency
b712a763d4: ci: use user friendly job names
f5cf4ec6c9: ci: initialize CVMFS only once
4044301494: Fix revert commit ce3927d7
ce3927d71a: Revert “poolmanager: delete property for switchingon/off caching for psu”
cc28966ee0: build(deps): bump org.postgresql:postgresql from 42.3.8 to 42.3.9
ba2a5bdf7d: build(deps): bump org.apache.commons:commons-compress
0764862633: pom: sync jersey-spring with spring (5) version
14d1acb020: ci: no special tags for non kubernetes jobs
d60eb0e135: ci: retry on transient errors
e07ee6d11c: ci: enable long running tests
3b7789d157: ci: use minimal almalinux–9 to upload packages
18a66bbb7d: ci,pom: add gitlab-code-quality-plugin, generate spotbugs report
78221848d3: pom: use latest maven-surefireand modernizer plugins
4e7ce95c09: ci: use dtzar helm-kubectl image
24e67db9b4: ci: use java17 image for deb packages
be2028c0f8: pom: use Jersey–2.41
af5ea1c266: pom: update datanucleus (and deps) to 6.0 release
c0d5c07aa1: Minor .gitignore tweaks based on IntelliJ IDEA experience on MacOS
12a2839be3: dcache-core: fix psu logging
426d7c4f7a: pool: mover grizzly IO buffer initialization into NfsTransferService
88881b9c80: webdav: fix slow listing
a8e5fec6d0: webdav: fix link header
0b330a038c: chimera: virdir, return only the file name
fea615f5ab: pool: re-introduce limit on number of concurrent p2p transfers
0bbf8f5d85: pool: remove IdleHandler from http and xroot movers
ffbbf5cad0: gplazma: fix NPE if gPlazma is rejecting all logins
adb184f757: chimera: add extra unit-tests for JdbcFs
95db4a770d: systemtest: support running clients in a container
7793086288: build: fix code-coverage reports
9074fa4aca: pool: use mover’s UUID to track xroot client reconnects
fd11f9381c: chimera: improve unit tests for jdbcfs rename
046f4a0bdb: ci: dont’t pull build artefact for kubernetes-based jobs
b79b9a7a02: Fix unit test for commit 6b47354
6b473540f4: gplazma: configured banfile plugin should ignore non-existent ban file
12a76a39dd: chimera: Revert “chimera: update FsSqlDriver#inodeOf to throw exception if file not found”
d63e2d9b93: system-test: increase direct memory
2d6fc8c575: xrootd: fix xrootd.root regression
88b73cd5af: dependencies: update modernizer plugin to be compatible with maven 3.9
b13f6d7bfd: webdav: wait for upload to complete
862963ec7e: pool:make readonly pool READONLY Motivation
523738e73c: add openssl installation hint
4e61c808fd: book: add transfermanager HA description
59ab8ffa6b: cleaner-disk: prevent ConcurrentModificationException in cleaning run
a5d6454498: dcache-xroot: fix effective root when subject is nobody
4facd1dcd3: dcache-xroot: check that descriptor is not null before calling close – fix
b00a5dc468: build(deps): bump ch.qos.logback:logback-classic
06f6c00979: build(deps): bump ch.qos.logback:logback-core from 1.2.10 to 1.2.13
73e683a66f: dcache-xroot: check that descriptor is not null before calling close
bb19a4850f: dcache-core: fix admin command completion
ffc5effbb9: qos: comment out property description
96893bcbac: ci: add exta helm ops to extend timeout and simplify retries
583330bd50: pom: Update mongodb-driver
3701b1329f: dcache-frontend: check for defined arguments with namespace qos
05d9c4cf29: webdav: do not log stacktrace if non-existing well-known is requested
87f196447b: pool: add option to use direct-io while checksum calculation
0f098e8a93: dcache-bulk,dcache-qos: repair mass cancellation issues
5da0a8d82c: dcache-bulk: allow reset to skip terminated targets
2c55595689: dcache-frontend: add hooks and context handler for dcache-view-admin
94e04e8193: pom.xml: update dcache-view to 2.1.0
9a8985598d: dcache-qos: cache modify requests until processed by executor
3a6fae5f39: info: fix issue 7327 adding nested pgGroup
ddb17205c5: pool: improve err msg on mover create if protocol is not supported
6412d58b5a: dcache-common: repaired unit test for QoSPolicy parsing (duration string error).
82922f20a0: Revert “dcache-frontend,common: check parsing of Duration in STAGE”
6ffe2cd10e: dcache-frontend,common: check parsing of Duration in STAGE
013db14c6b: dcache-bulk: fix handling of uncaught exceptions
bd1abd31fc: pool: make metadata directory location configurable
afb15bb1db: ci: generate release-notes template
f49389bb91: webdav: fix infinite recursion in Requests.stripToPath
830a8a1469: webdav: fix parsing of urls with two slashes in the path
8aa9e6af41: chimera: fix rollback of 9.1 to 9.0 db schema
6bd65801c8: gplazma: enhance help of explain login command
7ca86f06be: Revert “ci: use rancher to create k8s namespace”
ed2d9a2307: docker: use exec for start java process
63a5d1d103: Fix ce3cfe3: remove superfluous brace
ce3cfe33a1: dcache: make admin pin command asynchronous
aec714e653: oidc: fix remove invalid testcase
83d1e77eed: oidc: reject storage scopes without path
c71edfb1f9: webdav: fix formatting broken by 2ddd74a780
2ddd74a780: door,pool: handle multiple possible KafkaExceptions
714d0f617a: libs: update asm to version 9.5 with java–17 support
dc6db510c8: ci: use java17 builds for rpm and tar
88a8028c06: pom: add exta java option to run powermock test under java17
81c7e944c7: ci: remove extra ‘*’ in rpm-sign command
6b49307ce4: fix issue #7326
3104def644: build(deps): bump org.json:json from 20230227 to 20231013
f7e19137e2: libs: fix unexpected transient dependencies of rados4j
2f268fb1d7: pool: drop RDB based CEPH support
2076857a4c: dcache-bulk: use rate limiter to throttle semaphore release
0476a3a8f8: config: fix unit test in ci
8c65ee7636: config: add support for environment variable injection
ad19184647: dcache-qos: correction to the threshold warning
00313591b2: dcache-qos: set default task thread pool sizes all to max concurrent running
49ab9ed9dd: dcache-bulk: fix bug in archiver deletion query
adaacd864a: dcache-bulk: fix thread executor injection
32c2127d1b: ci: use pynfs:0.5, enable LOCK24 test
9698289eb3: ci: add webdav test
523654c774: doc: add QoS policy and role documentation to the dCache Book
e779d3802a: build(deps): bump org.apache.zookeeper:zookeeper from 3.7.1 to 3.7.2
42cc3d4ad9: pool: cms info should display configured checkums
c65c0071e8: wevdav: fix redirect path
ce9734a123: dcache-bulk: fix cancellation issues
76fde730c2: dcache-bulk: give directory listing a separate executor
3cbe3f486f: nfs: prepare code for removal of redundant stateid4 class
3d0a0bd56d: book: fix typo
e957031622: book: split srm chapter into srm, srmmanager, spacemanager
5172554462: ci: use rancher to create k8s namespace
f26fe0809d: ci: fix typos
e80d893e69: book: pinmananger, describe new default unpins/time
533c363872: dcache-core: increase default unpins per time
c370ba25f7: Support additional Kafka parameters for ftp and dcap.
60dd0baca4: dcache-bulk: add count and clear to archive admin commands
1df7a1800a: dcache-bulk: cancel activity future on target cancel
0b4140b454: dcache-bulk: refine container executor model
07492e6e94: docs: userguide fix example domains names to follow RFC 2606
5c9d04fff3: docs: UserGuide fix angle-braket in metalink
4c731ab8cf: docs: add description of webdav’s metalink support
094c1714c1: gplazma: oidc update explicit AuthZ parsing
49661e6b57: gplazma: oidc increase cache duration for OP public key material
8a5c358af4: dcache-bulk: container rewrite to optimize threading
3d23228988: dcache-bulk: add admin command and query to reset all requests with failed targets
a95a9edb5e: dcache-bulk: add convenience admin command for state counts
8b3ff0d2b4: dcache-qos: fix scanner operation completion logic
cf8e57a687: ci: by-pass docker.io for bitnami charts
7873035615: dcache-bulk: implement HA
bf70db77c5: dcache-cli: convert IllegalArgumentException to CommandException on call()
f9fbb002e1: dcache-bulk: do not PIN or STAGE files with AL ONLINE
271b32a07d: dcache-bulk: only set request status to QUEUED when permissions and targets are all inserted
b37a79a5b8: [maven-release-plugin] prepare for next development iteration
43a3e4482b: build(deps): bump org.eclipse.jetty:jetty-servlets

What’s new in dCache 10.0 Release notes