Highlights

• Added pool metadata directory configuration option
• Use environment variables as configuration properties

Incompatibilities

• dropped native CEPH support. Sites must migrate their pools before updating dCache
• dropped idle timeout handler in netty-based movers (xroot, http)

Acknowledgments

Many Thanks to Petter A. Urkedal and Lukas Mansour for their contributions.

Release 10.0.16

bulk

Restore the ability to use absolute paths when using bulk REST API.

cells

Cells will always try to re-establish dead tunnel connections.

qos

This fix will take care of the ‘Attribute is not defined: QOS_POLICY’ error in logs.

Changelog 10.0.15..10.0.16

9d00696bb1

[maven-release-plugin] prepare release 10.0.16

38ec0d0059

bulk: handle absolute/relative paths in uniform fashion

72e5752762

qos: QOS fails with ’Attribute is not defined: QOS_POLICY’

4a3e915912

cells: always try to re-establish dead tunnel, unless stopped

48d312b229

[maven-release-plugin] prepare for next development iteration

Release 10.0.15

jvm

Removed default JVM option UseCompressedOops so dCache works with large heap size.

Changelog 10.0.14..10.0.15

11e2ba3530

[maven-release-plugin] prepare release 10.0.15

f04bc70af4

jvm: drop UseCompressedOops JVM option

6d31a13780

[maven-release-plugin] prepare for next development iteration

Release 10.0.14

cells

When Zookeeper updates core domain infos, dCache will first kill the existing cell tunnels and then later try to read and parse the new value. If the new value is an empty string (for whatever reason), parsing will fail, but a new connection will not be established. This now fixed.

Changelog 10.0.13..10.0.14

f5570e31c3

[maven-release-plugin] prepare release 10.0.14

95c4792c1f

cells: ignore empty core domain uris propagated by zk

368e8e5502

[maven-release-plugin] prepare for next development iteration

Release 10.0.13

tape

Users reported 2 day pin lifetime on staged files (which is a default) despite specifying different values. This is now fixed.

Changelog 10.0.12..10.0.13

6286a554bd

[maven-release-plugin] prepare release 10.0.13

deee0f826d

tape REST api: additional fix tohandling of prefixed paths

aa38343e56

[maven-release-plugin] prepare for next development iteration

Release 10.0.12

xroot

Return destination address (that is, the haproxy address) if xrootd.enable.proxy-protocol=true is set, instead of the actual door address.

Changelog 10.0.11..10.0.12

699ba968f8

[maven-release-plugin] prepare release 10.0.12

159a48c761

xroot: handle haproxy and checksum command

1ceddef66f

[maven-release-plugin] prepare for next development iteration

Release 10.0.11

CI

Pipeline optimizations.

pool

Fix double decrement on active hsm requests. This addresses the issue of pools stopping flushing to tape with “Negative number of active requests”.

When a thread performing I/O gets interrupted, then an InterruptedIOException might be thrown. A DCAP mover will treat and propagate such an exception as a disk I/O error, thus disabling the pool. This fix reduces false positive disk I/O errors.

Changelog 10.0.10..10.0.11

59e5f659ae

[maven-release-plugin] prepare release 10.0.11

8377fb87ea

pool: don’t treat InterruptedIOException as a disk IO error

dd52056f98

pool: fix double decrement of hsm requests

06ffe15ff4

ci: split container image registry and repository

536684fde1

[maven-release-plugin] prepare for next development iteration

Release 10.0.10

bulk

The current release fixed broken command activities.

Changelog 10.0.9..10.0.10

beea266384

[maven-release-plugin] prepare release 10.0.10

5e8991675f

ci: run spotbugs only on master branch

97ace7d80c

bulk: fix broken command activities

54df6ada6a

[maven-release-plugin] prepare for next development iteration

Release 10.0.9

bulk

When specifying an empty target, bulk proceeded to process the request instead of failing fast. This is now fixed.

gplazma

A previous commit, leading to the last bugfix releases being blacklisted, introduced a regression in the multimap plugin. Where the ‘op’ principal type is used, logins will fail with dCache logging a stacktrace like java.lang.RuntimeException: Failed to create principal: java.lang.NoSuchMethodException: org.dcache.auth.OAuthProviderPrincipal.<init>(java.lang.String). This is now fixed.

Changelog 10.0.8..10.0.9

d547d4a58e

[maven-release-plugin] prepare release 10.0.9

7a2bfe0a2f

gplazma: fix broken commit d74d9568167f4

d74d956816

gplazma: multimap fix op regression

1aff06fe09

bulk: check targets for empty strings

4678e1b5ae

[maven-release-plugin] prepare for next development iteration

Release 10.0.8

doc

Better documentation clarifying OIDC provider ID and issue claim.

Changelog 10.0.7..10.0.8

0e22cbe693

[maven-release-plugin] prepare release 10.0.8

7ebab3e236

github: add action for atumatic github-release

3e749187ff

gplazma alise initial version of plugin

e659623da3

common: add issuer URI to OAuthProviderPrincipal

9468a857c5

docs: clarify OIDC provider ID and issue claim

fa8b721504

[maven-release-plugin] prepare for next development iteration

Release 10.0.7

CI

Improve our CI pipleline.

core

Pool migration : fixed behavior of migration copy with -replicas=n (where n>1) to behave as expected.

Libs

Keeping libraries up to date.

TAPE-API

Release by relative path works.

Changelog 10.0.6..10.0.7

7abb042c1a

[maven-release-plugin] prepare release 10.0.7

422aabb2a9

WLG TAPE REST API: fix handling of frontend.root in release API

9c52354c65

gitlab: mirror tags

350ae2b32b

libs: update mina-sshd to version 2.13.1

f06f71cbf0

Fix issue with infinite replicas when replicas > 1

516ba4b573

github: change mirroring action

8e80889481

ci: use shorter k8s namespace names

0eb89d1f0d

ci: sync CI pipeline with master

169454d112

ci: use desy nims repo for CentOS7

88ce98874d

[maven-release-plugin] prepare for next development iteration

Release 10.0.6

chimera

Now it is possible to n read/write CTA and migrated Enstore files without having to set hsmInstance tag in the full directory tree.

frontend

The current release fixed the issue about failures to invoke WLCG tape API to stage files on files relative to {webdav,frontend}.root

It is now possible to call stage API successully on paths ralative to frontend.root.

Changelog 10.0.5..10.0.6

87b2451956

[maven-release-plugin] prepare release 10.0.6

113335cac5

pom: compile code in the same jvm as maven

3f116a5169

chimera: add CTA HSM StorageInfo extractor

935670ec59

frontend: handle frontend.root variable properly

379449ae6e

[maven-release-plugin] prepare for next development iteration

Release 10.0.5

chimera

Adapt existing uri_encode function so that the Enstore HSM script does not fail and the files go to tape successfully.

Changelog 10.0.4..10.0.5

44699ad4ed

[maven-release-plugin] prepare release 10.0.5

37fc60f08f

chimera: fix uri_encode to handle special characters

1183d69440

[maven-release-plugin] prepare for next development iteration

Release 10.0.4

ci

Add option to pipeline to control upload options.

Add comments to various pipeline stages.

Changelog 10.0.3..10.0.4

6f8c437a72

[maven-release-plugin] prepare release 10.0.4

954828e9d0

ci: add comments to various pipeline stages

61d8a22c30

ci: add property to control upload options

339aa608a0

[maven-release-plugin] prepare for next development iteration

Release 10.0.3

bulk

Bulk truncates path to 256 characters and this seems to be causing problemsd.

This is now fixed.

pool

With CentOS7, the client sends the last request, such that offset+count == filesize. REHL9 sends the last request so that the count is multiple of 4096, which is legal.

The current release fixed miscalculation of offset on short read.

Changelog 10.0.2..10.0.3

323581948e

[maven-release-plugin] prepare release 10.0.3

13d66ba99e

ci: disable srmcp test

db51b8baa0

ci: use python3 for robot test

a8ef6f5a1b

bulk: do not truncate target paths

3895809e98

ci: pinpoint postgres helm version

3c827ab76c

pool: fix miscalculation of offset on short read.

8fe1fa05ec

[maven-release-plugin] prepare for next development iteration

Release 10.0.2

chimera

Fixed a bug that made it possible to create a loop on directory move.

Changelog 10.0.1..10.0.2

7d72588073

[maven-release-plugin] prepare release 10.0.2

625691108f

chimera: fix loop creation on directory move

15f32f5d69

ci: drop –ftp-create-dirs option (as we switch to https)

865fcb0b03

[maven-release-plugin] prepare for next development iteration

Release 10.0.1

rpm

Java 11 will be installed as dependency when Java 17 is not already present.

webdav

When running multiple Remote Transfermanager, two transfer that were started simultaneously had the same ID which led to the second transfer becoming orphan after the first one completed. This is fixed now.

Changelog 10.0.0..10.0.1

1bbfbbf8b9

[maven-release-plugin] prepare release 10.0.1

afd5dd5f6e

ci: use almalinux9 for rpm install test

d9f3706793

docs: update oidc chapter to explain trust anchors

3c15025ab8

rpm: don’t force java–11 it java–17 is installed

61e29f8397

ci: eplocitly specify kubernetes namespace

9762947f8a

ci: fix typo

b6b8386a19

webdav: use transfermanager+id to identify TPC transfer

da81c86739

[maven-release-plugin] prepare for next development iteration

Release 10.0.0

Billing

The property billing.format.json was added to write the logs in JSON format instead of plain text if it’s set to true.

Configuration

In the container world, it’s common to use environment variables to pass the configuration parameters. With release 10.0 dCache support injects of environment variables into a configuration with special values prefixed with env.. For example, to control wan port numbers with environment variables DCACHE_WAN_PORT_MIN and DCACHE_WAN_PORT_MAX the layout file might have the following configuration:

dcache.net.wan.port.min=${env.DCACHE_WAN_PORT_MIN}
dcache.net.wan.port.max=${env.DCACHE_WAN_PORT_MAX}

> NOTE: the nested variables are not allowed. E.g. ${env.$VAR1-$VAR2} will not be resolved.

DCAP

Added support for additional Kafka parameters.

Frontend

FTP

Added support for additional Kafka parameters.

gplazma

Update OIDC-related documentation.

Pinmanager

A pin request via admin interface no longer waits for the pin to be established before returning.

Pool

Sites that have tried to run dCache with built-in CEPH decided to go for CephFS (mounted as regular FS) as it provides better performance, scalability and erasure encoding. Thus, built-in CEPH is not recommended for deployments and is removed in version 10.0. Sites that have such a setup should migrate pools before updating the dCache.

Admin command csm info will display enabled digests for checksum calculation.

Added property pool.path.meta to specify the location of pools metadata directory, defaults to pool.path.

Added pool command csm use directio on|off to enable/disable O_DIRECT to bypass the filesystem cache during checksum calculation.

Prior version 10.0 for xroot and http movers dcache hat two ways to detect and cancel idle connections: Job Timeout Manager (jtm)and idleHandler. Now, the latter one is removed and only jtm-baes one is used. The following pool properties are obsolete:

pool.mover.xrootd.timeout.idle
pool.mover.xrootd.timeout.idle.unit
pool.mover.http.timeout.idle
pool.mover.http.timeout.idle.unit

Starting version 3.0 has dropped the limit on the maximal number of concurrent p2p transfers for a single pool. This is typically not a problem, however, in some situations, namely multiple transfers from many pools to a single one, during rebalancing, for example, a pool might become unresponsive due to IO starvation. With version 10.0 the maximal number of concurrent p2p transfers was re-introduced and pp set max active command is undeprecated. The default value is unlimited.

Dependencies

Updated external dependencies in preparation for Java17 migration.

Changelog from 9.2.0 to 10.0.0

51631eb409

qos: add flag to enable/disable role based authorization for transitions

f3ad57bfe2

common-security: add custom version of ProxyCSRGenerator#generate

3dafdddcb5

[maven-release-plugin] prepare branch @{releaseLabel}

4bc0305ae7

config: group logging configs in system-test

83ced04067

billing: Add option for json format

8f3b984fab

nfs: calculate desired memory fraction for correct memory allocation

dce4087df6

info: use DGA refresh rate as message timeout

bdb8da5cd3

Revert “chimera: make list method to support virtual directory listing”

824275ebe0

pool: make pool_read only if fs is ro

1fbd7ae388

chimera: make list method to support virtual directory listing

030d5ce33b

xroot: switch to xrootd4j–4.6.0

c357ac60d0

libs: update apache-curator version to 5.6.0

d78a4805ed

tape-api: map request supplied paths based on frontend.root

6b9e1c6b9f

srr: fix “empty path annotation” warning

4bace63646

notifications: Add BillingMessageSerializerVisitor

00ba8c7bff

bulk: fix divergent command params format strings

b5fc763f43

core: remove unintended changes to listPoolGroupXml

48e5cafb68

cells: fix error reporting on tunnel disconnect

a244b9f10a

Remove dependency on GNU patch.

55f5091d0f

ci: fix job dependency

b712a763d4

ci: use user friendly job names

f5cf4ec6c9

ci: initialize CVMFS only once

4044301494

Fix revert commit ce3927d7

ce3927d71a

Revert “poolmanager: delete property for switchingon/off caching for psu”

cc28966ee0

build(deps): bump org.postgresql:postgresql from 42.3.8 to 42.3.9

ba2a5bdf7d

build(deps): bump org.apache.commons:commons-compress

0764862633

pom: sync jersey-spring with spring (5) version

14d1acb020

ci: no special tags for non kubernetes jobs

d60eb0e135

ci: retry on transient errors

e07ee6d11c

ci: enable long running tests

3b7789d157

ci: use minimal almalinux–9 to upload packages

18a66bbb7d

ci,pom: add gitlab-code-quality-plugin, generate spotbugs report

78221848d3

pom: use latest maven-surefireand modernizer plugins

4e7ce95c09

ci: use dtzar helm-kubectl image

24e67db9b4

ci: use java17 image for deb packages

be2028c0f8

pom: use Jersey–2.41

af5ea1c266

pom: update datanucleus (and deps) to 6.0 release

c0d5c07aa1

Minor .gitignore tweaks based on IntelliJ IDEA experience on MacOS

12a2839be3

dcache-core: fix psu logging

426d7c4f7a

pool: mover grizzly IO buffer initialization into NfsTransferService

88881b9c80

webdav: fix slow listing

a8e5fec6d0

webdav: fix link header

0b330a038c

chimera: virdir, return only the file name

fea615f5ab

pool: re-introduce limit on number of concurrent p2p transfers

0bbf8f5d85

pool: remove IdleHandler from http and xroot movers

ffbbf5cad0

gplazma: fix NPE if gPlazma is rejecting all logins

adb184f757

chimera: add extra unit-tests for JdbcFs

95db4a770d

systemtest: support running clients in a container

7793086288

build: fix code-coverage reports

9074fa4aca

pool: use mover’s UUID to track xroot client reconnects

fd11f9381c

chimera: improve unit tests for jdbcfs rename

046f4a0bdb

ci: dont’t pull build artefact for kubernetes-based jobs

b79b9a7a02

Fix unit test for commit 6b47354

6b473540f4

gplazma: configured banfile plugin should ignore non-existent ban file

12a76a39dd

chimera: Revert “chimera: update FsSqlDriver#inodeOf to throw exception if file not found”

d63e2d9b93

system-test: increase direct memory

2d6fc8c575

xrootd: fix xrootd.root regression

88b73cd5af

dependencies: update modernizer plugin to be compatible with maven 3.9

b13f6d7bfd

webdav: wait for upload to complete

862963ec7e

pool:make readonly pool READONLY Motivation

523738e73c

add openssl installation hint

4e61c808fd

book: add transfermanager HA description

59ab8ffa6b

cleaner-disk: prevent ConcurrentModificationException in cleaning run

a5d6454498

dcache-xroot: fix effective root when subject is nobody

4facd1dcd3

dcache-xroot: check that descriptor is not null before calling close – fix

b00a5dc468

build(deps): bump ch.qos.logback:logback-classic

06f6c00979

build(deps): bump ch.qos.logback:logback-core from 1.2.10 to 1.2.13

73e683a66f

dcache-xroot: check that descriptor is not null before calling close

bb19a4850f

dcache-core: fix admin command completion

ffc5effbb9

qos: comment out property description

96893bcbac

ci: add exta helm ops to extend timeout and simplify retries

583330bd50

pom: Update mongodb-driver

3701b1329f

dcache-frontend: check for defined arguments with namespace qos

05d9c4cf29

webdav: do not log stacktrace if non-existing well-known is requested

87f196447b

pool: add option to use direct-io while checksum calculation

0f098e8a93

dcache-bulk,dcache-qos: repair mass cancellation issues

5da0a8d82c

dcache-bulk: allow reset to skip terminated targets

2c55595689

dcache-frontend: add hooks and context handler for dcache-view-admin

94e04e8193

pom.xml: update dcache-view to 2.1.0

9a8985598d

dcache-qos: cache modify requests until processed by executor

3a6fae5f39

info: fix issue 7327 adding nested pgGroup

ddb17205c5

pool: improve err msg on mover create if protocol is not supported

6412d58b5a

dcache-common: repaired unit test for QoSPolicy parsing (duration string error).

82922f20a0

Revert “dcache-frontend,common: check parsing of Duration in STAGE”

6ffe2cd10e

dcache-frontend,common: check parsing of Duration in STAGE

013db14c6b

dcache-bulk: fix handling of uncaught exceptions

bd1abd31fc

pool: make metadata directory location configurable

afb15bb1db

ci: generate release-notes template

f49389bb91

webdav: fix infinite recursion in Requests.stripToPath

830a8a1469

webdav: fix parsing of urls with two slashes in the path

8aa9e6af41

chimera: fix rollback of 9.1 to 9.0 db schema

6bd65801c8

gplazma: enhance help of explain login command

7ca86f06be

Revert “ci: use rancher to create k8s namespace”

ed2d9a2307

docker: use exec for start java process

63a5d1d103

Fix ce3cfe3: remove superfluous brace

ce3cfe33a1

dcache: make admin pin command asynchronous

aec714e653

oidc: fix remove invalid testcase

83d1e77eed

oidc: reject storage scopes without path

c71edfb1f9

webdav: fix formatting broken by 2ddd74a780

2ddd74a780

door,pool: handle multiple possible KafkaExceptions

714d0f617a

libs: update asm to version 9.5 with java–17 support

dc6db510c8

ci: use java17 builds for rpm and tar

88a8028c06

pom: add exta java option to run powermock test under java17

81c7e944c7

ci: remove extra ‘*’ in rpm-sign command

6b49307ce4

fix issue #7326

3104def644

build(deps): bump org.json:json from 20230227 to 20231013

f7e19137e2

libs: fix unexpected transient dependencies of rados4j

2f268fb1d7

pool: drop RDB based CEPH support

2076857a4c

dcache-bulk: use rate limiter to throttle semaphore release

0476a3a8f8

config: fix unit test in ci

8c65ee7636

config: add support for environment variable injection

ad19184647

dcache-qos: correction to the threshold warning

00313591b2

dcache-qos: set default task thread pool sizes all to max concurrent running

49ab9ed9dd

dcache-bulk: fix bug in archiver deletion query

adaacd864a

dcache-bulk: fix thread executor injection

32c2127d1b

ci: use pynfs:0.5, enable LOCK24 test

9698289eb3

ci: add webdav test

523654c774

doc: add QoS policy and role documentation to the dCache Book

e779d3802a

build(deps): bump org.apache.zookeeper:zookeeper from 3.7.1 to 3.7.2

42cc3d4ad9

pool: cms info should display configured checkums

c65c0071e8

wevdav: fix redirect path

ce9734a123

dcache-bulk: fix cancellation issues

76fde730c2

dcache-bulk: give directory listing a separate executor

3cbe3f486f

nfs: prepare code for removal of redundant stateid4 class

3d0a0bd56d

book: fix typo

e957031622

book: split srm chapter into srm, srmmanager, spacemanager

5172554462

ci: use rancher to create k8s namespace

f26fe0809d

ci: fix typos

e80d893e69

book: pinmananger, describe new default unpins/time

533c363872

dcache-core: increase default unpins per time

c370ba25f7

Support additional Kafka parameters for ftp and dcap.

60dd0baca4

dcache-bulk: add count and clear to archive admin commands

1df7a1800a

dcache-bulk: cancel activity future on target cancel

0b4140b454

dcache-bulk: refine container executor model

07492e6e94

docs: userguide fix example domains names to follow RFC 2606

5c9d04fff3

docs: UserGuide fix angle-braket in metalink

4c731ab8cf

docs: add description of webdav’s metalink support

094c1714c1

gplazma: oidc update explicit AuthZ parsing

49661e6b57

gplazma: oidc increase cache duration for OP public key material

8a5c358af4

dcache-bulk: container rewrite to optimize threading

3d23228988

dcache-bulk: add admin command and query to reset all requests with failed targets

a95a9edb5e

dcache-bulk: add convenience admin command for state counts

8b3ff0d2b4

dcache-qos: fix scanner operation completion logic

cf8e57a687

ci: by-pass docker.io for bitnami charts

7873035615

dcache-bulk: implement HA

bf70db77c5

dcache-cli: convert IllegalArgumentException to CommandException on call()

f9fbb002e1

dcache-bulk: do not PIN or STAGE files with AL ONLINE

271b32a07d

dcache-bulk: only set request status to QUEUED when permissions and targets are all inserted

b37a79a5b8

[maven-release-plugin] prepare for next development iteration

43a3e4482b

build(deps): bump org.eclipse.jetty:jetty-servlets