Highlights

  • Added pool metadata directory configuration option
  • Use environment variables as configuration properties

Incompatibilities

  • dropped native CEPH support. Sites must migrate their pools before updating dCache
  • dropped idle timeout handler in netty-based movers (xroot, http)

Acknowledgments

Many Thanks to Petter A. Urkedal and Lukas Mansour for their contributions.

Release 10.0.12

xroot

Return destination address (that is, the haproxy address) if xrootd.enable.proxy-protocol=true is set, instead of the actual door address.

Changelog 10.0.11..10.0.12

699ba968f8
[maven-release-plugin] prepare release 10.0.12
159a48c761
xroot: handle haproxy and checksum command
1ceddef66f
[maven-release-plugin] prepare for next development iteration

Release 10.0.11

CI

Pipeline optimizations.

pool

Fix double decrement on active hsm requests. This addresses the issue of pools stopping flushing to tape with “Negative number of active requests”.

When a thread performing I/O gets interrupted, then an InterruptedIOException might be thrown. A DCAP mover will treat and propagate such an exception as a disk I/O error, thus disabling the pool. This fix reduces false positive disk I/O errors.

Changelog 10.0.10..10.0.11

59e5f659ae
[maven-release-plugin] prepare release 10.0.11
8377fb87ea
pool: don’t treat InterruptedIOException as a disk IO error
dd52056f98
pool: fix double decrement of hsm requests
06ffe15ff4
ci: split container image registry and repository
536684fde1
[maven-release-plugin] prepare for next development iteration

Release 10.0.10

bulk

The current release fixed broken command activities.

Changelog 10.0.9..10.0.10

beea266384
[maven-release-plugin] prepare release 10.0.10
5e8991675f
ci: run spotbugs only on master branch
97ace7d80c
bulk: fix broken command activities
54df6ada6a
[maven-release-plugin] prepare for next development iteration

Release 10.0.9

bulk

When specifying an empty target, bulk proceeded to process the request instead of failing fast. This is now fixed.

gplazma

A previous commit, leading to the last bugfix releases being blacklisted, introduced a regression in the multimap plugin. Where the ‘op’ principal type is used, logins will fail with dCache logging a stacktrace like java.lang.RuntimeException: Failed to create principal: java.lang.NoSuchMethodException: org.dcache.auth.OAuthProviderPrincipal.<init>(java.lang.String). This is now fixed.

Changelog 10.0.8..10.0.9

d547d4a58e
[maven-release-plugin] prepare release 10.0.9
7a2bfe0a2f
gplazma: fix broken commit d74d9568167f4
d74d956816
gplazma: multimap fix op regression
1aff06fe09
bulk: check targets for empty strings
4678e1b5ae
[maven-release-plugin] prepare for next development iteration

Release 10.0.8

doc

Better documentation clarifying OIDC provider ID and issue claim.

Changelog 10.0.7..10.0.8

0e22cbe693
[maven-release-plugin] prepare release 10.0.8
7ebab3e236
github: add action for atumatic github-release
3e749187ff
gplazma alise initial version of plugin
e659623da3
common: add issuer URI to OAuthProviderPrincipal
9468a857c5
docs: clarify OIDC provider ID and issue claim
fa8b721504
[maven-release-plugin] prepare for next development iteration

Release 10.0.7

CI

Improve our CI pipleline.

core

Pool migration : fixed behavior of migration copy with -replicas=n (where n>1) to behave as expected.

Libs

Keeping libraries up to date.

TAPE-API

Release by relative path works.

Changelog 10.0.6..10.0.7

7abb042c1a
[maven-release-plugin] prepare release 10.0.7
422aabb2a9
WLG TAPE REST API: fix handling of frontend.root in release API
9c52354c65
gitlab: mirror tags
350ae2b32b
libs: update mina-sshd to version 2.13.1
f06f71cbf0
Fix issue with infinite replicas when replicas > 1
516ba4b573
github: change mirroring action
8e80889481
ci: use shorter k8s namespace names
0eb89d1f0d
ci: sync CI pipeline with master
169454d112
ci: use desy nims repo for CentOS7
88ce98874d
[maven-release-plugin] prepare for next development iteration

Release 10.0.6

chimera

Now it is possible to n read/write CTA and migrated Enstore files without having to set hsmInstance tag in the full directory tree.

frontend

The current release fixed the issue about failures to invoke WLCG tape API to stage files on files relative to {webdav,frontend}.root

It is now possible to call stage API successully on paths ralative to frontend.root.

Changelog 10.0.5..10.0.6

87b2451956
[maven-release-plugin] prepare release 10.0.6
113335cac5
pom: compile code in the same jvm as maven
3f116a5169
chimera: add CTA HSM StorageInfo extractor
935670ec59
frontend: handle frontend.root variable properly
379449ae6e
[maven-release-plugin] prepare for next development iteration

Release 10.0.5

chimera

Adapt existing uri_encode function so that the Enstore HSM script does not fail and the files go to tape successfully.

Changelog 10.0.4..10.0.5

44699ad4ed
[maven-release-plugin] prepare release 10.0.5
37fc60f08f
chimera: fix uri_encode to handle special characters
1183d69440
[maven-release-plugin] prepare for next development iteration

Release 10.0.4

ci

Add option to pipeline to control upload options.

Add comments to various pipeline stages.

Changelog 10.0.3..10.0.4

6f8c437a72
[maven-release-plugin] prepare release 10.0.4
954828e9d0
ci: add comments to various pipeline stages
61d8a22c30
ci: add property to control upload options
339aa608a0
[maven-release-plugin] prepare for next development iteration

Release 10.0.3

bulk

Bulk truncates path to 256 characters and this seems to be causing problemsd.

This is now fixed.

pool

With CentOS7, the client sends the last request, such that offset+count == filesize. REHL9 sends the last request so that the count is multiple of 4096, which is legal.

The current release fixed miscalculation of offset on short read.

Changelog 10.0.2..10.0.3

323581948e
[maven-release-plugin] prepare release 10.0.3
13d66ba99e
ci: disable srmcp test
db51b8baa0
ci: use python3 for robot test
a8ef6f5a1b
bulk: do not truncate target paths
3895809e98
ci: pinpoint postgres helm version
3c827ab76c
pool: fix miscalculation of offset on short read.
8fe1fa05ec
[maven-release-plugin] prepare for next development iteration

Release 10.0.2

chimera

Fixed a bug that made it possible to create a loop on directory move.

Changelog 10.0.1..10.0.2

7d72588073
[maven-release-plugin] prepare release 10.0.2
625691108f
chimera: fix loop creation on directory move
15f32f5d69
ci: drop –ftp-create-dirs option (as we switch to https)
865fcb0b03
[maven-release-plugin] prepare for next development iteration

Release 10.0.1

rpm

Java 11 will be installed as dependency when Java 17 is not already present.

webdav

When running multiple Remote Transfermanager, two transfer that were started simultaneously had the same ID which led to the second transfer becoming orphan after the first one completed. This is fixed now.

Changelog 10.0.0..10.0.1

1bbfbbf8b9
[maven-release-plugin] prepare release 10.0.1
afd5dd5f6e
ci: use almalinux9 for rpm install test
d9f3706793
docs: update oidc chapter to explain trust anchors
3c15025ab8
rpm: don’t force java–11 it java–17 is installed
61e29f8397
ci: eplocitly specify kubernetes namespace
9762947f8a
ci: fix typo
b6b8386a19
webdav: use transfermanager+id to identify TPC transfer
da81c86739
[maven-release-plugin] prepare for next development iteration

Release 10.0.0

Billing

The property billing.format.json was added to write the logs in JSON format instead of plain text if it’s set to true.

Configuration

In the container world, it’s common to use environment variables to pass the configuration parameters. With release 10.0 dCache support injects of environment variables into a configuration with special values prefixed with env.. For example, to control wan port numbers with environment variables DCACHE_WAN_PORT_MIN and DCACHE_WAN_PORT_MAX the layout file might have the following configuration:

dcache.net.wan.port.min=${env.DCACHE_WAN_PORT_MIN}
dcache.net.wan.port.max=${env.DCACHE_WAN_PORT_MAX}

> NOTE: the nested variables are not allowed. E.g. ${env.$VAR1-$VAR2} will not be resolved.

DCAP

Added support for additional Kafka parameters.

Frontend

FTP

Added support for additional Kafka parameters.

gplazma

Update OIDC-related documentation.

Pinmanager

A pin request via admin interface no longer waits for the pin to be established before returning.

Pool

Sites that have tried to run dCache with built-in CEPH decided to go for CephFS (mounted as regular FS) as it provides better performance, scalability and erasure encoding. Thus, built-in CEPH is not recommended for deployments and is removed in version 10.0. Sites that have such a setup should migrate pools before updating the dCache.

Admin command csm info will display enabled digests for checksum calculation.

Added property pool.path.meta to specify the location of pools metadata directory, defaults to pool.path.

Added pool command csm use directio on|off to enable/disable O_DIRECT to bypass the filesystem cache during checksum calculation.

Prior version 10.0 for xroot and http movers dcache hat two ways to detect and cancel idle connections: Job Timeout Manager (jtm)and idleHandler. Now, the latter one is removed and only jtm-baes one is used. The following pool properties are obsolete:

pool.mover.xrootd.timeout.idle
pool.mover.xrootd.timeout.idle.unit
pool.mover.http.timeout.idle
pool.mover.http.timeout.idle.unit

Starting version 3.0 has dropped the limit on the maximal number of concurrent p2p transfers for a single pool. This is typically not a problem, however, in some situations, namely multiple transfers from many pools to a single one, during rebalancing, for example, a pool might become unresponsive due to IO starvation. With version 10.0 the maximal number of concurrent p2p transfers was re-introduced and pp set max active command is undeprecated. The default value is unlimited.

Dependencies

Updated external dependencies in preparation for Java17 migration.

Changelog from 9.2.0 to 10.0.0

51631eb409
qos: add flag to enable/disable role based authorization for transitions
f3ad57bfe2
common-security: add custom version of ProxyCSRGenerator#generate
3dafdddcb5
[maven-release-plugin] prepare branch @{releaseLabel}
4bc0305ae7
config: group logging configs in system-test
83ced04067
billing: Add option for json format
8f3b984fab
nfs: calculate desired memory fraction for correct memory allocation
dce4087df6
info: use DGA refresh rate as message timeout
bdb8da5cd3
Revert “chimera: make list method to support virtual directory listing”
824275ebe0
pool: make pool_read only if fs is ro
1fbd7ae388
chimera: make list method to support virtual directory listing
030d5ce33b
xroot: switch to xrootd4j–4.6.0
c357ac60d0
libs: update apache-curator version to 5.6.0
d78a4805ed
tape-api: map request supplied paths based on frontend.root
6b9e1c6b9f
srr: fix “empty path annotation” warning
4bace63646
notifications: Add BillingMessageSerializerVisitor
00ba8c7bff
bulk: fix divergent command params format strings
b5fc763f43
core: remove unintended changes to listPoolGroupXml
48e5cafb68
cells: fix error reporting on tunnel disconnect
a244b9f10a
Remove dependency on GNU patch.
55f5091d0f
ci: fix job dependency
b712a763d4
ci: use user friendly job names
f5cf4ec6c9
ci: initialize CVMFS only once
4044301494
Fix revert commit ce3927d7
ce3927d71a
Revert “poolmanager: delete property for switchingon/off caching for psu”
cc28966ee0
build(deps): bump org.postgresql:postgresql from 42.3.8 to 42.3.9
ba2a5bdf7d
build(deps): bump org.apache.commons:commons-compress
0764862633
pom: sync jersey-spring with spring (5) version
14d1acb020
ci: no special tags for non kubernetes jobs
d60eb0e135
ci: retry on transient errors
e07ee6d11c
ci: enable long running tests
3b7789d157
ci: use minimal almalinux–9 to upload packages
18a66bbb7d
ci,pom: add gitlab-code-quality-plugin, generate spotbugs report
78221848d3
pom: use latest maven-surefireand modernizer plugins
4e7ce95c09
ci: use dtzar helm-kubectl image
24e67db9b4
ci: use java17 image for deb packages
be2028c0f8
pom: use Jersey–2.41
af5ea1c266
pom: update datanucleus (and deps) to 6.0 release
c0d5c07aa1
Minor .gitignore tweaks based on IntelliJ IDEA experience on MacOS
12a2839be3
dcache-core: fix psu logging
426d7c4f7a
pool: mover grizzly IO buffer initialization into NfsTransferService
88881b9c80
webdav: fix slow listing
a8e5fec6d0
webdav: fix link header
0b330a038c
chimera: virdir, return only the file name
fea615f5ab
pool: re-introduce limit on number of concurrent p2p transfers
0bbf8f5d85
pool: remove IdleHandler from http and xroot movers
ffbbf5cad0
gplazma: fix NPE if gPlazma is rejecting all logins
adb184f757
chimera: add extra unit-tests for JdbcFs
95db4a770d
systemtest: support running clients in a container
7793086288
build: fix code-coverage reports
9074fa4aca
pool: use mover’s UUID to track xroot client reconnects
fd11f9381c
chimera: improve unit tests for jdbcfs rename
046f4a0bdb
ci: dont’t pull build artefact for kubernetes-based jobs
b79b9a7a02
Fix unit test for commit 6b47354
6b473540f4
gplazma: configured banfile plugin should ignore non-existent ban file
12a76a39dd
chimera: Revert “chimera: update FsSqlDriver#inodeOf to throw exception if file not found”
d63e2d9b93
system-test: increase direct memory
2d6fc8c575
xrootd: fix xrootd.root regression
88b73cd5af
dependencies: update modernizer plugin to be compatible with maven 3.9
b13f6d7bfd
webdav: wait for upload to complete
862963ec7e
pool:make readonly pool READONLY Motivation
523738e73c
add openssl installation hint
4e61c808fd
book: add transfermanager HA description
59ab8ffa6b
cleaner-disk: prevent ConcurrentModificationException in cleaning run
a5d6454498
dcache-xroot: fix effective root when subject is nobody
4facd1dcd3
dcache-xroot: check that descriptor is not null before calling close – fix
b00a5dc468
build(deps): bump ch.qos.logback:logback-classic
06f6c00979
build(deps): bump ch.qos.logback:logback-core from 1.2.10 to 1.2.13
73e683a66f
dcache-xroot: check that descriptor is not null before calling close
bb19a4850f
dcache-core: fix admin command completion
ffc5effbb9
qos: comment out property description
96893bcbac
ci: add exta helm ops to extend timeout and simplify retries
583330bd50
pom: Update mongodb-driver
3701b1329f
dcache-frontend: check for defined arguments with namespace qos
05d9c4cf29
webdav: do not log stacktrace if non-existing well-known is requested
87f196447b
pool: add option to use direct-io while checksum calculation
0f098e8a93
dcache-bulk,dcache-qos: repair mass cancellation issues
5da0a8d82c
dcache-bulk: allow reset to skip terminated targets
2c55595689
dcache-frontend: add hooks and context handler for dcache-view-admin
94e04e8193
pom.xml: update dcache-view to 2.1.0
9a8985598d
dcache-qos: cache modify requests until processed by executor
3a6fae5f39
info: fix issue 7327 adding nested pgGroup
ddb17205c5
pool: improve err msg on mover create if protocol is not supported
6412d58b5a
dcache-common: repaired unit test for QoSPolicy parsing (duration string error).
82922f20a0
Revert “dcache-frontend,common: check parsing of Duration in STAGE”
6ffe2cd10e
dcache-frontend,common: check parsing of Duration in STAGE
013db14c6b
dcache-bulk: fix handling of uncaught exceptions
bd1abd31fc
pool: make metadata directory location configurable
afb15bb1db
ci: generate release-notes template
f49389bb91
webdav: fix infinite recursion in Requests.stripToPath
830a8a1469
webdav: fix parsing of urls with two slashes in the path
8aa9e6af41
chimera: fix rollback of 9.1 to 9.0 db schema
6bd65801c8
gplazma: enhance help of explain login command
7ca86f06be
Revert “ci: use rancher to create k8s namespace”
ed2d9a2307
docker: use exec for start java process
63a5d1d103
Fix ce3cfe3: remove superfluous brace
ce3cfe33a1
dcache: make admin pin command asynchronous
aec714e653
oidc: fix remove invalid testcase
83d1e77eed
oidc: reject storage scopes without path
c71edfb1f9
webdav: fix formatting broken by 2ddd74a780
2ddd74a780
door,pool: handle multiple possible KafkaExceptions
714d0f617a
libs: update asm to version 9.5 with java–17 support
dc6db510c8
ci: use java17 builds for rpm and tar
88a8028c06
pom: add exta java option to run powermock test under java17
81c7e944c7
ci: remove extra ‘*’ in rpm-sign command
6b49307ce4
fix issue #7326
3104def644
build(deps): bump org.json:json from 20230227 to 20231013
f7e19137e2
libs: fix unexpected transient dependencies of rados4j
2f268fb1d7
pool: drop RDB based CEPH support
2076857a4c
dcache-bulk: use rate limiter to throttle semaphore release
0476a3a8f8
config: fix unit test in ci
8c65ee7636
config: add support for environment variable injection
ad19184647
dcache-qos: correction to the threshold warning
00313591b2
dcache-qos: set default task thread pool sizes all to max concurrent running
49ab9ed9dd
dcache-bulk: fix bug in archiver deletion query
adaacd864a
dcache-bulk: fix thread executor injection
32c2127d1b
ci: use pynfs:0.5, enable LOCK24 test
9698289eb3
ci: add webdav test
523654c774
doc: add QoS policy and role documentation to the dCache Book
e779d3802a
build(deps): bump org.apache.zookeeper:zookeeper from 3.7.1 to 3.7.2
42cc3d4ad9
pool: cms info should display configured checkums
c65c0071e8
wevdav: fix redirect path
ce9734a123
dcache-bulk: fix cancellation issues
76fde730c2
dcache-bulk: give directory listing a separate executor
3cbe3f486f
nfs: prepare code for removal of redundant stateid4 class
3d0a0bd56d
book: fix typo
e957031622
book: split srm chapter into srm, srmmanager, spacemanager
5172554462
ci: use rancher to create k8s namespace
f26fe0809d
ci: fix typos
e80d893e69
book: pinmananger, describe new default unpins/time
533c363872
dcache-core: increase default unpins per time
c370ba25f7
Support additional Kafka parameters for ftp and dcap.
60dd0baca4
dcache-bulk: add count and clear to archive admin commands
1df7a1800a
dcache-bulk: cancel activity future on target cancel
0b4140b454
dcache-bulk: refine container executor model
07492e6e94
docs: userguide fix example domains names to follow RFC 2606
5c9d04fff3
docs: UserGuide fix angle-braket in metalink
4c731ab8cf
docs: add description of webdav’s metalink support
094c1714c1
gplazma: oidc update explicit AuthZ parsing
49661e6b57
gplazma: oidc increase cache duration for OP public key material
8a5c358af4
dcache-bulk: container rewrite to optimize threading
3d23228988
dcache-bulk: add admin command and query to reset all requests with failed targets
a95a9edb5e
dcache-bulk: add convenience admin command for state counts
8b3ff0d2b4
dcache-qos: fix scanner operation completion logic
cf8e57a687
ci: by-pass docker.io for bitnami charts
7873035615
dcache-bulk: implement HA
bf70db77c5
dcache-cli: convert IllegalArgumentException to CommandException on call()
f9fbb002e1
dcache-bulk: do not PIN or STAGE files with AL ONLINE
271b32a07d
dcache-bulk: only set request status to QUEUED when permissions and targets are all inserted
b37a79a5b8
[maven-release-plugin] prepare for next development iteration
43a3e4482b
build(deps): bump org.eclipse.jetty:jetty-servlets