What’s new in dCache 10.0
Release notes
Highlights
- Added pool metadata directory configuration option
- Use environment variables as configuration properties
Incompatibilities
- dropped native CEPH support. Sites must migrate their pools before updating dCache
- dropped idle timeout handler in netty-based movers (xroot, http)
Acknowledgments
Many Thanks to Petter A. Urkedal and Lukas Mansour for their contributions.
Release 10.0.12
xroot
Return destination address (that is, the haproxy address) if xrootd.enable.proxy-protocol=true
is set, instead of the actual door address.
Changelog 10.0.11..10.0.12
- 699ba968f8
- [maven-release-plugin] prepare release 10.0.12
- 159a48c761
- xroot: handle haproxy and checksum command
- 1ceddef66f
- [maven-release-plugin] prepare for next development iteration
Release 10.0.11
CI
Pipeline optimizations.
pool
Fix double decrement on active hsm requests. This addresses the issue of pools stopping flushing to tape with “Negative number of active requests”.
When a thread performing I/O gets interrupted, then an InterruptedIOException might be thrown. A DCAP mover will treat and propagate such an exception as a disk I/O error, thus disabling the pool. This fix reduces false positive disk I/O errors.
Changelog 10.0.10..10.0.11
- 59e5f659ae
- [maven-release-plugin] prepare release 10.0.11
- 8377fb87ea
- pool: don’t treat InterruptedIOException as a disk IO error
- dd52056f98
- pool: fix double decrement of hsm requests
- 06ffe15ff4
- ci: split container image registry and repository
- 536684fde1
- [maven-release-plugin] prepare for next development iteration
Release 10.0.10
bulk
The current release fixed broken command activities
.
Changelog 10.0.9..10.0.10
- beea266384
- [maven-release-plugin] prepare release 10.0.10
- 5e8991675f
- ci: run spotbugs only on master branch
- 97ace7d80c
- bulk: fix broken command
activities
- 54df6ada6a
- [maven-release-plugin] prepare for next development iteration
Release 10.0.9
bulk
When specifying an empty target, bulk proceeded to process the request instead of failing fast. This is now fixed.
gplazma
A previous commit, leading to the last bugfix releases being blacklisted, introduced a regression in the multimap plugin. Where the ‘op’ principal type is used, logins will fail with dCache logging a stacktrace like
java.lang.RuntimeException: Failed to create principal: java.lang.NoSuchMethodException: org.dcache.auth.OAuthProviderPrincipal.<init>(java.lang.String)
.
This is now fixed.
Changelog 10.0.8..10.0.9
- d547d4a58e
- [maven-release-plugin] prepare release 10.0.9
- 7a2bfe0a2f
- gplazma: fix broken commit d74d9568167f4
- d74d956816
- gplazma: multimap fix op regression
- 1aff06fe09
- bulk: check targets for empty strings
- 4678e1b5ae
- [maven-release-plugin] prepare for next development iteration
Release 10.0.8
doc
Better documentation clarifying OIDC provider ID and issue claim.
Changelog 10.0.7..10.0.8
- 0e22cbe693
- [maven-release-plugin] prepare release 10.0.8
- 7ebab3e236
- github: add action for atumatic github-release
- 3e749187ff
- gplazma alise initial version of plugin
- e659623da3
- common: add issuer URI to OAuthProviderPrincipal
- 9468a857c5
- docs: clarify OIDC provider ID and issue claim
- fa8b721504
- [maven-release-plugin] prepare for next development iteration
Release 10.0.7
CI
Improve our CI pipleline.
core
Pool migration : fixed behavior of migration copy
with
-replicas=n
(where n>1
) to behave as expected.
Libs
Keeping libraries up to date.
TAPE-API
Release by relative path works.
Changelog 10.0.6..10.0.7
- 7abb042c1a
- [maven-release-plugin] prepare release 10.0.7
- 422aabb2a9
- WLG TAPE REST API: fix handling of frontend.root in release API
- 9c52354c65
- gitlab: mirror tags
- 350ae2b32b
- libs: update mina-sshd to version 2.13.1
- f06f71cbf0
- Fix issue with infinite replicas when replicas > 1
- 516ba4b573
- github: change mirroring action
- 8e80889481
- ci: use shorter k8s namespace names
- 0eb89d1f0d
- ci: sync CI pipeline with master
- 169454d112
- ci: use desy nims repo for CentOS7
- 88ce98874d
- [maven-release-plugin] prepare for next development iteration
Release 10.0.6
chimera
Now it is possible to n read/write CTA and migrated Enstore files without having to set hsmInstance tag in the full directory tree.
frontend
The current release fixed the issue about failures to invoke WLCG tape API to stage files on files relative to {webdav,frontend}.root
It is now possible to call stage API successully on paths ralative to frontend.root.
Changelog 10.0.5..10.0.6
- 87b2451956
- [maven-release-plugin] prepare release 10.0.6
- 113335cac5
- pom: compile code in the same jvm as maven
- 3f116a5169
- chimera: add CTA HSM StorageInfo extractor
- 935670ec59
- frontend: handle frontend.root variable properly
- 379449ae6e
- [maven-release-plugin] prepare for next development iteration
Release 10.0.5
chimera
Adapt existing uri_encode
function so that the Enstore HSM script does not fail and the files go to tape successfully.
Changelog 10.0.4..10.0.5
- 44699ad4ed
- [maven-release-plugin] prepare release 10.0.5
- 37fc60f08f
- chimera: fix uri_encode to handle special characters
- 1183d69440
- [maven-release-plugin] prepare for next development iteration
Release 10.0.4
ci
Add option to pipeline to control upload options.
Add comments to various pipeline stages.
Changelog 10.0.3..10.0.4
- 6f8c437a72
- [maven-release-plugin] prepare release 10.0.4
- 954828e9d0
- ci: add comments to various pipeline stages
- 61d8a22c30
- ci: add property to control upload options
- 339aa608a0
- [maven-release-plugin] prepare for next development iteration
Release 10.0.3
bulk
Bulk truncates path to 256 characters and this seems to be causing problemsd.
This is now fixed.
pool
With CentOS7, the client sends the last request, such that offset+count == filesize. REHL9 sends the last request so that the count is multiple of 4096, which is legal.
The current release fixed miscalculation of offset on short read.
Changelog 10.0.2..10.0.3
- 323581948e
- [maven-release-plugin] prepare release 10.0.3
- 13d66ba99e
- ci: disable srmcp test
- db51b8baa0
- ci: use python3 for robot test
- a8ef6f5a1b
- bulk: do not truncate target paths
- 3895809e98
- ci: pinpoint postgres helm version
- 3c827ab76c
- pool: fix miscalculation of offset on short read.
- 8fe1fa05ec
- [maven-release-plugin] prepare for next development iteration
Release 10.0.2
chimera
Fixed a bug that made it possible to create a loop on directory move.
Changelog 10.0.1..10.0.2
- 7d72588073
- [maven-release-plugin] prepare release 10.0.2
- 625691108f
- chimera: fix loop creation on directory move
- 15f32f5d69
- ci: drop –ftp-create-dirs option (as we switch to https)
- 865fcb0b03
- [maven-release-plugin] prepare for next development iteration
Release 10.0.1
rpm
Java 11 will be installed as dependency when Java 17 is not already present.
webdav
When running multiple Remote Transfermanager, two transfer that were started simultaneously had the same ID which led to the second transfer becoming orphan after the first one completed. This is fixed now.
Changelog 10.0.0..10.0.1
- 1bbfbbf8b9
- [maven-release-plugin] prepare release 10.0.1
- afd5dd5f6e
- ci: use almalinux9 for rpm install test
- d9f3706793
- docs: update oidc chapter to explain trust anchors
- 3c15025ab8
- rpm: don’t force java–11 it java–17 is installed
- 61e29f8397
- ci: eplocitly specify kubernetes namespace
- 9762947f8a
- ci: fix typo
- b6b8386a19
- webdav: use transfermanager+id to identify TPC transfer
- da81c86739
- [maven-release-plugin] prepare for next development iteration
Release 10.0.0
Billing
The property billing.format.json
was added to write the logs in JSON format instead of plain text if it’s set to true.
Configuration
In the container world, it’s common to use environment variables to pass the
configuration parameters. With release 10.0 dCache support injects of environment
variables into a configuration with special values prefixed with env.
. For example,
to control wan port numbers with environment variables DCACHE_WAN_PORT_MIN and
DCACHE_WAN_PORT_MAX the layout file might have the following configuration:
dcache.net.wan.port.min=${env.DCACHE_WAN_PORT_MIN}
dcache.net.wan.port.max=${env.DCACHE_WAN_PORT_MAX}
> NOTE: the nested variables are not allowed. E.g. ${env.$VAR1-$VAR2}
will not be resolved.
DCAP
Added support for additional Kafka parameters.
Frontend
FTP
Added support for additional Kafka parameters.
gplazma
Update OIDC-related documentation.
Pinmanager
A pin request via admin interface no longer waits for the pin to be established before returning.
Pool
Sites that have tried to run dCache with built-in CEPH decided to go for CephFS (mounted as regular FS) as it provides better performance, scalability and erasure encoding. Thus, built-in CEPH is not recommended for deployments and is removed in version 10.0. Sites that have such a setup should migrate pools before updating the dCache.
Admin command csm info
will display enabled digests for checksum calculation.
Added property pool.path.meta
to specify the location of pools metadata directory, defaults to pool.path
.
Added pool command csm use directio on|off
to enable/disable O_DIRECT to bypass the filesystem cache during checksum calculation.
Prior version 10.0 for xroot and http movers dcache hat two ways to detect and cancel idle connections: Job Timeout Manager (jtm)and idleHandler. Now, the latter one is removed and only jtm-baes one is used. The following pool properties are obsolete:
pool.mover.xrootd.timeout.idle
pool.mover.xrootd.timeout.idle.unit
pool.mover.http.timeout.idle
pool.mover.http.timeout.idle.unit
Starting version 3.0 has dropped the limit on the maximal number of concurrent p2p transfers for a single pool. This is typically not a problem,
however, in some situations, namely multiple transfers from many pools to a single one, during rebalancing, for example, a pool might become
unresponsive due to IO starvation. With version 10.0 the maximal number of concurrent p2p transfers was re-introduced and pp set max active
command is undeprecated. The default value is unlimited
.
Dependencies
Updated external dependencies in preparation for Java17 migration.
Changelog from 9.2.0 to 10.0.0
- 51631eb409
- qos: add flag to enable/disable role based authorization for transitions
- f3ad57bfe2
- common-security: add custom version of ProxyCSRGenerator#generate
- 3dafdddcb5
- [maven-release-plugin] prepare branch @{releaseLabel}
- 4bc0305ae7
- config: group logging configs in system-test
- 83ced04067
- billing: Add option for json format
- 8f3b984fab
- nfs: calculate desired memory fraction for correct memory allocation
- dce4087df6
- info: use DGA refresh rate as message timeout
- bdb8da5cd3
- Revert “chimera: make list method to support virtual directory listing”
- 824275ebe0
- pool: make pool_read only if fs is ro
- 1fbd7ae388
- chimera: make list method to support virtual directory listing
- 030d5ce33b
- xroot: switch to xrootd4j–4.6.0
- c357ac60d0
- libs: update apache-curator version to 5.6.0
- d78a4805ed
- tape-api: map request supplied paths based on frontend.root
- 6b9e1c6b9f
- srr: fix “empty path annotation” warning
- 4bace63646
- notifications: Add BillingMessageSerializerVisitor
- 00ba8c7bff
- bulk: fix divergent command params format strings
- b5fc763f43
- core: remove unintended changes to listPoolGroupXml
- 48e5cafb68
- cells: fix error reporting on tunnel disconnect
- a244b9f10a
- Remove dependency on GNU patch.
- 55f5091d0f
- ci: fix job dependency
- b712a763d4
- ci: use user friendly job names
- f5cf4ec6c9
- ci: initialize CVMFS only once
- 4044301494
- Fix revert commit ce3927d7
- ce3927d71a
- Revert “poolmanager: delete property for switchingon/off caching for psu”
- cc28966ee0
- build(deps): bump org.postgresql:postgresql from 42.3.8 to 42.3.9
- ba2a5bdf7d
- build(deps): bump org.apache.commons:commons-compress
- 0764862633
- pom: sync jersey-spring with spring (5) version
- 14d1acb020
- ci: no special tags for non kubernetes jobs
- d60eb0e135
- ci: retry on transient errors
- e07ee6d11c
- ci: enable long running tests
- 3b7789d157
- ci: use minimal almalinux–9 to upload packages
- 18a66bbb7d
- ci,pom: add gitlab-code-quality-plugin, generate spotbugs report
- 78221848d3
- pom: use latest maven-surefireand modernizer plugins
- 4e7ce95c09
- ci: use dtzar helm-kubectl image
- 24e67db9b4
- ci: use java17 image for deb packages
- be2028c0f8
- pom: use Jersey–2.41
- af5ea1c266
- pom: update datanucleus (and deps) to 6.0 release
- c0d5c07aa1
- Minor .gitignore tweaks based on IntelliJ IDEA experience on MacOS
- 12a2839be3
- dcache-core: fix psu logging
- 426d7c4f7a
- pool: mover grizzly IO buffer initialization into NfsTransferService
- 88881b9c80
- webdav: fix slow listing
- a8e5fec6d0
- webdav: fix link header
- 0b330a038c
- chimera: virdir, return only the file name
- fea615f5ab
- pool: re-introduce limit on number of concurrent p2p transfers
- 0bbf8f5d85
- pool: remove IdleHandler from http and xroot movers
- ffbbf5cad0
- gplazma: fix NPE if gPlazma is rejecting all logins
- adb184f757
- chimera: add extra unit-tests for JdbcFs
- 95db4a770d
- systemtest: support running clients in a container
- 7793086288
- build: fix code-coverage reports
- 9074fa4aca
- pool: use mover’s UUID to track xroot client reconnects
- fd11f9381c
- chimera: improve unit tests for jdbcfs rename
- 046f4a0bdb
- ci: dont’t pull build artefact for kubernetes-based jobs
- b79b9a7a02
- Fix unit test for commit 6b47354
- 6b473540f4
- gplazma: configured banfile plugin should ignore non-existent ban file
- 12a76a39dd
- chimera: Revert “chimera: update FsSqlDriver#inodeOf to throw exception if file not found”
- d63e2d9b93
- system-test: increase direct memory
- 2d6fc8c575
- xrootd: fix xrootd.root regression
- 88b73cd5af
- dependencies: update modernizer plugin to be compatible with maven 3.9
- b13f6d7bfd
- webdav: wait for upload to complete
- 862963ec7e
- pool:make readonly pool READONLY Motivation
- 523738e73c
- add openssl installation hint
- 4e61c808fd
- book: add transfermanager HA description
- 59ab8ffa6b
- cleaner-disk: prevent ConcurrentModificationException in cleaning run
- a5d6454498
- dcache-xroot: fix effective root when subject is nobody
- 4facd1dcd3
- dcache-xroot: check that descriptor is not null before calling close – fix
- b00a5dc468
- build(deps): bump ch.qos.logback:logback-classic
- 06f6c00979
- build(deps): bump ch.qos.logback:logback-core from 1.2.10 to 1.2.13
- 73e683a66f
- dcache-xroot: check that descriptor is not null before calling close
- bb19a4850f
- dcache-core: fix admin command completion
- ffc5effbb9
- qos: comment out property description
- 96893bcbac
- ci: add exta helm ops to extend timeout and simplify retries
- 583330bd50
- pom: Update mongodb-driver
- 3701b1329f
- dcache-frontend: check for defined arguments with namespace qos
- 05d9c4cf29
- webdav: do not log stacktrace if non-existing
well-known
is requested - 87f196447b
- pool: add option to use direct-io while checksum calculation
- 0f098e8a93
- dcache-bulk,dcache-qos: repair mass cancellation issues
- 5da0a8d82c
- dcache-bulk: allow reset to skip terminated targets
- 2c55595689
- dcache-frontend: add hooks and context handler for dcache-view-admin
- 94e04e8193
- pom.xml: update dcache-view to 2.1.0
- 9a8985598d
- dcache-qos: cache modify requests until processed by executor
- 3a6fae5f39
- info: fix issue 7327 adding nested pgGroup
- ddb17205c5
- pool: improve err msg on mover create if protocol is not supported
- 6412d58b5a
- dcache-common: repaired unit test for QoSPolicy parsing (duration string error).
- 82922f20a0
- Revert “dcache-frontend,common: check parsing of Duration in STAGE”
- 6ffe2cd10e
- dcache-frontend,common: check parsing of Duration in STAGE
- 013db14c6b
- dcache-bulk: fix handling of uncaught exceptions
- bd1abd31fc
- pool: make metadata directory location configurable
- afb15bb1db
- ci: generate release-notes template
- f49389bb91
- webdav: fix infinite recursion in Requests.stripToPath
- 830a8a1469
- webdav: fix parsing of urls with two slashes in the path
- 8aa9e6af41
- chimera: fix rollback of 9.1 to 9.0 db schema
- 6bd65801c8
- gplazma: enhance help of
explain login
command - 7ca86f06be
- Revert “ci: use rancher to create k8s namespace”
- ed2d9a2307
- docker: use exec for start java process
- 63a5d1d103
- Fix ce3cfe3: remove superfluous brace
- ce3cfe33a1
- dcache: make admin pin command asynchronous
- aec714e653
- oidc: fix remove invalid testcase
- 83d1e77eed
- oidc: reject storage scopes without path
- c71edfb1f9
- webdav: fix formatting broken by 2ddd74a780
- 2ddd74a780
- door,pool: handle multiple possible KafkaExceptions
- 714d0f617a
- libs: update asm to version 9.5 with java–17 support
- dc6db510c8
- ci: use java17 builds for rpm and tar
- 88a8028c06
- pom: add exta java option to run powermock test under java17
- 81c7e944c7
- ci: remove extra ‘*’ in rpm-sign command
- 6b49307ce4
- fix issue #7326
- 3104def644
- build(deps): bump org.json:json from 20230227 to 20231013
- f7e19137e2
- libs: fix unexpected transient dependencies of rados4j
- 2f268fb1d7
- pool: drop RDB based CEPH support
- 2076857a4c
- dcache-bulk: use rate limiter to throttle semaphore release
- 0476a3a8f8
- config: fix unit test in ci
- 8c65ee7636
- config: add support for environment variable injection
- ad19184647
- dcache-qos: correction to the threshold warning
- 00313591b2
- dcache-qos: set default task thread pool sizes all to max concurrent running
- 49ab9ed9dd
- dcache-bulk: fix bug in archiver deletion query
- adaacd864a
- dcache-bulk: fix thread executor injection
- 32c2127d1b
- ci: use pynfs:0.5, enable LOCK24 test
- 9698289eb3
- ci: add webdav test
- 523654c774
- doc: add QoS policy and role documentation to the dCache Book
- e779d3802a
- build(deps): bump org.apache.zookeeper:zookeeper from 3.7.1 to 3.7.2
- 42cc3d4ad9
- pool: cms info should display configured checkums
- c65c0071e8
- wevdav: fix redirect path
- ce9734a123
- dcache-bulk: fix cancellation issues
- 76fde730c2
- dcache-bulk: give directory listing a separate executor
- 3cbe3f486f
- nfs: prepare code for removal of redundant stateid4 class
- 3d0a0bd56d
- book: fix typo
- e957031622
- book: split srm chapter into srm, srmmanager, spacemanager
- 5172554462
- ci: use rancher to create k8s namespace
- f26fe0809d
- ci: fix typos
- e80d893e69
- book: pinmananger, describe new default unpins/time
- 533c363872
- dcache-core: increase default unpins per time
- c370ba25f7
- Support additional Kafka parameters for ftp and dcap.
- 60dd0baca4
- dcache-bulk: add count and clear to archive admin commands
- 1df7a1800a
- dcache-bulk: cancel activity future on target cancel
- 0b4140b454
- dcache-bulk: refine container executor model
- 07492e6e94
- docs: userguide fix example domains names to follow RFC 2606
- 5c9d04fff3
- docs: UserGuide fix angle-braket in metalink
- 4c731ab8cf
- docs: add description of webdav’s metalink support
- 094c1714c1
- gplazma: oidc update explicit AuthZ parsing
- 49661e6b57
- gplazma: oidc increase cache duration for OP public key material
- 8a5c358af4
- dcache-bulk: container rewrite to optimize threading
- 3d23228988
- dcache-bulk: add admin command and query to reset all requests with failed targets
- a95a9edb5e
- dcache-bulk: add convenience admin command for state counts
- 8b3ff0d2b4
- dcache-qos: fix scanner operation completion logic
- cf8e57a687
- ci: by-pass docker.io for bitnami charts
- 7873035615
- dcache-bulk: implement HA
- bf70db77c5
- dcache-cli: convert IllegalArgumentException to CommandException on call()
- f9fbb002e1
- dcache-bulk: do not PIN or STAGE files with AL ONLINE
- 271b32a07d
- dcache-bulk: only set request status to QUEUED when permissions and targets are all inserted
- b37a79a5b8
- [maven-release-plugin] prepare for next development iteration
- 43a3e4482b
- build(deps): bump org.eclipse.jetty:jetty-servlets