Highlights

  • dCache now supports Java 17 as its platform

Incompatibilities

  • the cleaner service, originally a single cell, now consists of two parts: one cell for disk cleaning (cleaner-disk), one for hsm cleaning (cleaner-hsm)
  • DCAP and NFS doors will fail the request if file’s storage unit is not configured in PoolManager
  • linklocal and localhost interfaces are not published by doors and pools
  • DCAP movers always start in passive mode
  • removed experimental message encoding format
  • removed default HSM operation timeout
  • With version 9.0.24+, empty and non-existent banfiles will be treated the same

Release 9.0.26

ci

CI pipeline improvements.

dependencies

Update build dependency to be compatible with modern tools.

Changelog 9.0.25..9.0.26

d238308384
[maven-release-plugin] prepare release 9.0.26
56acff9750
dependencies: update modernizer plugin to be compatible with maven 3.9
93b95e3a9f
ci: add property to control upload options
6ebc3eea28
[maven-release-plugin] prepare for next development iteration

Release 9.0.25

chimera

Fixed a bug that made it possible to create a loop on directory move.

Changelog 9.0.24..9.0.25

4c73ad6598
[maven-release-plugin] prepare release 9.0.25
383de0d3d7
chimera: fix loop creation on directory move
32c778ea93
ci: drop –ftp-create-dirs option (as we switch to https)
d213a956a4
[maven-release-plugin] prepare for next development iteration

Release 9.0.24

gplazma

Previously, a missing banfile would result in every successive login attempt failing with an NPE. Now, even if configured, an empty or inexistent banfile will be ignored and logins should succeed. WARNING: This changes the banfile plugin behaviour! Empty and non-existent banfiles will be treated the same.

Changelog 9.0.23..9.0.24

594a5ecedf
[maven-release-plugin] prepare release 9.0.24
f2a60a3e70
Fix unit test for commit 6b47354
b7bd246cff
gplazma: configured banfile plugin should ignore non-existent ban file
1accfd13f3
[maven-release-plugin] prepare for next development iteration

Release 9.0.23

cleaner-disk

No more occasional ConcurrentHashMap exceptions in cleaner-disk runs due to concurrent pool status changes.

xroot

Regression eliminated, previous behavior with anonymous read restored.

Fixes previous fix for xroot descriptor: NPE risk mitigated.

Changelog 9.0.22..9.0.23

5b9380c05f
[maven-release-plugin] prepare release 9.0.23
befcb48a10
dcache-xroot: check that descriptor is not null before calling close – fix
68dbc75d77
dcache-xroot: check that descriptor is not null before calling close
d412c22478
cleaner-disk: prevent ConcurrentModificationException in cleaning run
a70934f17f
[maven-release-plugin] prepare for next development iteration

Release 9.0.22

dependencies

The MongoDB driver was updated. This might lead to different log messages.

Changelog 9.0.21..9.0.22

5ee504f4d6
[maven-release-plugin] prepare release 9.0.22
64b2b72b61
pom: Update mongodb-driver
c036f17312
[maven-release-plugin] prepare for next development iteration

Release 9.0.21

bulk

The current releae fixed handling of uncaught exception during perform.

Changelog 9.0.20..9.0.21

9df360ef99
[maven-release-plugin] prepare release 9.0.21
9d3295b813
dcache-frontend,common: check parsing of Duration in STAGE
2e559affff
dcache-bulk: fix handling of uncaught exception during perform
46b687f78a
[maven-release-plugin] prepare for next development iteration

Release 9.0.20

webdav

The urls similar to this https://door.domain.foo:1234//pnfs/domain.foo/path/to/file were strip into /domain.foo/path/to/file.

The current release fixed the parsing of urls with two slashes in the path.

Changelog 9.0.19..9.0.20

73cb1abb7c
[maven-release-plugin] prepare release 9.0.20
6bc9c4f656
webdav: fix infinite recursion in Requests.stripToPath
b94b640a97
webdav: fix parsing of urls with two slashes in the path
9d6944931a
[maven-release-plugin] prepare for next development iteration

Release 9.0.19

door

Exception handling is improuved for Kafka.

oidc

Storgae scopes without path will be rejected now.

Changelog 9.0.18..9.0.19

3fe7dde81a
[maven-release-plugin] prepare release 9.0.19
cf53c01c2a
oidc: fix remove invalid testcase
63aca37fa0
oidc: reject storage scopes without path
e2e61d73c1
door,pool: handle multiple possible KafkaExceptions
40bf212e34
[maven-release-plugin] prepare for next development iteration

Release 9.0.18

webdav

This patch fixes a bug that led to wrong paths on redirects.

Changelog 9.0.17..9.0.18

e6ff5edbbf
[maven-release-plugin] prepare release 9.0.18
915bfaf407
dcache-qos: correction to the threshold warning
25bf9b2ae8
wevdav: fix redirect path
1b3f229da3
[maven-release-plugin] prepare for next development iteration

Release 9.0.17

bulk

No more NPE/stack trace in the bulk PIN activity.

Online files should not accumulate useless user pins anymore.

cli

No more false stack trace/bug output on the admin shell.

gplazma

dCache will now accept non-targeted explicit AuthZ statements in the scope claim (e.g., storage.read). dCache will consider tokens containing compute explicit AuthZ statements but without any storage explicit AuthZ statements as tokens with explicit authorisation; the lack of any storage explicit authorisation statements will result in all requests with that token being denied.

dCache should continue to accept JWTs for longer, if there is a problem with the OP.

pool

No more logged stack traces due to IllegalStateException: Attribute is not defined: SIZE errors.

Changelog 9.0.16..9.0.17

9b8aa4d839
[maven-release-plugin] prepare release 9.0.17
427cbbf201
dcache-bulk: do not PIN or STAGE files with AL ONLINE
bfb8d25092
dcache-bulk: guard against erroneous argument names
bea4ad38ea
pool: on failed upload set file size to zero for space reporting
13c53cdd6d
dcache-cli: convert IllegalArgumentException to CommandException on call()
b124873c19
gplazma: oidc increase cache duration for OP public key material
f543d709f9
gplazma: oidc update explicit AuthZ parsing
c2c4327779
[maven-release-plugin] prepare for next development iteration

Release 9.0.16

packages

A problem was fixed that prevented system-test to run on MacOS.

webdav

This patch reverts the efficiency improvement of the last patch as it may lead to failing requests if set to PERFORMANCE.

Changelog 9.0.15..9.0.16

c1e38ba5be
[maven-release-plugin] prepare release 9.0.16
bf2140b95c
packages: fix system test populate sed expression
fc344605b0
dcache-webdav: revert improve efficiency of directory listing (14085/14088)
a6ec880729
doc: clean up the multiple protocol configuration explanation for xroot
205c735df3
ci: pin all jobs to dcache-dev runners (temporary)
fcf6c836b9
[maven-release-plugin] prepare for next development iteration

Release 9.0.15

logback

Redundant replications of unused log files are removed and the qos activity is now recorded as well.

qos

A bug was fixed that let to the message queue being blocked.

webdav

This patch speeds up listing of directories through webdav. All attributes are no longer supported/returned from PROPFIND, unless you set the following property on the WebDav door:

#  --- Default PROPFIND properties
#
#  The PROPFIND request allows a client to discover information
#  (properties) of dCache content.  When making a PROPFIND request,
#  the client normally indicates which properties are of interest.  If
#  not specified then the WebDAV server (dCache) is free to return a
#  default set of information.
#
#  Certain clients make PROPFIND requests without specifying
#  the desired set of properties, triggering a default set of
#  properties.  There are two problems with this: first, the
#  clients may react badly if information is missing from this default;
#  second, some of the required properties may have an adverse
#  performance impact for dCache.
#
#  This property controls whether to support client-side requests for
#  properties beyond a minimal set or not.
#
#      PERFORMANCE -- always return only a minimal set of information that does
#                     not incur any additional overhead. (These are basically
#                     the same as what a POSIX list request would proviode).
#
#      CLIENT_COMPATIBLE -- return the set of information required by
#                     the clients.  If the client does not specify the properties,
#                     it will get a maximal default set.
#
#  Because the performance impact of the latter under the most common usage scenarios
#  could be considerable, the default is set to PERFORMANCE.
#
(one-of?PERFORMANCE|CLIENT_COMPATIBLE)webdav.default-propfind-properties = PERFORMANCE

Changelog 9.0.14..9.0.15

91a68c92f7
[maven-release-plugin] prepare release 9.0.15
b3c04bc5cd
dcache-webdav: fix incorrect parameter value given to DcacheDirectoryResource constructor
c95ffc5a18
dcache-webdav: improve efficiency of directory listing
190474bf02
dcache-qos,dcache-bulk: allow qos update to call engine asynchronously
029a753ca3
skel,logback: make the .resilience and .qos log files singletons
ca24625745
[maven-release-plugin] prepare for next development iteration

Release 9.0.14

bulk

Each time a bulk request is started, the container is given a new semaphore based on the number of permits defined for the activity in question. This patch sets the default per-container permit values to something more reasonable.

Changelog 9.0.13..9.0.14

e82f3e7f67
[maven-release-plugin] prepare release 9.0.14
3c2fed0587
dcache-bulk: adjust semaphore permits to something more reasonable
fe7849fa7b
[maven-release-plugin] prepare for next development iteration

Release 9.0.13

dcache-bulk

Changes were recently made to hold statistical counts of requests and targets in memory for the sake of efficiency. However, running target counts are still problematic. This is especially true if a request is cancelled.

This is now fixed and the running count, which can be confusing and inaccurate, is not longer displayed.

RT 10496: Odd errors in Bulk logs for some requests in 9.1.0

This ticket uncovered a bug in the abort procedure.

The stored targets were not cancelled on abort. this is now fixed.

dcache-xroot

Improve efficiency of stat list (ls -l) has been introduced.

ls -l returns in approximately the same amount of time as a non-stat listing ls.

xrootd4j

Upgraded to 4.5.8 which improves handling/logging of Ssl exceptions.

Changelog 9.0.12..9.0.13

b54eec8f54
[maven-release-plugin] prepare release 9.0.13
b503a1c9f9
docs: fix external link to nfsmapid and DNS TXT Records docs
8c377d64b1
dcache-bulk: cancel all stored targets on abort
bfa9569455
dcache-bulk: remove in memory running state counts
b17e6b15a7
common: modify the way the flag on isRestricted works
ff1889f226
dcache-xroot: improve efficiency of stat list (ls -l)
162cba0996
pom.xml: upgrade to xrootd4j 4.5.8
c255444de4
[maven-release-plugin] prepare for next development iteration

Release 9.0.12

pnfsmanager

List scheduling behavior is now optional.

pnfsmanager.enable.parallel-listing property variable allows to restore previous behavior - listing requests are dispatched to a single queue that is prrocessed by multiple parallel workers.

dCache admin can change behavior of listing scheduler.

Changelog 9.0.11..9.0.12

1c4f133766
[maven-release-plugin] prepare release 9.0.12
e8163995ba
pnfsmanager: make list scheduling behavior optional (selectable)
c0f18dde41
[maven-release-plugin] prepare for next development iteration

Release 9.0.11

gplazma

gPlazma incompatibiliti detween version before and after 8.2.13 has been detected.

This is now fixed.

pool

If http client issues two GET requests then zero-copy enabled http mover will try to remove the IdleStateHandler twice and fail with NoSuchElementException.

This is now fixed.

qos

A bug in storing and retrieving tried value for verify operation has been fixed.

xroot

Modify mkdir to ignore dir exists on make parent option. The result conforms to vanilla xroot server.

Changelog 9.0.10..9.0.11

6668baca28
[maven-release-plugin] prepare release 9.0.11
1ace470660
dcache-gplazma: add serialVersionUID to MultiTargetedRestriction
e2e56d5120
pool: handle double remove of IdleStateHandler
ce81176547
add desc to OIDC properties
5fbfe12412
dcache-qos: fix bug in storing and retrieving ‘tried’ value for verify operation
7abb11cea5
dcache-xroot: modify mkdir to ignore dir exists on make parent option
167b1040f1
[maven-release-plugin] prepare for next development iteration

Release 9.0.10

vehicles

This patch fixes a bug where an Exception was thrown while handling relative paths in resolve symlink messages.

Changelog 9.0.9..9.0.10

a4f47e76f4
[maven-release-plugin] prepare release 9.0.10
ee96a529f0
dcache-vehicles: fix NPE in resolve symlink message
2db0433ac9
[maven-release-plugin] prepare for next development iteration

Release 9.0.9

frontend

If the user root contains a symlink, or the target path contains a symlink pertaining to the user root, some REST calls to /api/v1/namespace and /api/v1/id failed. This is fixed now.

All of the REST GET methods now allow anonymous users to get the data if the global property allows it. In the case of NONE, 401 not authorized is returned at login.

nfs

A race condition was fixed that let to sending a READ request to a pool when the mover was alreay stopped.

qos

The behaviour was changed to not scan all pools on startup or periodic rescanning of all pools outside the periodic window.

A bug was fixed that prevented the bulk request to complete.

The error reporting of aborted verification was improved to get a better understanding, what went wrong.

xrootd

A missing vomsdir is now throwing an exception instead of a stacktrace.

xrootd4j

Update xrootd4j to a new version, that fixes a byte buffer leak in ZTNCredentialUtils.

Changelog 9.0.8..9.0.9

10b667f296
[maven-release-plugin] prepare release 9.0.9
2a5f1b4696
nfs: fix race condition of LAYOUTRETURN and LAYOUTGET
c1fa09759c
xrootd: throw FileNotFoundException if vomsdir doesn’t exist
99a42835c6
dcache-qos: fix adjuster task cloning to reset state
3d42d5dda2
dcache-qos: improve exception reported on aborted verification
281893fb3e
ci: add gitlab-ci.yml
d3ea78616f
dcache-frontend: add symlink resolution to /api/v1/id GET and /api/v1/namespace
4175d9b8d3
dcache-frontend: remove authz checks in Quota GET methods
722dd95a11
dcache-qos: remove trigger to rescan pools on tag change
ff448bd04f
Revert “dcache-qos: remove trigger to rescan pools on tag change”
50de16ccdb
pom.xml: bump xrootd4j to next version (4.5.7, 4.4.8, 4.3.9, 4.2.13)
3623752212
dcache-qos: remove trigger to rescan pools on tag change
2c528cf1fd
[maven-release-plugin] prepare for next development iteration

Release 9.0.8

bulk

RuntimeExceptions should not cause the request ls to fail globally.

other

Update our citations file.

pnfs

A modest speed-up of directory listing by not always resolving symlinks when checking restrictions.

qos

Two small incompatibilities with subject and attributes are fixed.

Improvements in triggering QoS operation queues, which reduces the potential for queue starvation.

Changelog 9.0.7..9.0.8

19b96b7292
[maven-release-plugin] prepare release 9.0.8
d17307e90d
dcache-qos(verifier,engine): fix incompatibility issues with Subject and attributes
ea451f6ab9
dcache-bulk: catch Exception from getSubject()
4c2be1a800
dcache-qos: repair faulty queue refresh algorithm in verifier
bb03f46404
dcache,pnfs: skip symlink resolution when checking restrictions on directory children during listing
4279f70f7d
Update the CITATION.cff version field and date-released field for dCache 9.0.
083c99897c
[maven-release-plugin] prepare for next development iteration

Release 9.0.7

webdav

Fixes the webdav dropdown menu layout so it stays in the page.

Changelog 9.0.6..9.0.7

894a847d71
[maven-release-plugin] prepare release 9.0.7
06b0996325
webdav: fix dropdown menu layout
a20c766669
[maven-release-plugin] prepare for next development iteration

Release 9.0.6

bulk

Now REST API (Frontend) distinguishes Not Found from Invalid request.

Double counting of RUNNING targets has been fixed.

dcache

netty library has been updated to 4.1.92 with minor bugfix release.

frontend

SWAGGER comments for bulk offset on POST and GET has been updated.

metadata

No user- or admin observable changes.

pnfsmanager

Multiple batch jobs executing listings of the same directories especially if directories contain many entries quickly exhaust active threads, each thread executing the same listing, denying listing to all other customers and creating significant load on namespace server (and DB). Clients time out and retry creating even more load on dCache.

This is now fixed and lists of large directories no longer hog PnfsManage denying service to all other list calls. Additionally observed significnat reduction of CPU and IO load on chimera DB host.

webdav

The current release fixed visual element regression after bootstrap 5 upgrade

Changelog 9.0.5..9.0.6

a4019359d0
[maven-release-plugin] prepare release 9.0.6
ebf9eeb234
webdav: fix regression of visual elements with bootstrap 5
37a1f2b958
webdav: fix visual element regression after bootstrap 5 upgrade
09527224f8
dcache-frontend: update SWAGGER comments for bulk offset on POST and GET
fd41399e3e
metadata: add (some) ORCIDs to CITATION.cff
7698265da1
dcache-bulk: increment/decrement in-memory counts exclusively in the target object
8e946cd47d
pnfsmanager: process listings of the same path sequentially, implement list message folding
b31e6e8c70
dcache-bulk: fix double counting of RUNNING targets
a86e1fde6a
dcache-bulk,dcache-frontend: distinguish “Not Found” from “Invalid” request
3c82a23def
libs: use netty–4.1.92
042c5cb491
[maven-release-plugin] prepare for next development iteration

Release 9.0.6

### #

Changelog 9.0.5..9.0.6

a4019359d0
[maven-release-plugin] prepare release 9.0.6
ebf9eeb234
webdav: fix regression of visual elements with bootstrap 5
37a1f2b958
webdav: fix visual element regression after bootstrap 5 upgrade
09527224f8
dcache-frontend: update SWAGGER comments for bulk offset on POST and GET
fd41399e3e
metadata: add (some) ORCIDs to CITATION.cff
7698265da1
dcache-bulk: increment/decrement in-memory counts exclusively in the target object
8e946cd47d
pnfsmanager: process listings of the same path sequentially, implement list message folding
b31e6e8c70
dcache-bulk: fix double counting of RUNNING targets
a86e1fde6a
dcache-bulk,dcache-frontend: distinguish “Not Found” from “Invalid” request
3c82a23def
libs: use netty–4.1.92
042c5cb491
[maven-release-plugin] prepare for next development iteration

Release 9.0.5

dcache

Symlinks on both constraints and targets are permitted; thus macaroons will also work.

The support for both absolute and relative paths does not fail if there are symlinks affecting the path prefix.

gplazma

If there is a login failure involving a OIDC JWT then potentially important information is now extracted from the JWT and included in the logged login failure report.

POSC works with tokens when user root is ‘/’.

kafka

Change the LoggingProducerListener so that when a TimeoutException is caught, the error message will indicate that there is a connection issue or the broker is down.

nfs

When on LAYOUTGET door returned an error back to the client, then, as the operation failed, the client will not issue LAYOUT_RETURN. Thus such transfers would stay in the door. This is now fixed.

Changelog 9.0.4..9.0.5

fc918d7eb9
[maven-release-plugin] prepare release 9.0.5
97e69442f5
kafka:fix issue 7123 Kafka events fail to post
d096a5b034
gplazma: print selected JWT claims for failed logins
bcf71f0c34
nfs: remove failed to start read transfers
b8d4f8e02f
Motivation:
ee41b788b8
dcache-core,dcache-xroot,dcache-bulk,dcache-frontend: resolve path prefixes and paths for symlinks
33307a22ee
commons,dcache: alternative symlinks resolution on restrictions
0a28438ee7
[maven-release-plugin] prepare for next development iteration

Release 9.0.4

libs

Library update due to CVE–2023–20863

systemtest

The populate script now generates a valid dCache configuration if oidc-agent is installed and at least one configuration has been loaded.

Changelog 9.0.3..9.0.4

6f65cd1500
[maven-release-plugin] prepare release 9.0.4
2e85b6357a
systemtest: fix support for oidc testing
9c8bc59f4a
libs: use spring–5.3.27
0b1d9891b3
[maven-release-plugin] prepare for next development iteration

Release 9.0.3

bulk

QoS properties have been set to default to WLCG values. Also decreased the number of targets in a shallow request and increased number of threads available for target activities to 200.

Correct paths for recursively found targets are returned; i.e., absolute if the user submitted absolute paths, relative is the user used relative paths on the initial targets.

qos

Proper handling of the QoSModify requests: propagate the subject downstream to the actual adjustment tasks.

webdav

Correct behaviour on redirect with prefix.

Changelog 9.0.2..9.0.3

ced92c441a
[maven-release-plugin] prepare release 9.0.3
dd1d3f89d6
dcache-bulk: reset policy defaults
7fe8b32e95
dcache-bulk: fix prefix-stripping logic when returning request info
21c8870c4d
webdav: preserve request URL in protocol info
0db4aed2f2
dcache-qos: propagate subject to QoS Adjuster
2c95f8bc00
[maven-release-plugin] prepare for next development iteration

Release 9.0.2

chimera

Fixed a bug that produced SQL errors.

libs

Update nfs4j to 0.24.3 that add fixes in open-stateid handling that aims to fix issues with concurrent opens of a single file by a same client.

nfs

Improve door behaviour.

Changelog 9.0.1..9.0.2

3797126061
[maven-release-plugin] prepare release 9.0.2
9bddef2438
bulk: let bulk propagate Subject for pin/stage
f8409ad409
libs: update nfs4j to 0.24.3
da14fc0347
Revocation entries are defunct since gplazma–1 is discontinued
a0001bb45c
nfs: bind layout disposal to layout state id
c3ee106351
[maven-release-plugin] prepare for next development iteration
2be8f4a505
chimera: fix bug in resolve_path stored procedure

Release 9.0.1

frontend

Better error message reporting.

The RESTful interface supports paths relative to the user root as well as absolute paths for requests to the bulk service.

Note: a separate solution will be necessary for the one-off namespace resource operations.

kafka

Ensure, that exception is converted to a string. Better logging.

migration

When broken files were discovered during migration the migration module was going into sleeping state. This is now fixed.

pnfsmanger

The current release fixed migration move/copy going into SLEEP mode when the file on the pool has been deleted.

pool

Now the reduce possibility of stacked movers in CANCELED state is ensured.

webdav

The observed NPE should disappear.

xrootd4j

update to next xrootd4j version (4.5.6, 4.4.7, 4.3.8, 4.2.12)

Changelog 9.0.0..9.0.1

c4f3fd08b3
[maven-release-plugin] prepare release 9.0.1
7a240d5236
pnfsmanager: fix migration move/copy going into SLEEP mode when the file on the pool has been deleted
2b05e2810d
migration: fixing sleeping state for broken files
ccafa110e1
nfs: ensure that mover always associated with an open-stateid
c3bfcbd0be
pool: ensure that nfs mover completion takes place
0ac0fdc786
dcache-frontend: support VO- or user-root relative paths in requests
04907035c6
dcache-frontend: improve Swagger annotations for release and archiveinfo
8adf780e4c
webdav: return empty list instead of null as an empty collection
92bf848b0b
dcache-frontend: improve error message for empty string or non-JSON payload
cb025ee250
pom.xml: update to next xrootd4j version (4.5.6, 4.4.7, 4.3.8, 4.2.12)
4e8ef37ba8
kafka: ensure that errors are logged properly
2afdc7fbe7
[maven-release-plugin] prepare for next development iteration

Release 9.0.0

Admin

Paging capability has been added to certain admin commands. This means that large lists of results that exceed a certain limit are returned in chunks <= to the limit, and the user is prompted as to whether more should be displayed [Y/N]. These commands (see, for instance, bulk’s target ls), when invoked via a non-terminal script, will continue automatically to stream chunks back until the end of the list.

bulk

  • [41a9e807d7] dcache-bulk: fix missing pnfsid on cancel of PIN/STAGE

The move to a revised (version 2) bulk service was made with 8.2, but since then there have been a number of optimizations and improvements some, but not all, of which were backported to 8.2.

NOTE: Most importantly, there was a significant refactoring of the database tables to provide for more efficient use of space (via normalization). To preserve existing data, several somewhat costly queries are thus run on restart after upgrade from 8.2 to 9.0. As a benchmark for the amount of downtime necessary for the upgrade, figure approximately one hour per every 20M rows in the request_target table. It is always possible, in order to avoid overly long downtime delays, is to eliminate all requests which have completed; in the postgres interpreter:

DELETE FROM bulk_requests WHERE status='COMPLETED';

An important bug fix which was included in 8.2 was to change the behavior of the initial submit to store explicit targets (not those found recursively, but specified in the target list of the request) immediately/synchronously. This was needed to support WLCG requirements.

We have dropped the ‘delay’ option for request clearing for generic bulk requests. The admin command for STAGE activities now also automatically sets prestore to true, as with the request submitted through the REST API.

The admin commands for request ls and target ls now use the new paging capability.

LINK types are now handled correctly when the paths are refetched from prestored targets.

The bulk request target list JSON was fixed to be backward compatible with 7.2 (it now accepts both a string and a JSON array).

The target-by-target error reporting in the bulk request GET response JSON object is now less confusing.

Cells

The expereimantal message encoding format is removed. As a result, the property dcache.broker.channel.msg-payload-serializer is immutable and set to value standard.

Cleaner

The cleaner service, originally a single cell, now consists of two parts: one cell for disk cleaning (cleaner-disk), one for hsm cleaning (cleaner-hsm). They can be deployed as desired, be assigned different resources and each run in HA mode. This will hopefully improve performance issues and help admins configure and understand cleaner behaviour.

Be aware that the property names have changed their prefixes from cleaner.<something> to cleaner-disk.<something> and cleaner-hsm.<something>, while some admin commands have lost the “hsm” String from their name. Note: As not all previously existing parameters were used to control the behaviour of both the disk and hsm parts of the old, combined cleaner cell, please check which parameters are carried over to cleaner-hsm and cleaner-disk, respectively.

Example setup:

[dCacheDomain]
[dCacheDomain/cleaner-disk]
cleaner-disk.cell.name=cleaner-disk1

[dCacheDomain/cleaner-disk]
cleaner-disk.cell.name=cleaner-disk2

[dCacheDomain/cleaner-hsm]

Also, an admin command was added to the hsm-cleaner cell that allows forgetting a tape-resident pnfsid, meaning removing any corresponding delete target entries from the cleaner’s trash table database.

Core

To reduce network-configuration-based issues, starting with version 9.0, dCache does not publish IPv6 link-local or locahost IP addresses in the login broker. However, the connections for those interfaces are acepted.

DCAP

The DCAP door will fail the request if a file’s storage unit is not configured at the PoolManager, as all other doors already do.

Since dCache 1.7.0 the DCAP door was supporting two modes of establishing data connection  between a client and a pool. Either a pool is connected to the client, or the client initiates the connection to a pool. The latter is a firewall-friendly behavior, but should be requested by the client. Starting with dCache 9.0, the DCAP movers will always start in a passive mode, independent of client preference.

Frontend

There have been several bug fixes to the frontend bulk API, particularly to make it backward compatible with 7.2 and to fix regressions in the new /api/v1/stage resource. A bug blocking STAGE submissions where fileLifetime was not specified has been fixed.

A bug with mv (rename) and relative paths in the /api/v1/namespace resource was corrected.

Support was added for .well-known/security.txt, exposed both on the frontend and webdav ports.

Support for maintaining the pre–8.2 implementation for /api/v1/namespace qos transitions (via a dcache property) was added; also, if this is not set to false and the QosEngine service is not running, the namespace qos transition will now fail fast.

The default type parameter to ``/api/v1/pools/{pool}/nearline/queues (=all types) now works.

It is now possible to filter and sort the results of the /api/v1/restores by path, owner and group. Since this, as with other RESTful commands, is implemented as a string ‘contains’, it is possible to match all restore activity for a directory.

The User Guide has been updated to reflect the correct WLCG URL prefix and to add more information concerning support for .well-known.

gplazma

Upload (as in xrootd’s -P / persist on successful close) permissions with JWT token claims now work correctly.

History

A bug handling NaN (not a number) errors in histogram conversion has been fixed, as well as one for index-out-of-bounds errors in timeseries histograms.

NFS

The NFS door will fail the request if a file’s storage unit is not configured in PoolManager, as all other doors already do. The new show open files command lists all open files whether or not a mover exists for them (the latter is displayed by show transfers).

Pool

Typically, when dCache interacts with an HSM, there is a timeout on how long such requests can stay in the HSM queue. Despite the fact that those timeouts are HSM-specific, dCache comes with its own default values, which are usually incorrect, so admins usually end up explicitly setting them.

Starting with version 9.0, the default timeout has been removed. This means that there is no timeout for HSM operations unless explicitly set by admins.

NOTE: this change is unlikely to break existing setups, as previous timeout values are already stored in the pools setup file.

In addtion, a new command sh|rh|rm unset timeout has been added to drop defined timeouts.

QoS Engine

Some regressions which occurred during the adaptation of resilience code have been fixed; these involved the monitoring of pool status changes and support for regex resolution in the poolmanager configuration.

Support for migration using a new pool mode, “DRAINING”, was added. As with Resilience, a separately configurable namespace endpoint is now possible.

Resilience

Now allows for a separately configurable namespace endpoint (for instance, if resilience were to use an RDBMs replica instance rather than the live instance of chimera).

A recipe for the migration/draining of resilient pools has been added to the dCache cookbook.

XRootD

There have been a number of bug fixes since 8.2.0. These have all been backported. They involve handling of the door address and proper shutdown on error for the door proxy, returning the correct error code when a file exists, making the handling of anonymous restrictions by the door consistent (for multiple authentication protocol doors), correct handling of error logging, and not publishing link local addresses.

Defaults have been changed for number of mover threads (now uses Netty defaults) and the order of the door’s gplazma plugins (ztn before gsi).

Improvements to documentation concerning TLS and host cert/key, recommendations for direct memory usage with door proxy; we have also added an Xroot User Guide page (only minimally populated at the moment).

Changelog from 8.2.0 to 9.0.0

(tbd as soon as the branching is done)

2afdc7fbe7
[maven-release-plugin] prepare for next development iteration
3547993ac2
[maven-release-plugin] prepare release 9.0.0
3e5ee5fbd7
pom.xml: update to next xrood4j version (4.5.5, 4.3.7, 4.2.11)
d551428088
chimera: fix PostgreSQL optimization for labels
54d4b86f80
gplazma: oidc add suppress for audience claim verification
c4be3b0e53
gplazma: oidc allow suppression of offline verification
0c011b7e73
gplazma: oidc add support for suppress option
eb763b5948
dcache,chimera: resolve symlink before applying Restriction
c51c5c89f1
dcache-bulk: small improvement to RB 13915
06074c5fa5
dcache-bulk: fix JSON object key not found exception
baa98e9231
dcache-frontend: add more detailed description of request objects for bulk and stage
a74b20052b
correct snapshot
845eeaa714
update snapshot
ac92269837
[maven-release-plugin] prepare branch @{releaseLabel}
75f90254d1
dcache,spacemanager: change deprecated jdbc method use
374686bd6d
multiple: change deprecated unit test utilities
84f1da69f5
nearline-storage: drop default timeouts HSM operations
b9a404a309
libs: update HikariCP to version 5.0.1
9c28ee5ea6
dcache-frontend: resolve on parent of source when renaming
cf75c83fff
Fix openssl error when no email address is found.
adb245a541
cells: remove FST related tests.
6ac6d13834
cells: remove FST based message serialization
baa726c345
Don’t use global git scope, use repository scope.
112929f80c
frotend: introduce pool migration support
29fbe4a85c
systemtest: auto-generate X.509 credentials
0c5491418d
systemtest: remove work-arounds for oidc-agent
3f4d27c8e0
Update maven plugins.
d80292b569
dcache-bulk: remove target error object and always convert to type + message
930b8e6a69
gplazma: x509 do not report bad .info files as a dCache bug
5cb03d996f
dcache-xroot: demote bad login token warning to debug
ba755aff7b
dcache-xroot: add missing shutdown on proxy inactive channel
6ad477bdc5
webdav: respect xattrs for COPY request
45f2ae2a03
dcache-bulk: split arguments out into a separate table
98dc9217ae
kafka: delete not important log message
d7a86f771c
dcache-bulk: fix system test hsqldb unique key problem
83d3296d06
docs: add Xroot User Guide page
9967a890ab
gplazma: oidc support ‘aud’ with spaces
9e5c9a0c1c
nfs: update door to report explicit layout errors
579f0c135f
dcache-qos,psu: allow getStorageUnit to resolve regex
0bc8780906
dcache-frontend: provide optional legacy qos request in namespace resource
7a7d5d20e1
dcache-qos,dcache-resilience: allow db endpoint to have separately configurable properties
d62166adc5
dcache-frontend: convert empty to null when processing TimeUnit attribute
2d8b15279c
dcache-bulk,dcache-frontend: fix bug in tape/stage and in argument extraction from DB
0dbe3086a1
docs: add more information concerning support for .well-known
3cb49637bc
dcache-bulk: fix faulty merge of database update
680c3588d1
dcache-xroot: return kXR_ItExists error code when file exists
7dedcf5e60
telemetry: Don’t log stacktrace for caught expceptions
c514d71b19
pool: log thread name on double close
f3db14cb4b
psu:fix issue for nested pgroups
5b4ceb80bf
common: provisional fix for out of bounds error in timeseries histogram
40dc86223b
gplazma: allow upload permissions with MultiTargetRestriction
cbc882c906
common: prevent NPE by explict cast of (Double)double in ternary condition
cadd250f6c
Motivation:
0a3685f8dd
commons: add null check to nanToZero method
ffd7352cd0
dcache-history: avoid NaN stack trace in histogram conversion
6ce991fd5a
pool: remove active dcap mover support
3d86a07073
skel: deprecate unused dcache path for bulk
713e29f8a4
docs: update dcap chapter
0eb7fbb441
dcap: always enforce passive mode
f3e00378e6
fix binary download URL in README.md
f33f9e7928
dcache-xroot: make handling of anonymous restrictions consistent
7c26954bd6
nfs: refactor nfs-proxyio
a4173937db
pom.xml: update to next xrood4j version (4.5.4, 4.4.5, 4.3.6, 4.2.10)
fde2e98795
dcache-chimera: reduce frequency of expensive cleaner inode deletion
cf11ab084a
Update dcache-minimal-installation.md
ea7b107ee8
Update dcache-minimal-installation.md
4ed325bfcc
Update dcache-minimal-installation.md
11af9efea1
Update dcache-minimal-installation.md
51226de089
Update dcache-minimal-installation.md
37963c3fb3
Update dcache-minimal-installation.md
e7ccb4c3bb
docs: update User Guide to reflect correct WLCG URL prefix
9010e8382c
pool: handle composite grizzly buffers in NFS mover
1569bc07e3
Update Enable OIDC authentication in dCache
9190d52e08
OIDC how to first draft
f09f587965
Update dcache-minimal-installation.md
81cc3b3ae9
Update dcache-minimal-installation.md
0177784185
Update dcache-minimal-installation.md
feecaddd5e
Update dcache-minimal-installation.md
8c6f198fcf
Update dcache-minimal-installation.md
ade5f2058e
skel: add missing property deprecations for cleaner-disk
46bb508d83
skel: add missing property deprecations for qos and bulk
a2b1b0e37c
nfs: use buffer pool and shallow encoding to avoid extra copy
bdc23822cf
libs: use nfs4j–0.24.1
665dea9456
dcache-bulk: use pnfsid instead of path to match requests (GET /api/v1/bulkrequests)
fd1997b48b
common,dcache-bulk: add join capability to jdbc criterion and utils
0ced07c65b
dcache-bulk: drop column delay_clear (no longer used)
4cec30405d
dcache-bulk: mark expanded directory skipped if activity does not apply to directories
e4d8b4e9a8
dcache-bulk: make sure LINK types are handled when refetched from prestored targets
2213544711
dcache-xroot: handle outbound errors on channel promise
aa348d48fa
pool: reload host certificate if changed
fc604fc0b9
pool: pass SSLContext to p2p Companion
5483f99200
Update dcache-minimal-installation.md
cd02b08ecf
Update dcache-minimal-installation.md
eee803fd64
Update dcache-minimal-installation.md
54522fc2aa
Update dcache-minimal-installation.md
8460a7d575
Update dcache-minimal-installation.md
00a039638d
Update dcache-minimal-installation.md
21b37274c1
Update dcache-minimal-installation.md
7aa3f1c3c7
Update dcache-minimal-installation.md
9e1c89c885
Update dcache-minimal-installation.md
4a7b45ed28
Update dcache-minimal-installation.md
7e6eb168ac
Update dcache-minimal-installation.md
2c8855930c
Update dcache-minimal-installation.md
08a3c571e8
Update dcache-minimal-installation.md
8fd3101316
dcache-chimera: cleaner-disk, replace custom pool blacklist with pool info base
a3c795f5af
Update dcache-minimal-installation.md
6d09693658
dcache-chimera: cleaner better handles stale delete reply
5e1f767827
Update dcache-minimal-installation.md
13e45c3024
Update dcache-minimal-installation.md
91a98374d3
Update dcache-minimal-installation.md
a152be8096
Update dcache-minimal-installation.md
b1d24f6b6c
Update dcache-minimal-installation.md
8ee73ce057
Update dcache-minimal-installation.md
19e8916509
Update dcache-minimal-installation.md
54ed9f64e0
correction with gplazma
d6b696e88e
book: cleaner correction
d9b5492666
book: cleaner section on important database fields
2b1f43fbbb
dcache-frontend,webdav: add support for .well-known/security.txt
3fde03789f
doc: add recommendation for direct memory setting on xrootd proxy door
f0506a818e
book: add NOTE on domain naming, uniqueness
f242e5275f
pool,chimera: fix pool pings having a null-valued hsm field
e275a5c9f6
dcache-bulk: check for null before truncating path
d39815e55a
dcache-qos: check for null info in PoolInfoMap.isDraining
e9fb629d67
dcache-qos,frontend: add exception to remote client method and return 500 if fails
223a9445a0
dcache-frontend: handle parse exceptions in bulk and tape resources
89e484e21e
dcache-bulk: modify request limits to be something more reasoned
2d0be2958b
dcache-bulk: manage executor and data source shutdown explicitly
2a27f58f88
book: add ref to debug section
4d99119896
dcache-bulk: repair faulty merge
a09884f885
utils: makesure that linklocal and localhost interfaces not published
6b9c49e3a7
dcache-chimera: prevent PoolInformationBase NPE
de213caf7a
pom.xml: update xrootd4j (4.5.3, etc.)
7823be4acb
dcache-xroot: send door address to pool as client for proxied transfers
fd42de77e4
pool: disable idle connection handler when zero-copy is used
2b21b126c7
Endit and Endit Provider github repos
f499320aed
dcache-bulk: truncate paths > 256 chars
1733104b59
dcache-xroot: fix the default order for the gplazma plugins
ed7f5bafb3
nearline-storage: resolve provider before constructing HsmInfo
f0e12425f6
dcache-bulk: fix misnamed table in changeset
4e97cc0f9d
dcache-bulk: change long => Long in DAO update to conform with target class
0ce680e359
utils: use Enumiration#asIterator instead of guava’s forEnumiration
a2315c1a39
dcache-bulk: maintain target state counts in memory
4d8d36119d
dcache-bulk: eliminate storage of initial targets as text column in bulk_requests
5facbafc05
dcache-bulk: propagate PID enum to distinguish between INITIAL and DISCOVERED paths
7dd2bdb60b
dcache-bulk: convert pid field/column to enum/integer expressing node type
4c1a62a0ee
Revert “dcache-bulk: convert pid field/column to enum/integer expressing node type”
f54b9995f0
Revert “dcache-bulk: propagate PID enum to distinguish between INITIAL and DISCOVERED paths”
286196d2e2
Revert “dcache-bulk: eliminate storage of initial targets as text column in bulk_requests”
d72ae356ff
dcache-bulk: eliminate storage of initial targets as text column in bulk_requests
1208ac9880
dcache-bulk: propagate PID enum to distinguish between INITIAL and DISCOVERED paths
1dd3b4cd00
dcache-bulk: convert pid field/column to enum/integer expressing node type
0d9397f4d3
dcache-bulk: remove redundant update of request target in container start
771db06b8a
dcache-bulk: insert targets immediately on submit
a5f6cf5154
dcache-xroot: do not publish link local address for door
08d9884d74
nfs-proxy: use open-stateid when proxying IO request to a pool
586920a1a4
dcache-bulk, frontend: fix two small bugs
c26e8ade2a
dcache-bulk: initialize/load and reset on executor thread
07f4ee166d
dcache-bulk: fix incorrect condition for EOL on request info paging
8733936dd0
dcache-bulk: reset paging limit back to 10000
85a4920cd1
dcache-qos: support migration using a new pool mode “DRAINNG”
81a07b3e93
nfs.properties: note about limitation of NFSv3
485a911d72
build(deps): bump postgresql from 42.3.7 to 42.3.8
3d6092f3af
dcache-bulk: fix liquibase add index statement to be non-postgresql-specific
b28634dec4
dcache-bulk: fix liquibase so that hsql ignores index CONCURRENTLY
53d98713cd
frontend.srr.shares text rewritten
86c4bab65f
dcache-bulk: fix type error in updateCounts
0ba439959e
dcache-bulk: add id,rid index on request target table
0314973c4d
dcache-frontend: make bulk resource target backward compatible with 7.2
9f47863bfd
dcache-chimera: make sure non-leader does not execute clean run
9255e773ce
benchmark: add logback-test.xml to suppress logs
94585a1fde
libs: update fasterxml to 2.14.0
d42b7101b2
dcache-cli,ssh,bulk: allow PagedCommandResult to indicate EOL
d6e95a3c21
bulk.xml: remove the scheduled executor for delayed clear
6fb0eac102
nearline-storage: don’t catch Runtime exceptions
620ec5d710
poolmanager: use net unit groups to identify the request
b708a6ce53
nearline-storage: fix NPE on error path
3dff15a4a1
nearline-storage: fix race in flushing removed file
b6bbfccbe4
dcache-bulk: fix message thread invocation of database query for statistics
a806be10c9
dcache-bulk: refit request ls and target ls to use paged admin command results.
e83f39ab5d
dcache-bulk: deprecate/no longer support ‘delay’ option on request clear
c8a82608b6
dcache-bulk: fix garbled request policy command output
892ad2cc5e
dcache-frontend,bulk: fix bulk paging in restful api
96dce3bee3
dcache-admin,ssh: Add paging capability to AnsiTerminalCommand (and DirectCommand)
4fd2d39b3d
dcache-bulk: fix admin command options (uppercase)
38be074600
dcache-qos: junit tests for utils (2)
ae76c469f4
pool: remove unused field in FileRepositoryChannel
5527ce0222
docs: add debugging section about JFR
92971cb7b1
dcache-qos: junit tests for utils (1)
e15e720e59
dcache-qos: add Junit test for PoolInfoChangeHandler
099d1a5f2f
srm-server: fix bug making trs unit tests more unstable
81654c01a6
dcache-chimera: remove unused cleaner-hsm properties
bf8d071ed9
srm-server: make trs unit tests more stable
b41546ff02
nfs: add admin command to show open files
f73a1407df
dcache-qos: junit tests: PoolOpChangeHandler, mock objects and resources
5e1308198f
pom: use nfs4j–0.24.0
6776e5006a
dcache-rest namespace resource: add optional query parameter to retrieve a wealth of information about a file
655a27be9c
dcache-qos: fix minor regressions in handling of pool info diffs
9a88795c74
dcache-chimera: allow both cleaner types to change clean interval via admin interface
c4d8fab661
book, cleaner: add details
7435d5d009
dcache-frontend: fix type default of RESTful pool/nearline/queues resource
93284a7e9c
chimera:optimise db call
14b9853636
transfer-manager: drop obsolete args by CopyManagerMessage constructor
f189210559
dcache-qos: junit test for pool.qos
9edd75763b
dcache-qos: fix two more regressions in the PoolOpChangeHandler
eba290aa72
skel: change mover thread defaults to Netty default
09668f444c
pool: don’t publish attached HSM if lfs mode is precious
fc792930c7
pom.xml: bump xrootd4j to 4.5.2
d0d9f6ce2c
dcache-qos: add missing pool tag comparison in PoolOpChangeHandler
531c86a232
pool: use zero-copy send for plaintext http transfers
c421b14512
common-security: do not fail in case of partial ssl encoding
0f5b88f443
pool: avoid exception based flow in case of nfs bad stateid error
8f540520fe
dcache-xroot: add note about TLS and host cert/key to documentation
337970c0c3
dcache-bulk: force admin command STAGE activity to prestore=true, as with REST API
41a9e807d7
dcache-bulk: fix missing pnfsid on cancel of PIN/STAGE
f56d43ae4e
dcache-core: let pinmanager stop retrying pinning upon unpin state
758c58aec1
dcache-core: add details to pinmanager unpin admin command
b0ede1a0e4
fix vexing typo in properties file
df70ffc051
delete “reedme”
d40bb6f347
delete test readme
c4bfb6227d
dcache-chimera: disk-cleaner, error handling in parallel pool deletes
795de5369e
libs: use aspectj 1.9.9.1 with java 17 support
c3e4222b02
dcache-chimera: hsm-cleaner, remove pool-reported failures to delete from cache
0bcd6aca25
poolmanager: DRY - reuse link comparator logic in LinkComparator
70cee0d866
poolmanager: update NetHandler to use Subnet class
12b41e0592
dcache-spacemanager: fix NPE when link-group-file-name is not set
b41eca34fa
dcache-chimera: fix logging with fewer arguments provided than specified
51a76c97d0
poolmanager: implement PoolPreferenceLevel#toString
f6ad267da0
poolmanager: use java8 stream to implement LinkMap#iterator
91467a22d2
dcache-chimera: add hsm admin command to forget a pnfsid
1723a4fc33
dcache: NPE on removal via WebDAV and token
0e9da5faf0
dcache-frontend: add filtering and sorting on owner, group and path to restore info API
120e4a42eb
dcache-frontend: modify API to respect new ac_psux_match parameters
0d3ef24ebf
dcache-core: add pnfsid and path options to ac psu match
81dd75fd08
dcache-core: refactor psu match method into sub-methods for legibility
4f145182bb
dcache-core: handle null/globbed values correctly in psu match
532c1c8350
webdav: ensure that HttpProtocolInfo is populated with correct client address
2e1ddf87ce
gplazma: grid switch gridmap file to use ParsableFile
c145bfe267
gplazma: grid switch authzdb plugin to use ParsableFile
c9863cb2b1
gplazma: grid switch vorole plugin to use ParsableFile
90175fb7ba
dcache-xroot: explictly close the proxy’s bound port channel
a59f6c1235
pnfsmanager: dispatch create entry messages to threads associated with parent entries
99cd0b7159
dcache-xroot: select proxy door address on basis of reachability for client
fdfc74f1c7
doc: add recipe to cookbook for migration/draining of resilient pools
13b0acba6c
libs: use jython-standalone to be independent of local python installation
2f21050d09
libs: use up-to-date version of Jython
867f779076
Update dcache-minimal-installation.md
0f529d7e61
Fix typo in SRR documentation
574ae86f70
book: ha page, change cleaner to cleaner-disk and -hsm
c8e04eb368
book: install page, change cleaner to cleaner-disk
45aca8a804
Update dcache-minimal-installation.md
e327e0599e
Update dcache-minimal-installation.md
448275cc7c
Update dcache-minimal-installation.md
af48b0677a
Update dcache-minimal-installation.md
4fa01ca77b
Update dcache-minimal-installation.md
b06e583610
Update dcache-minimal-installation.md
2122f54809
Update dcache-minimal-installation.md
5d1dc89df9
adding gsi missing folder
6694551a0c
build(deps): bump postgresql from 42.3.3 to 42.3.7
1b4c549050
gplazma: grid drops the MapPredicate interface
c5de48f099
gplazma: grid add interface to abstract SourceBackedPredicateMap
9c91096aeb
dcache-chimera: documentation for splitting cleaner into two cells
aa58f5673e
dcache-chimera: separate disk and hsm cleaner into new cells
d33ed35775
qos-engine: prevent NPE if pool monitor not available
a5369100b6
gplazma2-grid: invalid IGTF policy file should fail with ParserException
b2bc8f3fad
webdav: update 3rd party js libraries
ff557a0e57
nfs4: return EACCESS when stage is not allowed
1c60d6fcac
dcap,nfs: handle storage class miss configuration
aa8ec769db
Book: describe r and g inheritance flags
d922611099
benchmarks: add PoolSelectionUnit example
179b34f051
dcache-chimera: cleaner code prettification and logging improvements
fb40802066
libs: update to spring–5.3.x
ae3621c57f
dcache-chimera: change admin command to show hsm delete location count
8b8589ce5b
gplamzma: vo-group plugin
4ffe93fa18
gplamzma: vo-group plugin
49c54f0b27
skel: update default java options to support JVM–17
5ad027c6f1
[maven-release-plugin] prepare for next development iteration
dbfabfc8e0
build(deps): bump netty-codec-http from 4.1.71.Final to 4.1.77.Final