Highlights

• Performance improvements for the concurrent directory creation and removal

Acknolagements

Many Thanks to Lars Jansse and Sandro Grizzo for their contributions.

Incompatibilities

• prior to version 9.2 dCache NFSv4.1 door will publish only nfs4_1_files layout. Now on the door publishes all available layout types
• dropped reference count tracking for directory tags
• dropped support of java options chimera_soft_update and chimera_lazy_wcc in favor of chimera.attr-consistency property
• If upgrading from 8.2, be sure to read also the release notes for 9.0 and 9.1, important changes are described there. See the 9.2 migration guide for more complete information
• With version 9.2.8+, empty and non-existent banfiles will be treated the same

Release 9.2.29

tape

Users reported 2 day pin lifetime on staged files (which is a default) despite specifying different values. This is now fixed.

Changelog 9.2.28..9.2.29

0d911615bc

[maven-release-plugin] prepare release 9.2.29

ab9a74538d

tape REST api: additional fix tohandling of prefixed paths

499959a572

[maven-release-plugin] prepare for next development iteration

Release 9.2.28

xroot

Return destination address (that is, the haproxy address) if xrootd.enable.proxy-protocol=true is set, instead of the actual door address.

Changelog 9.2.27..9.2.28

57579ad5de

[maven-release-plugin] prepare release 9.2.28

95cad94b15

xroot: handle haproxy and checksum command

89133016c8

[maven-release-plugin] prepare for next development iteration

Release 9.2.27

CI

Pipeline optimizations.

pool

Fix double decrement on active hsm requests. This addresses the issue of pools stopping flushing to tape with “Negative number of active requests”.

When a thread performing I/O gets interrupted, then an InterruptedIOException might be thrown. A DCAP mover will treat and propagate such an exception as a disk I/O error, thus disabling the pool. This fix reduces false positive disk I/O errors.

Changelog 9.2.26..9.2.27

c2f32fc913

[maven-release-plugin] prepare release 9.2.27

d8d5541241

pool: don’t treat InterruptedIOException as a disk IO error

8e376b7181

pool: fix double decrement of hsm requests

1215548076

ci: split container image registry and repository

0863dacffa

[maven-release-plugin] prepare for next development iteration

Release 9.2.26

bulk

The current release fixed broken command activities.

Changelog 9.2.25..9.2.26

9e9ea4fa6d

[maven-release-plugin] prepare release 9.2.26

6664395575

ci: run spotbugs only on master branch

b075c04d19

bulk: fix broken command activities

9c0b160e42

[maven-release-plugin] prepare for next development iteration

Release 9.2.25

bulk

When specifying an empty target, bulk proceeded to process the request instead of failing fast. This is now fixed.

gplazma

A previous commit, leading to the last bugfix releases being blacklisted, introduced a regression in the multimap plugin. Where the ‘op’ principal type is used, logins will fail with dCache logging a stacktrace like java.lang.RuntimeException: Failed to create principal: java.lang.NoSuchMethodException: org.dcache.auth.OAuthProviderPrincipal.<init>(java.lang.String). This is now fixed.

Changelog 9.2.24..9.2.25

8fecee871f

[maven-release-plugin] prepare release 9.2.25

4d14890f7d

gplazma: fix broken commit d74d9568167f4

729ce8f76a

gplazma: multimap fix op regression

5708994ab6

bulk: check targets for empty strings

6043a20af4

[maven-release-plugin] prepare for next development iteration

Release 9.2.24

doc

Better documentation clarifying OIDC provider ID and issue claim.

webdav

Bumps org.eclipse.jetty:jetty-servlets from 9.4.51.v20230217 to 9.4.52.v20230823. Thi has fixed java.lang.NullPointerException: null error on WebDAV Domain due to jetty 9.4.51 bug.

Changelog 9.2.23..9.2.24

bfaeae101e

[maven-release-plugin] prepare release 9.2.24

6394fbecd4

github: add action for atumatic github-release

6ab84e51ea

gplazma alise initial version of plugin

db6ea0f464

common: add issuer URI to OAuthProviderPrincipal

09e5a49937

build(deps): bump org.eclipse.jetty:jetty-servlets

eb858c9725

docs: clarify OIDC provider ID and issue claim

7e6098c30e

[maven-release-plugin] prepare for next development iteration

Release 9.2.23

CI

Improve our CI pipleline.

core

Pool migration : fixed behavior of migration copy with -replicas=n (where n>1) to behave as expected.

Libs

Improve and maintain our testing library.

Keeping libraries up to date.

TAPE-API

Release by relative path works.

Changelog 9.2.22..9.2.23

63803072d4

[maven-release-plugin] prepare release 9.2.23

a7aa1b3689

WLG TAPE REST API: fix handling of frontend.root in release API

a219e4457a

gitlab: mirror tags

3de9e9301c

libs: update mina-sshd to version 2.13.1

96ef3777ca

Fix issue with infinite replicas when replicas > 1

6b3ff705aa

github: change mirroring action

7bd322edc4

ci: use shorter k8s namespace names

abe0f45324

pom: add exta java option to run powermock test under java17

118a5cc34d

ci: sync CI pipeline with master

fb954fb8ef

ci: use desy nims repo for CentOS7

562668fc1e

ci: use almalinux9 for rpm install test

2991f5ef12

[maven-release-plugin] prepare for next development iteration

Release 9.2.22

chimera

Now it is possible to n read/write CTA and migrated Enstore files without having to set hsmInstance tag in the full directory tree.

frontend

The current release fixed the issue about failures to invoke WLCG tape API to stage files on files relative to {webdav,frontend}.root

It is now possible to call stage API successully on paths ralative to frontend.root.

Changelog 9.2.21..9.2.22

b7f34147e4

[maven-release-plugin] prepare release 9.2.22

743c52948e

pom: compile code in the same jvm as maven

8a428d3316

chimera: add CTA HSM StorageInfo extractor

69f2f109ca

frontend: handle frontend.root variable properly

5652f98acf

[maven-release-plugin] prepare for next development iteration

Release 9.2.21

chimera

Adapt existing uri_encode function so that the Enstore HSM script does not fail and the files go to tape successfully.

ci

CI pipeline improvements.

dependencies

Update build dependency to be compatible with modern tools.

Changelog 9.2.20..9.2.21

f48df60d4e

[maven-release-plugin] prepare release 9.2.21

9691e0096c

dependencies: update modernizer plugin to be compatible with maven 3.9

6e2c1186bd

chimera: fix uri_encode to handle special characters

f52a0ee2f4

ci: add property to control upload options

24759f2e3c

[maven-release-plugin] prepare for next development iteration

Release 9.2.20

bulk

Bulk truncates path to 256 characters and this seems to be causing problemsd.

This is now fixed.

pool

With CentOS7, the client sends the last request, such that offset+count == filesize. REHL9 sends the last request so that the count is multiple of 4096, which is legal.

The current release fixed miscalculation of offset on short read.

Changelog 9.2.19..9.2.20

f3b6d8e7c9

[maven-release-plugin] prepare release 9.2.20

8d7d4661d2

ci: disable srmcp test

a710037a84

ci: use python3 for robot test

4158c80ae5

bulk: do not truncate target paths

9b9714f267

ci: pinpoint postgres helm version

15ea025a3f

pool: fix miscalculation of offset on short read.

39bb2f532d

[maven-release-plugin] prepare for next development iteration

Release 9.2.19

chimera

Fixed a bug that made it possible to create a loop on directory move.

Changelog 9.2.18..9.2.19

368ce200f4

[maven-release-plugin] prepare release 9.2.19

465e00e82a

chimera: fix loop creation on directory move

9a46793677

ci: drop –ftp-create-dirs option (as we switch to https)

c0357f4f81

[maven-release-plugin] prepare for next development iteration

Release 9.2.18

webdav

When running multiple Remote Transfermanager, two transfer that were started simultaneously had the same ID which led to the second transfer becoming orphan after the first one completed. This is fixed now.

Changelog 9.2.17..9.2.18

9cb77bc505

[maven-release-plugin] prepare release 9.2.18

b0e6eaf759

ci: eplocitly specify kubernetes namespace

3f06a7a6ad

ci: fix typo

5cf1c757c9

webdav: use transfermanager+id to identify TPC transfer

73fbb8b77f

docs: update oidc chapter to explain trust anchors

25fe810ba9

[maven-release-plugin] prepare for next development iteration

Release 9.2.17

common

The out-of-box version of ProxyCSRGenerator#generate from CAnL uses SHA1 for proxy delegation, which is banned by modern OSes.

this now fixed and RHEL9 clients works with proxy delegation without enabling SHA1.

xroot

Switch to xrootd4j–4.6.0, ew major release with bug fixes and enhancements including the reload TLS certificate.

Changelog 9.2.16..9.2.17

42faeb1884

[maven-release-plugin] prepare release 9.2.17

973a06353d

xroot: switch to xrootd4j–4.6.0

3eeefe01c2

common-security: add custom version of ProxyCSRGenerator#generate

177d75f696

[maven-release-plugin] prepare for next development iteration

Release 9.2.16

qos

Admins now have the option to disable role based authorization for QoS transitions.

webdav

dCache now supports HTTP-TPC transfers where the dCache-local path includes HTTP reserved characters.

Changelog 9.2.15..9.2.16

c4035193dd

[maven-release-plugin] prepare release 9.2.16

4f8f2099ed

qos: add flag to enable/disable role based authorization for transitions

92c2350067

webdav: httptpc percent-decode local path

e08dc2a171

[maven-release-plugin] prepare for next development iteration

Release 9.2.15

bulk

Some of the format strings that manage the formatting of command results were sending to the user expect a different number of parameters than provided. This is now fixed and the returned bulk info strings return all expected values.

info

DGAs send messages to other dCache cells to discover their current status. These messages have a hard-coded one second timeout.

Some queries are data-intensive and could take longer than one second to build the answer, resulting in no information being provided.

this is now fixed and the info service will now wait longer for a cell to respond to a query for information.

nfs

Grizzly memory management uses heap memory size fraction even if direct memory is used (issue 7529). This issue has been fixed and dCache starts with 256m of direct memory.

Changelog 9.2.14..9.2.15

04a4d03830

[maven-release-plugin] prepare release 9.2.15

253e07ab7d

info: use DGA refresh rate as message timeout

58cc1f30f5

nfs: calculate desired memory fraction for correct memory allocation

f51fcde6db

bulk: fix divergent command params format strings

cfa39a1960

[maven-release-plugin] prepare for next development iteration

Release 9.2.14

cells

The error reporting on tunnel disconnect has been fixed.

srr

The current release fixed empty path annotation warning.

tape-api

Request supplied paths are now mapped based on frontend.root.

Changelog 9.2.13..9.2.14

7ff7f5ba6a

[maven-release-plugin] prepare release 9.2.14

f9ef94a8dd

tape-api: map request supplied paths based on frontend.root

2c8a17f3ad

srr: fix “empty path annotation” warning

fa7cd0ec2e

cells: fix error reporting on tunnel disconnect

bb4b067fb9

[maven-release-plugin] prepare for next development iteration

Release 9.2.13

core

This patch has been shown to fix the issue in which poolmanager will in some instances not load parts of its configuration.

packaging

Packaging infrastructure improvements.

Changelog 9.2.12..9.2.13

3d735b9766

[maven-release-plugin] prepare release 9.2.13

23469021c9

Revert “poolmanager: delete property for switchingon/off caching for psu”

861b5a104c

ci: use minimal almalinux–9 to upload packages

453fa408a8

[maven-release-plugin] prepare for next development iteration

Release 9.2.12

pool

Buffer initialization is now done at pool start instead of first NFS read request. This leads to more predictable buffer initialization.

Changelog 9.2.11..9.2.12

7ee8097efc

[maven-release-plugin] prepare release 9.2.12

fee2057919

ci: use dtzar helm-kubectl image

481408f770

dcache-core: fix psu logging

764779040b

pool: mover grizzly IO buffer initialization into NfsTransferService

82cd451b32

[maven-release-plugin] prepare for next development iteration

Release 9.2.11

webdav

Users have reported severe slow down when runnig listing using webdav.

This is currently fixed.

Changelog 9.2.10..9.2.11

46f47153a8

[maven-release-plugin] prepare release 9.2.11

da22a29c73

webdav: fix slow listing

2d73d8ba98

[maven-release-plugin] prepare for next development iteration

Release 9.2.10

webdav

A recent path introduced support for adding the Link HTTP response header, according to RFC 6249. Unfortunately, the code added the Link header for all requests, not just the intended GET and HEAD requests.

Additionally, due to peculiarities of how Milton generates PROPFIND results, the Link header is added multiple times: once for each subdirectory.

This results in the HTTP response headers taking up too much space and dCache failing the request, returning a 500 status code.

This regression has been now fixed where PROPFIND requests would fail if the directory contains too many subdirectories. The cut-off point depends on the length of the URL the PROPFIND request targets.

Changelog 9.2.9..9.2.10

f58389b35b

[maven-release-plugin] prepare release 9.2.10

f15e622cb4

webdav: fix link header

3125694915

[maven-release-plugin] prepare for next development iteration

Release 9.2.9

gplazma

A bug is fixed in gPlazma that is triggered when users attempt to authenticate with broken gPlazma configuration or a gPlazma plugin’s configuration is broken.

Changelog 9.2.8..9.2.9

ffef10d130

[maven-release-plugin] prepare release 9.2.9

c7993fe0f3

gplazma: fix NPE if gPlazma is rejecting all logins

42d497e172

[maven-release-plugin] prepare for next development iteration

Release 9.2.8

chimera

A recent commit as a side effect resulted in failure to upload a file to an xroot door with the overwrite (“-f”) option, returning okay but removing the original file without creating the new one. This patch reverts the offending commit: copy with an overwrite flag works again.

CI

Improve build pipeline.

gplazma

Previously, a missing banfile would result in every successive login attempt failing with an NPE. Now, even if configured, an empty or inexistent banfile will be ignored and logins should succeed. WARNING: This changes the banfile plugin behaviour! Empty and non-existent banfiles will be treated the same.

system-test

Increases the small default direct memory value for system-test: NFS on system-test works again.

webdav

Update webdav door to retry the request to the namespace if checksum is not present in files attributes.

xroot

A recent commit introduced handling of relative paths in xrootd with inadvertent side effect of dropping support xroot.root variable. This patch reverts that commit: pre–9.2 behavior is restored.

Changelog 9.2.7..9.2.8

63c0dfbd70

[maven-release-plugin] prepare release 9.2.8

7dcd2cc2fd

ci: dont’t pull build artefact for kubernetes-based jobs

1af0614d0c

Fix unit test for commit 6b47354

0deb226953

gplazma: configured banfile plugin should ignore non-existent ban file

2ac2180724

chimera: Revert “chimera: update FsSqlDriver#inodeOf to throw exception if file not found”

38d2cae1ff

system-test: increase direct memory

3840f87195

xrootd: fix xrootd.root regression

baa135bab9

webdav: wait for upload to complete

16d3bb94a3

[maven-release-plugin] prepare for next development iteration

Release 9.2.7

ci

Build and test system improvements.

cleaner-disk

No more occasional ConcurrentHashMap exceptions in cleaner-disk runs due to concurrent pool status changes.

core

Admin commands with hyphen-containing arguments are no longer trunkated prematurely, they are now properly shown and autocompleted.

qos

Prevent PropertySetterException in qos.

xroot

The change to use effectiveRoot did not take into account Subject = Nobody.

Regression eliminated, previous behavior with anonymous read restored.

Fixes previous fix for xroot descriptor: NPE risk mitigated.

Changelog 9.2.6..9.2.7

a91a3bbb1c

[maven-release-plugin] prepare release 9.2.7

8a60a76165

dcache-xroot: check that descriptor is not null before calling close – fix

84e5af54cd

dcache-xroot: check that descriptor is not null before calling close

711b4a38f8

cleaner-disk: prevent ConcurrentModificationException in cleaning run

84026b3779

dcache-xroot: fix effective root when subject is nobody

95590b4a5f

dcache-core: fix admin command completion

e8bc2c0274

qos: comment out property description

c2022ea407

ci: add exta helm ops to extend timeout and simplify retries

53afeee632

[maven-release-plugin] prepare for next development iteration

Release 9.2.6

dependencies

The MongoDB driver was updated. This might lead to different log messages.

frontend

A bug was fixed that led to an Exception when using qos without policy.

webdav

If a non-existing well-known resource is requested, it won’t log a stacktrace anymore.

Changelog 9.2.5..9.2.6

c0bc7fa627

[maven-release-plugin] prepare release 9.2.6

a83cc7fbfd

pom: Update mongodb-driver

1b10c15afb

dcache-frontend: check for defined arguments with namespace qos

1748b3b887

webdav: do not log stacktrace if non-existing well-known is requested

9ad93f688c

[maven-release-plugin] prepare for next development iteration

Release 9.2.5

bulk

The current releae fixed handling of uncaught exception.

dcache-view

dCache-view has been updated to 2.1.0 and it includes modifications to use new RolePrincipal instead of the roles plugin role, eliminating the need for additional logins for admin privileges.

qos

With ingest queues servicing large numbers of requests, as we have for performance reasons configured as defaults on QoS and Bulk, there was a potential for a race between the processing of the original modify request and a subsequent cancellation request, such that cancellation could not find the request as it was still in the executor queue.

This is fixed now and are is a trailing stream of requests still being processed after cancellation (from Bulk) completes.

Changelog 9.2.4..9.2.5

7969d6bbd7

[maven-release-plugin] prepare release 9.2.5

dee8cbc483

dcache-qos: cache modify requests until processed by executor

3a255bb92c

dcache-bulk,dcache-qos: repair mass cancellation issues

4bed61af7e

dcache-bulk: fix handling of uncaught exceptions

73b49ce730

pom.xml: update dcache-view to 2.1.0

ea69bea591

Fixed QoSPolicyTest duration strings

347c668ce9

dcache-frontend,common: check parsing of Duration in STAGE

470e0f65a3

[maven-release-plugin] prepare for next development iteration

Release 9.2.4

chimera

The rollback of 9.1 to 9.0 db schema was fixed and possibility to rollback namespace db to an earlier schema version is restored now.

webdav

The urls similar to this https://door.domain.foo:1234//pnfs/domain.foo/path/to/file were strip into /domain.foo/path/to/file.

The current release fixed the parsing of urls with two slashes in the path.

Changelog 9.2.3..9.2.4

cd710f3992

[maven-release-plugin] prepare release 9.2.4

cefea2f3cb

chimera: fix rollback of 9.1 to 9.0 db schema

95fcd5d230

webdav: fix infinite recursion in Requests.stripToPath

0e33db68c5

webdav: fix parsing of urls with two slashes in the path

428f7c77e5

ci: generate release-notes template

8fff819a39

Revert “ci: use rancher to create k8s namespace”

9e4b101bcf

[maven-release-plugin] prepare for next development iteration

Release 9.2.3

dcache-bulk

Performance and stability improved, but throughput continues when the submitted task activities are in a state of waiting for future completion.

door

Exception handling is improuved for Kafka.

oidc

Storgae scopes without path will be rejected now.

Changelog 9.2.2..9.2.3

b8134f8db3

[maven-release-plugin] prepare release 9.2.3

ec5783ca89

dcache-bulk: use rate limiter to throttle semaphore release

4e9864a336

oidc: fix remove invalid testcase

8059c50501

oidc: reject storage scopes without path

af96f5c6d6

door,pool: handle multiple possible KafkaExceptions

c9fe2f61f9

docker: use exec for start java process

674bb383df

[maven-release-plugin] prepare for next development iteration

Release 9.2.2

bulk

A bug was fixed that may lead to the archiver deleting ongoing requests.

qos

The default size of the task queues are set to “qos.limits.verifier.max-running-operations” now to increase the throughput. IMPORTANT: The default size of the QoS Engine’s modify thread pool was also updated to 500 as the current value, 32, is too little.

webdav

This patch fixes a bug that led to wrong paths on redirects.

Changelog 9.2.1..9.2.2

16ffa5d557

[maven-release-plugin] prepare release 9.2.2

3278975650

ci: use pynfs:0.5, enable LOCK24 test

09c872a80a

ci: use rancher to create k8s namespace

38678c9209

dcache-qos: correction to the threshold warning

1a8bb02376

dcache-qos: set default task thread pool sizes all to max concurrent running

b3a1bb7b6b

dcache-bulk: fix bug in archiver deletion query

ea7e26199b

dcache-bulk: fix thread executor injection

d466c360c0

[maven-release-plugin] prepare for next development iteration

08c0ac9c60

wevdav: fix redirect path

Release 9.2.1

bulk

Correct bulk cancellation semantics.

The problem with cancellation (hanging perpetually in the CANCELLING state) was solved.

Return directory listing to its own executor.

Same performance as before, but with a total memory footprint at a little more than half available physical memory instead of being very close to it.

Add commands to display the count instead of the actual entry (as with requests and targets), and also a command to clear/delete from the archive table.

documentation

Add commands to display the count instead of the actual entry (as with requests and targets), and also a command to clear/delete from the archive table.

Fixes link in documentation.

Add section in WebDAV door chapter on metalink.

Changelog 9.2.0..9.2.1

8d4bf82df5

[maven-release-plugin] prepare release 9.2.1

8cac6fa53e

dcache-bulk: fix cancellation issues

66b67eabe3

dcache-bulk: give directory listing a separate executor

151b4cb86a

doc: add QoS policy and role documentation to the dCache Book

0ffd722bd6

docs: UserGuide fix angle-braket in metalink

02924858c5

docs: add description of webdav’s metalink support

be306f450e

dcache-bulk: refine container executor model

3cd0912aca

dcache-bulk: cancel activity future on target cancel

201359918b

dcache-bulk: add count and clear to archive admin commands

95f53d8478

[maven-release-plugin] prepare for next development iteration

Release 9.2.0

Known issues

Cancel is currently broken in bulk service and recursive requests could freeze up.

Admin

Paging capability was added to the AnsiTerminalCommand (and DirectCommand), where results exceeding 10K lines prompt the user with [Y/N] for further results. When executing a shell command in non-terminal mode, all results are streamed back continuously. The Bulk commands have been converted to use this feature.

Bulk

There have been many small fixes and improvements to this service since 8.2.

The most significant changes or additions include:

• The storage layer has been redesigned to conserve space and for efficiency/throughput. Please note that moving up to the new database schema may require some time.
  The amount of time can be generally computed in terms of the number of entries in the request_target table; figure about 1 hour for every 10 million. If it is necessary to maintain all such entries, we recommend you do the upgrade offline, using the dcache database update command-line tool. If there is no need to keep completed requests in the database, we would advise truncation or at least deletion of the completed requests before the upgrade.

• Periodic archiving of requests has been added; this is configurable via properties and admin commands. The archive table maintains an abbreviated summary of the requests, and can be purged via admin command as well.

• The container job has been rewritten to bring it in line with SRM bringonline performance; multithreaded directory listing has also been implemented to improve recursive request time-to-completion.

• Path resolution (for relative paths) has been integrated into bulk request processing.

• Activity providers now can capture the environment so that defaults can be customized (this presently pertains only to pin and stage lifetime attributes).

• Support has been added for the QoS update request to handle policy arguments.

• Bulk has been made replicable (HA). Please read the requirements in the cookbook section on High Availability.

• The delay clear option has been eliminated from bulk requests.

• The prestore option has also been eliminated, as it is no longer necessary (since all initial targets are immediately batch stored synchronously) before the submission request returns.

A few minor points:

• Default limits for request size and number of requests per user have been set on the basis of WLCG requirements.

• The counts displayed by the info command are from startup and are cumulative; for actual (current) counts based on the data store, use the status counts command.

There are numerous properties which have been deprecated, particularly those dealing with limits. Please review the bulk.properties file for details. It should not be necessary under normal usage to adjust the limits for thread pools, database connections and semaphores from the defaults. Depending on the volume of activity and the site requirements, the archiver period and window may need to be shortened from the default. Note that the options for clearOnSuccess and clearOnFailure that can be included in an individual bulk request to indicate immediate deletion are not available for STAGE requests as this feature is not part of the WLCG specification, so the only way to remove such requests automatically is through the archiver.

Chimera

With dcache version 9.0 chimera has introduced chimera_soft_update and chimera_lazy_wcc Java properties to control the behavior of the parent directory attribute update policy. Now those properties are obsolete and replaced by a regular dcache configuration property chimera.attr-consistency, which takes the following values:

policy	behaviour
strong	a creation of a filesystem object will right away update parent directory’s mtime, ctime, nlink and generation attributes
weak	a creation of a filesystem object will eventually update (after 30 seconds) parent directory’s mtime, ctime, nlink and generation attributes. Multiple concurrent modifications to a directory are aggregated into a single attribute update.
soft	same as weak, however, reading of directory attributes will take into account pending attribute updates.

Read-write exported NFS doors SHOULD run with strong consistency or soft consistency to maintain POSIX compliance. Read-only NFS doors might run with weak consistency if non-up-to-date directory attributes can be tolerated, for example, when accessing existing data, or soft consistency, if up-to-date information is desired, typically when seeking newly arrived files through other doors.

Frontend

Aside from bug fixes, the following changes should be noted:

• Support for .well-known/security.txt was added to both the frontend and WebDav ports.
• More detailed description of request objects for bulk and stage.
• Improved error messages in several places.
• Support for relative paths and symlink prefix resolution for bulk and namespace resources.
• The frontend.wellknown paths have been deprecated in favor of dcache.wellknown.
• Authz checks have been removed in Quota GET methods.
• Use of RolePrincipal (see under gPlazma) replaces reliance on LoginAttributes and the old admin role (special gid) as defined by the roles plugin.
• Support for QoS Rule Engine policies. A new qos-policy resource allows one to add, remove, list and retrieve policy definitions. See the Swagger pages for details.
• An -optional query parameter was added to the namespace resource to retrieve extra information about a file; the new QoS Policy file attributes (QOS_POLICY and QOS_STATE) are included with this option.
• Support for pool migration has been introduced (migrations resource). See Swagger pages for details.

gplazma

A RolePrincipal has been added to support the use of role definitions in the multimap file. Currently the available roles are: admin, qos-user and qos-group. The second allows the user to transition files owned by the user’s uid; the third allows the user to transition files whose group is the user’s primary gid. The two qos roles can be combined. Admin grants full admin privileges. A multimap example:

dn:"/DC=org/DC=cilogon/C=US/O=Fermi National Accelerator Laboratory/OU=People/CN=Al Rossi/CN=UID:arossi" username:arossi  uid:8773  gid:1530,true roles:admin

These roles do not depend on the presence of the roles plugin. Since the frontend has been changed to use the new RolePrincipal and not the old role definitions (where a special gid must be defined), one can easily drop that plugin from the gplamza.conf file.

There was also an important fix for a bug that was preventing upload (e.g., xroot POSC) when using tokens.

The gplazma-xacml plugin has been dropped.

NFS

Prior to version 9.2 dCache, to support RHEL6 based clients, if no species export options are specified, the NFSv4.1 door were publishing only nfs4_1_files layout. Now on the door publishes all available layout types. If for whatever reason RHEL6 clients are still used, the old behavior can be enforced by lt=nfsv4_1_files export option.

PNFS Manager

To remove unused directory tags chimera keeps the reference count (nlink) of tags. This approach creates a ‘hot’ record that serializes all updates to a given top-level tag. Starting 9.2 dCache doesn’t rely on ref count anymore and uses conditional DELETE, which should improve the concurrent directory creation/deletion rate.

Pool

The mover ls command is updated to display with the -u option the subject associated with the mover.

Added pool.mover.https.port.min and pool.mover.https.port.max to control TPC port number used by HTTPS mover.

Pools info command displays HTTP, HTTPS, NFS and XROOT movers listen ports and interface.

Billing information from DCAP and FTP movers now on include local socket end point

Poolmanager

Pnfsid and path options have been added to ac psu match.

Re-introduce wrandom partition type - a weighted-random pool selection partition, which works as a WASS partition, but ignores LRU metric and number of movers.

QoS

Aside from bug fixes, the following significant changes to QoS should be noted:

• QoS was made to support migration using a new pool mode, “DRAINING”; please consult the book chapter for further details.

• The DB namespace endpoint can now be configured to be separate from the main Chimera database (originally introduced/changed for Resilience). In this way, the scanner, whose namespace queries are read-only, could be pointed at a database replica.

  This remains possible for QoS, even though the QoSEngine now also is responsible for updating Chimera with file policy state. These writes are all done via messaging (PnfsHandler) rather than by direct DB connection, so they go to the master Chimera instance.

• Requests for QoS transitions are now authorized on the basis of role (see under gPlazma).

• The first version of the QoS Rule Engine has been added. With this, one can define a QoS policy to apply to files either through a directory tag or via a requested transition; the engine tracks the necessary changes in state over time. A new database table has been added to the qos database. Remember that if you are deploying QoS for the first time, you need to create the database:

  createdb -U <user> qos

  Fuller explanation of the rule/policy engine is forthcoming in the Book chapter.

• In conformity with the new rule engine changes, the scanner has been modified in terms of how it runs scans. There are now two types of system scans, the (NEARLINE) QoS scans (to ensure that files with a policy requiring them to be flushed have indeed been written to tape) and a system-wide ONLINE scan (there are two versions of this and an option to choose which one). Details in the Book; also refer to the relevant admin commands.

Note that the singleton QoS service (where all four components are plugged into each other directly) is no longer available; the four services can, however, still be run together or in separate domains, as with any dCache cell.

Resilience

Resilience is still available in 9.2, but should be considered as superseded by the QoS services. We encourage you to switch to the latter as soon as is feasible. Remember not to run Resilience and QoS simultaneously.

A recipe has been added to the cookbook for migration/draining of resilient pools. This continues to apply in QoS.

WebDAV

The WebDAV door now provides limited support for metalink format. Metalink is an XML format, described by RFC 5854, that describes how to download multiple files. The initial support is limited to describing the files within the targeted directory: there is no support for recursion. The metalink information is available via HTTP Content Negotiation (client sends a Accept: application/metalink4+xml header) or via a Link HTTP response header (see RFC 6249) when generating the normal HTML output.

XRootD

Aside from bug fixes and various documentation additions and clarifications, the following should be noted:

• Proxying through the xroot door is now available.
• Relative paths are supported in the xroot URL.
• Resolution of symlinks in path prefixes and paths is supported.
• The efficiency of the stat list (ls -l) has been greatly improved.

An Xroot User Guide page has been started.

Changelog from 9.1.0 to 9.2.0

95f53d8478

[maven-release-plugin] prepare for next development iteration

86ab6a2f5d

[maven-release-plugin] prepare release 9.2.0

8d0d926ab1

gplazma: oidc update explicit AuthZ parsing

11ceff7a92

gplazma: oidc increase cache duration for OP public key material

b640b5e92c

dcache-bulk: container rewrite to optimize threading

b2b76e9e15

dcache-bulk: add admin command and query to reset all requests with failed targets

74bab6447e

dcache-bulk: add convenience admin command for state counts

12a1fbf5f4

dcache-qos: fix scanner operation completion logic

28b0bebc2d

ci: by-pass docker.io for bitnami charts

ab16db9102

dcache-bulk: implement HA

1378259897

dcache-cli: convert IllegalArgumentException to CommandException on call()

2507eea428

dcache-bulk: do not PIN or STAGE files with AL ONLINE

a29351aae1

dcache-bulk: only set request status to QUEUED when permissions and targets are all inserted

8479f44e1e

Create test.txt

d23083495f

[maven-release-plugin] prepare branch @{releaseLabel}

707cf4f1cb

pool: on failed upload set file size to zero for space reporting

afe112ed48

dcache-bulk: guard against erroneous argument names

bf9bb6c5e3

dcache-qos: remove entry from rule engine table if file not found or other namespace exception

34b411557f

dcache-frontend: include new QoS Policy attributes in -optional

805735ff23

book: remove v6.2 references in HA chapter

449651ff22

poolmanager: fix wrandom partition incompatibilities

fb04d1c990

Revert “poolmanager: remove wrandom partition type”

3a33ae774c

Add a correct check for the existance of the specified link, poolgroup or HSM when using Migration via REST.

0fea8a858f

docs: gplazma document oidc plugin

67508e933b

pnfsmanager: inroduce limit on number of concurrent listing of the same directory

c1e3728412

packages: fix system test populate sed expression

2a01b13aa3

dcache-webdav: revert improve efficiency of directory listing (14085/14088)

29f340d3c0

dcache-qos,system-test: override db property to hsqldb

8ee98957db

ci: collect billing records from kafka

a518359f78

doc: clean up the multiple protocol configuration explanation for xroot

1266bb7372

dcache-bulk: fix warning for premature stop of job container

17453da77a

dcache (qos): move required attributes to Transfer.readNamespaceAttributesAsync

a1140fceff

pool: include local endpoint for ftp transfers

effabfe737

pool: report dcap local endpoint

52602da836

ci: pin all jobs to dcache-dev runners (temporary)

a8de5478b5

dcache-frontend,dcache-bulk,dcache-qos: regularize observance of the admin role

49cf4062a9

dcache-bulk: handle InterruptedException in DirListTask

41f34d8b43

ci: trace node where gitlab agent is running

af134b683e

ci: use egi-trust repo

560dce66e4

ci: use almalinux9-minimal images for rpm signing

3567dde692

dcache-webdav: fix incorrect parameter value given to DcacheDirectoryResource constructor

253859a532

dcache-qos: drop subject from qos_operation table (qos engine 9)

8cb60b96cd

dcache-frontend: add support for qos policies (qos rule engine 8)

13580d972c

dcache-bulk: modify qos update activity to support policies (qos rule engine 7)

0ebfa0553b

dcache-qos: add policy support to scanner (qos rule engine 6)

710ee615e2

dcache-qos: throttle verifier requests to engine (qos rule engine 5)

77bd7a1fb9

dcache-qos: add support for policy handling in qos-engine (qos rule engine 4)

e2a1f371eb

dcache,pnfs: implement qos policy support (qos engine 3)

e3d67b8c30

chimera,dcache-chimera: add support for inode policy info (qos engine 2)

21af809d58

chimera: implement qos policy storage (qos engine 1)

1f79ce55ce

webdav: add metalink support

707000c026

dcache-webdav: improve efficiency of directory listing

7e691a4f53

dcache-frontend,dcache-webdav: use RolePrincipal instead of LoginAttributes roles

7609229430

common,gplazma-multimap,dcache-qos: qos role authorization, revised

fe522a5409

ci: enable srm tests

603ab40ccc

ci: run junit test during rpm build

ec1ff2b5f4

ci: add srmls tests

2a40be361c

ci: report gridtest test results

2fb9170283

chimera: update FsSqlDriver#inodeOf to throw exception if file not found

c425a62257

ci: add first robot test

09e8e32390

dcache-qos: rework verifier operation handling so most of it is in memory (as in resilience)

8c92f2c413

common: fix NPE in AbstractUidPrincipal#equals

2b83b2c135

ci: upload_rpm must match all other upload rules

410762eb55

common,gplazma-multimap,dcache-qos: authorize qos transition based on role

de47e03e3a

dcache-qos,dcache-bulk: allow qos update to call engine asynchronously

4af8e26ce0

skel,logback: make the .resilience and .qos log files singletons

87dc1a4ea5

build(deps): bump org.springframework.kafka:spring-kafka

6643935656

pool: remove hidden defaults for rh/rm/sh operations

0dde36e585

ci: use CI_REGISTRY_IMAGE variable to address container image

99bfff3c52

dcache-bulk: adjust semaphore permits to something more reasonable

b5e8dd5e3f

ci: add xroot based tests

9e88a1ae07

ci: better readablity and only logic based refactoring

841d53731c

dcache-bulk: reconfigure activity providers to capture environment

a32aa038a6

rpm: add required packages

84c2caa61d

docs: fix external link to nfsmapid and DNS TXT Records docs

4071315e36

docs: add dcache door generic workflow

21f441b4ee

Revert “ci: pass extra pgk and rpm to dcache container”

1398b38eef

dcache-bulk: cancel all stored targets on abort

ef26f40e9a

dcache-bulk: implement periodic archiving of old completed requests (part 2)

e36714d42a

CI: Inheritance for Kubernetes

d5c7c9d3c6

ci: fix image labels

ccb6c8d491

ci: add labels to produced containers

f7b8b4de3a

ci: pass extra pgk and rpm to dcache container

e5fdc5d741

ci: fix manual upload logic

8dcb21e0a2

ci: install required OS tools for dcache container

9cc8f8a376

ci: add workaround installation of shadow-utils

a62142e6f9

pool: Save hsm load provider to setup file

02e0ef6028

ci: use dependencies instread of command not found: needs

1de87b4131

remove extra listing of workspace

e8d7e9fc10

ci: use pynfs–0.4 container

b396a798ab

ci: collect logs before cleanup

e007ede840

dcache-bulk: implement periodic archiving of old completed requests (part 1)

a29d8f4f08

dcache-xroot: improve efficiency of stat list (ls -l)

83e5974301

dcache-qos: remove singleton service configuration

d09c80074a

dcache-bulk: remove in memory running state counts

c0021411ec

pom.xml: upgrade to xrootd4j 4.5.8

ab7c3b1a4e

ci: install shadow-utils before other packages

4f7f78160f

ci: add possibility to manually publish non tagged packages

27d348c711

ci: almalinux:9-minimal as based image with java17

a565601190

common: modify the way the flag on isRestricted works

aa0ce5b75e

ci: add publishing to github

e1a450dc80

ci: initialize worker node

f9622f2fb2

ci: deply test WN with shared cvmfs

801358fc31

pool: show the listen port of nfs mover with admin info command

2cbd89fdf9

pool: introduce property for https mover port range

dab3637221

pnfsmanager: make list scheduling behavior optional (selectable)

a848b94a1c

chimera: rename FsSqlDriver#removeTag into removeAllTags

77c2dd60ce

chimera: fix commit 8fe67b2046

8fe67b2046

chimera: introduce attribute to control attribute update consistency

f56ac72e4b

Fixed string name in deserializer

cdcb84ae3a

Revert “Revert ”dcache-common: add QoS Json policy objects“”

fa9ffb95cf

Revert “dcache-common: add QoS Json policy objects”

ea28525852

dcache-common: add QoS Json policy objects

ff857831df

scripts: update default heap and mempry sizes

b9dd2f410e

ci: fix vesion of kafka helm chart (v2)

13de8e7a1a

ci: fix vesion of kafka helm chart

7cbee56df8

allow action only main repo

d9ed86fcde

docs: more pnfs flow diagrams

5eb4e70418

docs: more pnfs flow diagrams

dbd85ee88c

docs: add pnfs flow diagram

cdc47c7287

github: add gitlab mirroring action

f34a0cb4fa

dcache-gplazma: add serialVersionUID to MultiTargetedRestriction

ff86d14551

chimera: drop nlink ref count for directory tags

2acfaf8d14

dcache-bulk: allow multi-threaded directory listing on expandDirectories

33a37d6774

fix bash-completion

4be34ff6c3

pool: handle double remove of IdleStateHandler

633d1333ed

nfs: expose dcache version in EXCHANGE_ID operation

572f76cf90

dcache-qos: fix bug in storing and retrieving ‘tried’ value for verify operation

9f6910ece9

dcache-bulk: deprecate “prestore” option and remove related container

5f5a5a96f8

libs: use nfs4j–0.25.x

3fb66514d9

pool: add option to display user subject with mover ls

c70248cc26

ci: fix negation of disabled tests

d767580166

ci: disable unsupported pynfs tests

587ab61da1

ci: envorce nfsv4.2 for pynfs tests

3c1cd449f1

ci: use latest pynfs test set

ae44287d91

ci: fix exclude/include for pynfs tests

d73359de02

ci: set pynfs test error code according to test results

25f6b1ea32

ci: fail build if pynfs tests are failing

17e8f603a1

Fix Migration Resources

6e9e80c1da

ci: remove obsolete kybernetes deployment script

2970ebaa37

ci: use helm based dcache deployment

ad253a69ef

dcache-xroot: modify mkdir to ignore dir exists on make parent option

9e1955734a

book: fix rpm download links

108f2f5a3a

Reformat code in HsmSet.java

1b7e07c08e

ci: add pynfs test into pipeline

88109a7d74

ci: remove unused pipeline job

0edcfb12d0

rpm: depend on java–11-headless

b406b78bdb

dcache-vehicles: fix NPE in resolve symlink message

421e1dc932

nfs: fix race condition of LAYOUTRETURN and LAYOUTGET

1e54a504f9

xrootd: throw FileNotFoundException if vomsdir doesn’t exist

a3ca140698

ci: deploy in kubernetes

a9ebbc044f

dcache-qos: fix adjuster task cloning to reset state

31346a795b

dcache-qos: improve exception reported on aborted verification

fd38a52850

ci: prepare k8s environment for deploy and test

1ae82cc59c

pool: remote http movers to provide local endpoint info

fbdc448493

ci: add gitlab-ci.yml

62e945c4e7

docs: add DOI lable/badge

7f6675d7db

gplazma-roles: add QoS role support

ae83a08319

dcache-http: add http request header for role assertion

8de990a241

dcache-frontend: add symlink resolution to /api/v1/id GET and /api/v1/namespace

024b1cf5c8

dcache-frontend: remove authz checks in Quota GET methods

98ad434afb

dcache-qos: remove trigger to rescan pools on tag change

0d682c68de

Revert “dcache-qos: remove trigger to rescan pools on tag change”

825c40d4a0

pom.xml: bump xrootd4j to next version (4.5.7, 4.4.8, 4.3.9, 4.2.13)

adda59a455

dcache-qos: remove trigger to rescan pools on tag change

ad2632019d

several: remove non-ASCII dash occurrences

9eae61bbac

dcache,pnfs: skip symlink resolution when checking restrictions on directory children during listing

19e5341e58

dcache-qos: repair faulty queue refresh algorithm in verifier

6dde34e643

dcache-qos(verifier,engine): fix incompatibility issues with Subject and attributes

34adbb0828

skel,dcache-frontend,dcache-webdav: use dcache.well-known for all doors

d85ec22dff

dcache-bulk: catch Exception from getSubject()

d8e1826672

book: update config-message-passing.md

5217a1da4a

build(deps): bump netty-handler from 4.1.92.Final to 4.1.94.Final

9d4c75a102

Docs: frontend.wellknown!wlcg-tape-rest-api.path also applies to WebDAV doors

93ed1709f4

[maven-release-plugin] prepare for next development iteration

80f485f13b

build(deps): bump guava from 31.1-jre to 32.0.0-jre