Highlights

• Optional TLS for p2p transfers
• Zone-aware cell messaging
• New serialization format

Incompatibilities

Starting release 6.1 the BerkeleyDBMetaDataRepository is the default metadata repository for the pools. Sites that was not explicitly specifying pool.plugins.meta property have to set the old default value: pool.plugins.meta=org.dcache.pool.repository.meta.file.FileMetaDataRepository

Pool in 6.1.x are not compatible with older dcap doors. However, new dcap doors can operate with existing pools.

Acknowledgments

(tbd, this section uses markdown formatting)

Release 6.1.36

acl

The group ACEs now apply to desired group instead of to a users with the same numeric id.

Changelog 6.1.35..6.1.36

e320f183a5

[maven-release-plugin] prepare release 6.1.36

0ce852a52d

acl: set IDENTIFIER_GROUP flag if WHO is GROUP or OWNER_GROUP

2f9ceb031c

[maven-release-plugin] prepare for next development iteration

Release 6.1.35

frontend

SRR returns data now with official json schema

webdav

Fix how sym-links are shown in the static HTML (web-browser) view from the WebDAV door.

Changelog 6.1.34..6.1.35

ac22afcd25

[maven-release-plugin] prepare release 6.1.35

6fa435b89e

frontend: add service to provide Storage resource reporting

df65e59b1b

webdav: update representation of symbolic links in HTML page

be4983de53

[maven-release-plugin] prepare for next development iteration

Release 6.1.34

dcache-xroot

The next release of xrootd4j is used now, which improves handling TPC read requests.

Changelog 6.1.33..6.1.34

60f18a8374

[maven-release-plugin] prepare release 6.1.34

22502ed813

dcache-xroot: bump dependency to next xrootd4j release

fb559b44ed

[maven-release-plugin] prepare for next development iteration

Release 6.1.33

srmmanager

A race condition is fixed that, if triggered, results in a memory leak. This leak can also affect the TURLs returned by SrmManager, where out-of-date information about doors is used.

Changelog 6.1.32..6.1.33

284f32fbd6

[maven-release-plugin] prepare release 6.1.33

8412e3563f

srmmanager: fix race condition in LoginBrokerSubscriber

1b33c2f3e4

[maven-release-plugin] prepare for next development iteration

Release 6.1.32

alarms

Clearer documentation for billing/alarms services regarding HA requirements.

skel

Whether a cell/service supports HA or not can now be derived from the properties file (except for doors).

Please see <name>.cell.replicable.

srm

The access log file for SRM requests now includes the client-supplied list of protocols, if any were provided.

Changelog 6.1.31..6.1.32

0ed78900af

[maven-release-plugin] prepare release 6.1.32

544331a36f

alarms, billing: specify shared rdbms in the case of replicated service

c106c3ea65

skel: add explicit replicable property even when value is false

bdbcfe7e4f

srm: log transfer protocols in access log

7fe8ab1d71

[maven-release-plugin] prepare for next development iteration

Release 6.1.31

dcache-core

The old web pages claim the transfer size and speed are reported in KB and KB/s, although the actually used units are KiB and KiB/s. Reporting the correct unit is important for monitoring. This is now fixed.

gplazma

dCache may now be configured so that the multimap oidc predicates match the sub claim value from a specific OAuth2 Provider using the format SUBVALUE@OP, where OP is the dCache-internal alias for the OP. Existing multimap configuration continues to work, but admin is warned to update the multimap configuration.

Changelog 6.1.30..6.1.31

22f5f8931c

[maven-release-plugin] prepare release 6.1.31

2d307b91cd

gplazma: multimap now supports OP in ‘oidc’ predicate

9295e69c52

dcache-core: correct reported units of transfer size and speed

64cbad4033

[maven-release-plugin] prepare for next development iteration

Release 6.1.30

xroot

The current release fixed the parsing to account for host/error pairs.

Upload transfer checksum failure is fixed and the gfal scenario now works.

Changelog 6.1.29..6.1.30

06c51b2822

[maven-release-plugin] prepare release 6.1.30

d981ec1a0b

dcache-xroot: parsing of ‘triedrc’ CGI lacks separation by optional comma

6b2efa35f1

dcache-xroot: fix upload transfer checksum failure (stable branches)

13d94929b0

[maven-release-plugin] prepare for next development iteration

Release 6.1.29

ftp

Globus transfer agent is now able to create symbolic links.

pool

dCacheView now works for redirected transfers for non-anonymous data access, provided the WebDAV door is using TLS encryption and is configured with ‘webdav.redirect.allow-https’ set to ‘true’.

poolmanager

A very rare race-condition is fixed that could be triggered if a pool completes a stage or pool-to-pool request (to satisfy a client’s read request) immediately before dying.

webdav

dCacheView is able to view and download files when authenticated and when the WebDAV door proxies the transfer.

Changelog 6.1.28..6.1.29

a1d13d4c26

[maven-release-plugin] prepare release 6.1.29

0fe2340945

webdav: use Jetty’s built-in support for CORS

9eff7ea1ed

poolmanager: fix _waitingFor access outside of synchronized block

a724dcd897

ftp: add support for the SITE SYMLINK command

44ba3ac891

pool: http allow client to send credentials when TLS is used

29ecae497c

[maven-release-plugin] prepare for next development iteration

Release 6.1.28

frontend

frontend/api readers and writers always zero issue is fixed now

gplazma

The LDAP plugin behaviuor now more closely follows that of other mapping plugins. This allows deployments where LDAP is tried first and, if that fails to identify the user, fall-back strategies are used.

Changelog 6.1.27..6.1.28

111a1acf66

[maven-release-plugin] prepare release 6.1.28

ba7873346e

gplazma: ldap throw exception if no principal is added

19d4f66382

poolV4: fix reader/writer counts

ecdf313739

[maven-release-plugin] prepare for next development iteration

Release 6.1.27

dcache-xroot

The cirrent release fixed memory leak, race and other small issues with reconnect to pool.

frontend

The dcache-view.endpoints.webdav property now overrides any auto-discovered WebDAV endpoint, making explicit configuration easier.

pnfsmanager

A regression is fixed the prevents creating a symbolic link.

Changelog 6.1.26..6.1.27

3082ef649a

[maven-release-plugin] prepare release 6.1.27

0ee97b506b

frontend: bump version of dCacheView

6559fcd58b

pnfsmanager: fix NPE regression from 3dfed7e8b0

09f09d99bc

dcache-xroot: fix memory leak, race and other small issues with reconnect to pool

cdd91e3a52

frontend: bump dCacheView to v1.6.2

cc1948c443

[maven-release-plugin] prepare for next development iteration

Release 6.1.26

poolmanager

When poolmanager requests that a pool stage a file, the pool may return the special error code HSM_DELAY_ERROR (10013). On receiving this error, poolmanager suspends the request.

This is now fixed.

xrootd

xrootd4j is updated to 3.5.10.

Changelog 6.1.25..6.1.26

8d04f449c1

[maven-release-plugin] prepare release 6.1.26

41dcbb8ea3

dcache-xrootd: update xrootd4j to 3.5.10

cf3ecd8722

poolmanager: fix NPE if pool delays stage without giving a reason

04efa36e50

[maven-release-plugin] prepare for next development iteration

Release 6.1.25

xrootd

Now the door will ignore xrootd.enable.tried-hosts property if property false instead of returning errror.

Changelog 6.1.24..6.1.25

653a926a08

[maven-release-plugin] prepare release 6.1.25

f17e25d5ff

dcache-xroot: (door) ignore tried if property false instead of returning error

0a4749af1c

Update install.md

43c3fe2cab

[maven-release-plugin] prepare for next development iteration

Release 6.1.25

xrootd

Now the door will ignore xrootd.enable.tried-hosts property if property false instead of returning errror.

Changelog 6.1.24..6.1.25

653a926a08

[maven-release-plugin] prepare release 6.1.25

f17e25d5ff

dcache-xroot: (door) ignore tried if property false instead of returning error

0a4749af1c

Update install.md

43c3fe2cab

[maven-release-plugin] prepare for next development iteration

Release 6.1.24

Changes affecting multiple services

Sometimes file checksums are missing in namespace. Transfer requiring checksum verification fails then with an error saying that “No checksums found”, but doesn’t log a stack trace server side anymore.

dcache-xroot

During periods of heavy usage on a pool node, if I/O stalls during read, the xroot client may try to reconnect and continue reading. This release fixes a bug which fails the transfer on the attempted reconnect with a “uuid no longer valid” error (because the file mover has already been closed and removed). Clients should now be able to resume reading the file on retry.

Changelog 6.1.23..6.1.24

a39c0de3f7

[maven-release-plugin] prepare release 6.1.24

da13dda0bc

dcache-xroot: Allow client to reattempt open on pool when I/O stalls

5f6015d700

check if checksums are present and throw CacheException if they are not

bd03a8a23e

[maven-release-plugin] prepare for next development iteration

Release 6.1.23

admin

The current release fixed doors login dump cache admin command to show the information sent by the door along with the login result.

common

Printing of bearer tokens is improved to provide better assurance that bearer tokens are not leaked.

gplazma

All login failures are now recoreded in pinboard as a single-line summary, explaining why the login attempt failed.

webdav

The current release fixed a regression where dCache would complain about badly formatted IPv6 addresses, preventing those addresses from being included in the access log files.

The current release fixed a regression where dCache would complain about badly formatted IPv6 addresses, preventing those addresses from being included in the access log files.

xroot

The current relase updated xrootd4j to 4.0.5 which fixes improper use of destination token when contacting source server during third-pary transfer.

Changelog 6.1.22..6.1.23

687de75ea5

[maven-release-plugin] prepare release 6.1.23

93d5d18b74

dcache-xroot: update to xrootd4j–4.0.5

e32de4e2c0

webdav/frontend: fix IPv6 address logging

5ff2e7c750

common: update how BearerToken is shown in Subjects#toString

8067df53f4

authn: add missing hashCode and equals methods for PasswordCredential

5e73199991

authn: fix “login dump cache” admin command output

911ab10158

gplazma: provide single-line summary in pinboard for each login failure

250748b310

authn: log failures in UnionLoginStrategy

04f3749dd0

authn: add missing hashCode and equals methods for BearerTokenCredential

29f094c523

common: Update BearerToken representation to include some information

53f362cf17

[maven-release-plugin] prepare for next development iteration

Release 6.1.22

webdav

Restarting transfermanager while there are ongoing HTTP-TPC transfers no longer results in an endless loop, with transfermanager logging a message every ~5 seconds per ongoing HTTP-TPC transfer.

Changelog 6.1.21..6.1.22

b44f6b5668

[maven-release-plugin] prepare release 6.1.22

5c5a996be9

webdav-transfermanager: fail gracefully if transfermanager restarted

704d9b5ccd

[maven-release-plugin] prepare for next development iteration

Release 6.1.21

biiling

The current relase fixed unit info for bandwidth measurements in billing.properties (bytes/sec, not MiB/sec).

frontend

Frontend now logs any time it returns 500 Internal Server Error to the client.

dcache-view Previously displayed MB/sec for transfer rate, but the value is the old KB/sec. This is now improved and pages display the values advertised.

pool

The current release assures now that checksum scanner wiill mark files broken and sends alarm.

webdav

An HTTP-TPC transfer will abort automatically if transfer-manager is restarted, without waiting for the client (FTS) to time-out.

Changelog 6.1.20..6.1.21

963dc9a352

[maven-release-plugin] prepare release 6.1.21

ccf4760d4b

dcache-pool: Assure checksum scanner marks files broken and sends alarm

57f9e32948

webdav/transfermanager: fail gracefully on restart, for perf marker

99654d07e1

default: fix unit info for bandwidth measurements in billing.properties

cb742ca1a9

frontend: create a log entry when we return 500 Internal Server Error

b95b867193

dcache-frontend: change transfer rate to compute what it advertised

c8a36e7b27

[maven-release-plugin] prepare for next development iteration

Release 6.1.20

book

A new section is added to the xrootd chapter explaining the need for SAN extensions on the hostcert in the case of hostname aliasing.

ftp

Globus transfer service can now list directories that contain incomplete files, those that are still being uploaded.

The FTP no longer leaking sockets when dynamically calculating file checksums (e.g., MD5). In practise, this issue only affects transfers via the Globus transfer service.

The current release fixed Globus transfer service directory listing for medium size directories.

The current release fixed Globus transfers hanging when verify file integrity after transfer is enabled and the pool is not configured to calculate an MD5 checksum.

Honour the ftp.net.internal configuration property for dynamic checksum calculation.

Changelog 6.1.19..6.1.20

fbd737286b

[maven-release-plugin] prepare release 6.1.20

656a324c27

ftp: do not leak sockets when calculating checksums dynamically

095b02d305

ftp: add work-around to calculate MD5 for Globus

98468678f7

ftp: add place-holder file size work-around for Globus directory listing

2783011bd3

rpm take build directory path into quotes

bfb3454e7b

ftp: fix MLSC support for medium-length directories

fc7564fb5a

ftp: fix on-the-fly checksum calculation

79f4c75126

book: document host alias issue with xroot TPC

ad4f26096e

[maven-release-plugin] prepare for next development iteration

Release 6.1.19

pool

The current release fix a problem where, under heavy load, redirected https transfers result in garbled SSL information, preventing the HTTP client from working.

services

Recovery processe in the case where a pool’s message queue is overloaded, as old (and now useless) requests may be simply discarded is improved.

webdav

There were reports of the access log file missing entries. This problem seems to be triggered if the client disconnects before dCache has finished processing the request, which is certainly true for cancelled HTTP-TPC transfers. This is now fixed.

xrootd

The xroot protocol access log entries for the xrootd door and pool have been enhanced to include information about the kXR_query requests.

Changelog 6.1.18..6.1.19

dfdaadc123

[maven-release-plugin] prepare release 6.1.19

6c2d8c9918

prepare for next development iteration

d261fe34cf

TransferManager: choose better pool query message TTL

bbae4d1252

webdav/frontend: ensure cancelled HTTP-TPC transfers are logged

d44f28bdc1

xrootd/pool: improve access log to record more details of kXR_query

8336e2cc85

pool: https avoid sharing SSLEngine between TCP connections

Release 6.1.18

The current release fixes EGI-SVG–2020–16939 vulnerabilities.

Release 6.1.17

xrootd

The xrootd4j version was updated to 3.5.8.

Changelog 6.1.16..6.1.17

fae8870df5

[maven-release-plugin] prepare release 6.1.17

23bf788f20

pom: update xrootd4j to 3.5.8

8c204ac235

[maven-release-plugin] prepare for next development iteration

Release 6.1.16

pool

The mover thread is not interrupted outside of IO activity to avoid disabling of repository.

HTTP-TPC transfers where dCache pushes data to an Apache server (for example as used by Dynafed) will now work for zero-length files.

webdav

HTTPS-PUT requests are now possible.

Changelog 6.1.15..6.1.16

bf73ac6530

[maven-release-plugin] prepare release 6.1.16

7c34f736ee

pool: http-tpc do not loop if HEAD ‘Content-Length’ response is missing

1a1975ad0e

webdav:HTTPS PUT req issue

279af5f5aa

pool: preserve CDC on p2p transfer

341de58525

pool: http-tpc drop ‘Accept-Encoding’ on HEAD requests

ab462b9a3d

pool: detach IO thread when mover finished

5e05bccba8

[maven-release-plugin] prepare for next development iteration

Release 6.1.15

pool

dCache pools now support GSI-based xrootd-TPC by default.

Changelog 6.1.14..6.1.15

1f61af5473

[maven-release-plugin] prepare release 6.1.15

b379cdefa4

pool: update xrootd-tpc authenication default

09ddc0fd90

[maven-release-plugin] prepare for next development iteration

Release 6.1.14

enstore

Enstore storage info provides works with URI based tape locations now.

pool

The HTTP-TPC PUSH requests (which use the HTTP PUT request to send data) are now more robust against slow remote servers that redirect the transfer.

The current release fixed NullPointerException bug in MongoDB-backed pool metadata storage and SpotBugs NORMAL-level warning.

With the chenges introduced in current release, when using HTTP-TPC, the timeouts dCache pool uses when requesting information from the remote server have been adjusted, based on operational experience. In general, dCache will now wait longer for the remote server to complete any post-processing before giving up and failing the transfer.

Changelog 6.1.13..6.1.14

2f5383a4e3

[maven-release-plugin] prepare release 6.1.14

5dcb95190c

chimera: update EnstoreStorageInfoExtractor to use FileState

bdb2a822f2

chimera: use UTF–8 constant instead of string in ChimeraEnstoreStorageInfoExtractor

f212438e29

enstore: populate the correct storage info information on set

b834910bbe

pool: http-tpc adjust GET and HEAD request timeouts

8d505d8c97

pool: http-tpc PUT request are repeatable

7ed0edb4e2

pool: support removing MongoDB storage-info entry

d8ba33ff6c

[maven-release-plugin] prepare for next development iteration

Release 6.1.13

dcap

Sometimes file restores on dCap door were hanging. This is now fixed.

Changelog 6.1.12..6.1.13

cd92673826

[maven-release-plugin] prepare release 6.1.13

c318c7b31f

[maven-release-plugin] prepare for next development iteration

7cb32b4ff5

dcap: fix automatic door retry on transient errors

Release 6.1.12

gplazma

X.509 certificates that use the “SHA–384 with RSA” algorithm are now logged with this name instead of the raw OID value.

httpd

The current release fix pool usage color scheme css used for UsageInto and PoolInfo pages.

pool

dCache pools that are undertaking HTTP-TPC transfers will now wait longer for the remote servers to complete any post-transfer activity. Transfers that involve heavily loaded remote servers are now more likely to succeed.

Changelog 6.1.11..6.1.12

a7fe40d85b

[maven-release-plugin] prepare release 6.1.12

ae6616d5db

pool: http-tpc increase timeout waiting for remote server post-processing

39952d5fbc

gplazma: include additional algorithm in LoginResultPrinter

bc2be58a21

httpd: fix pool usage color scheme css

d85bdc4d42

correct snapshot version for logback-test-config

a73c8d4074

correct snapshot version

Release 6.1.11

nfs

The current release fixed previously observed infinite WRITE+COMMIT loop.

Changelog 6.1.10..6.1.11

0d59a8a090

[maven-release-plugin] prepare release 6.1.11

0aae861a17

nfs-proxy: op WRITE must use the same verifier as COMMIT

9fd3c6d73a

[maven-release-plugin] prepare for next development iteration

Release 6.1.10

dcache

There have been several occasions where in a dCache instance using an embedded ZooKeeper server such HA services stopped working because an ephemeral node was elected leader that has no associated service anymore; it should have been deleted. This state might go undetected for a longer time, and removing such stale nodes in order for the services to continue functioning as intended requires manual intervention.

This is fixed now and the functioning of HA services using ZooKeeper-based leader elections is more reliable.

resilience

A more robust handling of PSU changes which remove pool, unit or group mappings was introduced.

Recently introduced improvements changed the way resilience handles file “removal” (no longer setting the repository entry to ‘removed’ but simply by caching the replica). This change, however, did not take into account the handling of broken files. Encountering a broken file, it would indiscriminately attempt to remove it, whether it was cached or not; this was leading to an infinite loop, with the file operation continuously iterating without doing any further work. This situation could potentially hang the pool scans (if there are as many broken files as there are scan threads), and even the file operation queue. The current release fixed this issue and there is no any potential for stalled operations when encountering broken replicas.

Changelog 6.1.9..6.1.10

e42123433e

[maven-release-plugin] prepare release 6.1.10

b70f8fc044

dcache-core: fix embedded ZooKeeper persisting stale ephemeral nodes

5b5e14425b

dcache-resilience: ignore broken cached files

0ba3237d8a

dcache-resilience: protect access of pool info map against NoSuchElementException

1e7bc1129d

[maven-release-plugin] prepare for next development iteration

Release 6.1.9

macaroons

Fixed macaroons validation where in case of multiple combined caveats for some of them only the digital signature was checked.

resilience

If a FileOperation is canceled while its subtask is running a NullPointerException was thrown. This is now fixed.

Changelog 6.1.8..6.1.9

45124a1c28

[maven-release-plugin] prepare release 6.1.9

3c2936e497

dcache-resilience: avoid NPE in file operation cancel

a104e9220a

macaroons: verifyCaveat should return false for unprocessed caveats

432e9508e3

[maven-release-plugin] prepare for next development iteration

Release 6.1.8

cleaner

Database passwords can now be stored in a dedicated file.

pool

Failing HTTP-TPC transfers where the remote party sends a malformed RFC 3230 checksum value are now fixed.

util

The current release fixed pgpass file handling for jdbc URLs with options, like jdbc:postgresql://dbhost:5432/foo?prepareThreshold=3

Changelog 6.1.7..6.1.8

5cab9be888

[maven-release-plugin] prepare release 6.1.8

652cf942d9

pool: do not fail transfer if RFC3230 is badly formed.

bb3f741184

util: fix pgpass support for jdbc urls with options

6bf2e0d1b1

cleaner: add support for db password file

bf6e354aec

[maven-release-plugin] prepare for next development iteration

Release 6.1.7

pool

The current release fixed interrupted p2p transfer issue and the pool should not be disabled.

Changelog 6.1.6..6.1.7

7a0816c

[maven-release-plugin] prepare release 6.1.7

864bcbd

pool: rework interrupt processing on p2p

71584b7

[maven-release-plugin] prepare for next development iteration

Release 6.1.6

common

dCache no longer logs OIDC access tokens or macaroon.

The current release fixed a bug where SRM complains enum constant IGTF_LOA_CEDER does not exist.

dcache

Priviously we have introduced support for the xrootd cgi tried=,, which was an optional feature because this exclusion can put access to the file into an indefinitely suspended state for all clients.

The current relase fixed the latter problem and when ‘tried’ is activated (the only time excluded hosts can be non-empty), failure to read the file will not put the request container into the SUSPENDED state.

dcache-xrootd

The version is upgrated to xrootd4j 3.5.7 with lossen username validation, so that

usernames like foo.1234:56 are no longer rejected.

The current release fixed java.lang.IllegalStateException: ChecksumChannel must not be written to after getChecksums.

TPC client first will be shut down when the pool netty channel goes inactive.

pnfsmanager

The current release fixed a NullPointerException when a file is deleted while a directory listing is being compiled.

pool

When a legacy mover is killed by job timeout manager thread interruption mechanism is used. As mover thread is taken from a thread pool, the next task might see this interrupted state, if not cleared. This state can be checked bu other components and trigger false errors: java.lang.InterruptedException THREAD_INTERRUPTED: InterruptedException.

This is now fixed and pool is not disabled itself after slow running mover is killed by jtm.

Changelog 6.1.4..6.1.6

bad07a9

[maven-release-plugin] prepare release 6.1.6

eb80e6c

pnfsmanager: avoid NPE if file is deleted during directory listing

7def465

common: do not log bearer token values.

9ee0b74

pool: rework interrupt processing in AbstractMoverProtocolTransferService

efd886b

common: fix LoAPrincipal deserialisation

09545db

dcache-xrootd: bump to xrootd4j 3.5.7

68a06d6

pool: clean interrupted state after mover complete (again)

a8c34de

pinmanager: use PoolManagerStub when talking to pool manager

aec2ba0

dcache (pool manager): override suspend when “excluded” (hosts) is not empty’

b5a45cc

pool: clear mover thread’s interrupted state before returning to pool

7071492

Wrap the shutDown in try … catch for InterruptedException.

d91aa82

dcache-xrootd: cancel TPC transfer when client disconnects unexpectedly from pool

7bab3ea

[maven-release-plugin] prepare for next development iteration

01521fe

[maven-release-plugin] prepare release 6.1.5

9e17bb0

pool: repository account capacity correctly if file channel closed

1f5b1ac

dcache-xrootd: make ‘tried’/excluded hosts an optional feature

df80b16

docs: TheBook update index to include the chapter on Kafka

d618231

[maven-release-plugin] prepare for next development iteration

Release 6.1.4

frontend

The frontend’s inotify-over-SSE now honour the ‘frontend.root’ configuration property and any user-specific root. This means that, for any inotify subscription, the path is calculated relative to the doors’ root and the user-specific root. For many dCache sites, this has no impact as they are using the default for both; however, the user root is used to implement the macaroon’s ‘root’ caveat. The current release fixed how inotify subscription requests are processed when made with a macaroon with a ‘root’ caveat.

pool

When making an HTTP third-party copy (HTTP-TPC), dCache no longer sends the ‘Authorization’ HTTP request header in any subsequent request when the remote server responds with a redirection.

The unix xrootd tpc security plugin was included in order to enable the dCache TPC client to use a dCache pool as source when signed hash verification is on. However, this is now fixed and no special configuration necessary for organizations (like Tier 1) needing to communicate with EOS.

Changelog 6.1.3..6.1.4

981eb95

[maven-release-plugin] prepare release 6.1.4

55499a0

pool (xrootd): make tpc security plugin default unix

ff320a7

docs: add missing architecture diagram

e49bc52

frontend: honour door and user root for inotify subscriptions

97838fa

frontend: refactor inotify client-path to dCache-path mapping

28d7d6b

frontend: add SelectionContext for event subscription

16396cf

docs: UserGuide fix missing back-tick

67d57d1

pool: drop Authorization HTTP header on redirected

4bedd32

[maven-release-plugin] prepare for next development iteration

Release 6.1.3

frontend

When frontend is run in a core domain and there is no history service reachable, the retry on no route to cell ends up spamming the message queues. This is now fixed.

login-broker

The lb set update command ignored the provided value and set the existing one again. This is now fixed and lb update times can be modified through admin interface.

xrootd

The current release fixed compatible level security for sigver.

The xrootd client has a command-line option, --path, which tells the server to create missing directories. This option is included in two-party copy, but setting it for TPC has no effect. TPCs which wish to write to dCache (as destination) to a non-existent subdirectory fail. This is now fixed and dCache no longer fails in these cases.

Changelog 6.1.2..6.1.3

00c175b

[maven-release-plugin] prepare release 6.1.3

de8e13a

login-broker: fix ‘lb set update’ command

db91dfd

dcache-xrootd: always create missing directories on write

60dd143

dcache-xrootd: fix compatible level security for sigver

5eb7b0d

dcache-frontend: remove retry flag on sendAndWait to history service

e6ca223

[maven-release-plugin] prepare for next development iteration

Release 6.1.2

nearline-storage

There were issues in pool space allocation/de-allocation in conjunction to HSM connectivity and handling restore errors resulting in failing stage end up with used space miscalculation: pool_read, fault occurred in repository: Internal repository error. Pool restart required:, cause: java.lang.IllegalArgumentException: Cannot set used space to a negative value.

This is now fixed and disk copy is removed if restore from HSM failed.

Changelog 6.1.1..6.1.2

9d24d15

[maven-release-plugin] prepare release 6.1.2

efe932b

nearline-storage: remove disk copy if restore from HSM failed

225cb96

[maven-release-plugin] prepare for next development iteration

Release 6.1.1

dcache

Qos migration policy engine was raising JVM error when no tape pool found. This is now fixed and the normal behavior is that HTTP error code reported back with No HSM pool found.

frontend

The current release improved error responses and they are more specific now.

The current release fixed the NPE stack trace arised because the pool data could be sent with a default sweeper data object.

Changelog 6.1.0..6.1.1

139eca2

[maven-release-plugin] prepare release 6.1.1

2e41f65

dcache-frontend: make ErrorResponseProvider return the more specific error message

8520249

dcache: qos migration policy engine should not raise JVM error when no tape pool found

beedd99

dcache-history,dcache-frontend: guard against unconfigured sweeper histogram

a347604

[maven-release-plugin] prepare for next development iteration

Release 6.1.0

Cells

Starting with release 6.1.0 dCache internal massage routing will prefer zone local cells when possible. This is benefitial in geo-distributed HA setups, where talking to remote cells may introduce additional latency.

A new serializer is available in dCache, which improves the speed of internal message passing between cells. As this feature has not been tested very extensively and it’s possible that sites may experience unforeseen behavior, the new serializer has to be explicitly enabled as:

dcache.broker.channel.msg-payload-serializer = experimental

NOTE: when different cells are not of the same bug-fix release, then dCache will always fall back to using the standard serialization protocol.

Chimera

Removed unneccesary requests to DB on new file creation. Fixed operation with read-only database cponnections.

Frontend

As part of the effort (in progress) to provide enhanced support for QoS, the RESTful QoS transition API and implementation has been revisited. First, the way QoS was being evaluated (counting all pins) has been corrected by using a QoS owned pin. Next, in addition to pinning,
QoS changes now involve modification of the access latency and
retention policy values for the file in the namespace. The frontend notifies PnfsManager of these transitions, and this notification then gets relayed to resilience. Some internal refactoring of the QoS logic out of frontend and into core was also done.

An inefficiency resulting in the double fetch of sweeper data from the pools was fixed.

The restores and transfers admin pages were erroneously requiring user login; this has been eliminated.

For the API enabling and disabling pools, a bug preventing the expression of the flag using a boolean rather than a string, e.g., pools/{name}/usage/mode -d {"rdonly":true} rather than pools/{name}/usage/mode -d {"rdonly":"true"}) has been fixed.

History

Some commands available on the frontend for refreshing and adjusting the timeout of pool data collection were missing from the history service; these have now been made available.

Also addressed were null pointer issues arising from the arrival of incompletely initialized data from the pools; code was refactored to make sure composed objects do not have empty fields; in addition, a few extra checks for missing data are now made.

NFS

The admin command ‘show clients’ is updated to accept address/netmask or pattern as a host argument, for example: admin > show clients 10.1.1.0/24 admin > show clients fe80::eef4:bbff:fed2:f9d0 admin > show clients wn*.dcache.org

PNFS Manager

Notification is now sent when access latency or retention policy changes for a file (in order to support the RESTful transition API and management of QoS by resilience).

Pool

Starting from 6.1.0 the pool to pool transfers can be optionaly configured to use TLS. A new propertry pool.enable.encrypted.p2p-transfers is introduced that accepts three values:

• NEVER: TLS is never used for p2p transfers
• ALWAYS: all p2p transfers aare using TLS
• CROSSZONES: TLS is used only when source and dstination pools are in different zones.

NOTE, that in the current implementation CROSSZONES has the same effect as ALWAYS.

The BerkeleyDBMetaDataRepository is now default format for pools metadata.

An inefficiency in the processing of sweeper data for frontend histogram display,
which could result in CPU spiking on the pool node, has been fixed via an in-memory optimization replacing callouts to the repository store.

The pool transfer service now incorporates both older and newer xrootd version numbers for compatibility (2.7, 4).

Pool Manager

The network hostname of the pool is now made available to the selection unit in order to support the xrootd ‘tried’ query element; a small fix was also added to restore backward compatibility with older pools that do not so advertise their host name to the pool monitor.

The selection algorithm has been modified to support excluding tried hosts.

Resilience

Modifications were made to couple resilience more tightly to QoS transition requests; it can now manage transitions for “resilient” files between tape and disk. This means that it will make replicas for a NEARLINE file which has become ONLINE, and can cache all replicas for an ONLINE file that has become NEARLINE, provided that the file belongs to a storage unit with a non-null ‘-required’ value.

A small bug (null pointer exception) which could occur if a file staged from tape no longer exists on any pool that is part of a resilient poolgroup has been fixed. There were also two minor code improvements involving object comparisons.

XRootD

The dCache XrootD door now observes the xrootd protocol ‘tried=host1,host2,…’ query element.
What this means is that if the xrootd client (xrdcp) should decide to retry a failed transaction by returning to the dCache door, the hosts listed as having been previously tried will be used to exclude potential pools for the transfer. Should no other pools be available, dCache will behave the same as when a file is unavailable (it will wait indefinitely until the client gives up).

Pool and door compatibility based on changed version numbers has now been fixed so that older doors will work with newer pools and vice versa.

Error handling (mapping of error types to codes returned to the xrootd client) was refined to make it more consistent and conform more closely with the xrootd protocol.

The version number indicated by dCache now should reflect the actual xrootd protocol version being used (it was formerly fixed, incorrectly, at 2.7).

The xrootd client is now redirected to the pool using an actual hostname rather than IP address, for the sake of compatibility with RFC 5280 (4.2.1.6.
Subject Alternative Name). This is essential for eventual support of TLS.

Two internal changes in advance of future features (TLS, SciTokens) have been made: the door now uses a caching login strategy, similarly to most of the other dCache doors; it also supports authorization handlers which need to access login strategies (both of these will be necessary for SciTokens).

A bug in the delegation provider requiring that a VOMS directory be defined even if the GSI module is not being used has been fixed.

A bug preventing read paths from being observed by the list command has been fixed.

A bug preventing xrdcp -C <checksum type> from returning the correct checksum when multiple checksums are defined on the pools has been fixed.

Batch/Startup scripts

The batch scripts for the alarms service and xrootd doors were erroneously configured not to exit and restart on fatal errors; if there were unhandled exceptions during startup, these two services would simply hang, making detection of their status (e.g., through external monitoring) difficult. The scripts have been fixed to conform to the exit/restart sequence used by all other services.

Changelog from 6.0.0 to 6.1.0

8520249

dcache: qos migration policy engine should not raise JVM error when no tape pool found

beedd99

dcache-history,dcache-frontend: guard against unconfigured sweeper histogram

a347604

[maven-release-plugin] prepare for next development iteration

f15e40b

[maven-release-plugin] prepare release 6.1.0

3448267

dcache-frontend: allow pool enable/disable to use boolean JSON value

e8a7705

dcache-frontend,history: protect against missing highest bin in histogram data

00883c2

[maven-release-plugin] prepare branch 6.1

8bbae91

vehicles: remove historic field from PoolPassiveIoFileMessage

0903034

build(deps): bump version.netty from 4.1.10.Final to 4.1.45.Final

f308a7f

dcache, frontend: release dcache-view version 1.6.1

41665a1

util: refactor task running to aid code reuse

2c3fd78

dcache: release liquibase locks only if update is requested

4fa3948

libs: remove dependency on jackson-mapper-asl

d9529e4

dcache-xrootd: change URL to use hostname on redirect to pool

7a17096

Revert “libs: remove dependency on jackson-mapper-asl”

aee153f

libs: remove dependency on jackson-mapper-asl

a9268f3

chimera: don’t try to access level2 for newly created files

8e11da1

ftp: avoid NullPointerException when client transfers without ALLO

0186ba2

skel: repair erroneous batch directives before cell creation

3512cdc

dcache-xrootd: don’t initialize delegation provider when there is no gsi module

3185607

dcache-frontend: remove unnecessary login requirement on restores and transfers

c385552

frontend: events inotify avoid holding global lock when sending events

11d21f1

gplazma: scitoken add partial support for WLCG Common JWT Profile scopes

4907261

pool: log when a queue starts to queue movers and when this stops

a48e5e0

nfs: don’t use stream to compute collection size

e73815c

srm: use host IP for comparison when determining if SURL is local

527e8d3

ftp: support zero allo values

bc451a5

dcache: increase metadata wait time for kafka producer config

5366aca

frontend: events inotify fix deadlock

7d78c80

dcache-xrootd: make door use CachingLoginStrategy

15e0c5d

dcache-xrootd: support authz handlers which need to access login strategies

0b5f8f1

dcache, frontend: release dcache-view version 1.6.0

6a87a2b

gplazma: scitoken refactor plugin

eb1b81d

Fix tape-reserved size calculation

dd00230

common: add missing toString method for MultiTargetedRestriction

753712a

gplazma: scitoken add SciTokenPlugin unit test

d02f314

nfs: extend host filter of ‘show clients’ command to accept patterns

4f6ef66

Update config-stage-protection.md

cbffb34

poolmanager: add affinity requirements to PoolManagerMessage

2328947

poolmanager: make RendezvousPoolManagerHandler#backendFor private

bff16d6

webdav: include DAV header in OPTIONS requests.

5ee32d5

pnfsmanager: better explain restriction-based permission problems

f1f1371

src: don’t use equals to compare Class instances

ccd6c3a

dcache, dcache-frontend: move QoS logic out of frontend

79b3f87

cells: prefer queue route within zone

959723b

srm-server: make powermock test java11 compatible

2623c05

pom: explicitly define dependency on javassist

68e577b

libs: explicitly add a dependency on asm library

ab90a12

nfs: rename NFSv41Door#ioMessages -> transfers

33f0468

cells: don’t call CellCuratorFramework#close on CellNucleus#shutdown

49bc104

cells: move CuratorFramework watchers initialization into Domain

9138f84

pom: update assembly plugin version to 3.2.0

7100e62

cells: do not re-define zookeeper watcher

69f9387

libs: use hazelcast–3.12.5

2bd1f71

cells: better logging of Acl message problem

648b880

gplazma: multimap support files with multiple primary gids

b46555d

gplazma: x509 avoid NPE if certificate has no EKU

63440f3

pom:Update kafka-client lib version to 2.4.0

170a33f

libs: use mockito 3.2.4, powermock 2.0.4 and hamcrest 2.2

370a512

gplazma: x509 only map IGTF AP LoAs to REFEDS if entity is natural person

2a31529

gplazma: scitoken add unit tests and fix SciTokenScope

31cbb8c

pool: make berkeleyDBMetaDataRepository default

20e47c9

webdav : set Access-Control-Allow-Credentials to true

a63c689

util: re-set pool selection on NoRouteToCellException

2c0d03e

gplazma: x509 add support for CAs asserting version-specific LoA policies

9cd645b

canl: update to version 2.5.1

4b28329

packages: don’t try to extract chimera’s legacy sql scripts

3df1cd4

dcache-history, dcache-frontend: do not allow null composed json objects

74e90ab

frontend: avoid NPE when request has no media type

6d15fbc

dcache: add null check to pool info collector util

e705a31

common: don’t use Object#equals to compare constants

90261b0

config: fix typo in property name

656dfa7

gplazma: x509 include DN when logging incompatible LoAs

39d66a2

frontend/webdav: disable fallback-to-anonymous by default

ac24bb0

gplazma: scitoken fix two issues with SciToken plugin

1dee55f

chimera: fix dot file parser to handle weird file names

a96c3a1

cells: Message payload serialization using FST

640d03a

dcache: restore compatibility of pool monitor with pre–6.1 pools

8b3e221

dcap: gsi bad client excessive logging

24a1c04

transfermanager: avoid logging in non-error condition

19a36cd

x509: include LoA from CA’s .info file

f7f4f75

x509: update documentation link

fbed8ec

webdav: fix CORS when all clients are allowed to connect

02d6bd8

Revert “srm: remove SharedMemoryCacheJobStorage”

056d144

Revert “cells: Message payload serialization using FST”

f0804a6

cells: Message payload serialization using FST

b318075

cells: remove historic ssh1 support code

215972a

srm: remove SharedMemoryCacheJobStorage

d71349f

gplazma: oidc add support EGI LoAs

2a2f90f

gplazma: x509 add heuristics to identify PERSON certificates

4adebcc

srm: minor refactor of setting scheduler

e719169

gplazma: oidc add support for eduperson_assurance claims

a848d5c

srm: rename method that accepts a new Job

c86bf9d

srm: remove deadcode from JobStorage classes

eb16021

packaging: move local.version property out of package format profiles

48a9fb2

SRM: refactor Scheduler to make acceptJob clearer

ec2db90

srm: avoid redundant saveJob call

d47b9ba

srm: use guava-style to check SURL validity

653983a

srm: remove dead code, restrict code access

daabc90

pool: encrypted p2p transfer modes

d911506

docs: remove whitespaces in cookbook-writing-hsm-plugins.md

be12f34

docs: fix reference to cookbook-writing-hsm-plugins.md

611f764

packaging: move build.id property out of package format profiles

b255159

nfs: avoid NPE on error path

dbd3b95

srm: introduce Job state change notification, update Scheduler to use it

7298a7f

srmmanager: move SURL resolution into dCache plugin

fcf3bf8

SrmManager: provide better logging about problems

4b438f9

SrmManager: fix handling of saved requests on start-up

8a9fb94

SrmManager: avoid spamming if PinManager is down

9b22564

cells: prefer default route within zone

73f5d01

cells: make CellRoute zone aware

2015acc

pool: refactor JTM to use ScheduledExecutorService

d060824

srmmanager: fix logged state transition description

dca3f17

resilience: fix broke commit cc4f25a3c8

cc4f25a

resilience: don’t use old data API

3d86f95

doors: fix “lb set tags” command with no arguments

db1e289

pool: improve messages when migration job is cancelled.

cec4de4

srm: propagate srmmanager id into requests

f817fa8

srm: refactor Job state-change

2667459

docs: TheBook add chapter on SRR

ebe05a1

scripts: fix variable ordering in dcache-storage-descriptor

ea27cdf

cells: add unit test for CellRoutingTable

48ea014

pnfsmanager: silence delivery failures for FileAttributesTopic

67a8a26

pool: update “rep ls -s” formatting

8899988

cells: use Objects#requireNonNull instead of guava’s alternative.

e482513

cells: fix javadoc

dd48978

gplazma2-ldap: remove obsolete scala based plugin

2175c3f

cells: remove historic code

c6b17a5

common: revamp the byte column type in ColumnWriter

6c68637

dcache-xrootd: refit checksum handling after xrootd4j bug fix

9c4338f

dcache-frontend: notify PnfsManager of QoS transitions

e6534d4

dcache-resilience: allow handling of QoS transitions for resilient files

3dfed7e

dcache-core: add file attribute notification when AL/RP changes

3d06ecc

webdav: avoid logging non-error as an error

23dae03

common: fix bug in header space calculation for ColumnWriter

e836fe1

gplazma: x509 add heuristics for entity defn principals

b184d20

pool: include pool name in health-check reports

a3c7979

build: do not attempt to attach dir assembly to srm-client

98cc34e

gplazma: scitoken add support for multiple audience claims

c267db5

build: revert surefire upgrade to allow dCache builds to succeed

6d27231

build(deps): bump jackson-databind from 2.9.10 to 2.9.10.1

a602c36

dcache: restore pool compatibility with doors sending ‘Xrootd–2’ protocol version

451fa8b

chimera: do not update access latency on updat/insert in level_4

8a2ab43

build: fix maven warning about missing maven-enforcer-plugin version

86e4e23

srm: allow client to control whether srmPrepareToGet can trigger staging

c754523

build: fix cpd config, disable src-xref and generated code checking

1a45f19

build: avoid shipping log4j with dCache

e29da0b

resilience: avoid NPE if pool removed from resilience poolgroup

0356bc5

build: fix DataNucleus maven plugin logging errors

16f9b35

build: avoid using prerequisite argument in pom

3b1aa0c

build: choose a specific maven-groovy-plugin version

20ee75d

build: fix warnings when compiling FHS-compliant packages

4188f0e

resilience: override Object::equals, complete equals contract

0005a2d

frontend: fix NPE if limit is not specified

8ee6476

build: make building docker image optional and disabled by default

8cef28d

info: fix array index out of bound exception for admin commands

2365d13

pool: fix annotation in PoolMigrationCopyReplicaMessage

43f6472

srm: log number of SURLs in an SRM request.

e228d41

srm: improve access logging of srmBringOnline requests.

c9b49c4

srmclient: complete and tidy-up storageSystemInfo support

224ece3

gplamza: fix SciToken fetching remote content

08c05ea

gplazma: add support for REFEDS LoAs and rework x509 to include them

f1ce0a4

dcache-frontend: fix QoS pin semantics

9d3b029

dcache-xrootd: honor read paths when listing directories

9f6cd9f

pool: clean up master/push fc8dd482d13b8d089cce68859cfbb038cc7d8d48

924295a

libs: use potgres jdbc driver 42.2.8

f065a6d

pom: use java 11 friendly maven-surefire-plugin (junit)

314e195

libs: use java 11 friendly version of JNA

fc8dd48

pool: encrypted p2p transfers

473972b

pool: fix the xrootd version number on pool transfer service

4aeef4a

poolmanager: fix pool’s canonicalHostName change detection (part2)

12b4066

pool: clean code for HttpsTransferService

053dd1f

resilience: don’t compare Integer objects by refference

71dfffa

dcache-xrootd: refine error handling

e90b647

poolmanager: fix pool’s canonicalHostName change detection

261e786

dcache-xrootd: support the ‘tried’ CGI in xrootd door

6dae458

Pool Monitor: add host excluded unit tests and refactor

a8cd588

xrootd: fix kill mover command

3bcb15a

PoolManager: support selection filter for excluding pools on given hosts

274c313

dcache-frontend: avoid double fetch of sweeper data from pools

15807c7

doors: initialize IdentityResolverFactory after dependency injection

b28f730

cells: remove dead code dmg.util.cdb

ef32161

Revert “pom: require Java–11 at compile and run time”

77fe78a

pom: require Java–11 at compile and run time

7a891f5

sweeper: use in-memory map instead of repository for histogram data

387d981

pom: update aspectj maven plugin to java 11 capable

f9226d3

scitoken: do not use javax.activation.MimeType

235c71d

gplazma: stop using sun.security.rsa.RSAPublicKeyImpl

d779f40

pool:restructure httpsprofile in pool.xml by adding ssl-context-builder bean

92e2926

poolmanager: scan for matching pools when dynamic group is created

6751152

poolmanager: drop tags map argument in DynamicPGroup#addIfMatches

59becc2

poolmanager: include pool tags into SelectionPool record

2eb1f70

libs: move to DN 5

49ae166

How to enable scitokens in gplazma.conf

5de5570

dcache-history: expose refresh and timeout through admin commands

c06f419

chimera: explicitly specify encoding when converting string to bytes

23bec2d

nfs: re-try stale mover

c95f73b

nfs: restore transfers debug context on client retry

ead00eb

dcache-xrootd: replace constants for version number

e3a4d13

dcache-xrootd: update protocol version numbers

8ea611d

Pool Monitor: refactor to be more DRY

594cd1e

libs: update apache.commons:commons-compress to 1.19

124bcfd

cells: fix broken test (commit f64778f15f)

f64778f

cells: store CellAddressCore as gateway in CellRoute

34561e5

pool selection: make actual network hostname of pool available

c6d60dd

poolmanager: always allow removal of dynamic pool group

f467472

chimera: fix ABBA db deadlock when mkdir and rmdir run concurrently

cd74de0

util:fix DM_DEFAULT_ENCODING warning

5b8c58b

cells:fix DM_DEFAULT_ENCODING warning in services.login

e389138

cells: use ThreadLocalRandom when selecting a route

9f90535

cell: fix DM_DEFAULT_ENCODING warning

22d935c

nfs: do not wait for mover to shutdown for reads

8398f5f

Revert “Revert ”docker: Add a way to create docker image“”

87ae552

dcap: restart pool selection on OUT-OF-DATE error

926287f

gplazma-ldap: avoid thread leak by explicitly close NamingEnumeration

5ff1cf8

cells: fix DM_DEFAULT_ENCODING warning

7ac620e

Revert “docker: Add a way to create docker image”

c31f77a

[maven-release-plugin] prepare for next development iteration