Highlights

  • Optional TLS for p2p transfers
  • Zone-aware cell messaging
  • New serialization format

Incompatibilities

Starting release 6.1 the BerkeleyDBMetaDataRepository is the default metadata repository for the pools. Sites that was not explicitly specifying pool.plugins.meta property have to set the old default value: pool.plugins.meta=org.dcache.pool.repository.meta.file.FileMetaDataRepository

Pool in 6.1.x are not compatible with older dcap doors. However, new dcap doors can operate with existing pools.

Acknowledgments

(tbd, this section uses markdown formatting)

Release 6.1.36

acl

The group ACEs now apply to desired group instead of to a users with the same numeric id.

Changelog 6.1.35..6.1.36

e320f183a5
[maven-release-plugin] prepare release 6.1.36
0ce852a52d
acl: set IDENTIFIER_GROUP flag if WHO is GROUP or OWNER_GROUP
2f9ceb031c
[maven-release-plugin] prepare for next development iteration

Release 6.1.35

frontend

SRR returns data now with official json schema

webdav

Fix how sym-links are shown in the static HTML (web-browser) view from the WebDAV door.

Changelog 6.1.34..6.1.35

ac22afcd25
[maven-release-plugin] prepare release 6.1.35
6fa435b89e
frontend: add service to provide Storage resource reporting
df65e59b1b
webdav: update representation of symbolic links in HTML page
be4983de53
[maven-release-plugin] prepare for next development iteration

Release 6.1.34

dcache-xroot

The next release of xrootd4j is used now, which improves handling TPC read requests.

Changelog 6.1.33..6.1.34

60f18a8374
[maven-release-plugin] prepare release 6.1.34
22502ed813
dcache-xroot: bump dependency to next xrootd4j release
fb559b44ed
[maven-release-plugin] prepare for next development iteration

Release 6.1.33

srmmanager

A race condition is fixed that, if triggered, results in a memory leak. This leak can also affect the TURLs returned by SrmManager, where out-of-date information about doors is used.

Changelog 6.1.32..6.1.33

284f32fbd6
[maven-release-plugin] prepare release 6.1.33
8412e3563f
srmmanager: fix race condition in LoginBrokerSubscriber
1b33c2f3e4
[maven-release-plugin] prepare for next development iteration

Release 6.1.32

alarms

Clearer documentation for billing/alarms services regarding HA requirements.

skel

Whether a cell/service supports HA or not can now be derived from the properties file (except for doors).

Please see <name>.cell.replicable.

srm

The access log file for SRM requests now includes the client-supplied list of protocols, if any were provided.

Changelog 6.1.31..6.1.32

0ed78900af
[maven-release-plugin] prepare release 6.1.32
544331a36f
alarms, billing: specify shared rdbms in the case of replicated service
c106c3ea65
skel: add explicit replicable property even when value is false
bdbcfe7e4f
srm: log transfer protocols in access log
7fe8ab1d71
[maven-release-plugin] prepare for next development iteration

Release 6.1.31

dcache-core

The old web pages claim the transfer size and speed are reported in KB and KB/s, although the actually used units are KiB and KiB/s. Reporting the correct unit is important for monitoring. This is now fixed.

gplazma

dCache may now be configured so that the multimap oidc predicates match the sub claim value from a specific OAuth2 Provider using the format SUBVALUE@OP, where OP is the dCache-internal alias for the OP. Existing multimap configuration continues to work, but admin is warned to update the multimap configuration.

Changelog 6.1.30..6.1.31

22f5f8931c
[maven-release-plugin] prepare release 6.1.31
2d307b91cd
gplazma: multimap now supports OP in ‘oidc’ predicate
9295e69c52
dcache-core: correct reported units of transfer size and speed
64cbad4033
[maven-release-plugin] prepare for next development iteration

Release 6.1.30

xroot

The current release fixed the parsing to account for host/error pairs.

Upload transfer checksum failure is fixed and the gfal scenario now works.

Changelog 6.1.29..6.1.30

06c51b2822
[maven-release-plugin] prepare release 6.1.30
d981ec1a0b
dcache-xroot: parsing of ‘triedrc’ CGI lacks separation by optional comma
6b2efa35f1
dcache-xroot: fix upload transfer checksum failure (stable branches)
13d94929b0
[maven-release-plugin] prepare for next development iteration

Release 6.1.29

ftp

Globus transfer agent is now able to create symbolic links.

pool

dCacheView now works for redirected transfers for non-anonymous data access, provided the WebDAV door is using TLS encryption and is configured with ‘webdav.redirect.allow-https’ set to ‘true’.

poolmanager

A very rare race-condition is fixed that could be triggered if a pool completes a stage or pool-to-pool request (to satisfy a client’s read request) immediately before dying.

webdav

dCacheView is able to view and download files when authenticated and when the WebDAV door proxies the transfer.

Changelog 6.1.28..6.1.29

a1d13d4c26
[maven-release-plugin] prepare release 6.1.29
0fe2340945
webdav: use Jetty’s built-in support for CORS
9eff7ea1ed
poolmanager: fix _waitingFor access outside of synchronized block
a724dcd897
ftp: add support for the SITE SYMLINK command
44ba3ac891
pool: http allow client to send credentials when TLS is used
29ecae497c
[maven-release-plugin] prepare for next development iteration

Release 6.1.28

frontend

frontend/api readers and writers always zero issue is fixed now

gplazma

The LDAP plugin behaviuor now more closely follows that of other mapping plugins. This allows deployments where LDAP is tried first and, if that fails to identify the user, fall-back strategies are used.

Changelog 6.1.27..6.1.28

111a1acf66
[maven-release-plugin] prepare release 6.1.28
ba7873346e
gplazma: ldap throw exception if no principal is added
19d4f66382
poolV4: fix reader/writer counts
ecdf313739
[maven-release-plugin] prepare for next development iteration

Release 6.1.27

dcache-xroot

The cirrent release fixed memory leak, race and other small issues with reconnect to pool.

frontend

The dcache-view.endpoints.webdav property now overrides any auto-discovered WebDAV endpoint, making explicit configuration easier.

pnfsmanager

A regression is fixed the prevents creating a symbolic link.

Changelog 6.1.26..6.1.27

3082ef649a
[maven-release-plugin] prepare release 6.1.27
0ee97b506b
frontend: bump version of dCacheView
6559fcd58b
pnfsmanager: fix NPE regression from 3dfed7e8b0
09f09d99bc
dcache-xroot: fix memory leak, race and other small issues with reconnect to pool
cdd91e3a52
frontend: bump dCacheView to v1.6.2
cc1948c443
[maven-release-plugin] prepare for next development iteration

Release 6.1.26

poolmanager

When poolmanager requests that a pool stage a file, the pool may return the special error code HSM_DELAY_ERROR (10013). On receiving this error, poolmanager suspends the request.

This is now fixed.

xrootd

xrootd4j is updated to 3.5.10.

Changelog 6.1.25..6.1.26

8d04f449c1
[maven-release-plugin] prepare release 6.1.26
41dcbb8ea3
dcache-xrootd: update xrootd4j to 3.5.10
cf3ecd8722
poolmanager: fix NPE if pool delays stage without giving a reason
04efa36e50
[maven-release-plugin] prepare for next development iteration

Release 6.1.25

xrootd

Now the door will ignore xrootd.enable.tried-hosts property if property false instead of returning errror.

Changelog 6.1.24..6.1.25

653a926a08
[maven-release-plugin] prepare release 6.1.25
f17e25d5ff
dcache-xroot: (door) ignore tried if property false instead of returning error
0a4749af1c
Update install.md
43c3fe2cab
[maven-release-plugin] prepare for next development iteration

Release 6.1.25

xrootd

Now the door will ignore xrootd.enable.tried-hosts property if property false instead of returning errror.

Changelog 6.1.24..6.1.25

653a926a08
[maven-release-plugin] prepare release 6.1.25
f17e25d5ff
dcache-xroot: (door) ignore tried if property false instead of returning error
0a4749af1c
Update install.md
43c3fe2cab
[maven-release-plugin] prepare for next development iteration

Release 6.1.24

Changes affecting multiple services

Sometimes file checksums are missing in namespace. Transfer requiring checksum verification fails then with an error saying that “No checksums found”, but doesn’t log a stack trace server side anymore.

dcache-xroot

During periods of heavy usage on a pool node, if I/O stalls during read, the xroot client may try to reconnect and continue reading. This release fixes a bug which fails the transfer on the attempted reconnect with a “uuid no longer valid” error (because the file mover has already been closed and removed). Clients should now be able to resume reading the file on retry.

Changelog 6.1.23..6.1.24

a39c0de3f7
[maven-release-plugin] prepare release 6.1.24
da13dda0bc
dcache-xroot: Allow client to reattempt open on pool when I/O stalls
5f6015d700
check if checksums are present and throw CacheException if they are not
bd03a8a23e
[maven-release-plugin] prepare for next development iteration

Release 6.1.23

admin

The current release fixed doors login dump cache admin command to show the information sent by the door along with the login result.

common

Printing of bearer tokens is improved to provide better assurance that bearer tokens are not leaked.

gplazma

All login failures are now recoreded in pinboard as a single-line summary, explaining why the login attempt failed.

webdav

The current release fixed a regression where dCache would complain about badly formatted IPv6 addresses, preventing those addresses from being included in the access log files.

The current release fixed a regression where dCache would complain about badly formatted IPv6 addresses, preventing those addresses from being included in the access log files.

xroot

The current relase updated xrootd4j to 4.0.5 which fixes improper use of destination token when contacting source server during third-pary transfer.

Changelog 6.1.22..6.1.23

687de75ea5
[maven-release-plugin] prepare release 6.1.23
93d5d18b74
dcache-xroot: update to xrootd4j–4.0.5
e32de4e2c0
webdav/frontend: fix IPv6 address logging
5ff2e7c750
common: update how BearerToken is shown in Subjects#toString
8067df53f4
authn: add missing hashCode and equals methods for PasswordCredential
5e73199991
authn: fix “login dump cache” admin command output
911ab10158
gplazma: provide single-line summary in pinboard for each login failure
250748b310
authn: log failures in UnionLoginStrategy
04f3749dd0
authn: add missing hashCode and equals methods for BearerTokenCredential
29f094c523
common: Update BearerToken representation to include some information
53f362cf17
[maven-release-plugin] prepare for next development iteration

Release 6.1.22

webdav

Restarting transfermanager while there are ongoing HTTP-TPC transfers no longer results in an endless loop, with transfermanager logging a message every ~5 seconds per ongoing HTTP-TPC transfer.

Changelog 6.1.21..6.1.22

b44f6b5668
[maven-release-plugin] prepare release 6.1.22
5c5a996be9
webdav-transfermanager: fail gracefully if transfermanager restarted
704d9b5ccd
[maven-release-plugin] prepare for next development iteration

Release 6.1.21

biiling

The current relase fixed unit info for bandwidth measurements in billing.properties (bytes/sec, not MiB/sec).

frontend

Frontend now logs any time it returns 500 Internal Server Error to the client.

dcache-view Previously displayed MB/sec for transfer rate, but the value is the old KB/sec. This is now improved and pages display the values advertised.

pool

The current release assures now that checksum scanner wiill mark files broken and sends alarm.

webdav

An HTTP-TPC transfer will abort automatically if transfer-manager is restarted, without waiting for the client (FTS) to time-out.

Changelog 6.1.20..6.1.21

963dc9a352
[maven-release-plugin] prepare release 6.1.21
ccf4760d4b
dcache-pool: Assure checksum scanner marks files broken and sends alarm
57f9e32948
webdav/transfermanager: fail gracefully on restart, for perf marker
99654d07e1
default: fix unit info for bandwidth measurements in billing.properties
cb742ca1a9
frontend: create a log entry when we return 500 Internal Server Error
b95b867193
dcache-frontend: change transfer rate to compute what it advertised
c8a36e7b27
[maven-release-plugin] prepare for next development iteration

Release 6.1.20

book

A new section is added to the xrootd chapter explaining the need for SAN extensions on the hostcert in the case of hostname aliasing.

ftp

Globus transfer service can now list directories that contain incomplete files, those that are still being uploaded.

The FTP no longer leaking sockets when dynamically calculating file checksums (e.g., MD5). In practise, this issue only affects transfers via the Globus transfer service.

The current release fixed Globus transfer service directory listing for medium size directories.

The current release fixed Globus transfers hanging when verify file integrity after transfer is enabled and the pool is not configured to calculate an MD5 checksum.

Honour the ftp.net.internal configuration property for dynamic checksum calculation.

Changelog 6.1.19..6.1.20

fbd737286b
[maven-release-plugin] prepare release 6.1.20
656a324c27
ftp: do not leak sockets when calculating checksums dynamically
095b02d305
ftp: add work-around to calculate MD5 for Globus
98468678f7
ftp: add place-holder file size work-around for Globus directory listing
2783011bd3
rpm take build directory path into quotes
bfb3454e7b
ftp: fix MLSC support for medium-length directories
fc7564fb5a
ftp: fix on-the-fly checksum calculation
79f4c75126
book: document host alias issue with xroot TPC
ad4f26096e
[maven-release-plugin] prepare for next development iteration

Release 6.1.19

pool

The current release fix a problem where, under heavy load, redirected https transfers result in garbled SSL information, preventing the HTTP client from working.

services

Recovery processe in the case where a pool’s message queue is overloaded, as old (and now useless) requests may be simply discarded is improved.

webdav

There were reports of the access log file missing entries. This problem seems to be triggered if the client disconnects before dCache has finished processing the request, which is certainly true for cancelled HTTP-TPC transfers. This is now fixed.

xrootd

The xroot protocol access log entries for the xrootd door and pool have been enhanced to include information about the kXR_query requests.

Changelog 6.1.18..6.1.19

dfdaadc123
[maven-release-plugin] prepare release 6.1.19
6c2d8c9918
prepare for next development iteration
d261fe34cf
TransferManager: choose better pool query message TTL
bbae4d1252
webdav/frontend: ensure cancelled HTTP-TPC transfers are logged
d44f28bdc1
xrootd/pool: improve access log to record more details of kXR_query
8336e2cc85
pool: https avoid sharing SSLEngine between TCP connections

Release 6.1.18

The current release fixes EGI-SVG–2020–16939 vulnerabilities.

Release 6.1.17

xrootd

The xrootd4j version was updated to 3.5.8.

Changelog 6.1.16..6.1.17

fae8870df5
[maven-release-plugin] prepare release 6.1.17
23bf788f20
pom: update xrootd4j to 3.5.8
8c204ac235
[maven-release-plugin] prepare for next development iteration

Release 6.1.16

pool

The mover thread is not interrupted outside of IO activity to avoid disabling of repository.

HTTP-TPC transfers where dCache pushes data to an Apache server (for example as used by Dynafed) will now work for zero-length files.

webdav

HTTPS-PUT requests are now possible.

Changelog 6.1.15..6.1.16

bf73ac6530
[maven-release-plugin] prepare release 6.1.16
7c34f736ee
pool: http-tpc do not loop if HEAD ‘Content-Length’ response is missing
1a1975ad0e
webdav:HTTPS PUT req issue
279af5f5aa
pool: preserve CDC on p2p transfer
341de58525
pool: http-tpc drop ‘Accept-Encoding’ on HEAD requests
ab462b9a3d
pool: detach IO thread when mover finished
5e05bccba8
[maven-release-plugin] prepare for next development iteration

Release 6.1.15

pool

dCache pools now support GSI-based xrootd-TPC by default.

Changelog 6.1.14..6.1.15

1f61af5473
[maven-release-plugin] prepare release 6.1.15
b379cdefa4
pool: update xrootd-tpc authenication default
09ddc0fd90
[maven-release-plugin] prepare for next development iteration

Release 6.1.14

enstore

Enstore storage info provides works with URI based tape locations now.

pool

The HTTP-TPC PUSH requests (which use the HTTP PUT request to send data) are now more robust against slow remote servers that redirect the transfer.

The current release fixed NullPointerException bug in MongoDB-backed pool metadata storage and SpotBugs NORMAL-level warning.

With the chenges introduced in current release, when using HTTP-TPC, the timeouts dCache pool uses when requesting information from the remote server have been adjusted, based on operational experience. In general, dCache will now wait longer for the remote server to complete any post-processing before giving up and failing the transfer.

Changelog 6.1.13..6.1.14

2f5383a4e3
[maven-release-plugin] prepare release 6.1.14
5dcb95190c
chimera: update EnstoreStorageInfoExtractor to use FileState
bdb2a822f2
chimera: use UTF–8 constant instead of string in ChimeraEnstoreStorageInfoExtractor
f212438e29
enstore: populate the correct storage info information on set
b834910bbe
pool: http-tpc adjust GET and HEAD request timeouts
8d505d8c97
pool: http-tpc PUT request are repeatable
7ed0edb4e2
pool: support removing MongoDB storage-info entry
d8ba33ff6c
[maven-release-plugin] prepare for next development iteration

Release 6.1.13

dcap

Sometimes file restores on dCap door were hanging. This is now fixed.

Changelog 6.1.12..6.1.13

cd92673826
[maven-release-plugin] prepare release 6.1.13
c318c7b31f
[maven-release-plugin] prepare for next development iteration
7cb32b4ff5
dcap: fix automatic door retry on transient errors

Release 6.1.12

gplazma

X.509 certificates that use the “SHA–384 with RSA” algorithm are now logged with this name instead of the raw OID value.

httpd

The current release fix pool usage color scheme css used for UsageInto and PoolInfo pages.

pool

dCache pools that are undertaking HTTP-TPC transfers will now wait longer for the remote servers to complete any post-transfer activity. Transfers that involve heavily loaded remote servers are now more likely to succeed.

Changelog 6.1.11..6.1.12

a7fe40d85b
[maven-release-plugin] prepare release 6.1.12
ae6616d5db
pool: http-tpc increase timeout waiting for remote server post-processing
39952d5fbc
gplazma: include additional algorithm in LoginResultPrinter
bc2be58a21
httpd: fix pool usage color scheme css
d85bdc4d42
correct snapshot version for logback-test-config
a73c8d4074
correct snapshot version

Release 6.1.11

nfs

The current release fixed previously observed infinite WRITE+COMMIT loop.

Changelog 6.1.10..6.1.11

0d59a8a090
[maven-release-plugin] prepare release 6.1.11
0aae861a17
nfs-proxy: op WRITE must use the same verifier as COMMIT
9fd3c6d73a
[maven-release-plugin] prepare for next development iteration

Release 6.1.10

dcache

There have been several occasions where in a dCache instance using an embedded ZooKeeper server such HA services stopped working because an ephemeral node was elected leader that has no associated service anymore; it should have been deleted. This state might go undetected for a longer time, and removing such stale nodes in order for the services to continue functioning as intended requires manual intervention.

This is fixed now and the functioning of HA services using ZooKeeper-based leader elections is more reliable.

resilience

A more robust handling of PSU changes which remove pool, unit or group mappings was introduced.

Recently introduced improvements changed the way resilience handles file “removal” (no longer setting the repository entry to ‘removed’ but simply by caching the replica). This change, however, did not take into account the handling of broken files. Encountering a broken file, it would indiscriminately attempt to remove it, whether it was cached or not; this was leading to an infinite loop, with the file operation continuously iterating without doing any further work. This situation could potentially hang the pool scans (if there are as many broken files as there are scan threads), and even the file operation queue. The current release fixed this issue and there is no any potential for stalled operations when encountering broken replicas.

Changelog 6.1.9..6.1.10

e42123433e
[maven-release-plugin] prepare release 6.1.10
b70f8fc044
dcache-core: fix embedded ZooKeeper persisting stale ephemeral nodes
5b5e14425b
dcache-resilience: ignore broken cached files
0ba3237d8a
dcache-resilience: protect access of pool info map against NoSuchElementException
1e7bc1129d
[maven-release-plugin] prepare for next development iteration

Release 6.1.9

macaroons

Fixed macaroons validation where in case of multiple combined caveats for some of them only the digital signature was checked.

resilience

If a FileOperation is canceled while its subtask is running a NullPointerException was thrown. This is now fixed.

Changelog 6.1.8..6.1.9

45124a1c28
[maven-release-plugin] prepare release 6.1.9
3c2936e497
dcache-resilience: avoid NPE in file operation cancel
a104e9220a
macaroons: verifyCaveat should return false for unprocessed caveats
432e9508e3
[maven-release-plugin] prepare for next development iteration

Release 6.1.8

cleaner

Database passwords can now be stored in a dedicated file.

pool

Failing HTTP-TPC transfers where the remote party sends a malformed RFC 3230 checksum value are now fixed.

util

The current release fixed pgpass file handling for jdbc URLs with options, like jdbc:postgresql://dbhost:5432/foo?prepareThreshold=3

Changelog 6.1.7..6.1.8

5cab9be888
[maven-release-plugin] prepare release 6.1.8
652cf942d9
pool: do not fail transfer if RFC3230 is badly formed.
bb3f741184
util: fix pgpass support for jdbc urls with options
6bf2e0d1b1
cleaner: add support for db password file
bf6e354aec
[maven-release-plugin] prepare for next development iteration

Release 6.1.7

pool

The current release fixed interrupted p2p transfer issue and the pool should not be disabled.

Changelog 6.1.6..6.1.7

7a0816c
[maven-release-plugin] prepare release 6.1.7
864bcbd
pool: rework interrupt processing on p2p
71584b7
[maven-release-plugin] prepare for next development iteration

Release 6.1.6

common

dCache no longer logs OIDC access tokens or macaroon.

The current release fixed a bug where SRM complains enum constant IGTF_LOA_CEDER does not exist.

dcache

Priviously we have introduced support for the xrootd cgi tried=,, which was an optional feature because this exclusion can put access to the file into an indefinitely suspended state for all clients.

The current relase fixed the latter problem and when ‘tried’ is activated (the only time excluded hosts can be non-empty), failure to read the file will not put the request container into the SUSPENDED state.

dcache-xrootd

The version is upgrated to xrootd4j 3.5.7 with lossen username validation, so that

usernames like foo.1234:56 are no longer rejected.

The current release fixed java.lang.IllegalStateException: ChecksumChannel must not be written to after getChecksums.

TPC client first will be shut down when the pool netty channel goes inactive.

pnfsmanager

The current release fixed a NullPointerException when a file is deleted while a directory listing is being compiled.

pool

When a legacy mover is killed by job timeout manager thread interruption mechanism is used. As mover thread is taken from a thread pool, the next task might see this interrupted state, if not cleared. This state can be checked bu other components and trigger false errors: java.lang.InterruptedException THREAD_INTERRUPTED: InterruptedException.

This is now fixed and pool is not disabled itself after slow running mover is killed by jtm.

Changelog 6.1.4..6.1.6

bad07a9
[maven-release-plugin] prepare release 6.1.6
eb80e6c
pnfsmanager: avoid NPE if file is deleted during directory listing
7def465
common: do not log bearer token values.
9ee0b74
pool: rework interrupt processing in AbstractMoverProtocolTransferService
efd886b
common: fix LoAPrincipal deserialisation
09545db
dcache-xrootd: bump to xrootd4j 3.5.7
68a06d6
pool: clean interrupted state after mover complete (again)
a8c34de
pinmanager: use PoolManagerStub when talking to pool manager
aec2ba0
dcache (pool manager): override suspend when “excluded” (hosts) is not empty’
b5a45cc
pool: clear mover thread’s interrupted state before returning to pool
7071492
Wrap the shutDown in try … catch for InterruptedException.
d91aa82
dcache-xrootd: cancel TPC transfer when client disconnects unexpectedly from pool
7bab3ea
[maven-release-plugin] prepare for next development iteration
01521fe
[maven-release-plugin] prepare release 6.1.5
9e17bb0
pool: repository account capacity correctly if file channel closed
1f5b1ac
dcache-xrootd: make ‘tried’/excluded hosts an optional feature
df80b16
docs: TheBook update index to include the chapter on Kafka
d618231
[maven-release-plugin] prepare for next development iteration

Release 6.1.4

frontend

The frontend’s inotify-over-SSE now honour the ‘frontend.root’ configuration property and any user-specific root. This means that, for any inotify subscription, the path is calculated relative to the doors’ root and the user-specific root. For many dCache sites, this has no impact as they are using the default for both; however, the user root is used to implement the macaroon’s ‘root’ caveat. The current release fixed how inotify subscription requests are processed when made with a macaroon with a ‘root’ caveat.

pool

When making an HTTP third-party copy (HTTP-TPC), dCache no longer sends the ‘Authorization’ HTTP request header in any subsequent request when the remote server responds with a redirection.

The unix xrootd tpc security plugin was included in order to enable the dCache TPC client to use a dCache pool as source when signed hash verification is on. However, this is now fixed and no special configuration necessary for organizations (like Tier 1) needing to communicate with EOS.

Changelog 6.1.3..6.1.4

981eb95
[maven-release-plugin] prepare release 6.1.4
55499a0
pool (xrootd): make tpc security plugin default unix
ff320a7
docs: add missing architecture diagram
e49bc52
frontend: honour door and user root for inotify subscriptions
97838fa
frontend: refactor inotify client-path to dCache-path mapping
28d7d6b
frontend: add SelectionContext for event subscription
16396cf
docs: UserGuide fix missing back-tick
67d57d1
pool: drop Authorization HTTP header on redirected
4bedd32
[maven-release-plugin] prepare for next development iteration

Release 6.1.3

frontend

When frontend is run in a core domain and there is no history service reachable, the retry on no route to cell ends up spamming the message queues. This is now fixed.

login-broker

The lb set update command ignored the provided value and set the existing one again. This is now fixed and lb update times can be modified through admin interface.

xrootd

The current release fixed compatible level security for sigver.

The xrootd client has a command-line option, --path, which tells the server to create missing directories. This option is included in two-party copy, but setting it for TPC has no effect. TPCs which wish to write to dCache (as destination) to a non-existent subdirectory fail. This is now fixed and dCache no longer fails in these cases.

Changelog 6.1.2..6.1.3

00c175b
[maven-release-plugin] prepare release 6.1.3
de8e13a
login-broker: fix ‘lb set update’ command
db91dfd
dcache-xrootd: always create missing directories on write
60dd143
dcache-xrootd: fix compatible level security for sigver
5eb7b0d
dcache-frontend: remove retry flag on sendAndWait to history service
e6ca223
[maven-release-plugin] prepare for next development iteration

Release 6.1.2

nearline-storage

There were issues in pool space allocation/de-allocation in conjunction to HSM connectivity and handling restore errors resulting in failing stage end up with used space miscalculation: pool_read, fault occurred in repository: Internal repository error. Pool restart required:, cause: java.lang.IllegalArgumentException: Cannot set used space to a negative value.

This is now fixed and disk copy is removed if restore from HSM failed.

Changelog 6.1.1..6.1.2

9d24d15
[maven-release-plugin] prepare release 6.1.2
efe932b
nearline-storage: remove disk copy if restore from HSM failed
225cb96
[maven-release-plugin] prepare for next development iteration

Release 6.1.1

dcache

Qos migration policy engine was raising JVM error when no tape pool found. This is now fixed and the normal behavior is that HTTP error code reported back with No HSM pool found.

frontend

The current release improved error responses and they are more specific now.

The current release fixed the NPE stack trace arised because the pool data could be sent with a default sweeper data object.

Changelog 6.1.0..6.1.1

139eca2
[maven-release-plugin] prepare release 6.1.1
2e41f65
dcache-frontend: make ErrorResponseProvider return the more specific error message
8520249
dcache: qos migration policy engine should not raise JVM error when no tape pool found
beedd99
dcache-history,dcache-frontend: guard against unconfigured sweeper histogram
a347604
[maven-release-plugin] prepare for next development iteration

Release 6.1.0

Cells

Starting with release 6.1.0 dCache internal massage routing will prefer zone local cells when possible. This is benefitial in geo-distributed HA setups, where talking to remote cells may introduce additional latency.

A new serializer is available in dCache, which improves the speed of internal message passing between cells. As this feature has not been tested very extensively and it’s possible that sites may experience unforeseen behavior, the new serializer has to be explicitly enabled as:

dcache.broker.channel.msg-payload-serializer = experimental

NOTE: when different cells are not of the same bug-fix release, then dCache will always fall back to using the standard serialization protocol.

Chimera

Removed unneccesary requests to DB on new file creation. Fixed operation with read-only database cponnections.

Frontend

As part of the effort (in progress) to provide enhanced support for QoS, the RESTful QoS transition API and implementation has been revisited. First, the way QoS was being evaluated (counting all pins) has been corrected by using a QoS owned pin. Next, in addition to pinning,
QoS changes now involve modification of the access latency and
retention policy values for the file in the namespace. The frontend notifies PnfsManager of these transitions, and this notification then gets relayed to resilience. Some internal refactoring of the QoS logic out of frontend and into core was also done.

An inefficiency resulting in the double fetch of sweeper data from the pools was fixed.

The restores and transfers admin pages were erroneously requiring user login; this has been eliminated.

For the API enabling and disabling pools, a bug preventing the expression of the flag using a boolean rather than a string, e.g., pools/{name}/usage/mode -d {"rdonly":true} rather than pools/{name}/usage/mode -d {"rdonly":"true"}) has been fixed.

History

Some commands available on the frontend for refreshing and adjusting the timeout of pool data collection were missing from the history service; these have now been made available.

Also addressed were null pointer issues arising from the arrival of incompletely initialized data from the pools; code was refactored to make sure composed objects do not have empty fields; in addition, a few extra checks for missing data are now made.

NFS

The admin command ‘show clients’ is updated to accept address/netmask or pattern as a host argument, for example: admin > show clients 10.1.1.0/24 admin > show clients fe80::eef4:bbff:fed2:f9d0 admin > show clients wn*.dcache.org

PNFS Manager

Notification is now sent when access latency or retention policy changes for a file (in order to support the RESTful transition API and management of QoS by resilience).

Pool

Starting from 6.1.0 the pool to pool transfers can be optionaly configured to use TLS. A new propertry pool.enable.encrypted.p2p-transfers is introduced that accepts three values:

  • NEVER: TLS is never used for p2p transfers
  • ALWAYS: all p2p transfers aare using TLS
  • CROSSZONES: TLS is used only when source and dstination pools are in different zones.

NOTE, that in the current implementation CROSSZONES has the same effect as ALWAYS.

The BerkeleyDBMetaDataRepository is now default format for pools metadata.

An inefficiency in the processing of sweeper data for frontend histogram display,
which could result in CPU spiking on the pool node, has been fixed via an in-memory optimization replacing callouts to the repository store.

The pool transfer service now incorporates both older and newer xrootd version numbers for compatibility (2.7, 4).

Pool Manager

The network hostname of the pool is now made available to the selection unit in order to support the xrootd ‘tried’ query element; a small fix was also added to restore backward compatibility with older pools that do not so advertise their host name to the pool monitor.

The selection algorithm has been modified to support excluding tried hosts.

Resilience

Modifications were made to couple resilience more tightly to QoS transition requests; it can now manage transitions for “resilient” files between tape and disk. This means that it will make replicas for a NEARLINE file which has become ONLINE, and can cache all replicas for an ONLINE file that has become NEARLINE, provided that the file belongs to a storage unit with a non-null ‘-required’ value.

A small bug (null pointer exception) which could occur if a file staged from tape no longer exists on any pool that is part of a resilient poolgroup has been fixed. There were also two minor code improvements involving object comparisons.

XRootD

The dCache XrootD door now observes the xrootd protocol ‘tried=host1,host2,…’ query element.
What this means is that if the xrootd client (xrdcp) should decide to retry a failed transaction by returning to the dCache door, the hosts listed as having been previously tried will be used to exclude potential pools for the transfer. Should no other pools be available, dCache will behave the same as when a file is unavailable (it will wait indefinitely until the client gives up).

Pool and door compatibility based on changed version numbers has now been fixed so that older doors will work with newer pools and vice versa.

Error handling (mapping of error types to codes returned to the xrootd client) was refined to make it more consistent and conform more closely with the xrootd protocol.

The version number indicated by dCache now should reflect the actual xrootd protocol version being used (it was formerly fixed, incorrectly, at 2.7).

The xrootd client is now redirected to the pool using an actual hostname rather than IP address, for the sake of compatibility with RFC 5280 (4.2.1.6.
Subject Alternative Name). This is essential for eventual support of TLS.

Two internal changes in advance of future features (TLS, SciTokens) have been made: the door now uses a caching login strategy, similarly to most of the other dCache doors; it also supports authorization handlers which need to access login strategies (both of these will be necessary for SciTokens).

A bug in the delegation provider requiring that a VOMS directory be defined even if the GSI module is not being used has been fixed.

A bug preventing read paths from being observed by the list command has been fixed.

A bug preventing xrdcp -C <checksum type> from returning the correct checksum when multiple checksums are defined on the pools has been fixed.

Batch/Startup scripts

The batch scripts for the alarms service and xrootd doors were erroneously configured not to exit and restart on fatal errors; if there were unhandled exceptions during startup, these two services would simply hang, making detection of their status (e.g., through external monitoring) difficult. The scripts have been fixed to conform to the exit/restart sequence used by all other services.

Changelog from 6.0.0 to 6.1.0

8520249
dcache: qos migration policy engine should not raise JVM error when no tape pool found
beedd99
dcache-history,dcache-frontend: guard against unconfigured sweeper histogram
a347604
[maven-release-plugin] prepare for next development iteration
f15e40b
[maven-release-plugin] prepare release 6.1.0
3448267
dcache-frontend: allow pool enable/disable to use boolean JSON value
e8a7705
dcache-frontend,history: protect against missing highest bin in histogram data
00883c2
[maven-release-plugin] prepare branch 6.1
8bbae91
vehicles: remove historic field from PoolPassiveIoFileMessage
0903034
build(deps): bump version.netty from 4.1.10.Final to 4.1.45.Final
f308a7f
dcache, frontend: release dcache-view version 1.6.1
41665a1
util: refactor task running to aid code reuse
2c3fd78
dcache: release liquibase locks only if update is requested
4fa3948
libs: remove dependency on jackson-mapper-asl
d9529e4
dcache-xrootd: change URL to use hostname on redirect to pool
7a17096
Revert “libs: remove dependency on jackson-mapper-asl”
aee153f
libs: remove dependency on jackson-mapper-asl
a9268f3
chimera: don’t try to access level2 for newly created files
8e11da1
ftp: avoid NullPointerException when client transfers without ALLO
0186ba2
skel: repair erroneous batch directives before cell creation
3512cdc
dcache-xrootd: don’t initialize delegation provider when there is no gsi module
3185607
dcache-frontend: remove unnecessary login requirement on restores and transfers
c385552
frontend: events inotify avoid holding global lock when sending events
11d21f1
gplazma: scitoken add partial support for WLCG Common JWT Profile scopes
4907261
pool: log when a queue starts to queue movers and when this stops
a48e5e0
nfs: don’t use stream to compute collection size
e73815c
srm: use host IP for comparison when determining if SURL is local
527e8d3
ftp: support zero allo values
bc451a5
dcache: increase metadata wait time for kafka producer config
5366aca
frontend: events inotify fix deadlock
7d78c80
dcache-xrootd: make door use CachingLoginStrategy
15e0c5d
dcache-xrootd: support authz handlers which need to access login strategies
0b5f8f1
dcache, frontend: release dcache-view version 1.6.0
6a87a2b
gplazma: scitoken refactor plugin
eb1b81d
Fix tape-reserved size calculation
dd00230
common: add missing toString method for MultiTargetedRestriction
753712a
gplazma: scitoken add SciTokenPlugin unit test
d02f314
nfs: extend host filter of ‘show clients’ command to accept patterns
4f6ef66
Update config-stage-protection.md
cbffb34
poolmanager: add affinity requirements to PoolManagerMessage
2328947
poolmanager: make RendezvousPoolManagerHandler#backendFor private
bff16d6
webdav: include DAV header in OPTIONS requests.
5ee32d5
pnfsmanager: better explain restriction-based permission problems
f1f1371
src: don’t use equals to compare Class instances
ccd6c3a
dcache, dcache-frontend: move QoS logic out of frontend
79b3f87
cells: prefer queue route within zone
959723b
srm-server: make powermock test java11 compatible
2623c05
pom: explicitly define dependency on javassist
68e577b
libs: explicitly add a dependency on asm library
ab90a12
nfs: rename NFSv41Door#ioMessages -> transfers
33f0468
cells: don’t call CellCuratorFramework#close on CellNucleus#shutdown
49bc104
cells: move CuratorFramework watchers initialization into Domain
9138f84
pom: update assembly plugin version to 3.2.0
7100e62
cells: do not re-define zookeeper watcher
69f9387
libs: use hazelcast–3.12.5
2bd1f71
cells: better logging of Acl message problem
648b880
gplazma: multimap support files with multiple primary gids
b46555d
gplazma: x509 avoid NPE if certificate has no EKU
63440f3
pom:Update kafka-client lib version to 2.4.0
170a33f
libs: use mockito 3.2.4, powermock 2.0.4 and hamcrest 2.2
370a512
gplazma: x509 only map IGTF AP LoAs to REFEDS if entity is natural person
2a31529
gplazma: scitoken add unit tests and fix SciTokenScope
31cbb8c
pool: make berkeleyDBMetaDataRepository default
20e47c9
webdav : set Access-Control-Allow-Credentials to true
a63c689
util: re-set pool selection on NoRouteToCellException
2c0d03e
gplazma: x509 add support for CAs asserting version-specific LoA policies
9cd645b
canl: update to version 2.5.1
4b28329
packages: don’t try to extract chimera’s legacy sql scripts
3df1cd4
dcache-history, dcache-frontend: do not allow null composed json objects
74e90ab
frontend: avoid NPE when request has no media type
6d15fbc
dcache: add null check to pool info collector util
e705a31
common: don’t use Object#equals to compare constants
90261b0
config: fix typo in property name
656dfa7
gplazma: x509 include DN when logging incompatible LoAs
39d66a2
frontend/webdav: disable fallback-to-anonymous by default
ac24bb0
gplazma: scitoken fix two issues with SciToken plugin
1dee55f
chimera: fix dot file parser to handle weird file names
a96c3a1
cells: Message payload serialization using FST
640d03a
dcache: restore compatibility of pool monitor with pre–6.1 pools
8b3e221
dcap: gsi bad client excessive logging
24a1c04
transfermanager: avoid logging in non-error condition
19a36cd
x509: include LoA from CA’s .info file
f7f4f75
x509: update documentation link
fbed8ec
webdav: fix CORS when all clients are allowed to connect
02d6bd8
Revert “srm: remove SharedMemoryCacheJobStorage”
056d144
Revert “cells: Message payload serialization using FST”
f0804a6
cells: Message payload serialization using FST
b318075
cells: remove historic ssh1 support code
215972a
srm: remove SharedMemoryCacheJobStorage
d71349f
gplazma: oidc add support EGI LoAs
2a2f90f
gplazma: x509 add heuristics to identify PERSON certificates
4adebcc
srm: minor refactor of setting scheduler
e719169
gplazma: oidc add support for eduperson_assurance claims
a848d5c
srm: rename method that accepts a new Job
c86bf9d
srm: remove deadcode from JobStorage classes
eb16021
packaging: move local.version property out of package format profiles
48a9fb2
SRM: refactor Scheduler to make acceptJob clearer
ec2db90
srm: avoid redundant saveJob call
d47b9ba
srm: use guava-style to check SURL validity
653983a
srm: remove dead code, restrict code access
daabc90
pool: encrypted p2p transfer modes
d911506
docs: remove whitespaces in cookbook-writing-hsm-plugins.md
be12f34
docs: fix reference to cookbook-writing-hsm-plugins.md
611f764
packaging: move build.id property out of package format profiles
b255159
nfs: avoid NPE on error path
dbd3b95
srm: introduce Job state change notification, update Scheduler to use it
7298a7f
srmmanager: move SURL resolution into dCache plugin
fcf3bf8
SrmManager: provide better logging about problems
4b438f9
SrmManager: fix handling of saved requests on start-up
8a9fb94
SrmManager: avoid spamming if PinManager is down
9b22564
cells: prefer default route within zone
73f5d01
cells: make CellRoute zone aware
2015acc
pool: refactor JTM to use ScheduledExecutorService
d060824
srmmanager: fix logged state transition description
dca3f17
resilience: fix broke commit cc4f25a3c8
cc4f25a
resilience: don’t use old data API
3d86f95
doors: fix “lb set tags” command with no arguments
db1e289
pool: improve messages when migration job is cancelled.
cec4de4
srm: propagate srmmanager id into requests
f817fa8
srm: refactor Job state-change
2667459
docs: TheBook add chapter on SRR
ebe05a1
scripts: fix variable ordering in dcache-storage-descriptor
ea27cdf
cells: add unit test for CellRoutingTable
48ea014
pnfsmanager: silence delivery failures for FileAttributesTopic
67a8a26
pool: update “rep ls -s” formatting
8899988
cells: use Objects#requireNonNull instead of guava’s alternative.
e482513
cells: fix javadoc
dd48978
gplazma2-ldap: remove obsolete scala based plugin
2175c3f
cells: remove historic code
c6b17a5
common: revamp the byte column type in ColumnWriter
6c68637
dcache-xrootd: refit checksum handling after xrootd4j bug fix
9c4338f
dcache-frontend: notify PnfsManager of QoS transitions
e6534d4
dcache-resilience: allow handling of QoS transitions for resilient files
3dfed7e
dcache-core: add file attribute notification when AL/RP changes
3d06ecc
webdav: avoid logging non-error as an error
23dae03
common: fix bug in header space calculation for ColumnWriter
e836fe1
gplazma: x509 add heuristics for entity defn principals
b184d20
pool: include pool name in health-check reports
a3c7979
build: do not attempt to attach dir assembly to srm-client
98cc34e
gplazma: scitoken add support for multiple audience claims
c267db5
build: revert surefire upgrade to allow dCache builds to succeed
6d27231
build(deps): bump jackson-databind from 2.9.10 to 2.9.10.1
a602c36
dcache: restore pool compatibility with doors sending ‘Xrootd–2’ protocol version
451fa8b
chimera: do not update access latency on updat/insert in level_4
8a2ab43
build: fix maven warning about missing maven-enforcer-plugin version
86e4e23
srm: allow client to control whether srmPrepareToGet can trigger staging
c754523
build: fix cpd config, disable src-xref and generated code checking
1a45f19
build: avoid shipping log4j with dCache
e29da0b
resilience: avoid NPE if pool removed from resilience poolgroup
0356bc5
build: fix DataNucleus maven plugin logging errors
16f9b35
build: avoid using prerequisite argument in pom
3b1aa0c
build: choose a specific maven-groovy-plugin version
20ee75d
build: fix warnings when compiling FHS-compliant packages
4188f0e
resilience: override Object::equals, complete equals contract
0005a2d
frontend: fix NPE if limit is not specified
8ee6476
build: make building docker image optional and disabled by default
8cef28d
info: fix array index out of bound exception for admin commands
2365d13
pool: fix annotation in PoolMigrationCopyReplicaMessage
43f6472
srm: log number of SURLs in an SRM request.
e228d41
srm: improve access logging of srmBringOnline requests.
c9b49c4
srmclient: complete and tidy-up storageSystemInfo support
224ece3
gplamza: fix SciToken fetching remote content
08c05ea
gplazma: add support for REFEDS LoAs and rework x509 to include them
f1ce0a4
dcache-frontend: fix QoS pin semantics
9d3b029
dcache-xrootd: honor read paths when listing directories
9f6cd9f
pool: clean up master/push fc8dd482d13b8d089cce68859cfbb038cc7d8d48
924295a
libs: use potgres jdbc driver 42.2.8
f065a6d
pom: use java 11 friendly maven-surefire-plugin (junit)
314e195
libs: use java 11 friendly version of JNA
fc8dd48
pool: encrypted p2p transfers
473972b
pool: fix the xrootd version number on pool transfer service
4aeef4a
poolmanager: fix pool’s canonicalHostName change detection (part2)
12b4066
pool: clean code for HttpsTransferService
053dd1f
resilience: don’t compare Integer objects by refference
71dfffa
dcache-xrootd: refine error handling
e90b647
poolmanager: fix pool’s canonicalHostName change detection
261e786
dcache-xrootd: support the ‘tried’ CGI in xrootd door
6dae458
Pool Monitor: add host excluded unit tests and refactor
a8cd588
xrootd: fix kill mover command
3bcb15a
PoolManager: support selection filter for excluding pools on given hosts
274c313
dcache-frontend: avoid double fetch of sweeper data from pools
15807c7
doors: initialize IdentityResolverFactory after dependency injection
b28f730
cells: remove dead code dmg.util.cdb
ef32161
Revert “pom: require Java–11 at compile and run time”
77fe78a
pom: require Java–11 at compile and run time
7a891f5
sweeper: use in-memory map instead of repository for histogram data
387d981
pom: update aspectj maven plugin to java 11 capable
f9226d3
scitoken: do not use javax.activation.MimeType
235c71d
gplazma: stop using sun.security.rsa.RSAPublicKeyImpl
d779f40
pool:restructure httpsprofile in pool.xml by adding ssl-context-builder bean
92e2926
poolmanager: scan for matching pools when dynamic group is created
6751152
poolmanager: drop tags map argument in DynamicPGroup#addIfMatches
59becc2
poolmanager: include pool tags into SelectionPool record
2eb1f70
libs: move to DN 5
49ae166
How to enable scitokens in gplazma.conf
5de5570
dcache-history: expose refresh and timeout through admin commands
c06f419
chimera: explicitly specify encoding when converting string to bytes
23bec2d
nfs: re-try stale mover
c95f73b
nfs: restore transfers debug context on client retry
ead00eb
dcache-xrootd: replace constants for version number
e3a4d13
dcache-xrootd: update protocol version numbers
8ea611d
Pool Monitor: refactor to be more DRY
594cd1e
libs: update apache.commons:commons-compress to 1.19
124bcfd
cells: fix broken test (commit f64778f15f)
f64778f
cells: store CellAddressCore as gateway in CellRoute
34561e5
pool selection: make actual network hostname of pool available
c6d60dd
poolmanager: always allow removal of dynamic pool group
f467472
chimera: fix ABBA db deadlock when mkdir and rmdir run concurrently
cd74de0
util:fix DM_DEFAULT_ENCODING warning
5b8c58b
cells:fix DM_DEFAULT_ENCODING warning in services.login
e389138
cells: use ThreadLocalRandom when selecting a route
9f90535
cell: fix DM_DEFAULT_ENCODING warning
22d935c
nfs: do not wait for mover to shutdown for reads
8398f5f
Revert “Revert ”docker: Add a way to create docker image“”
87ae552
dcap: restart pool selection on OUT-OF-DATE error
926287f
gplazma-ldap: avoid thread leak by explicitly close NamingEnumeration
5ff1cf8
cells: fix DM_DEFAULT_ENCODING warning
7ac620e
Revert “docker: Add a way to create docker image”
c31f77a
[maven-release-plugin] prepare for next development iteration