Highlights

  • As a Golden Release, the focus of 5.2 is on robustness and stability. Many small changes (not all mentioned here) target handling rare error conditions, and logging has become more detailed across the board.

  • XRootD third-party copying support has matured, including credential delegation support, and can be used in production.

  • This release introduces the new concept of “zones” to improve networking performance in geographically distributed dCache instances. While 5.2 does not offer a full, admin-configurable system yet, this is an important milestone for future installation types.

Incompatibilities

  • The order in which fields are logged in the frontend log for WebDAV accesses has changed (but the set of fields itself has remained the same). This may require tuning of any custom log processors that rely on the field ordering.

  • The default setting for logging aborted FTP transfers was changed, so that sites upgrading to dCache v5.2 will now automatically log aborted transfers.

  • Pool tags may no longer use the literal string “zone”.

Acknowledgments

We gratefully acknowledge code and documentation contributions from Christoph Anton Mitterer and Onno Zweers.

Release 5.2.30

chimera

Filenames containing brackets are no longer treated as magic dot files.

dcap

Sometimes file restores on dCap door were hanging. This is now fixed.

Changelog 5.2.29..5.2.30

41bda767cb
[maven-release-plugin] prepare release 5.2.30
d2fefff34d
[maven-release-plugin] prepare for next development iteration
36ee505c16
dcap: fix automatic door retry on transient errors
73e6adba0f
chimera: fix dot file parser to handle weird file names

Release 5.2.29

gplazma

X.509 certificates that use the “SHA–384 with RSA” algorithm are now logged with this name instead of the raw OID value.

pool

dCache pools that are undertaking HTTP-TPC transfers will now wait longer for the remote servers to complete any post-transfer activity. Transfers that involve heavily loaded remote servers are now more likely to succeed.

Changelog 5.2.28..5.2.29

7c683c7860
[maven-release-plugin] prepare release 5.2.29
7b8aabe43f
pool: http-tpc increase timeout waiting for remote server post-processing
1b72a1696d
gplazma: include additional algorithm in LoginResultPrinter
5a791039b5
[maven-release-plugin] prepare for next development iteration

Release 5.2.28

nfs

The current release fixed previously observed infinite WRITE+COMMIT loop.

Changelog 5.2.27..5.2.28

8106b2761c
[maven-release-plugin] prepare release 5.2.28
70626265fe
nfs-proxy: op WRITE must use the same verifier as COMMIT
8d542a3cef
[maven-release-plugin] prepare for next development iteration

Release 5.2.27

dcache

There have been several occasions where in a dCache instance using an embedded ZooKeeper server such HA services stopped working because an ephemeral node was elected leader that has no associated service anymore; it should have been deleted. This state might go undetected for a longer time, and removing such stale nodes in order for the services to continue functioning as intended requires manual intervention.

This is fixed now and the functioning of HA services using ZooKeeper-based leader elections is more reliable.

resilience

A more robust handling of PSU changes which remove pool, unit or group mappings was introduced.

Recently introduced improvements changed the way resilience handles file “removal” (no longer setting the repository entry to ‘removed’ but simply by caching the replica). This change, however, did not take into account the handling of broken files. Encountering a broken file, it would indiscriminately attempt to remove it, whether it was cached or not; this was leading to an infinite loop, with the file operation continuously iterating without doing any further work. This situation could potentially hang the pool scans (if there are as many broken files as there are scan threads), and even the file operation queue. The current release fixed this issue and there is no any potential for stalled operations when encountering broken replicas.

Changelog 5.2.26..5.2.27

76048255b3
[maven-release-plugin] prepare release 5.2.27
837e45d090
dcache-core: fix embedded ZooKeeper persisting stale ephemeral nodes
53f0401450
dcache-resilience: ignore broken cached files
f0fa8ac73b
dcache-resilience: protect access of pool info map against NoSuchElementException
0da727a6ec
[maven-release-plugin] prepare for next development iteration

Release 5.2.26

macaroons

Fixed macaroons validation where in case of multiple combined caveats for some of them only the digital signature was checked.

resilience

If a FileOperation is canceled while its subtask is running a NullPointerException was thrown. This is now fixed.

Changelog 5.2.25..5.2.26

4d0802bc06
[maven-release-plugin] prepare release 5.2.26
fbdc2791e6
dcache-resilience: avoid NPE in file operation cancel
6cd2960032
macaroons: verifyCaveat should return false for unprocessed caveats
7faa85e9c8
[maven-release-plugin] prepare for next development iteration

Release 5.2.25

cleaner

Database passwords can now be stored in a dedicated file.

pool

Failing HTTP-TPC transfers where the remote party sends a malformed RFC 3230 checksum value are now fixed.

util

The current release fixed pgpass file handling for jdbc URLs with options, like jdbc:postgresql://dbhost:5432/foo?prepareThreshold=3

Changelog 5.2.24..5.2.25

b5862742aa
[maven-release-plugin] prepare release 5.2.25
6bc9442a7f
pool: do not fail transfer if RFC3230 is badly formed.
b691a090ad
util: fix pgpass support for jdbc urls with options
140d894ae0
cleaner: add support for db password file
2340df1866
[maven-release-plugin] prepare for next development iteration

Release 5.2.24

pool

The current release fixed interrupted p2p transfer issue and the pool should not be disabled.

Changelog 5.2.23..5.2.24

be7786d
[maven-release-plugin] prepare release 5.2.24
48a57b6
pool: rework interrupt processing on p2p
0e34a61
[maven-release-plugin] prepare for next development iteration

Release 5.2.23

common

dCache no longer logs OIDC access tokens or macaroon.

dcache-xrootd

The version is upgrated to xrootd4j 3.5.7 with lossen username validation, so that

usernames like foo.1234:56 are no longer rejected.

The current release fixed java.lang.IllegalStateException: ChecksumChannel must not be written to after getChecksums.

TPC client first will be shut down when the pool netty channel goes inactive.

pnfsmanager

The current release fixed a NullPointerException when a file is deleted while a directory listing is being compiled.

pool

When a legacy mover is killed by job timeout manager thread interruption mechanism is used. As mover thread is taken from a thread pool, the next task might see this interrupted state, if not cleared. This state can be checked bu other components and trigger false errors: java.lang.InterruptedException THREAD_INTERRUPTED: InterruptedException.

This is now fixed and pool is not disabled itself after slow running mover is killed by jtm.

Changelog 5.2.21..5.2.23

7efd6d4
[maven-release-plugin] prepare release 5.2.23
c5be7e9
pnfsmanager: avoid NPE if file is deleted during directory listing
a2499b4
common: do not log bearer token values.
299b60d
pool: rework interrupt processing in AbstractMoverProtocolTransferService
158e153
dcache-xrootd: bump to xrootd4j 3.5.7
fcf57f0
pool: clean interrupted state after mover complete (again)
a6d5a2d
pinmanager: use PoolManagerStub when talking to pool manager
c3b1d2c
pool: clear mover thread’s interrupted state before returning to pool
bd1ff48
wrap the client shutdown with a try … catch for InterruptedException
ae4d50a
dcache-xrootd: cancel TPC transfer when client disconnects unexpectedly from pool
952a2ef
[maven-release-plugin] prepare for next development iteration
0157190
[maven-release-plugin] prepare release 5.2.22
626080c
pool: repository account capacity correctly if file channel closed
8fad854
[maven-release-plugin] prepare for next development iteration

Release 5.2.21

frontend

The frontend’s inotify-over-SSE now honour the ‘frontend.root’ configuration property and any user-specific root. This means that, for any inotify subscription, the path is calculated relative to the doors’ root and the user-specific root. For many dCache sites, this has no impact as they are using the default for both; however, the user root is used to implement the macaroon’s ‘root’ caveat. The current release fixed how inotify subscription requests are processed when made with a macaroon with a ‘root’ caveat.

pool

When making an HTTP third-party copy (HTTP-TPC), dCache no longer sends the ‘Authorization’ HTTP request header in any subsequent request when the remote server responds with a redirection.

The unix xrootd tpc security plugin was included in order to enable the dCache TPC client to use a dCache pool as source when signed hash verification is on. However, this is now fixed and no special configuration necessary for organizations (like Tier 1) needing to communicate with EOS.

Changelog 5.2.20..5.2.21

86cf625
[maven-release-plugin] prepare release 5.2.21
b6f83b6
pool (xrootd): make tpc security plugin default unix
5931f53
docs: add missing architecture diagram
2c0713b
frontend: honour door and user root for inotify subscriptions
f48066b
frontend: refactor inotify client-path to dCache-path mapping
7503540
frontend: add SelectionContext for event subscription
d4d4e26
pool: drop Authorization HTTP header on redirected
eb1a180
[maven-release-plugin] prepare for next development iteration

Release 5.2.20

frontend

When frontend is run in a core domain and there is no history service reachable, the retry on no route to cell ends up spamming the message queues. This is now fixed.

login-broker

The lb set update command ignored the provided value and set the existing one again. This is now fixed and lb update times can be modified through admin interface.

nfs

The current release added modifications such as do not wait for mover to shutdown for reads, which was blocking client and wait until mover is actually stopped. This modification resulted slight performance improvement for IO on small files.

xrootd

The current release fixed compatible level security for sigver.

The xrootd client has a command-line option, --path, which tells the server to create missing directories. This option is included in two-party copy, but setting it for TPC has no effect. TPCs which wish to write to dCache (as destination) to a non-existent subdirectory fail. This is now fixed and dCache no longer fails in these cases.

Changelog 5.2.19..5.2.20

8016208
[maven-release-plugin] prepare release 5.2.20
62780c7
login-broker: fix ‘lb set update’ command
1c65a81
nfs: do not wait for mover to shutdown for reads
df87ccd
dcache-xrootd: always create missing directories on write
27ebaee
dcache-xrootd: fix compatible level security for sigver
d6cb494
dcache-frontend: remove retry flag on sendAndWait to history service
bd99bf4
[maven-release-plugin] prepare for next development iteration

Release 5.2.19

nearline-storage

There were issues in pool space allocation/de-allocation in conjunction to HSM connectivity and handling restore errors resulting in failing stage end up with used space miscalculation: pool_read, fault occurred in repository: Internal repository error. Pool restart required:, cause: java.lang.IllegalArgumentException: Cannot set used space to a negative value.

This is now fixed and disk copy is removed if restore from HSM failed.

Changelog 5.2.18..5.2.19

5cda208
[maven-release-plugin] prepare release 5.2.19
8b69757
nearline-storage: remove disk copy if restore from HSM failed
45c56be
[maven-release-plugin] prepare for next development iteration

Release 5.2.18

dcache

QOS migration policy engine was raising JVM error when no tape pool found. This is now fixed and the normal behavior is that HTTP error code reported back with No HSM pool found.

frontend

The current release improved error responses and they are more specific now.

The current release fixed the NPE stack trace arised because the pool data could be sent with a default sweeper data object.

Changelog 5.2.17..5.2.18

498232c
[maven-release-plugin] prepare release 5.2.18
1a4a541
dcache-frontend: make ErrorResponseProvider return the more specific error message
03b6a51
dcache: qos migration policy engine should not raise JVM error when no tape pool foundMotivation:
dc146fe
dcache-history,dcache-frontend: guard against unconfigured sweeper histogram
d8c769b
[maven-release-plugin] prepare for next development iteration

Release 5.2.17

frontend

From RESTful admin API, the POST to pools/{name}/usage/mode -d {"rdonly":true} failed but {"rdonly": "true"} succeeded.

This is fixed now and both boolean and string work.

The current release fixed a bug where an SSE client that receives many events without the connection breaking will eventually exhause all memory in the JVM.

The current release fixed stack trace in logs.

Changelog 5.2.16..5.2.17

e7a24a8
[maven-release-plugin] prepare release 5.2.17
f1d3759
dcache-frontend: allow pool enable/disable to use boolean JSON value
fcfe98a
dcache-frontend,history: protect against missing highest bin in histogram data
d80363a
frontend: fix memory leak in long-running SSE connections
35014bb
[maven-release-plugin] prepare for next development iteration

Release 5.2.17

frontend

From RESTful admin API, the POST to pools/{name}/usage/mode -d {"rdonly":true} failed but {"rdonly": "true"} succeeded.

This is fixed now and both boolean and string work.

The current release fixed a bug where an SSE client that receives many events without the connection breaking will eventually exhause all memory in the JVM.

The current release fixed stack trace in logs.

Changelog 5.2.16..5.2.17

e7a24a8
[maven-release-plugin] prepare release 5.2.17
f1d3759
dcache-frontend: allow pool enable/disable to use boolean JSON value
fcfe98a
dcache-frontend,history: protect against missing highest bin in histogram data
d80363a
frontend: fix memory leak in long-running SSE connections
35014bb
[maven-release-plugin] prepare for next development iteration

Release 5.2.16

frontend

The current release added support for OIDC names and Client-IDs with spaces.

Changelog 5.2.15..5.2.16

b69aa19
[maven-release-plugin] prepare release 5.2.16
1ce186a
dcache, frontend: release dcache-view version 1.5.7
af1c9ca
[maven-release-plugin] prepare for next development iteration

Release 5.2.15

dcache-xrootd

Now door does not fail to start if there is no voms directory on the host.

frontend

The current release removed unnecessary login requirement on restores and transfers.

The current release fixed a bug in the frontend if the inotify events are used.

skel

The current release repaired erroneous batch directives before cell creation.

Now it is fixed and domain is not left in zombie state after a fatal error, but restarts, as it should.

srm

Now host IP is used for comparison when determining if SURL is local.

Changelog 5.2.14..5.2.15

a178d50
[maven-release-plugin] prepare release 5.2.15
93f930b
skel: repair erroneous batch directives before cell creation
264117c
dcache-xrootd: don’t initialize delegation provider when there is no gsi module
173c4af
dcache-frontend: remove unnecessary login requirement on restores and transfers
38b27a9
srm: use host IP for comparison when determining if SURL is local
2296705
frontend: events inotify fix deadlock
35fed0c
dcache,frontend: release dcache-view version 1.5.6
9cc78e8
[maven-release-plugin] prepare for next development iteration

Release 5.2.14

cell

Curator client was not able to restore the connection to ZK server after network partitioning. The is now fixed.

skel

The current relase fixed tape-reserved size calculation.

webdav

The current release fixed, where the WebDAV door failed to follow RFC 4918. This make some clients reject dCache WebDAV door as a valid WebDAV endpoint.

Changelog 5.2.13..5.2.14

fd8e9e9
[maven-release-plugin] prepare release 5.2.14
5e2f441
Fix tape-reserved size calculation
e53ea3a
webdav: include DAV header in OPTIONS requests.
ef58be4
cells: do not re-define zookeeper watcher
da8aa45
[maven-release-plugin] prepare for next development iteration

Release 5.2.13

canl

The current release updated lib version to 2.5.1.

gplazma

The current release fixed URL-prefix SciToken parsing and error handling if JWT contains malformed SciToken scopes.

webdav

When making a cross-origin call to webdav door from e.g. the frontend by using dcache-view, this call would be blocked if the client want to forward the user certificate. This is now fixed.

Changelog 5.2.12..5.2.13

5bcc20e
[maven-release-plugin] prepare release 5.2.13
cb79d64
gplazma: scitoken add unit tests and fix SciTokenScope
1d0a126
webdav : set Access-Control-Allow-Credentials to true
5713218
canl: update to version 2.5.1
6b66cf2
[maven-release-plugin] prepare for next development iteration

Release 5.2.12

chimera

Enstore client, encp, stores information in layer 4 (level_4). It was natural to assume that files written directly by encp would have NEARLINE/CUSTODIAL AccessLatenct/RetentionPolicy. Although, setting of access latency interferes with QoS.

This is fixed no and encp does not interfer with file access latency (say if it is specified to be ONLINE).

config

A typo in the dcap config file was fixed correcting dcacp.enable.kafka to dcap.enable.kafka.

frontend

The current release fixed a NPE bug (which is then logged as a stack-trace) that is triggered when the reply does not contain a media type.

gplazma

The SciToken plugin will now reject any JWT where there is none of the expected scopes defined. This allows dCache to support both OpenID-Connect and SciTokens.

webdav

The current release fixed an issue of transfers through dCacheView when the webdav door is configured with empty webdav.allowed.client.origins value, which is the default value.

Changelog 5.2.11..5.2.12

2859046
[maven-release-plugin] prepare release 5.2.12
a07b585
frontend: avoid NPE when request has no media type
f42f51d
dcache: add null check to pool info collector util
289ea51
config: fix typo in property name
a4dc031
chimera: do not update access latency on updat/insert in level_4
2e037c0
gplazma: scitoken fix two issues with SciToken plugin
aeb2c30
webdav: fix CORS when all clients are allowed to connect
fb453df
[maven-release-plugin] prepare for next development iteration

Release 5.2.11

srm

The current release fixed a problem resulting in high CPU use in SrmManager if clients are attempting to pin a file and PinManager is unavailable.

A regression fixed where SrmManager will reject all QUEUED jobs and INPROGRESS BringOnline requests on restart, if there are no SRM doors running when SrmManager starts.

Changelog 5.2.10..5.2.11

b0dba87
[maven-release-plugin] prepare release 5.2.11
0e57e10
SrmManager: fix handling of saved requests on start-up
009f623
SrmManager: avoid spamming if PinManager is down
8db6df3
[maven-release-plugin] prepare for next development iteration

Release 5.2.10

doors

The current release fixed a bug where running the lb set tags admin command without any arguments triggers a NullPointerException.

pool

The current release improved error messages about jobs cancellation.

scripts

The dcache-storage-descriptor command no longer requires a URL argument.

Changelog 5.2.9..5.2.10

d5096b2
[maven-release-plugin] prepare release 5.2.10
dabb072
doors: fix “lb set tags” command with no arguments
686be7b
pool: improve messages when migration job is cancelled.
34b50e3
scripts: fix variable ordering in dcache-storage-descriptor
9a71d94
docs: TheBook add chapter on SRR
fd4e37d
[maven-release-plugin] prepare for next development iteration

Release 5.2.9

dcache

The current release restored pool compatibility with doors sending Xrootd-2 protocol version.

gplazma

The SciToken gplazma plugin now supports the audience (aud) claim where the claim’s value is an array. This allows dCache to support SciTokens with multiple audience values.

pool

Pool health-check log messages now include the pool’s name.

webdav

On an unsuccessful HTTP-TPC pull request, dCache will delete the file. If this deletion did not work then an error was logged. This is fixed now and failures to delete the incomplete file from a failed HTTP-TPC pull request, where the incomplete file has been deleted by some other means are now logged at DEBUG level, rather than WARN level.

xrootd

The current release refited checksum handling after xrootd4j bug fix.

Changelog 5.2.8..5.2.9

1f8567d
[maven-release-plugin] prepare release 5.2.9
847222b
dcache-xrootd: refit checksum handling after xrootd4j bug fix
5419fda
webdav: avoid logging non-error as an error
ca96e1d
pool: include pool name in health-check reports
b1f4aa4
gplazma: scitoken add support for multiple audience claims
63a2620
build(deps): bump jackson-databind from 2.9.10 to 2.9.10.1
e289df7
dcache: restore pool compatibility with doors sending ‘Xrootd–2’ protocol version
3d64e71
[maven-release-plugin] prepare for next development iteration

Release 5.2.8

frontend

The current release fixed QoS pin semantics.

A bug is fixed in frontend that results in a NullPointerException for billing queries where no limit is specified.

srm

Investigation into tape carousel has highlighted limitations in what dCache logs. In particular, information about the pin lifetime for srmBringOnline is missing.

This is now fixed and dCache SRM access logging contains information about client’s desired pin lifetime in srmBringOnline requests.

xrootd

The current release refined error handling by providing more meaningful (or at least consistent) error responses to the client.

Changelog 5.2.7..5.2.8

7a67863
[maven-release-plugin] prepare release 5.2.8
7da9409
frontend: fix NPE if limit is not specified
1956d34
srm: improve access logging of srmBringOnline requests.
b081289
dcache-frontend: fix QoS pin semantics
9b1f54a
dcache-xrootd: refine error handling
865710c
[maven-release-plugin] prepare for next development iteration

Release 5.2.7

Changes affecting multiple services

The Apache Commons Compress library used in dCache was updated to version 1.19.

A rare deadlock situation in the Chimera database was eliminated. In cases where, within the same directory, concurrent mkdir and rmdir events happened, transactions within the database could deadlock. This would be indicated by the message

ERROR: deadlock detected

in the logs.

pool

A regression introduced in the last release caused a ClassNotFoundException on pool startup with a message like

[ERROR] Server responded with an error: [3012] Failed to open file (Protocol Xrootd-4.0:123.32.123.32:38374 is not supported [27])

appearing in the logs. This release fixes that issue.

There were reports of extraordinarily high CPU usage on pool nodes with a large number of cached files. Through an optimization of the sweeper, CPU usage was reduced significantly.

xrootd

This release fixes a small regression in the kill mover command that would occasionally fail to kill all targeted movers.

This release fixes a vulnerability in dCache’s XRootD protocol implementation. We recommend that all sites update their XRootD doors. Details will be made available through EGI Security and, in a week’s time, through an update to these release notes.

Changelog 5.2.6..5.2.7

2e98b03a4f
[maven-release-plugin] prepare release 5.2.7
47ab7704ba
dcache-xrootd: honor read paths when listing directories
6a62dac6dc
pool: fix the xrootd version number on pool transfer service
3950434553
resilience: don’t compare Integer objects by refference
85d2e6b470
xrootd: fix kill mover command
f13708238d
doors: initialize IdentityResolverFactory after dependency injection
18f63397cf
sweeper: use in-memory map instead of repository for histogram data
76d52b26ac
dcache-xrootd: replace constants for version number
44ec827db0
dcache-xrootd: update protocol version numbers
0c4f84f18c
libs: update apache.commons:commons-compress to 1.19
0d8144d766
chimera: fix ABBA db deadlock when mkdir and rmdir run concurrently
37daa02f8a
[maven-release-plugin] prepare for next development iteration

Release 5.2.6

dcap

dcap door could not handle out-of-date errors. This is now fixed.

gplazma

The current release fixed thread leak by explicitly close NamingEnumeration

Changelog 5.2.5..5.2.6

5ab8c5a
[maven-release-plugin] prepare release 5.2.6
cc6e167
dcap: restart pool selection on OUT-OF-DATE error
74e08dc
gplazma-ldap: avoid thread leak by explicitly close NamingEnumeration
14e67d1
libs: update jackson-databind to 2.9.10
df306ed
[maven-release-plugin] prepare for next development iteration

Release 5.2.5

chimera

Admins may require to change top-level tags in the middle of a directory tree and those tags should be propagated to all subdirectories. To support this a recursive update tag command is required.

Admins now can set/update and push directory tags as:

````

chimera:/# writetag /exports/data foo-tag foo chimera:/# pushtag /exports/data foo-tag

nfs

The current release added ability to set file checksum(s) via dot command.

pool

The current releasse improved metadata check speed on pool re-start.

resilience

If there was two replicas, one of which was precious, and some change was triggered which reduces the required count for the file to 1, the removal operation was failing. This behavior is now fixed.

srm

A new user community requires the srm tools to be able to handle very large file listings. During preliminary tests, OutOfMemory errors from the srmls tool were observed. This is now fixed and srm can now support operations on very large file lists without running out of memory.

webdav

The current release added allow header to list of response headers for OPTION method request.

Changelog 5.2.4..5.2.5

4faec49
[maven-release-plugin] prepare release 5.2.5
8f217bc
httpd: escape status field in HttpPoolMgrEngineV3
def422e
webdav: add allow header to OPTION method request
a8edd03
nfs: support ability to set file checksum(s) via dot command
fc0bd33
chimera: add command to push tags into subdirectories
cbd868d
libs: use h2 version 1.4.199
ad0a8e6
srm: Remove JVM memory limits
eed5b71
resilience: fix remove count when replica is precious
1ce9a9d
chimera: update ctime on checksum set and remove
c01691c
Update config-zookeeper.md
3e4ab82
dcache: fix RepositorySubsystem unit test NPE
73eb4d4
pool-repository: Improve metadata check speed on pool re-start.
33a8aaa
[maven-release-plugin] prepare for next development iteration

Release 5.2.4

common

The current release fixed formatting of error message in Checksum.

frontend

Admins may now configure frontend to specify in which country (or countries) data may be stored. This information is visible through dCacheView.

gplazma

The current release fixe remote reading of JSON with UTF–8 CharSet and dCache can now work with OPs that use utf-8 charset.

nfs

The current release added ability to get valid checksum types via NFS. Usage cat ".(checksums)()". Returns a new-line delimited list of checksum type names.

util

The current release improved log messaging so that CDC context is available in during retries.

Changelog 5.2.3..5.2.4

b5c6b56
[maven-release-plugin] prepare release 5.2.4
17032a6
nfs: checksum types dot command
4f31c1c
frontend: make geographic placement configurable
6e85512
common: fix formatting of error message in Checksum
069d9eb
scitoken: fix remote reading of JSON with UTF–8 CharSet
f457d8a
util: decorate retry executor with CDC-aware wrapper
774fe07
[maven-release-plugin] prepare for next development iteration

Release 5.2.3

nfs

NPE on “show transfers” command is now fixed.

webdav

The current release fixed CORS for WebDAV doors that do not allow anonymous access; in particular, to support dCacheView uploading and downloading files with such authentication-required WebDAV doors.

Changelog 5.2.2..5.2.3

cf0ed9a
[maven-release-plugin] prepare release 5.2.3
4dfd693
nfs: fix NPE on “show transfers” command
db319cb
docs: Update UserGuide to use guide-specific navigation header
638ef78
webdav: fix cross origin resources sharing issue
f2d488f
[maven-release-plugin] prepare for next development iteration

Release 5.2.2

chimera

The shell infrastructure supports commands being given interactively, on the commandline (e.g., ‘chimera mkdir /path/to/dir’) and from stdin (e.g., ‘echo “mkdir /path/to/dir” | chimera’). chimera now supports the latter case and properly shows command output when invoked in that fashion.

frontend

This release updates dCache View to 1.5.5.

nfs

The NFS door now correctly handles situations where newly created read-only files could occasionally not be written into:

f = os.open('test.txt', os.O_WRONLY|os.O_CREAT, 0400)
os.write(f,"Hello pNFS!")
os.fsync(f)
os.close(f)

will now succeed.

webdav

A client may issue a PUT request that targets an existing collection resource; i.e., attempt to write a file as a path that is a directory. dCache, until now, responded with an incorrect status code of 500. This release changes the status code for this operation to 405 (Method not allowed), thus keeping closer to RFC 4918.

xrootd

This release improves compatibility with the xrdcp client in versions >4.9 by responding correctly to query strings requesting a specific checksum type.

Changelog 5.2.1..5.2.2

f69d003419
[maven-release-plugin] prepare release 5.2.2
e7023b90cf
dcache, frontend: release dcache-view version 1.5.5
231163b4bf
nfs: introduce workaround ‘permission deny’ on layout commit
5b01c29f8a
chimera: chimera shell should show output when commands come from stdin.
9712188578
webdav: return 405 status code for PUT requests targeting collections
a0cb14ce62
dcache-xrootd: add checksum cgi handling to door query
2c89042850
[maven-release-plugin] prepare for next development iteration

Release 5.2.1

frontend

dCache View was updated to version 1.5.4.

ftp

This release fixes a regression that would cause erroneous NullPointerExceptions when FTP doors were probed by HAProxy.

many

The dcache pool ls command now provides correct output even if the pool is defined with a single-digit number of bytes.

pool

The default value for the xrootd Third-Party Copying server response timeout, pool.mover.xrootd.tpc-server-response-timeout, was increased from 2 to 30 seconds to provide more robust behaviour in the face of high loads and network congestion.

transfermanager

Error messages where the TransferManager is unable to discover the current status of the pool mover now include the pool’s name.

xrootd

This release fixes an issue that would cause high CPU load on xrootd.

TPC lite (delegation) is now more robust when communicating with EOS destinations.

XRootD doors now correctly communicate their TPC capabilities to clients.

Changelog 5.2.0..5.2.1

16792c33ed
[maven-release-plugin] prepare release 5.2.1
b5d705b92b
scripts: avoid copy-n-paste error when calculating pool size
f71b0a14de
dcache-xrootd: check the session for credential on source open
7d32476800
dcache-xrootd: respond to tpc query correctly*
a4749b2843
dcache-xrootd: compute VOMs CA refresh interval using unit
ba7b902a4e
dcache, frontend: release dcache-view version 1.5.4
d60f840f3a
pools: make the xrootd tpc response timeout less aggressive
d13d5a9a45
transfermanager: include pool name in error for ‘mover ls’ failures
969af6d25e
ftp: avoid NPE on HA-Proxy probes
5b3cf0e620
[maven-release-plugin] prepare for next development iteration

Release 5.2.0

Admin

The interactive admin shell gained two improvements in logging: Log entries from admin commands now include the SSH session-ID, and commands that were executed through the admin shell are logged in detail. An example:

level=INFO ts=2019-05-02T12:49:58.518+0200 event=org.dcache.services.ssh2.connect remote.socket=/2001:638:700:xx:52224
level=INFO ts=2019-05-02T12:49:58.561+0200 event=org.dcache.services.ssh2.login username=admin remote.socket=/2001:638:7xx:52224 method="PublicKey (RSA SHA256:al9RPdXETJXX)" successful=true
level=INFO ts=2019-05-02T12:50:06.657+0200 event=org.dcache.services.ssh2.exec username=admin cmd.args="\\c PoolManager"
level=INFO ts=2019-05-02T12:50:12.000+0200 event=org.dcache.services.ssh2.exec username=admin cmd.destination=[>PoolManager@core-dcache-lab] cmd.args="rc ls"

Alarms

The alarms system gained some improvements in logging, such as reporting the originating pool name in POOL_DEAD messages. Additionally, messages can be sent directly to Kafka. The new properties dcache.log.kafka.topic, for setting the topic name, and dcache.log.level.kafka to set the log level, were added to configure this feature.

Billing

The database tables billinginfo, storageinfo, doorinfo and hitinfo can, if desired, be automatically truncated. This allows a logrotate-style approach to limiting the growth of the billing database, which can fill up quickly over the course of time if there is a lot of door activity.

A built-in cron can be set to run every 24 hours in order to remove older rows from these “fine-grained” tables.

The following properties are relevant:

billing.enable.db-truncate (default = false)
billing.db.fine-grained-truncate-before (default = 365)
billing.db.fine-grained-truncate-before.unit (default = DAYS)

Chimera

The Chimera database now keeps track of the time when a file was added to the trash table. This timestamp is stored in the ictime field of t_locationinfo_trash.

Cleaner

The cleaner now can wait a specified amount of time after deletion before file finally get removed from pools and HSM. Two new properties were added to control this behaviour:

cleaner.service.grace-period = 0
cleaner.service.grace-period.unit = SECONDS

Frontend

The HTTP access log now includes HTTP Entities (if they were present in the request) to facilitate debugging.

Additionally, the frontend now provides useful, minimal logging of API requests in the access log file.

Periodic activity associated with the frontend door is now logged with the door’s cell name. Such messages will also appear in the door’s pinboard.

FTP

The default settings for the logging of aborted FTP transfers were changed, so that Sites that upgrade to dCache v5.2 will now automatically log aborted transfers. The reason for that change is that it provides post mortem information about transfers where the client received less data than it requested. Enabling it only after someone reports a problem could be too late (if the problem is not reproducible).

The log entries for aborted transfers now also include detailed information about the cause of the issue, as well as being more compact.

gplazma

As for several other services, logging was improved to make troubleshooting easier. Failed login attempts using OpenID-Connect, particularly from dCacheView, are now logged.

Info

The info service now list a domain’s zone as the metric ‘domains.domain-name.static.zone’, if the domain is configured to have a zone. If the domain is not configured with a zone then this metric is not published.

NFS

When a pNFS client using the flexfile layout reports a connection error to a pool, then, as an alternative to just logging the issue, an alarm will be raised.

Periodic activity associated with the NFS door is now logged with the door’s cell name. Such messages will also appear in the door’s pinboard.

Pool

The timeout for XRootD 3rd-party-copying responses was increased to 30 seconds, making operations more robust.

Pool start-up logging is now recorded against the corresponding pool cell name.

A pool now includes a ‘zone’ tag where the value is zone within which the pool resides. Pools no longer accept the tag ‘zone’.

Various internal changes should make pool operations in general more robust.

Pool Manager

In a situations, where for a given READ access two links are available and there are no pools available at link with a higher priority, then a pool from a lower link priority will be selected if file is available.

Resilience

Resilience is now able to use a CACHEd file as a replication source in cases where no STICKY copy is available. This also means that a cached initial copy (such as if a resilient pool is marked lfs=volatile and is chosen by the pool manager) will not stop resilience from creating the requisite number of permanent copies.

Various other changes make Resilience’s operations more robust.

Transfer Manager

Error messages where the TransferManager is unable to discover the current status of the pool mover now include the pool’s name.

An optimization to retrying mover operations makes Transfer Manager’s operations more efficient: If a pool does not support a particular transfer type, retries on that pool will not even be attempted.

WebDAV

Periodic activity associated with the WebDAV door is now logged with the door’s cell name. Such messages will also appear in the door’s pinboard.

XRootD

The client timeout is now configurable through two new properties or, alternatively, an interactive admin command.

The Third-party embedded client has a timer which will interrupt and return an error if the response from the server does not arrive after a given amount of time.

The default values for this can be controlled by the properties:

pool.mover.xrootd.tpc-server-response-timeout
pool.mover.xrootd.tpc-server-response-timeout.unit

These are set to 2 seconds iby default, matching the comparatively aggressive behavior of the SLAC implementation. However, dCache allows you to control this dynamically as well, using the admin command

\s <xrootd-door> xrootd set server response timeout

This could conceivably be necessary under heavier load.

Zookeeper

The Zookeeper library was updated to version 3.4.14, addressing CVE–2019–0201.

Changelog from 5.1.0 to 5.2.0

a4749b2843
dcache-xrootd: compute VOMs CA refresh interval using unit
ba7b902a4e
dcache, frontend: release dcache-view version 1.5.4
d60f840f3a
pools: make the xrootd tpc response timeout less aggressive
d13d5a9a45
transfermanager: include pool name in error for ‘mover ls’ failures
969af6d25e
ftp: avoid NPE on HA-Proxy probes
5b3cf0e620
[maven-release-plugin] prepare for next development iteration
a329e2d7a6
[maven-release-plugin] prepare release 5.2.0
2804dcdcb9
Revert “docker: Add a way to create docker image”
4fb9567392
[maven-release-plugin] prepare branch 5.2
796e4bf6fb
core: add support for Properties prefix configuration
a1c615ae50
pool: include diagnostic context when logging why a replica was removed
481e5ec180
libs: use jersey–2.28
4e3e760523
pom: move dependency version information into parent pom
de09a3a6af
nfs: raise an alarm when client reports connection issues with a pool
c1f7183f3c
poolmanager: don’t fallback selection links by default
bb94ab1599
UnpinProcessor: add logging
2377c86beb
frontend: add HTTP entities in the access log file.
300db762c5
cells: remove safety buffer and enforce exit on OOM
9a85d76cce
core: fix pool selection in killAll command of TransferManager
3dc0af4cb8
nfs: enforce charset encoding when processing dot files
487dac8e81
chimera: remove historic code
f20fa88ed2
pool: avoid IllegalStateException in ‘csm check *’ command
650063856b
frontend: add basic access logging
388e6d2b6a
dcap: fix premature close of kafka sender
7885b185b6
sweeper: compute now after the values have been fetched
f853024421
dcache-xrootd: add ability to override default timeout for server response (TPC)
2244b9a5ff
dcache: abstract common features of HTTP access logging
21b1650492
frontend: include CDC in scheduled activity
f41b4a9d1b
nfs: include CDC in scheduled activity
c334ea1c58
webdav: include CDC in scheduled activity
61de56a69c
pool: ensure initialisation thread has correct CDC information
71723b8c39
dcache-xrootd: fix delegation client lifecycle management
768ff98f1a
pinmanager: add LoggingDao to monitor changes in persistent state.
1461b4c3f1
gplazma: consider bearer token when deciding whether to log failures
bf7e4ea5b5
gplazma: add extra OIDs for Key-Usage
f424d8b376
cells: remove unused constructor for AbstractCell
0d5ac9cb3b
pinmanager: fix issues with logging context of background activity
8d4db2d0f2
cells: work-around Curator creating custom ThreadFactory
b528098af8
cleaner: add grace period before files get cleaned
d34292dc30
pool: update replica commands to use CommandException
654bde3daf
TimeUtils: use ‘now’ when describing the current time
cd2d04e72d
cleaner: stop using deprecated guava API
e12f41993d
chimera: keep track when a file was added into trash table
4c99ed65c5
libs: use zookeeper–3.4.14
4009541521
common: allow counting histogram to accept negative values
040c86b1d6
pool (sweeper): catch negative file lifetime value
bdaf63f0a1
dcache: cleaned up pool admin command outputs
a770085352
PinManager: change info message
b36b612add
pinmanager: add logging info
bab39ae28f
pom: use jackson-databind/annotations/core version 2.9.9 CVE–2019–12086
1faec76731
UnpinProcessor: fix assumed typo {)
f77fd77e27
webdav: allow transfers as user with role ‘admin’
cbf920fd55
pinmanager: avoid NPE if no argument given for ‘bulk ls’ command
8225651652
pool: add ‘zone’ tag
563aaf9273
Pgpass file permission for POSIX (Issue #4670) (#4682)
b1dfe06a79
alarms: add pool name to POOL_DEAD alarm
2c5abae30e
cells: add support classes to learn in which zone they reside
aea1c4b4a5
cells: make tunnels aware in which zone the remote domain resides
9379199b77
info: publish a domain’s zone
27cb88fdfd
pool: fix reordering of removable replicas on access
d2472e4edc
pool: fix storage of replica last access time
b03d514de4
ftp/pool: enable aborted transfer logging by default
619d9ee193
pool: ftp include cause in aborted transfer report
be1f0e1a99
pool: ftp avoid printing useless information in abort log
c05ccad2e0
cells: add zone concept
d5e308f90d
Revert “dcap: fix premature close of kafka sender:”
68c2ddc60d
dcap: fix premature close of kafka sender:
12d872d53e
dcache-resilience: account for waiting states
babd708540
dcap/pinmanager: stage request for unknown location results in NPE
571ad60872
admin: include SSH session-ID in admin command logs
2852519e21
pool: inotify avoid race in IN_CLOSE_WRITE event
ca21bdd8f6
libs: use jetty 9.4.18.v20190429
9e34434e35
dcap batch : fix handling of dcap.kafka.topic variable
348b459990
pool: remove dummy VFS used by NFSv4MoverHandler
8ad4ca3a7a
jetty: make CanlContextFactory subclass of jetty.ssl.SslContextFactory.Server
b91240daff
frontend: fix race on client reconnecting
3823f5008d
docs: UserGuide reformat JSON to be more consistent, fix some JSON errors
2ae15e3138
docs: UserGuide override prism’s CSS to leave space for drop-shadow
2ba2a67d9b
ssh: add session to access log file
be520165b4
dcache (billing db): add capability for automatic truncation of fine-grained info tables
1f3424199d
admin: log executed commands into access log
abe853ea47
docs: Added link to Transport Security chapter
4f8b8ba486
11667: dcache-xrootd: add full proxy delegation to door
c68f470a80
docs: Added link to Transport Security chapter
d4f481c36e
11667: dcache-xrootd: add full proxy delegation to door
b02a110df8
dcache-xrootd: implement proxy delegation client
35cd7adc11
dcache-resilience: allow cached file as source when no sticky available
a389f5c6e7
pom: jackson.databind, jackson.annotations, jackson.core version 2.9.8 CVE–2018–14719
0b43b86d73
pom: use jackson-databind, jackson.annotations, jackson.core to 2.9.6
5b5d951096
alarms: sending alarms events in json format
120f702739
pom: use jetty 9.4.17.v20190418
4ed4cd0474
Update frontend.md
4ca91914cb
Update frontend.md
265917aeb0
Update frontend.md
1d30b6bb8a
Update frontend.md
3bb992ff60
Update frontend.md
a4d9cc6abe
Update frontend.md
701929dc9d
Update frontend.md
5c60f86127
Update frontend.md
6d00809bbd
Update frontend.md
128336e247
Update frontend.md
ebacac4297
Update frontend.md
bfe4f5070f
Update frontend.md
f71577d640
Update frontend.md
844fd839da
Update frontend.md
1149f0595f
Update frontend.md
f5298f20c7
Update frontend.md
1e2a4ca139
Update frontend.md
8eccff1f8a
Update frontend.md
b07d3faf68
docs: UserGuide add drop-shadow for images and pre-formatted text
3bdb9a9ce5
docs: UserGuide add initial skeleton version of macaroons chapter
7e2bfeb16e
docs: UserGuide frontend tidy up minor problems
5e78c3b5bf
docs: UserGuide frontend add SwaggerUI images
2f4db801bd
docs: UserGuide drop the excessive inter-line spacing.
d5a35fd6ab
docs: UserGuide add support for console output
30b0f423a4
docs: UserGuide frontend avoid too many console copy-n-paste, focus on JSON
c8943140ab
resilience: add unit tests for new task operations and to check non-sticky processing and removal semantics
124b3e92aa
resilience: eliminate the ‘new pool’ flag (no longer necessary)
c367e9fad8
resilience: do full pool replica state verification and change remove semantics
232604b78e
resilience: adjust synchronization of file operation removal from map
5ffed3902d
pool: avoid throwing a RuntimeException for non-bugs
dbea103273
docs: mode info on dynamic pool groups
ef7870fd6e
Add initial support for code highlighting with prism.js
04bd4708e0
alarms: add logback Kafka Appender
9e1ec493c7
docs: TheBook UserGuide fix location of edit-me-on-github ribbon
ae8ba0b3ab
docs: TheBook update heading importance to be more consistent
e8092bd281
transfermanager: do not retry starting mover if transfer is not supported
e6d48431b9
pool: avoid log-and-throw anti-pattern
ed41f74110
frontend: ensure client is disconnected when shutting down channel
82adce7015
frontend: avoid race on cancelling channel garbage-collect task
a06637cf15
transfermanager: avoid NPE on shutdown
0c80d91596
pool: throw exception with meaningful error message
ead326ffea
docs: UserGuide frontend add section on doors, minor tidy up in events
01a4ef3f12
docs: UserGuide frontend add information about client identifier
c4af495fb9
docs: UserGuide frontend add links between the various QoS activities
cc249e0ae7
docs: UserGuide frontend add examples showing events in action
3fbbaec073
docs: UserGuide add second-level in table-of-contents
fc9351c833
docs: UserGuide add missing information about namespace stat flags
8a99fe1540
docs: UserGuide frontend add index to aid navigation
f58d3cf3ad
docs: UserGuide frontend add more information about inotify
36d22e25c1
docs: UserGuide frontend be more consistent with line splitting
8e77a71058
docs: UserGuide frontend add details about event channels
54de2c9d32
docs: UserGuide frontend add initial description of events
72d233d6cf
[maven-release-plugin] prepare for next development iteration