Incompatibilities

• The space manager database schema has changed. It will be updated on upgrade, which may take some time on large databases. Previous versions of dCache are not forward compatible with the new schema and the schema has to be explicitly rolled back before downgrading dCache.

• The Chimera database schema has changed. It will be updated on upgrade, which may take some time on large databases.

• The chimera-cli utility has been replaced by chimera.

• In DCAP, ctime is now last attribute change time (like in Posix), not creation time.

• The output format of the metadataof command in PNFS manager has changed.

• Link group attributes are no longer supported.

• The space manager admin commandline interface has been rewritten, with several commands being removed, replaced, or modified.

• The interpretation of the spacemanager.enable.reserve-space-for-non-srm-transfers property has changed.

• If the NFS door and PnfsManager are hosted in the same domain then the layouts file must have the ‘pnfsmanager’ service before the ‘nfs’ service.

Release 2.8.31

Changes affecting multiple services

dCache uses a standard format to monitor the performance of various components: in the srm door to record how quickly SRM requests are processed (print srm counters command), the generic cell message monitoring (monitoring info command), and pnfsmanager service (the “Statistics” section in info). This release fixes a rounding error that prevents these statistics from including long-lived requests.

pinmanager

The pinmanager service has the ls command that allows the admin to limit the results to a specific pin or all pins against some PNFS-ID. This release fixes listing by pin id.

srm

This release fixes the srm service so it cancels corresponding pinmanager requests when an SRM client aborts a bring-online request.

Changelog 2.8.30..2.8.31

17d3d93

[maven-release-plugin] prepare release 2.8.31

4240870

common: Fix division by zero regression in gauges

15fb0f3

common: Fix rounding error in request gauge

2501714

srm: Abort pinning when cancelling bring-online requests

98f1f30

pinmanager: Fix listing by id

b2a625f

system-test: add missing dCache disposible CA certificate

250d073

system-test: add regenerated host and user credentials

c1dc098

[maven-release-plugin] prepare for next development iteration

Release 2.8.30

dcap

Update dcap door to avoid possible NullPointerException when describing an error to the client.

Changelog 2.8.29..2.8.30

4fc4fac

[maven-release-plugin] prepare release 2.8.30

ea0e4a8

dcap: do not call toString() on error object

4740b77

[maven-release-plugin] prepare for next development iteration

Release 2.8.29

pool

When a client reads a file, the pool reads blocks of data from the local filesystem. When reading such a block, the pool could receive fewer bytes than requested. Previously, the pool assumed that this only happens when the end-of-file is reached; however, this is not guaranteed. Should this assumption be violated then the data sent to the client will be corrupt. In practise, the pool’s assumption is true for Linux and local filesystems; however, the code has been updated to remove this theoretical cause of corruption.

Changelog 2.8.28..2.8.29

91215c0

[maven-release-plugin] prepare release 2.8.29

93a0cac

pool: Fix read corruption in HTTP mover

1331e5c

[maven-release-plugin] prepare for next development iteration

Release 2.8.28

pool

When attempting to upgrade non-precious and non-cached files (e.g. a file marked broken), the receiving end of the migration module would answer twice: first (correctly) with a failure and then (incorrectly) with a success. This is now fixed.

Changelog 2.8.27..2.8.28

ac4af38

[maven-release-plugin] prepare release 2.8.28

09a439f

pool: Fix bug in migration module upgrade logic

841f3bd

[maven-release-plugin] prepare for next development iteration

Release 2.8.27

Changes affecting multiple services

Various scripts, including the dcache command, invoke the java command with a list of directories in which Java should look for support libraries. Previously, the current working directory was (mistakenly) included in that list. This could lead to odd behaviour; one particular example is running a dcache database command from the /etc/dcache directory. This release fixes this problem by excluding the current directory.

When there is some problem in the communication between domains and error message is logged. Previously the explanation for the problem was logged as “null”. With this release, a more descriptive explanation is provided.

Changelog 2.8.26..2.8.27

6971f30

[maven-release-plugin] prepare release 2.8.27

01415de

tunnel: use toString if IOException#getMessage returns null

d810424

Exclude cwd from classpath

4a6b7cc

[maven-release-plugin] prepare for next development iteration

Release 2.8.26

Changes affecting multiple services

This patch fixes a race condition in Chimera that affects the nfs door and the pnfsmanager service. The effect is that, if two clients attempt to delete the same target (a file, link or directory) at the same time then the nlink count for the parent directory is decreased twice. “At the same time” means within the time taken to process the deletion; this is instance-specific but should be much less than 1 ms for well-configured systems. Sites can repare any incorrect nlinks with the following SQL:

UPDATE t_inodes SET inlink = (
    SELECT COUNT(*) FROM t_dirs  WHERE t_inodes.ipnfsid = t_dirs.iparent
) WHERE itype = 16384;

This is safe to run on a running production instance, but may take some time and will affect dCache’s responsiveness while running.

gplazma

The description for how to migrate away from using the forbidden useGPlazmaAuthorizationModule and useGPlazmaAuthorizationCell properties had caused confusion. The description has now been updated to be more explicit.

httpd

Fix filtering boxes and sorting on Pool Admin, Pool Usage, Poolgroups, Space Tokens and Tape Transfer Queue.

srm

Uploading a file with SRM involves three steps: preparing for the upload (srmPrepareToPut), uploading the file, marking the upload finished (srmPutDone). The third step can fail but previously the response from dCache is always Upload failed.. With this release, a meaningful error message is returned.

Changelog 2.8.25..2.8.26

a9d4f3c

[maven-release-plugin] prepare release 2.8.26

18ca9d0

gplazma: update error message for forbidden properties

486aeb5

(2.8) webadmin: make jquery selector specific to individual tables

b1142b0

(2.8) webadmin: restore missing components to respect jquery script options

c93f214

srm: include the reason why upload failed

a3c3ae6

chimera: fix race condition on remove

92251fd

webadmin: ensure unique id attributes for all (currently) tested UI elements

4cf15d9

[maven-release-plugin] prepare for next development iteration

Release 2.8.25

ftp

Fix default value for ftp.authz.readonly for plain (unencrypted) doors. This restores the default value to the dCache v2.6 default value of true.

The response from the plain (unencrypted) ftp door if the user specifies the wrong password is badly formed. Althogh it is possible that some clients are robust against such incorrect responses, with this release the ftp door responds correctly.

webdav

Fixes a bug where, if a double-slash is present, all parts of the path leading up to the double-slash are ignored; for example, with the bug, a path like /a/b//c/d is handled as if /c/d was specified. With this release, double-slashes are treated like single slashes; the above example is handled as if /a/b/c/d was specified.

Fix the NullPointerException triggered if client attempts to upload a file as a child of some existing file.

Changelog 2.8.24..2.8.25

80d7ff0

[maven-release-plugin] prepare release 2.8.25

81c0b88

ftp: fix response if user fails to authenticate to weak FTP door

bcc51c3

ftp: fix invalid default for ftp.authz.readonly property

c6ce396

webdav: fix double-slash bug by upgrading to patched milton

1a3360d

webdav: fix NullPointerException when PUT as a child of a file

8c8c2c9

[maven-release-plugin] prepare for next development iteration

Release 2.8.24

Changes affecting multiple services

Previously releases of dCache claimed to support Java–8; however, this is not true. For Java–8 support, sites must upgrade to dCache v2.10 or newer.

httpd

Fix two minor issues when authenticating with the webadmin interface: “unauthorised access” and being redirected to the home page. The unauthorised access error can occur when selecting “Login” under the bird logo (top right corner); this is now fixed. The redirection problem occurs when selecting a tab that requires administrative privileges while not logged in; this redirects the browser to the login page. Previously, after a successful login, the browser was redirected to the home page. Now the browser is redirected to the selected tab.

webdav

A previous bug-fix release fixed how dCache responds when the client attempts to DELETE a non-existent file. Unfortunately, this triggered a different problem where such activity results in a stack-trace that starts java.lang.ClassCastException: java.lang.String cannot be cast to javax.security.auth.Subject. This second problem is now fixed.

Changelog 2.8.23..2.8.24

d12c8a8

[maven-release-plugin] prepare release 2.8.24

547f243

Disallow Java 8

aacde44

webdav: Alternative to fixing return code of DELETE of absent file

29fbab9

(2.8) webadmin: fix login redirect bug

6ac1b0c

[maven-release-plugin] prepare for next development iteration

Release 2.8.23

info-provider

In previous releases, the info-provider assumed the broker domain is dCacheDomain. This assumption has been removed.

nfs

Update to nfs4j v0.7.9. Avoid errors when the door is heavily loaded and the client closes a file. Avoid potential deadlock under the same circumstances. Add the pnfs and nopnfs options for controlling whether PNFS is supported. Fix parsing the exports when a host is mentioned multiple times. Note that localhost must now have an explicit entry in the exports file.

pool

A recent release fixed a bug that caused pool.mover.ftp.allow-incoming-connections to be ignored. Fixing that bug revealed another that caused the property to have the opposite effect. This is now fixed.

webdav

Although dCache behaves correctly if the client interrupts a proxied transfer; however, this is logged as a bug. This is now fixed.

Changelog 2.8.22..2.8.23

e7c2da3

[maven-release-plugin] prepare release 2.8.23

fddb44a

webdav: don’t log a stack-track when proxy transfer is interrupted

3ec0541

info-provider: remove dCacheDomain assumption.

8d751e9

(2.8) gitignore additions for IntelliJ

639e52d

libs: update to nfs4j–0.7.9

277f3cf

pool: Fix regression causing FTP movers to default to proxy mode

127a01f

[maven-release-plugin] prepare for next development iteration

Release 2.8.22

Changes affecting multiple services

Although dCache system configuration property names do not contain spaces, it is possible to define such properties. Previously, doing so breaks the dcache command. This is now fixed.

If the number of concurrent connections is set to -1 then the ftp and dcap doors will leak memory, eventually triggering an out-of-memory error that will restart the domain. This is now fixed.

dcap

Fix regression against v2.6 and earlier dCache in how gsidcap doors are known to SRM and how they are published in BDII/GLUE.

pool

The pool.mover.ftp.allow-incoming-connections property had no effect. This is now fixed.

spacemanager

Adjust spacemanager schema migration to be aware of earlier dCache bugs and to work-around site-local indexes that clash with new indexes that dCache needs.

srm

Previously, if the SRM client requests listing a directory, specifies a non-zero offset and does not limit the response size then dCache would fail this request with an IllegalArgumentException. This is now fixed.

webdav

With the recent upgrade of the Milton library some new behaviour was introduced. One example is that, under certain circumstances (and to support certain clients) the Milton library returns a 401 (not authorised) when attempting to delete a non-existing file. Unfortunately, this change then broke ATLAS clients. This patch updates dCache so it returns 404 (not found) under these circumstances.

Fix the response when a client requests a byte-range beyond the end of a file. This is necessary for compatibly with ARC clients.

Changelog 2.8.21 to 2.8.22

1778656

[maven-release-plugin] prepare release 2.8.22

8d7c305

webdav: Fix reported content length for partial GETs

8c80c91

webdav: Fix return code on DELETE of absent file

651a59b

srm: fix semi-infinite ls range with non-zero offset

c65a728

pool: Fix typo that breaks pool.mover.ftp.allow-incoming-connections

c5983ea

shell: fix shell oracle for configuration keys with a space

1cdab85

dcap: Fix regression in published protocol family

4ff3a11

loginmanager: Fix leak caused by absent child limit

f3917d1

spacemanager: Fix null constraints and other schema migration issues

b1e26a8

[maven-release-plugin] prepare for next development iteration

Release 2.8.21

nfs

Under heavy load, if the nfs door is restarted, the new door may receive response messages from requests send by the previous door. In previous versions of dCache, this situation triggered a stack-trace. This is now fixed.

pool

The NFS specification allows the server to specify multiple addresses when telling the client where to connect; for example, specifying both an IPv4 and an IPv6 address, or both addresses for multi-homed machines. This requires the client to choose the appropriate interface. For Scientific Linux 6, the kernel client will always use the first supplied address in the list and fail if it cannot access the pool with that address. With this release, pools will order the list, using heuristics to select which IPv4 address is “correct” and list it first.

poolmanager

With previous dCache versions, the WAAS selection algorithm had a bug where it could (mistakenly) consider pools full if all pools had very fresh files. This is fixed with this release.

webdav

Upgrade to Milton v2.6. This fixes the buffering problem where a proxied vector read request results in the entire file being written to a tmp directory and not deleted. With this release, requests for 100 kiB or less data result in no data being written to disk; requests for more than 100 kiB are still written to disk, but only the data needed to satisfy the request is stored and the file is deleted once the response has been sent. Some issues persist: data isn’t deleted if there is a failure sending it to the client and the whole file is requested from the pool.

Changelog 2.8.20 to 2.8.21

adfe43f

[maven-release-plugin] prepare release 2.8.21

267e203

poolmanager: Fix full pool detection for WASS

2bad558

Upgrade to Milton 2.6

dfd91c6

pool: reorder ip addresses returned to NFS client

8e9bfbf

nfs: fix NPE on door reboot

8c3b6b3

[maven-release-plugin] prepare for next development iteration

Release 2.8.20

Changes affecting multiple services

Restarting a domain within an active dCache instance can lead to a domain receiving messages for a cell as it is starting. Strict control is require to avoid the cell attempting to process messages before it is ready. This release fixes one place where this control was missed, which could lead to a NullPointerException. While this problem can affect any core dCache service, it was noticed with the spacemanager service.

httpd

dCache versions including and after 2.11.0, 2.10.9, 2.9.12, 2.8.16, 2.7.21 and 2.6.36 required sites to delete existing RRD files when upgrading; i.e., run the command rm -f /var/lib/dcache/plots/*.rrd when the domain hosting the httpd service is stopped. This release reverts that change, but requires sites that have already upgraded to repeat the rm command. Sites upgrading from an earlier dCache version do not need to delete anything.

nfs

Protect against a NullPointerException if the client attempts to read the contents of a file’s level where that level exists in the database but contains a Nil value. This does not happen under normal circumstances.

pool

The Berkeley DB, which may be used to store file metadata on the pool, does not like being interrupted. The pool tries hard to avoid interrupting reading or writing; this release fixes one place that slipped through.

Changelog 2.8.19 to 2.8.20

8b245bc

[maven-release-plugin] prepare release 2.8.20

bb202c0

imera: protect against NPE in FsSqlDriver#read

4da50fc

Fix NPE in cell initialization

92db6bd

pool: Avoid interrupting Berkeley DB in migration module server

ac7a972

dcache-webadmin: revert rrd data source names

da00de9

[maven-release-plugin] prepare for next development iteration

Release 2.8.19

httpd

In previous versions of dCache, the webadmin war file is automatically unpacked. This has been problematic as a dCache upgrade did not always trigger updating the unpacked webadmin, resulting in dCache running the older webadmin. With this release, dCache no longer unpacks the war file; the webadminWarunpackdir and httpd.container.webapps.tmp-dir properties are now obsolete.

Changelog from 2.8.18 to 2.8.19

9143af5

[maven-release-plugin] prepare release 2.8.19

e6255d2

[maven-release-plugin] prepare for next development iteration

79b8708

(2.8) dcache-webadmin: change Jetty setting so .war is not unpacked

Release 2.8.18

Changes affecting multiple services

A bug was discovered that resulted in the nlink count becoming negative. While wrong, this had a knock-on effect that prevented pnfsmanager and nfs services from listing the affected directory. With this release, directory listing are robust against such problems.

alarms

Fix alarm definition that triggers when the pool discovers a file with the wrong checksum. Previously it mistakenly triggered if an upload was incomplete.

nfs

Update nfs4j to v0.7.8. This brings some small performance benefits when listing a directory and for reading data through NFS v3.

Fix bug where, when a client read activity is proxied, the corresponding dcap mover wasn’t removed if client didn’t wait for the queued mover to start. Previously, such movers would accumulate on the pool.

pnfsmanager

Prevent attempts to remove the . and .. directories.

webadmin

Display the command’s output in the cell admin page to use a monospace font.

Remove a javascript error due to missing clojure dependency.

Changelog from 2.8.17 to 2.8.18

1398703

[maven-release-plugin] prepare release 2.8.18

c93431e

alarms: fix regex for checksum alarm

19282f9

webadmin: remove redundant head element in alarms panel html

c5de8f4

dcache-webadmin: change output field of cell admin page to monospace font

687aa99

(2.11) dcache-webadmin: eliminate clojure dependency

0751511

nfs-proxy: kill mover if we get a timeout on redirect

90fe062

libs: update to nfs4j–0.7.8

e584a4c

chimera: protect list initialization from FS inconsistencies

d19ce90

webadmin: tidy up unavailable page slightly

12ca4fd

chimera: prevent attempts to remove ‘.’ and ‘..’

16ca7af

[maven-release-plugin] prepare for next development iteration

Release 2.8.17

Fixes affecting multiple services

Update dcache check-config to print an error if site-local configuration contains scoped properties: one that contain a ‘/’ character.

httpd

Fix pool queue plots in webadmin.

nfs

Performance improvements.

pnfsmanager

Fix the message reported when moving (or renaming) a file or directory and certain problems were found: the destination directory isn’t a directory, the source doesn’t exist, or overwriting with different types (e.g., overwriting a directory with a file).

poolmanager

If a site using WAAS specifies too large a Space Cost Factor then the algorithm used for write pool selection breaks down, logging “Unreachable statement.” Starting with this release, a warning is logged indicating the cause of the problem.

Changelog from 2.8.16 to 2.8.17

6e4fb2a

[maven-release-plugin] prepare release 2.8.17

17b6c99

chimera: Add messages to JdbcFs#move exceptions

f4c7a80

poolmanager: Warn when spacecostfactor is too big

575bb58

check-config: Produce error when using scoped properties

ac38688

(2.8) pool queue plots - fix partial package refactoring and neglected removal of jndi arguments

5e25480

pom: update to nfs4j–0.7.7

f8471dc

[maven-release-plugin] prepare for next development iteration

Release 2.8.16

Fixes affecting multiple services

Running the eval admin command so it returns a non-zero return-code would respond indicating that a bug had been found. This has been fixed.

An named environment may be executed and can contain a reference to itself. If this recursion is unchecked, eventually memory is exhausted and the domain would restart. Now an error is logged and the domain is not killed.

Previously, some exceptions were logged with the wrong cell context or without any context. This has been fixed.

billing

Billing records issued by doors never included the file’s size. This is now fixed for all doors except NFS.

gplazma

The xacml plugin now supports the gums server returning GID values.

pool

Fix internal copying of files (triggered by the SRM copy command) so they respect the LAN port range.

Fix migration module’s random pool selection so that it does not select pools that are full.

Fix NullPointerException triggered when using migration module’s proportional pool selection and all pools are full.

The rep ls command can calculate per-storage-class statistics. This release fixes a problem where files without a storage-info would trigger an IllegalStateException when calculating these statistics.

Fix sweeping of files without a storage-info. Previously an IllegalStateException was thrown; this could be triggered by the admin interface or by the sweeper.

Fix a problem where the pool would register free space before actually deleting the file from the file system; for a brief moment, the pool would appear to have more free space that is actually available.

Periodically, pools check that dCache’s internal accounting of total and free capacity does not exceed the OS supplied values for the partition (i.e., the output from the df command); if they do then the dCache internal accounting is adjusted to match. The dCache.org team have observed that, after a file is deleted, some seconds may elapse before the corresponding extra free capacity is reported by the OS. To protect pools from this effect, the total and free capacity check is suppressed for 60 seconds after a file was deleted.

spacemanager

If a transient deadlock resolution error occurs, unconditionally retry the operation and do not log it. Such errors are part of RDBMS design when resolving concurrent updates.

srm

Fix the srm service’s admin interface ls command so that listed jobs can optionally include only those that have failed, have completed or were cancelled.

webadmin

Webadmin periodically creates billing graphs by querying the billing service for the information it needs. Previously, if this query failed (e.g., the billing service was restarted or failed to start first) then no further billing graphs are generated and a domain restart is needed. As a result of this work, the properties poolqplots.refresh-interval, poolqplots.refresh-interval-unit, httpd.plots.pool-queue.refresh and httpd.plots.pool-queue.refresh.unit are no longer supported. The httpd.plots.pool-queue.min-time-step and httpd.plots.pool-queue.min-time-step.unit properties now also cover this configuration.

Changelog from 2.8.15 to 2.8.16

3d8c195

[maven-release-plugin] prepare release 2.8.16

034fae2

gplazma-xacml: Add handling of GID returned by GUMS in gPlazma2 XACML plugin

4296a50

chimera: fix unit tests

87d63e1

pool: Fix pool size health check in case of asynchronous release of space

99d67fd

spacemanager: Suppress transient deadlock resolution errors

e71060a

pool: Fix race leading to false positices in pool size health checks

9f2575e

pool: Fix ISE in CacheEntryImpl#toString

d99e691

pool: Fix ISE in ‘rep ls’

a6d1a8c

Made PoolQueuePlotData enum compatible with java 8.

5f575c7

(2.8) webadmin: fix exit login in billing refresh loop

eb1d50a

Marked refresh properties obsolete.

477cb5a

cell: Log exceptions within the correct cell context

a2815df

pool: Fix pool selection bugs in migration module

a197381

billing: Add file size to request records

eecfb6c

(2.8) webadmin: minor improvements to rrd4j-based pool-queue plots

4bf9cc7

Backport PortRange extensions

2326ac3

pool: Respect LAN port range for internal srmcp transfers

92c2f28

srm: Fix listing of failed, done and cancelled jobs

a4d1212

cell: Prevent interpreter stack overflow from killing the domain

20efd5e

cell: Declassify eval failure as a bug

96574ce

[maven-release-plugin] prepare for next development iteration

Release 2.8.15

Fixes affecting multiple services

The nfs and srm both cache replies from gPlazma to increase the speed of authenticating and identifying the user. Previously transient errors when communicating with gPlazma were also cached, delaying the recovery time from such errors. Now, such errors are not cached.

Update the comments within the configuration property files to reflect the new configuration property names.

Fix OpenMQ communications to use new configuration property names.

dcap

Fix opening a file when space-manager enabled.

nfs

No longer trigger an NullPointerException when opening the .(parent)() dot-command file in Chimera’s root directory.

The dcache ports command now shows the NFS port based on the new configuration properties.

pool

Fix xrootd support for vector read. The problem was discovered with ROOT v6, which reports errors like Single readv transfer is too large.

Fix the dcap mover to use the new pool.mover.dcap.port configuration property.

srm

Fix veracity of information stored about jobs.

xrootd

Fix problems when calculating a response to the kXR_set request. If encountered, the problem results in the following being logged, where ‘nnn’ in ‘xrootd-disk-nnn’ is some integer number:

Uncaught exception in thread xrootd-disk-nnn java.lang.IllegalStateException: null

Changelog from 2.8.14 to 2.8.15

b867278

[maven-release-plugin] prepare release 2.8.15

0f6bb89

dcap: fix interaction with Spacemanager

25c1c0c

xrootd: Upgrade to xrootd4j 1.3.5

6e224f4

The gPlazma cache used by NFS and SRM caches both positive and negative replies. Unfortunately it also caches failures to communicate with gPlazma. This means that a transient timeout would be cached too, thus increasing the effect of the transient error.

b4d77a9

pool: Fix xrootd vector read limits

404be1b

chimera: fix NPE on ‘.(parent)()’ for root inode

f232324

srm: Fix asynchroneous job storage leak

a72fefe

configuration: adjust references to deprecated properties

6808dc8

unittests: increase timeouts and fix race in DiskSpaceAllocatorTest

6d84856

[maven-release-plugin] prepare for next development iteration

Release 2.8.14

Fixes affecting multiple services

Two problems are fixed that affected those services that make use of the grid trust store (/etc/grid-security/certificates): gplazma, srm, gsiftp, webdav and xrootd:

• When a CA is found not to have a signing-policy file, subsequent attempts to discover a signing-policy file will automatically fail for one hour. Previously, if any CA was removed since the service started then certain CAs (those using UTF–8 encoded subject RDNs) would be placed in this not-found cache by mistake. This resulted users with certificates from such CAs would succeed the first time they used dCache but all subsequent attempts would fail until the cache expires or the trust-store is refreshed.

• A minor memory leak is fixed. The amount of memory leaked was proportial to the number of different CAs in use.

Two problems are fixed that affected services providing GSI or plain SSL authentication using the jGlobus library:

• dCache now supports TLS v1.2 clients when using GSI or plain SSL using the jGlobus library.

• Signal end-of-stream when remote sends a CLOSE notification. The remote party normally sends a CLOSE notification before terminating the TCP connection, typically send from server to client. Previously, receiving such a notification was treated as an error.

Fixed thread safetly of timestamp formatting, principally used in logging and generating the output of admin commands.

billing

Fixes a bug where the bytes read graph can show inflated values.

nfs

Fix permission check when client requests a list of some directory’s contents.

Add support for mounting when the server-side mount-point is a sym-link.

Changelog from 2.8.13 to 2.8.14

b3effdd

[maven-release-plugin] prepare release 2.8.14

9a079a6

common: Fix race condition in AtomicCounter unit test

406e03d

billing plots: use ‘transferred’ rather than ‘size’ for bytes

26389c8

libs: update jglobus to 2.0.6-rc8.d

6a094e0

Avoid thread-unsafe use of SimpleDateFormatter

78aa6fb

libs: update to jglobus–2.0.6-rc7.d

5848f58

libs: update to nfs4j–0.7.6

299d896

[maven-release-plugin] prepare for next development iteration

6ba648a

build: Update to findbugs 3

Release 2.8.13

pnfsmanager

Previously, the automatic schema management for Chimera failed to create an index on the iparent column of t_dirs table. The lack of this index results in deletions becoming progressively slower as dCache stores more files. If your dCache instance was created with dCache v2.3.0 or newer then it is likely affected by this problem. Instances created with an earlier version are not affected. The pnfsmanager service will check for the index when it starts and add the index if it is missing. Alternatively, the check may be performed manually using the dcache database update command.

pool

This release fixes several issues with the sweeper:

• Previously, the sweeper consumed considerable amount of memory while freeing up space; in the worse case, this could result in the domain running out of memory and restarting; this has been fixed.

• The sweeper ls command would make the pool unresponsive while the output is generated; now, running this command no longer prevents the pool from taking on more work.

• As the sweeper purge, sweeper free and rep rmclass commands can take a long time to finish, the admin interface does not wait for them to finish; instead, a quick response is returned and the command continues in the background. Previously there was no indication when such commands had finished; now, the pool will log when the command has completed.

• The pool logs regular sweeper operations at lower priority; this makes the pool logs less noisy when operating under normal conditions.

poolmanager

In previous versions, poolmanager would log a stack-trace if a user attempts to open a file for reading and no read pool could be found because a unit failed to match. Now the failure is logged as a normal message.

xrootd

The xrootd door now informs billing when a user uses the xrootd protocol to delete a file.

gplazma

The ldap plugin allows a door to request converting a uid or gid back to a username; currently, only the nfs door makes use of this functionality, when generating a directory list. In previous dCache releases, the plugin behaved incorrectly if the uid or gid is unknown; this is now fixed.

Changelog from 2.8.12 to 2.8.13

1a80f0b

[maven-release-plugin] prepare release 2.8.13

9b78495

chimera: mark as-run if i_dirs_ipnfsid exists

8faa2ae

solaris: fix solaris package script and add work-around for pkgmk bug

874994d

chimera: create missing index i_dirs_iparent

08600a0

pool: Resolve high memory usage and other issues in sweeper

4abcbb3

info: Fix HashMap ordering assumption in unit test

71b74e1

Make JDK byte code verification bug workaround Java 8 compatible

493c063

gplazma: Fix JVM implementation dependency in unit test

9ebb958

poolmanager: Suppress stack trace in case of unmatch units

c28fed3

xrootd: Add billing entry on delete

6342a51

bugfix: fix reverseMap not throwing NoSuchPrincipalExceptions

04a058b

[maven-release-plugin] prepare for next development iteration

Cell startup

Release 2.8.12

Cell startup

Fix potential dead-lock when a dCache domain is starting. The problem is triggered when a cell receives a message while initialising. If dCache is started “from cold” then this cannot happen. However, if a domain that contains well-known cell restarts while the rest of dCache is operational then the problem could be triggered. For example, if a door domain ran out of memory then there is a small risk that the domain would not restart correctly. The problem was present since dCache v2.8.0 and is fixed with this release.

Changelog from 2.8.11 to 2.8.12

d6214bc

[maven-release-plugin] prepare release 2.8.12

6abe91a

Fix compilation error

60019a9

cells: Fix deadlock during startup

e4b1314

[maven-release-plugin] prepare for next development iteration

Release 2.8.11

Packaging

Add python-psycopg2 as recommended dependency to debian package; it’s needed by the dcache-star command.

Database Schema

Add work-around for bug in liquibase. Previous 2.8 releases failed the database upgrade for the alarms and billing database schema due to incorrect SQL being generated when modifying a CHARACTER VARYING column.

alarms

The alert should be generated if a pool’s background checksum scanning (“scrubbing”) discovers a broken file. With this release, such alerts are now generated.

poolmanager

Stage and pool-to-pool transfers, triggered by poolmanager, are monitored by poolmanager querying the source pool to check if the request is still active. In previous versions of dCache, for n transfers from a pool, this monitoring generated O(n*n) of network traffic. In particular, poolmanager’s monitoring of bulk staging requests can result in sufficient network traffic between the pool and poolmanager that other dCache activity is impacted. With this release, the monitoring traffic is reduced in general, most prominently for large concurrent activity, such as bulk stage requests.

The rc destroy command has been removed. The implementation was incomplete and the operation was dangerous.

If the door resubmits a request to poolmanager that triggers staging of a file, poolmanager uses additional memory while waiting for the stage to complete. For bulk staging, this additional memory usage can be significant, potentially resulting in poolmanager exhausting the available memory. This release updates poolmanager so a door resubmitting a stage request does not increase the memory footprint of poolmanager.

The poolmanager has an incomplete feature called ‘clumping’. This is where poolmanager can handle multiple select-pool-for-read requests for the same file as a single request. There is a hard-coded clumping limit that prevents too many files from being handled together; this allows poolmanager to choose an alternative pool when staging or replicating to make the file available. Once this limit is reached, subsequent open requests for the same file will fail in poolmanager. Doors handle this failure differently; the xrootd and webdav doors propagates this failure back to the client, while the dcap and ftp doors will resubmit the request to poolmanager. This release increases the clumping limit from 1 to 20; it also changes the behaviour of the doors so all will retry the pool-selection if the clumping limit is reached. This fixes the ‘request clumping limit reached’ failures.

This release fixes a race-condition between two clients opening the same file. Previously, if poolmanager starts processing the second open request as it is finishing processing the first then there is a tiny chance that the second request is lost.

Changelog from 2.8.10 to 2.8.11

f134d52

[maven-release-plugin] prepare release 2.8.11

09f5d0a

build: add work-around for JDK bug and PowerMock

42b3aca

poolmanager: Fix ‘request clumping limit reached’ failures

12e7aea

poolmanager: Avoid request leak

55bdb0a

alarms, billing: fix liquibase character varying and cleanup

dc234ef

alarms: Fix checksum alarm filter.

f6854df

deb: Add python-psycopg2 as a recommended dependency

c675c42

[maven-release-plugin] prepare for next development iteration

Release 2.8.10

webadmin

Fixes a regression that prevented the admin from deleting alerts in the alarm system.

Improve esthetics of the pool queue plot grid.

pool

Fixes a problem where, under specific circumstances, a pool with a file’s data will erroneously claim it does not have that file’s data. This problem occurs only if an HTTP or xrootd client attempts to read a file immediately after uploading it; it does not affect other protocols. If a client triggers the problem then PnfsManager will remove the pool as the location of the file’s data with two consequences: first, the file cannot be read; second, the orphaned entry will not be removed if the file is deleted. With this release, the problem is fixed; we recommend running the ‘pnfs register’ command on all pools as a precaution against orphaned entries.

info-provider

Suppress publishing NFS door endpoints. Publishing these endpoints can result in sites failing functional tests since dCache running an NFS endpoint does not mean that files are accessible from the test machine.

Changelog from 2.8.9 to 2.8.10

b521308

[maven-release-plugin] prepare release 2.8.10

80b6149

Suppress nfs’ file protocol from being published

85662bc

pool: Fix race condition that leads to orphaned files

fd0d755

dcache-webadmin: fix pool queue plot grid placeholder image

2aea823

(2.8) webadmin: fix regression in alarm deletion filter

8f1774a

cells: remove System time dependency in unit-test

ac34f59

[maven-release-plugin] prepare for next development iteration

Release 2.8.9

pool

The background checksum scanner now treats files that cannot be accessed as an error rather than skipping them.

Fix the response headers when HTTP clients (such as Davix) make a request for multiple fragments of a file.

pnfsmanager

For systems using enstore, fix restoring of files with filenames that contain a ‘+’ in the filename.

srm

Increase scalability of SRM by allowing work to spread over more CPU cores during file upload and download.

xrootd

Honour the WriteToken directory tag and implicit reservation when writing data.

nfs

Honour the WriteToken directory tag and implicit reservation when writing data.

Changelog from 2.8.8 to 2.8.9

2d266de

[maven-release-plugin] prepare release 2.8.9

034527b

pool: log with error if checksum scanner gets an IO error

fccdfee

Fix ArrayIndexOutOfBoundsException when processing multiple surls invoking srmget.

46aab4d

Fix issue with incorrect determination of whether or not a string is encoded by replacing URLDecoder with import org.spri

56d9c68

http: do not use new API

191001d

http: fix multipart response size

07cd525

xrootd,nfs: Add support for space management

84c56a0

srm: Fix lock congestion

dca739b

http: add minimal HEAD support in the mover

dc759fb

[maven-release-plugin] prepare for next development iteration

Release 2.8.8

xrootd

Fix redirection to IPv6 pools.

Report read requests that suffer a permission denied failure as “Read permission denied” rather than “Write permission denied”.

To allow xrootd plugins to intercept and translate errors.

Update dCache responses to avoid intermittent failures with xrdfs.

webadmin

Fix login bug that results in Access Denied page.

dcap

Provide more robust cleanup if there is a problem after the client finishes reading or writing data. The symptom is an increasing number of “active transfers” reported in the web monitoring pages. Eventually, the httpd and webadmin services consume all available memory and the domain(s) hosting them will die with an OutOfMemoryError.

Scripts

Update dCache scripts to allow running with Java 8 runtime.

Changelog from 2.8.7 to 2.8.8

02d1c62

[maven-release-plugin] prepare release 2.8.8

5b562d9

xrootd: Quote IPv6 addresses in kXR_redirect replies

1d127d0

xrootd: Fix error message for read patch check

fc93339

dcache-webadmin: fix login redirect bug which misleadingly results in Access Denied Page

5b90dd5

xrootd: Upgrade to xrootd4j 1.3.3

896a850

scripts: allow java8 runtime

4613a8d

dcap: ensure that we remove session on DoorTransferFinishedMessage

cc6952c

[maven-release-plugin] prepare for next development iteration

Release 2.8.7

spacemanager

Fix the column formatting for various admin commands should one of the numerical values be negative.

srm

Update the SRM to use the SRM-specific configuration property (‘srm.enable.space-reservation’) rather than the generic property (‘dcache.enable.space-reservation’). As the default value for srm.enable.space-reservation is the value of dcache.enable.space-reservation, this change should not affect sites.

ftp

Don’t export child doors. When a client connects, the ftp door creates a child cell that handles the client requests. Changes in 2.7 meant that these child cells mistakenly became well-known cells. This is now fixed.

dcap

Don’t export child doors. When a client connects, the dcap door creates a child cell that handles the client requests. Changes in 2.7 meant that these child cells mistakenly became well-known cells. This is now fixed.

info-provider

Update the use of xmllint to include the ‘–noent’ option; mitigation against CVE–2014–0191 resulted in xmllint behaving differently when ‘–noent’ is omitted.

pool

Add support for additional xrootd client commands: querying the server configuration and a file’s checksum, and locating a file. The pool now redirects the client back to the xrootd door if the opaque part of URL is missing.

xrootd

Add support for additional xrootd client commands: querying the server configuration and a file’s checksum, and locating a file.

libraries

Update jGlobus version. This results in better error reporting for FTP transfers and X.509 certificates with a comma in the DN are handled correctly.

Changelog from 2.8.6 to 2.8.7

96de256

[maven-release-plugin] prepare release 2.8.7

9d7c002

info-provider: fix works-by-accent bug when importing dCache config

ffe41ed

xrootd: Backport xrootd door and mover from dCache 2.9

dcde73b

pom: update jglobus library

aa75ad4

srm: Use srm.enable.space-reservation rather than dcache.enable.space-reservation

0d282de

Fix rendering of negative numbers in columm layout

f9cd5b5

ftp,dcap: Don’t export child doors

9c030cf

[maven-release-plugin] prepare for next development iteration

Release 2.8.6

httpd

Prevent reading of configuration files.

srm

Fix how the status of requests that can contain multiple files (Get, Put, Ls, BringOnline, Copy) are calculated and avoid “illegal state transition” errors that are sometimes logged.

Ensure that request history information is listed chronologically.

webdav

split protocol name from client IP address when logging transfers

Work-around bug in StringTemplate library that can result in NullPointerException being logged when generating an error response for the client.

xrootd

split protocol name from client IP address when logging transfers

pnfsmanager

Don’t log “duplicate key” errors if the error is understood.

messaging

Report correct statistics about messages even after a cell’s message queue overflows.

pool

Make pools more robust against the unexpected. Files that appear unexpectedly (e.g., an HSM script delivers a file after dCache has cancelled the request) do not trigger the pool from disabling itself but are deleted; requests to create a file that already exists trigger re-registration of that file in pnfsmanager.

Log a stack-trace for pool-to-pool transfers that encounter an extreme problem with the virtual machine.

gplazma

ldap plugin allows ldap server’s URL to be specified.

Changelog from 2.8.5 to 2.8.6

35eec61

[maven-release-plugin] prepare release 2.8.6

633ec78

httpd: Fix a couple of bugs

4347384

srm: Avoid duplicate status code update on status query

90c1a21

core: split protocol name form the client IP in HttpProtocolInfo

7f4b6ba

xrootd: split protocol name form the client IP in XrootdProtocolInfo

5c37c7a

chimera: Fix duplicate key supression

bc965d0

cells: Fix message counting in case of queue overflow

94069bb

pool: Improve how we deal with unexpected files in pools

a17158d

pool: Log P2P fatal errors with stack trace

f487ac7

srm: Fix transition ordering for restored requests

719e6c1

webdav: work-around race-condition in StringTemplate library

b106907

systemtest: regenerate host credentials, add user credentials

71eca5f

remove deprecated annotation for back-port

1d18115

gplazma2-ldap: use url rather than host+port

7c871af

fhs: Use gzip compression for deb package

43ea0c0

[maven-release-plugin] prepare for next development iteration

Release 2.8.5

alarms

Add missing configuration elements and documentation to support sending an email when an alert is received.

statistics

Fix NullPointerException in statistics service

pool

Report correct client IP to billing for passive FTP transfers; previously the proxy address of the FTP door was reported, even when there is a direct connection between client and pool.

poolmanager

Fix infinite loop when selecting a pool from the lru partition type.

nfs

Ensure error messages are logged with the diagnostic context (the information in the square brackets).

Changelog from 2.8.4 to 2.8.5

f7d0ef7

[maven-release-plugin] prepare release 2.8.5

97daab6

dcache alarms: add missing configuration elements and documentation for SMTP appender

8e417ea

statistics: Fix NPE

71445eb

pool: Report correct client IP to billing for passive FTP transfers

dbff8b6

poolmanager: Fix infinite loop in lru partition type

c4fef0c

nfs: do not use CDC as try-with-resource

b717020

[maven-release-plugin] prepare for next development iteration

Release 2.8.4

ftp

A dCache cell is created each time an FTP client connects to the control channel. This release fixes the problem that these cells are not killed after the client disconnects.

nfs

Fix reading and writing to the non-zero levels (i.e., file metadata) through NFS.

Fix directory listings so a user attempting to list a directory that they are not allowed to see will receive a “permission denied” error rather than an IO error.

info-provider

Fix how the SRM is published within the GLUE v1.3 tree.

poolmanager

Fix a problem with WASS pool selection where the selection of a pool with a large number of writers (typical of a tape read pool) would be artificially inhibited. If enough eligible pools suffer this then pool manager would report that no pool is available for staging.

Changelog from 2.8.3 to 2.8.4

356f121

[maven-release-plugin] prepare release 2.8.4

057b538

poolmanager,pool: Fix precision underflow in WASS

242b848

scripts: fix host-credential generation

bd66d48

info-provider: fix capitalisation of SRM in ControlProtocol

10591be

libs: update to nfs4j–0.7.5

9515e97

ftp: fix unit tests so they run on IPv4-only machine

87f3768

Kill worker when calling SequentialExecutor#shutdownNow

2c51030

[maven-release-plugin] prepare for next development iteration

Release 2.8.3

ftp

Fix the CKSM command for paths containing a space.

Solaris

Reintroduce the option to build Solaris packaging; it is still the “old” package format, but this is supported by Oracle Solaris 11.

srm

Fix a problem where the order in which SRM loaded information back from the database on startup was not as expected, resulting in increased start-up time.

Fix how often the SRM checks for expired jobs: before it was every 60 milliseconds, now it is once a minute.

Prevent a deadlock if the SRM is stopped too quickly after being started.

billing

Make the pattern matcher in the billing file parser case insensitive. This solves a bug in which old billing records were not indexed, so sites using the indexer are recommended to reindex using the ‘/usr/sbin/dcache-billing-indexer -all’ command.

pool

Avoid that fast xrootd or HTTP clients can trigger out-of-memory exception if they write too quickly. This can happen if the pool is slow or runs out of disk space.

Fix the responsiveness of ‘mover ls’ when a pool is full.

loginbroker

Avoid sending a flood of “Failed to send update to..” messages during shutdown.

Fix the ‘lb set update’ command

Changelog from 2.8.2 to 2.8.3

7f1a529

[maven-release-plugin] prepare release 2.8.3

3bb9bfb

ftp: fix CKSM command for files with white-space

7978931

pkg: fix solaris packaging

d1b2bc9

srm: Avoid loading the same job multiple times

527ec9c

srm: Fix scheduling period for expiration task

4b0031f

billing: Make indexer case insensitive

269e625

pool: Avoid memory exhaustion problems on write with xrootd and http

0abcccb

pool: Fix lock problems in xrootd and http movers

2d0f67f

loginbroker: Fix login broker registration retry

b9b757f

srm: fix race condition if stop is called too soon after start

f3bbf3c

[maven-release-plugin] prepare for next development iteration

Release 2.8.2

Pool

Fix problem with ‘pf’ admin command that leads to a SerializationException.

Fixed WebDAV and HTTP support for libneon clients. This includes ROOT as it now uses the davix library, which is built on libneon.

SpaceManager

Fix liquibase schema migration that prevents startup if dCache has only one link-group

FTP, dcap, HTTP and xrootd

Fix regression in transfers that prevents (dCache internal) retries on the transient error when a pool is temporarily offline. Also remove the mistaken retry for certain permanent problems.

NFS

Fail an attempt to open file if the pool reports that this file is on a broken tape.

Changelog from 2.8.1 to 2.8.2

9d62b12

[maven-release-plugin] prepare release 2.8.2

c0edd89

nfs: return NFSERR_IO is we detect broken file on tape

464ed8c

http: adjust unit test to multirage changes

d7d8bc0

http: fix multipart reply

62a8224

pool: Fix pf command regression

aac2e98

spacemanager: Fix database sequence initialization

0a2cd55

Allow retry on transient transfer failures

cd8da58

[maven-release-plugin] prepare for next development iteration

Release 2.8.1

Admin

Fix typo that prevented the use of a history file for SSH v1 service.

Fix logged error when typing ‘save’ in the admin interface for a service has no configuration file.

FTP

Fix occational hanging FTP sessions.

PnfsManager

Fix compatibility with 2.6 pools.

xrootd

Improve compatibility with the xrdfs, the newer xrootd client.

Fixes dCache’s response to the LOCATE, LOGIN and OPEN requests.

Pool

Fixed when a replica’s access-time is updated. Previously it was updated on pool-to-pool transfers and not updated for regular downloads, the opposite of what is supposed to happen. This bug affected pool garbage collection order, space cost calculation and updating the file’s access time in the namespace.

Fixed logging messages from migration tasks, pool-to-pool tasks, and sticky bit expiration.

gPlazma

Fix XACML authentication.

SRM

Fix a number of problems when undertaking a third-party copy with SRM v1.1: the client ignored failures to set the file status to ‘Done’, it also changed the file status to ‘Done’ even when then transfer failed, and the server did not accept ‘Failed’ as a file status.

NFS

Add support for open-less read requests. Under certain recovery conditions the Linux kernel client will use a special NFS operation to read a file without opening it first. This is now supported.

Avoid leaving dead records if the NFS door was unable to start a mover.

Fix log messages to include context information when client uses proxied IO.

Fix lookup on the root of the tree.

Fix handling of illegal operations.

Translate problems with underlying namespace into more exact NFS errors.

Fix obscure error triggered by attempting to move a file as a parent of another file.

Changelog from 2.8.0 to 2.8.1

bbb8177

[maven-release-plugin] prepare release 2.8.1

a3411eb

pnfsmanager: Fix compatibility with 2.6 pools

670fff1

ftp: Fix race condition in command dispatching

d87c627

ssh1: Fix typo in service batch file

191ef0c

xrootd: Upgrade to xrootd4j 1.2.5

cb71fde

pool: Fix logging context for thread pool

2deb42f

srm,srm-client: Report failure for failed SRM v1 transfers

da66a27

srm-client: fix adler32

cf1338e

chimera: fix rename into a file

9f565cf

libs: update nfs to 0.7.4

2b1bb8f

nfs-proxy: fix logging CDC

89778b2

nfs4: clean failed request from pending queue

0c0f17c

nfs-proxy: add support for open-less read requests

4f7d02d

Fix NPE in save command

d4b4a71

gplazma2-xacml: revert version of opensaml

e4cda2e

pool: Fix access time update

07349b3

xrootd: Upgrade to xrootd4j v1.2.4

ac5c844

core, billing: remove unit test

92fb0d3

[maven-release-plugin] prepare for next development iteration

Release 2.8.0

Changes affecting several services

Chimera

Chimera has been updated to track the creation time of files. This is different from the change time (ctime). On upgrade the Chimera database schema will be updated, which may take some time for large databases (half an hour at NDGF for 50 million files).

The creation time for existing files is obviously unknown. When updating the schema, existing files will receive a creation time being the oldest of the mtime, ctime, and atime fields.

The SQL scripts that used to be used to initialize a Chimera database are no longer shipped with dCache. The Chimera schema is automatically initialized and maintained by dCache.

The Chimera commandline tool chimera-cli has been replaced with chimera. The new tool offers an interactive shell when invoked without arguments and can execute scripts. It can also be invoked with a command on the commandline like the old chimera-cli tool. Note that some commands have a different argument order compared to the old tool (they now use the same order as used by similar tools defined in the Posix standard).

To lean about the supported commands, start the Chimera shell by running chimera and then type help.

Libraries

The JDBC driver for Postgresql 9.3 is now included.

Upgraded to version 3 of the Liquibase database schema management library.

Logging

Added the admin commandline command log set pattern and log get pattern to every cell. These allow the log format to be altered at runtime. To make persistent changes to the log format, updating the properties dcache.log.format.file and dcache.log.format.pinboard in dcache.conf is preferred.

Statistics counters

The request execution time tracking code, used in services like pnfsmanager, srm, and nfs, has been improved. It is now possible to reset these counters through JMX. An external JMX client is needed to make use of this functionality.

FTP, DCAP, messaging

Several changes have been made to logic common to both the FTP and DCAP doors, and the low level connection handling code of the message passing code used in dCache. These fixes address several rare race conditions and failures in error recovery, as well as reduce resource consumption.

Changes to services

webdav

Limited support for WebDAV properties has been implemented. WebDAV provides an extensible mechanism for reporting additional meta data about files and directories. These are known as properties and can be queried through the PROPFIND HTTP method - the same mechanism used to list WebDAV collection resources (directories in dCache). Currently only support for checksums, access latency and retention policy is provided.

An example follows.

$ curl -X PROPFIND -H Depth:0 http://localhost:2880/public/test-1390395373-1 \
--data '<?xml version="1.0" encoding="utf-8"?>
        <D:propfind xmlns:D="DAV:">
            <D:prop xmlns:R="http://www.dcache.org/2013/webdav"
                    xmlns:S="http://srm.lbl.gov/StorageResourceManager">
                <R:Checksums/>
                <S:AccessLatency/>
                <S:RetentionPolicy/>
            </D:prop>
        </D:propfind>'

<?xml version="1.0" encoding="utf-8" ?>
<d:multistatus xmlns:cs="http://calendarserver.org/ns/" xmlns:d="DAV:" xmlns:cal="urn:ietf:params:xml:ns:caldav" xmlns:ns1="http://srm.lbl.gov/StorageResourceManager" xmlns:ns2="http://www.dcache.org/2013/webdav" xmlns:card="urn:ietf:params:xml:ns:carddav">
    <d:response>
        <d:href>/public/test-1390395373-1</d:href>
        <d:propstat>
            <d:prop>
                <ns1:AccessLatency>ONLINE</ns1:AccessLatency>
                <ns2:Checksums>adler32=6096c965</ns2:Checksums>
                <ns1:RetentionPolicy>REPLICA</ns1:RetentionPolicy>
            </d:prop>
            <d:status>HTTP/1.1 200 OK</d:status>
        </d:propstat>
    </d:response>
</d:multistatus>

Support for WLCG federated HTTP transfers has been added. This allows the webdav door to redirect a client to another replica if a local copy was not found. Support for this relies on special URLs generated by a central catalogue service. For all other URLs the behaviour is unchanged.

billing

The billing text file indexing tool, which was added in 2.7, has been improved in several ways:

• Indexing and compression speed has been improved by using multiple CPU cores.

• Billing files are now traversed in lexicographic order, which means that search results are reported in chronological order.

• The search tool now outputs the date of the billing file in front of every match.

• Support for compressing billing error files has been added.

• Improved commandline interface to allow multiple arguments where this makes sense.

dcap

DCAP was updated to report last attribute change time in the ctime field of a stat request. This was the intended semantics from the beginning and is what ctime means in the Posix standard, but in dCache ctime has in some places been confused with creation time.

ftp

Added IPv6 support by implementing RFC 2428. Note that IPv6 support will negate any advantages GridFTP 2 might have provided, which means that in many cases IPv6 data transfers will be proxied through the FTP door. The JVM has to be IPv6 enabled by adding -Djava.net.preferIPv4Stack=false to the dcache.java.options.extra property in dcache.conf. RFC 2428 is only supported on IPv6. When clients connect over IPv4 we pretend not to support the IPv6 extensions.

The FTP door has been extensively refactored to use thread pools shared between several sessions and to reduce the number of temporary threads being created at the beginning of each session. This should reduce resource consumption and reduce latency.

Improved error messages when no write pools could be found for an upload. This includes correctly classifying these events as transient or permanent errors.

gplazma

The LDAP plugin has been improved. Previous versions hard coded values for the home and root directory. The root directory was set to / and the home directory was set to the home directory as stored in the LDAP server. In dCache 2.8, the behaviour was made configurable by introducing two new properties, gplazma.ldap.home-dir and gplazma.ldap.root-dir. For those two properties it is now possible to use keywords that will be substituted by the corresponding attribute value in LDAP. For example %homeDirectory% will be replaced by the value of attribute homeDirectory in the people tree.

info

Only permanent space reservations, i.e. reservations without an expiration time, are published. This prevents implicit space reservations from bloating the xml output with information that is usually short lived and uninteresting.

nfs

Added the -c option to the nfs door’s stat command to reset statistics counters.

A new NFS dot-command has been added to retrieve all the existing checksums for a given file. Reading the files .(get)(filename)(checksum) or .(get)(filename)(checksums) returns a comma-delimited list of type:value pairs for all checksums stored in the Chimera database.

NFS doors now verify whether the Chimera database schema is up to date. If not, the quit. NFS doors do not support updating the schema automatically. Use PnfsManager or the manual dcache database update command to update the schema.

pnfsmanager

Minor performance improvements in file creation and file removal.

Limited support for symlinks has been added.

The output of the metadataof command has been updated to provide ISO8601 formatted date and time values. The old format did not include the year.

pool

Minor latency reduction on mover creation under light load.

The rep ls command now lists all its arguments even when some of the replicas do not exist on the pool.

poolmanager

Improved error messages when no write pools are found for a transfer.

LinkGroup attributes have never been used in dCache. Therefore we have removed the admin commandline commands psu remove linkGroup attribute and psu set linkGroup attribute. Since the poolmanager.conf file may contain the latter command, the command is still accepted, although it doesn’t do anything besides printing psu set linkGroup attribute is obsolete. If you see this message in the log file, please resave your pool manager setup.

spacemanager

Space manager has been heavily refactored and modified in this release. The changes almost amount to a rewrite, though the overall architecture of how space manager is used in dCache is preserved. We did this to improve the performance, robustness, logging and ease of use.

Database changes

The space manager database is now managed with the Liquibase schema management system. This is the same library also used for Chimera, pin manager and billing. Upon upgrade, the database schema will automatically be updated. Several changes have been made to the schema, and for large installations the first restart after upgrade will thus perform a lengthy schema update. It is possible to perform the schema update without starting dCache by using the dcache database update SrmSpaceManager@DOMAIN command.

Due to the schema changes, once updated, it is not possible to downgrade dCache without rolling back the schema changes. dCache can perform the rollback using the dcache database rollbackToDate TIMESTAMP SrmSpaceManager@DOMAIN and dcache database rollbackToTag TAG SrmSpaceManager@DOMAIN commands. For the latter, the database must have been tagged using the dcache database tag TAG SrmSpaceManager@DOMAIN command prior to upgrading the schema.

Third party alterations to the schema should be carefully inspected before upgrading, and possibly be rolled back if they conflict with the schema changes. In particular workarounds for a problem related to outdated statistics for the srmspacefile table are no longer necessary.

Many fields now have non-null constraints, and some have unique constraints. Several indexes have been dropped.

The database schema contains several fields which are aggregates of other fields. In earlier versions, these aggregated fields were maintained by dCache, adding considerable overhead in the form of database round trips and locks. This logic has now been embedded in the database in the form of database triggers. Be aware that these triggers are essential and must be restored if the database is restored from backup.

The file system path for non-SRM transfers is no longer registered in the space manager database. Such files are only registered by their PNFS ID. For SRM transfers, the path is only retained until the file has been uploaded. Once uploaded, the path is cleared as it is no longer used and only bloats the database.

Space manager now performs more garbage collection to remove expired or released reservations from its database. This should reduce the amount of database bloat.

Space manager acts as a proxy service to pool manager. That is, it intercepts all messages from doors to pool manager. In particular it adds additional processing to the pool selection and mover creation messages. That these messages are intercepted has not changed; however, the processing that occurs when these messages are received has changed. In general, database updates are postponed until the latest possible moment in the process of selecting a pool and starting a mover. This improves robustness in case of errors, as less state has to be rolled back. In some cases these changes also allow us to skip processing the message in space manager, allowing such messages to be forwarded to pool manager right away. This reduces latency.

Admin shell changes

The space manager admin commandline interface has been completely rewritten. All commands have been modified. Please consult the build in help output for details. The commands fix missing size and removeFilesFromSpace have been removed, while the commands ls, ls file space tokens, listInvalidSpaces, removeFilesFromSpace, remove file, reserve, release, and update space reservation have been either renamed or modified. The new commands are

ls link groups

list link groups

ls spaces

list space reservations

ls files

list files in space reservations

reserve space

create new space reservation

update space

alter space reservation

release space

remove space reservation

purge file

remove file from space reservation

purge spaces

Garbage collect removed and expired space reservations

These commands accept a great many options that alter their behaviour, so make sure to read the help output offered by each of them. The new commands no longer expose the auto-generated link group id and file id database keys. Instead link groups are identified by their name, and files by their PNFS ID or path.

Configuration changes

Database connection pool settings are now configurable using the spacemanager.db.connections.min and spacemanager.db.connections.max properties.

Space manager no longer uses the ThreadManager cell for task management. Instead it has its own thread pool. This thread pool can be configured using the spacemanager.limits.threads property. This change decouples the space manager from the SRM service.

Several issues related to non-SRM transfers have been fixed. In particular the interpretation of spacemanager.enable.reserve-space-for-non-srm-transfers has changed. When enabled, space manager will create a space reservation in a link group for the file being uploaded. Previously, if creating the space reservation failed because no suitable link group was found for the file, the upload would fail. Starting with 2.8, the upload will fall back to a link not in a linkgroup.

Other changes

Several cases of wrong behaviour on failed uploads have been fixed. These affect both non-SRM transfers and SRM transfer. The bugs could cause the retried upload to fail too.

Space manager in some cases used to query pool manager for additional information during link group selection. This has now been replaced by a periodic import of the entire pool selection state from pool manager. The benefit is that space manager now can select link groups without querying pool manager, thus reducing load on pool manager and reducing latency in space manager.

srm

A new log file was added to the SRM. This log file contains an entry for every SRM protocol level access being made and is thus analogous to the common access logs generated by most web servers. The log contains a timestamp, the client address, the SRM request type, the SRM client, and enough information to identify the SRM request in the SRM database. The log is created in /var/log/dcache/ and is named after the domain followed by the string .access. The file is created for every domain, but it will be empty unless the domain contains an SRM service. Future updates may add support for other protocols. The log is rotated daily and kept for thirty days (this is configurable through logback).

Several minor improvements to logging have been implemented.

webadmin

The space reservation view now only lists permanent spaces, i.e. spaces without an expiration time. This suppresses implicit space reservations in the list, which makes the output more compact.

Changelog from 2.7 to 2.8.0

4cf2ca0

ftp: Reenable support for RFC 2428

bb867f7

chimera: Optimized creation time column changeset

6ed05d6

dcache-webadmin: fix ArrayIndexOutOfBounds exception in ListView (Tape Transfers page)

800a33e

ftp: Disable ipv6 support

9a09439

ftp: Disable ipv6 support

6ad46ba

spacemanager: Fix liquibase changeset for postgresql

10acbdc

spacemanager: Fix link group overallocation trigger

a03b060

spacemanager: Fix unlock of allocated files on delete

f5c4e29

cells,ftp: Take care to initialize the CDC

c1e7329

spacemanager: Fix NPE when creating infinite reservations

2661b84

spacemanager: Improve CLI errors

51f3855

nfs: Prevent NFS door from starting if schema is not up to date

5679ce0

spacemanager: Remove file count from space reservation list

8d2006f

Restore compatibility with 2.6 pools

5b4d3a6

spacemanager: Fix extraction of null values

5a70f1f

spacemanager: Fix queries searching for glob match

947d1c0

spacemanager: Fix expiration check

1156be3

spacemanager: Fix option value in ls files command

e51e287

chimera: Optimize schema migration

ead6c2b

gplazma2-ldap: Ignore junit tests

f3b9125

spacemanager,pinmanager,xrootd: Don’t block during startup if pool manager is down

d84e71e

pom: Fix duplicate include of release plugin

7f0ee6b

spacemanager: Avoid aggressive error output on cell communication

c93d11e

spacemanager: Fix case statements in db triggers

bb50822

spacemanager: Fix update of file entity

3bffb67

spacemanager: Fix slow SQL query

85fba26

deb: Append timestamp to snapshot builds

12709ae

spacemanager: Make help strings less ambiguous

bc4b71d

spacemanager: Rename release file to purge file

ecf5ed4

spacemanager: Limit ls files command to transient states by default

3121bdb

spacemanager: Replace lifetime column with expirationtime column

6660dc3

spacemanager: Make changesets drop indexes conditional

940254b

spacemanager: Reorder liquibase changesets to make updates faster

c6ea3fb

spacemanager: Use HikariCP

f9d063d

spacemanager,webadmin,info: Only publish permanent space reservations

09bb5e7

spacemanager: Delete dead code

a4127e1

spacemanager: Turn state singletons into enums

d06daa7

spacemanager: Clear path when file is stored and make path unique

bd27597

spacemanager: Simplify updateFile method

1cdb0a4

spacemanager: Use FsPath to represent file system paths

4a03ed3

spacemanager: Add automatic and manual purge of perished spaces

f3eaba9

spacemanager: Add not null and unique constraints to link group name

6550cec

spacemanager: More fluent API and less business logic in data access layer

5a5ccd2

pom: enforce maven-release-plugin 2.4.2

dfcee57

spacemanager: Fix precondition check for plpgsql language check

dfeaa7c

libs: update nfs library to 0.7.1

9662041

srm: revert “refactor SchedulerFactory”

b2c4f15

spacemanager: Fix trigger name in HSQLDB triggers

979e8e4

glue: do not assume PoolManager runs in dCacheDomain

0231563

chimera: add an rm command to shell

de7d6e3

chimera: make lstag argument optional

cf963c4

chimera: refactor PGET, part 2: eliminate caching of file locality node

5e38de9

chimera: refactor PGET into abstract node

f547ecd

backoff-controller: update unit-tests, refactor code

1767266

pool: Follow up cd0c482af5338249c064f0cbe889bdc2c1790c0f - Make rep ls output more efficient

3bd9089

spacemanager: Minor UI improvements

e51ad32

spacemanager: Fix help output of reserve space

1165969

spacemanager: Don’t accept output retention policy

3c09ed7

spacemanager: Add missing update of deleted flag

739b60c

Drop chimera-cli script

b9d945f

nfs4: fix orphan proxy movers

cd0c482

pool: Fixing rep ls output, RT 8203

02a022d

chimera: tigger action with FsInode_PSET#setStat()

f6423b2

spacemanager: Delete flushed file reservations when name space entry is deteled

0fb5840

spacemanager: Fix expiration output on ‘ls files’ command

2330db5

spacemanager: Refresh pool monitor on ‘update link groups’

98a721a

spacemanager: Rewrite command line interface

c9332c7

chimera: Fix help output of chimera CLI

804d13a

chimera: Add more documentation and error checking to chimera shell

39fa0fe

spacemanager: Propagate SpaceException error to SRM

aa01662

spacemanager: Fix syntax error in SQL expression

76ead5a

spacemanager: Fix capitalization bug in liquibase changeset

9e4e85b

webdav: support redirecting to next replica for NOT FOUND or FORBIDDEN responses

9ba476a

pool: Do not save terminated jobs to setup file

3475087

spacemanager: Fix capitalization bug in liquibase changeset

1750f0b

Fix text width calculation for ANSI formatted help output

548b541

spacemanager: Retry on transient errors

da4d83c

dcache-chimera: Add interactive chimera shell

62e5dec

cns: make unit tests less sensitive to speed of computer

b30cb25

build: fix RPM builds

7279741

Reduce class path length on liquibase invocation

02e47c4

poolmanager: Remove link group attributes

c3c4ab7

transfermanagers: Send subject and updated file attributes to pool

68909a2

spacemanager: Split space manager into service layer and data access layer

972f92c

spacemanager: Convert tabs to spaces.

c53dbc1

spacemanager: Use JdbcTemplate and annotation based transaction demarcation

064b6cd

Fix gitignore

be1b718

nfs: fix compatibility with 2.6

c7b6aa3

Per ApplicationContext @Transactional configuration

46c37f2

spacemanager: Add changeset to install plpgsql

827b2f8

Use ajc to compile aspects

50084af

cells: Catch UnresolvedAddressException

c6949ce

admin: fix batch file of acm cell

b006e05

chimera: add “.(get)(filename)(checksum)” command

9657929

chimera: fix get locality to work with id-type FsInode

623a93b

nfs4: fix IO on special files

aea2010

spacemanager: Add missing rollback in liquibase

1f0c1a5

spacemanager: Use PoolMonitor to extract link groups

b6ec60c

spacemanager: Do not use ThreadManager

5c786c2

spacemanager: Make database connection pool configurable

44bab04

spacemanager: More code cleanup

7fb7684

spacemanager: Remove dead code and other minor cleanup

222cae0

spacemanager: Maintain aggregate columns with database triggers

79cad7f

spacemanager: Drop ‘fix missing size’ command

877a497

spacemanager: Mover DB schema management to Liquibase

2cbcb17

spacemanager: Do not delete SRM managed reservation entries on file deletion

e199cbf

spacemanager: Postpone DB update to intercepted pool accept message

48255d9

Upgrade dependencies

b3b5a29

spacemanager: Refactor message dispatching code

98012be

Add preallocation field to PoolAcceptFileMessage

4c41bd1

spacemanager: Fix use of global property in spacemanager configuration

93ec46a

nfs4: implement proxy-io on read

3f0703b

libs: update nfs4j to 0.7.0

77a7423

Fix PoolRemoveFilesMessage#toString

f151dab

pnfsmanager: Use ISO8601 compatible date for metadataof output

91a2ed3

Upgrade to DataNucleus 3.3

ff628f7

Upgrade to liquibase 3

5b538ed

Upgrade to Spring Framework 4

21588b7

poolmanager: Add named error codes for no-pool-configured and no-pool-online

5967210

nfsv41: remove unused field in NFS4ProtocolInfo

94030f5

dcache-core (billing): modify changesets to do concurrent index creation

d07d1bb

nfsv41: simplify internal Transfer class

ce9d61d

pom: scala-reflect depenencies

cad663d

gplazma: ldap: make user home and root configurable

c1eed0c

poolmanager: Replace StorageInfo by FileAttributes in PSU

3ee3a23

spacemanager: Allow unreserved transfers even when space reservation for non-srm transfers is enabled

8c44bdf

spacemanager: Reduce latency for some pool selection messages

e105197

spacemanager: Postpone DB update on pool selection

d54d1eb

spacemanager: Use imported PoolMonitor to select link group

5273cfd

ftp: Use shared thread pool for various processing threads

8642945

cells: Further refine LoginManager startup and shutdown

41d2970

defaults: fix typo in common header

0a18178

spacemanager: Fix typo in property name

5c32ce9

spacemanager: Split selectPool method

4755b31

defaults: add common explanation of annotations

09769a3

nfs: introduce PoolDeviceMap for pool to deviceid mapping

d6b2d87

nfs: remove obsolete comment

0a9e1c9

nfs: do not create an instance of StripingPattern per pool

3933ee7

logback shell: add set pattern command

322adcb

transfermanager: formatting patch

85994d4

transfermanager: white-space only fix

0bd6afa

dcache-webadmin: put limit on entries in alarms table

e30a4ae

dcache-webadmin: wget xml statepath (Info page) fails with HTML 500 error

0200256

dcache-core (statistics): fix yet another potential NPE in tree creation

f6855ee

dcache-core: fix statistics totals for top, year and month

64fb8a6

common: simplify gauge logic

c547ddd

srm: fix Copy scheduling

90e584f

ftp: Refactor command queue into sequential executor

1193536

srm: refactor SchedulerFactory

d5512f6

commons: use long to store updates count of a gauge

9cf2b89

ftp: Fix race conditions caused by message reordering

d885e74

ftp: Fix race that lead to erroneous error message in log file

25e91f9

ftp: Fix deadlock introduced in recent refactoring

97c8ac6

ftp: Add LoginCellProvider for FTP

641a644

pool: Fix race condition in migration module

a6ec39a

nfsv41: use sort timeout between door and pool manager

7114b90

scripts: fix dcache script for Solaris machines

2ff8818

webdav: log why request is rejected with permission denied

143b96c

srm: refactor part of Scheduler to use guava Service

fd77a0e

chimera: avoid extra db lookups on create and remove

8b70f9e

pnfsmanager: add support for symlink creation

bd4aa6c

namespace: add support for link creation

d80c824

srm: add missing information from “ls queues” command

5c788be

dcache-webadmin: fix comparison semantics

110a50c

Clean service shutdown order in LoginManager

70974a3

ftp: Reorganize ftp classes

465a782

Implement EPSV and EPRT commands

3834456

pool: bypass IO queue if there is a slot available

3caea80

nfsv41: garbage-collect expired moved

4395eca

ftp: Refactor FTP door to separate FTP interpreter from I/O logic

1e7f211

ftp: Minor cleanup of FTP doors

e541396

cells: Add factory for login cell creation

77787ee

cells: Use common LoginBrokerHandler in LoginManager

f0decb0

ftp: Rename ac_ accessors to ftp_

74fb60c

system-test: add nfsv41 into default configuration

87dab31

cells: Move dCache’s cell interfaces into cells

5b4d856

cells: Associate threads and initialization messages with their cell

19c2b52

Fix configuration cache invalidation

b1c04d4

chimera: use path to get files parent on remove

e99720f

cells: Refactor LoginManager

35c7970

loginmanager: Automatic reformatting of code

1ac339c

pnfsmanager: Drop PnfsGetStorageInfoMessage

e7742ac

pnfsmanager: Drop legacy FileMetaData class

edd8ad8

pnfsmanager: Drop PnfsSetFileMetaDataMessage

fbae6b4

pnfsmanager: Drop legacy PnfsGetFileMetaDataMessage

8747a50

srm: Avoid use of legacy PnfsGetFileMetaDataMessage

6ace876

Add support for simple read-only WebDAV properties

021b410

Remove legacy logging in AbstractCell

a6500bb

pool: Fix command string output for migration commands

c5ab9ad

Add help text for rebalancer.

c8587e6

namespace: use FileType to specify type of create object

661a15e

gplazma: add public key size information to gPlazma results printer

efd4dee

Fix log message containing invalid access latency and retention policy values

ccd9cb1

libs: update nfs library

d88993d

transfermanagers: Avoid use of legacy pnfsmanager message

4609b07

transfermanagers, replica, admin: Fix pool to pool transfers

110be4f

gPlazma-ldap: fix non-serializable Set problem

f6afcf7

libs: update jglobus libraries

61e9859

billing: Parallelize billing search

c115138

billing: Sorted file tree traversal

dfaee9e

log: Rename access and events logs

fd74e03

srm: Supress stack trace on EofException in Axis 1

6217cee

srm: Supress stack trace on EofException in Axis 1

9708aad

billing: Add date prefix to output of search command

6350056

ldap: fix filter property

752e058

dcap: use CHANGE_TIME to provide ctime on stat

7501da4

namespace: use correct DB field for CREATION_TIME

9bc76df

chimera: update DB schema to support CREATION_TIME

ee05697

chimera: remove historic SQL code

9a30984

namespace: use liquibase for chimera based tests

63ff338

dcap: separate protocol name and client ip in DCapProtocolInfo#toString()

6aa5a40

chimera: compact hadles passed to NFS server

71c667a

nfs: add -c option to statics command to reset counters

d73114f

billing: Add shebang header to indexer cronjob

2d8a981

srm: Fix compilation error introduced by bad merge

442cde2

srm: Some more refactoring of Scheduler

59a6828

srm: More Scheduler refactoring

28ff5c5

srm: Get rid of dead exception code in Scheduler

f0bd233

srm: Refactor Scheduler

7c34310

pool: Fix -sticky option in migration module

893e167

srm-client: fix building of tagged RPMs

fb55fb8

info-provider: Fix regression in publishing GlueServiceEndpoint

8608b98

ssh2: fix int overflow with timeout

a818b5d

srm: Improve logging on pin failure

ca9b2c2

srm: Fix a few IllegalStateTransition errors in srmLs

fb28839

commons: add a way to reset gauges and counters

12b8206

correcting IoMode check for SEEK_AND_WRITE

b8907ec

billing: Fix compression of error files

000c505

Fixed typo in logback configuration

7717ac1

commons: make use try-with-resource in couters

9904943

xrootd: Upgrade to xrootd4j 1.2.2

6a0fc40

spacemanager: Fix S2 usecase.CheckGarbageSpaceCollector regression

59440b0

Fix build failure in EventLogger class.

cc106ce

srm: Add an access log

1142ec3

libs: update to nfs4j–0.6.2

9099693

srm: Improve error message on file_busy and duplication_errors

74a070b

info-provider: Fix publishing of SRM

b995602

chimera: fix handling of bytea data in path2inode and path2inodes stored procedures

7782b92

srm: Fix space expiration check in space query

264ebd2

srm: Fix regression in srmGetSpaceMetaData

a16d3eb

info-provider: Fix regression caused by info-provider using a deprecated property

cf33d78

deb: Include cron job for billing compression and indexing

eb26c21

doors: Allow loginbroker property to be empty

ca144ff

srm: Fix NPE and timer thread kill when restoring jobs

7e448d0

spacemanager: Fix setter for cleanupExpiredSpaceFiles property

5190748

srmspacemanager: catch and ignore SQL exception on error handling path when processing PoolMgrSelectWritePool on reply from PoolManager