1.9.12 Release Notes
dCache 1.9.12 focuses on:
- Completing the transition to the new configuration system and improving the script to migrate old configurations.
- Adding GLUE 2 support.
- Replacing the pin manager with a faster and more robust implementation.
- Extending the dcache script.
- Adding HTTP support to third party srmCopy transfers.
- Adding proxy certificate and VOMS support to the HTTP/WebDAV door.
- Reducing file open latency.
- gPlazma 2 with ARGUS black listing.
Table of contents
Upgrade Instructions
Please use The Ultimate Golden Release Upgrade Guide to upgrade from 1.9.5 to 1.9.12.
Incompatibilities
Please consider the following changes when upgrading from a version before 1.9.12-1:
- The arguments and behaviour of /opt/d-cache/bin/dcache create pool have changed.
- The output of the /opt/d-cache/bin/dcache create pool has changed.
- The script /opt/d-cache/install/install.sh has been removed.
- The SRM no longer submits PostgreSQL specific vacuum commands to the database.
- The output format of the PNFS manager info command has changed.
- The info provider configuration file /opt/d-cache/etc/glue-1.3.xml was replaced by /opt/d-cache/etc/info-provider.xml. The file formats of the two files are different.
- The database schema of the pin manager has changed and will be updated on first start.
- The command line interface of the pin manager has changed slightly.
- Several DNS lookups and reverse lookups have been eliminated. This may affect installations on multihomed hosts.
- The configuration parameters httpHost, xsltProcessor, xylophoneConfigurationDir, xylophoneConfigurationFile, xylophoneXSLTDir, saxonDir, srmJdbcMonitoringLogEnabled, srmNumberOfDaysInDatabaseHistory, srmOldRequestRemovalPeriodSeconds, srmJdbcSaveCompletedRequestsOnly, srmJdbcEnabled, and srmCleanPendingRequestsOnRestart are deprecated.
- The configuartion parameters srmVacuum, srmVacuumPeriod, webdav.images.logo, webdav.images.directory, webdav.images.file, webdav.style.css, webdavLogoPath, webdavDirIconPath, webdavFileIconPath, webdavCssPath, dirIconPath, fileIconPath, and cssPath are obsolete.
- The values of the configuration property broker.scheme have changed: jms and hybrid have been replaced by amq and cells+amq-embedded, respectively.
- SSH keys used by the admin door have moved from /opt/d-cache/config/ to /opt/d-cache/etc/.
- Static content used for the httpd service has moved from /opt/d-cache/docs/ to /opt/d-cache/share/httpd/static/. Deployments that replace any of these files have to be updated.
- The new pin manager requires at least PostgreSQL 8.3. The upgrade will fail with older versions. Notice that PostgreSQL 8.1 is no longer support by the PostgreSQL community and that version 8.2 will reach end of life ultimo 2011. We recommend using at least PostgreSQL 8.4 for all dCache databases.
Compatibility
It is safe to mix pools of releases 1.9.5, 1.9.6, 1.9.7, 1.9.8, 1.9.9, 1.9.10, 1.9.11 and 1.9.12. Head nodes and doors must be upgraded to 1.9.12 together and cannot be mixed with head nodes or doors of releases before 1.9.12. Components of different 1.9.12 releases can be mixed freely.
Compatibility Matrix
We distinguish between pool components and head nodes. Any component which is not a pool is considered a head node, including doors. The following table shows compatibility between different releases.
1.9.12 Head | 1.9.12 Pool | ||
---|---|---|---|
Head | 1.9.5 | no | no |
1.9.7 | no | no | |
1.9.8 | no | no | |
1.9.9 | no | no | |
1.9.10 | no | no | |
1.9.11 | no | no | |
1.9.12 | yes | yes | |
Pool | 1.9.5 | yes | yes |
1.9.7 | yes | yes | |
1.9.8 | yes | yes | |
1.9.9 | yes | yes | |
1.9.10 | yes | yes | |
1.9.11 | yes | yes | |
1.9.12 | yes | yes |
dCache 1.9.12-26
Miscellaneous
Updated JGlobus libraries to support jumbo proxies.
Changelog 1.9.12-25 to 1.9.12-26
- [r18216] libs: update globus library
dCache 1.9.12-25
Service: pool
Fixed timeout bug in migration module
Service: httpd
Fixed a problem with usage charts on the pool and pool groups pages not being shown when using certain settings for system locals.
Service: srm
Fixed a bug that prevented clean shutdown of the SRM.
Changelog 1.9.12-24 to 1.9.12-25
- [r18089] pool: Fix timeout bug in migration module (Merge of r18029)
- [r18090] srm: Use correct scheduler for space reservation requests (Merge of r18062)
- [r18091] httpd: Don't use system locale when generating HTML (Merge of r18063)
dCache 1.9.12-24
Miscellanious
Fixed the storage-info check which is needed for the migration from PNFS to chimera.
Changelog 1.9.12-23 to 1.9.12-24
dCache 1.9.12-23
Chimera
Fixed incorrect reply on database errors which may cause data loss on flush. The situation may occur if chimera's postgresql database is not available during flush to the HSM.
Service: xrootd
Fixed an NPE in xrootd mover.
Changelog 1.9.12-22 to 1.9.12-23
- [r17877] pool: Fix NPE in xrootd mover
- [r17880] chimera: fix FsInode.exists() function by throwing exception
- [r17891] cells: fix bad merge r17845
- [r17896] release 1.9.12-23
dCache 1.9.12-22
Incompatibility
Please consider the following change when upgrading from a version before 1.9.12-22
Due to a change in the pnfsmanager the
chimera stored procedures have to be recreated. Execute psql
-U
This change does also require a synchronized upgrade of
- pnfsmanager
- nfsv3
- nfsv41
- cleaner
Chimera
Added support to change the mode, the group and the owner of a directory tag if the tag permission allows this operation.
Service: pool
Fixed a race condition in pools affecting srmCopy transfers.
Fixed formatting of date strings.
Fixed logging of erroneous context information in pools for HTTP and xrootd transfers. The problem was attempted fixed in an earlier release, however the prior fix did not work as intended.
Fixed xrootd vector read bug affecting reads of more than one file
Service: pnfsmanager
Fixed a null pointer exception that typically occurs when reading files migrated from PNFS.
Added an option to verify lookup permissions of the entire
path.
For performance reasons dCache with PNFS only verified the lookup
permissions of the directory containing the file system entry
corresponding to the path. I.e., only the lookup permissions for the
last parent dirctory of the path were enforced. For compatibility
reasons Chimera inherited these semantics.
When this option is set to true, Chimera will verify the
lookup permissions of all directories of a path.
This change requires that chimera users recreate the stored
procedures by psql -U
Service: billing
Fixed formatting of date strings.
Added a NULL check to avoid NPE.
Service: httpd
Fixed formatting of date strings.
Service: gplazma
Fixed longest prefix match in vorolemap.
Service: srm
Fixed a potential race condition.
Fixed formatting of date strings.
Improved error messages if the host certificate, the host key or the ca certificates are are owned by a wrong user.
Service: xrootd
Fixed a potential race condition.
Fixed hanging downloads during or shortly after a file is being uploaded.
Fixed bug which produced wrong TTL values when sending requests to pool manager and other cells.
The properties xrootdAllowedPaths and xrootdIsReadOnly are forbidden now. The same functionality can be achieved by setting xrootdAllowedWritePaths and xrootdAllowedReadPaths. The configuration of existing deployments may have to be adjusted on upgrade.
Fixed a bug that prevented GSI xrootd and the gPlazma2 kpwd plugin from working together.
Service: ftp
Fixed a potential race condition.
Fixed hanging downloads during or shortly after a file is being uploaded.
Fixed bug which produced wrong TTL values when sending requests to pool manager and other cells.
Service: gridftp
Fixed the issue that mkdir did not work with UberFTP.
Service: webdav
Fixed a potential race condition.
Fixed hanging downloads during or shortly after a file is being uploaded.
Fixed bug which produced wrong TTL values when sending requests to pool manager and other cells.
Service: nfsv
Fixed hanging downloads during or shortly after a file is being uploaded.
Service: nfsv4.1
Fixed a potential race condition.
Fixed bug which produced wrong TTL values when sending requests to pool manager and other cells.
Service: dcap
Fixed a potential race condition.
Miscellanious
Fixed messaging bug that potentially can affect all services.
Fix display of arrays for bean property and bean properties commands.
Reordered the classpath such that plugin jars are at the front of the classpath. This allows plugins to replace components already shipped with dCache.
Properties
New property:- pnfsVerifyAllLookups
- xrootdAllowedPaths
- xrootdIsReadOnly
Changelog 1.9.12-21 to 1.9.12-22
- [r17623] CellMessageDispatcher: Fix lost reply bug in message dispatching (Merge of r17576)
- [r17628] UniversalSpringCell: Fix inspection commands for arrays (Merge of r17577)
- [r17632] dcache: Remove ThreadManager unit tests (Merge of r17355)
- [r17638] unit-tests: ignore broken unit-test (Merge of r17634)
- [r17642] pool: Fix race condition for srmCopy grid ftp transfers (Merge of r17583)
- [r17645] pnfsmanager: Fix AL and RP lookup for file without AL and RP (Merge of r17606, r17605, r17604)
- [r17651] Fix array copy in IoDoorInfo (Merge of r17646, r17585)
- [r17654] Add missing chaining of exceptions
- [r17655] boot: Search plugin directories to locate service files (Merge of r16510)
- [r17668] unit-test: ignore time-critical test (Merge of r17660)
- [r17672] Fix race conditions in use of SimpleDateFormat (Merge of r17594)
- [r17687] jetty-GSI: provide meaningful errors if files are owned by wrong user (Merge of r17602)
- [r17691] xrootd,webdav,nfs,ftp: Fix download hang on incomplete file (Merge of r17616)
- [r17695] doors: Fix treatment of Long.MAX_VALUE as timeout value (Merge of r17619)
- [r17707] pool: Fix CDC context - again (Merge of r17705, r17704, r17661)
- [r17719] pool: Fix xrootd vector read bug affecting reads of more than one file (Merge of r17706)
- [r17727] gplazma: Fix longest prefix match in vorolemap (Merge of r17726, r17725, r17715)
- [r17728] xrootd: Fix interpretation of xrootdAllowed*Paths and xrootdIsReadOnly (Merge of r17624)
- [r17734] update HSQLDB version on supported branches
- [r17736] xrootd: Fix GSI principal extraction
- [r17738] Remove file mistakenly committed ages ago
- [r17741] Reorder classpath in the presence of plugins
- [r17747] NPE in billingDB
- [r17748] Fix build after broken commit
- [r17796] chimera : show 1Exabyte as available space (Merge of r17774)
- [r17826] pnfsmanager: add option to verify lookup permissions of entire path (Merge of r15551, r15567, r15568)
- [r17834] chimera: add support for change mode, owner and group of a tag (Merge of r17078)
- [r17839] pnfsmanager: tidy-up after broken commit (Merge of r15551)
- [r17845] routing: fix concurrent modification exception under heavy load
- [r17847] pnfsmanager: further tidy-up after broken commit (Merge of r15551)
- [r17855] gridftp: mkdir did not work with UberFTP; work-around + fix successful reply (Merge of r17850)
dCache 1.9.12-21
Service: info
Modified the logging such that the IllegalArgumentException, which used to be logged if info checks for the current status of a poolgroup, link ... that does not exist anymore, is now logged at debug level.
Fixed a bug that caused the info service to report cannot find StateChangeSet for path object if object is a non-existent part of the info tree.
Service: webdav
Fixed bugs that prevented downloads from working correctly. These were the lack of proper escaping of the redirect URL, of proper UTF-8 encoding of the displayable file name, and of marking the output as UTF-8 encoded and proper encoding of the URLs for the links in the listing, which prevented correctly displaying non-ASCII file- and directory names.
Provided a work-around for a Java bug that would cause any attempt to view a directory to fail with a 500 Internal Error page being shown, if this directory or a file inside the directory triggers the bug.
Chimera
Improving the identification which dCache service is using a particular database connection.
Service: gplazma
Fixed a bug that occurred when two gplazma cells were configured in the same domain and dCache tries to terminate one during startup.
Fixed XACML plugin by upgrading to latest versions of EMI voms and trustmanager libraries. Symptoms of the bug were NoClassDefFoundError error messages in the log files.
Increased available space from 10 Petabyte to 1 Exabyte.
Service: xrootd
Removed a restriction that prevented an xrootd authentication plugin and an xrootd authorization plugin to be used at the same time. Removed a limitation on xrootd authentication bucket size. The limit would have resulted in error messages like Caused by: java.io.IOException: Encountered a bucket with length 11088. Maximum allowed bucket length 10000.
Modified the ALICE authorization token plugin for xrootd to no longer require an authorization token for directory listing. Directory listing can thus be performed without explicit authorization. The change only affects users that use the ALICE authorization token plugin. The change was requested by the ALICE project.
Service: ftp
The different kinds of ftp doors now all support an ftp.read-only property. Defaults are as before.
Fixed a regression that prevented read-only only anonymous users in the plain FTP door.
Fixed Kerberos FTP, which could not start.
Service: pool
Fixed performance issues that affected FTP downloads of files bigger than 2GB. These issues have been observed on Solaris, but may have affected other operating systems too.
Fixed several xrootd releated race conditions and protocol violations in the pool.
Upraded Berkeley DB Java Edition to 4.1.21. This library may be used by pools to store meta data. The upgrade fixes numerous bugs in the library.
Fixed a null pointer exception in the xrootd mover. The exception was triggered by clients contacting a pool directly without being redirected by an xrootd door first.
Fixed several race conditions in the xrootd mover that would be triggered when the mover is killed. The symptoms of the race condition were stack traces in the pool log.
Fixed erroneous context information in log messages generated by xrootd and http movers.
Service: srm
Fixed mistake in error message that could lead to misleading and ambiguous error messages being logged.
Service: spacemanager
Enabling the spacemanager service with srmSpaceManagerEnabled=true used to cause a misconfiguration of the space manager. Fixed a test such that it is now possible to choose one of the allowed values (yes, true, on and enabled) to enable the spacemanager service without misconfiguration.
All domains
Fixed a race condition in the route command of the System cell, that would cause a ConcurrentModificationException to be logged.
Fixed a race condition in the ls command of the RoutingManager.
dcache script
A check to all dcache commands has been added to check whether the hostkey.pem is in PKCS#8 format, if present. If so, a warning is emitted and the command returns with an error code. The warning includes instructions describing how to fix the problem. Unfortunately this new feature introduces a regression that causes grep: /etc/grid-security/hostkey.pem: Permission denied to be written to the console when the dcache script is used by a user without read access to the hostkey. The bug can safely be ignored and will be fixed in 1.9.12-22.
dcache pool create
Fixed a bug in dcache pool create that prevented the script to change the owner of the pool directory.
dcache pool ls
Fixed a bug that prevented dcache pool ls from listing pools without a setup file.
dcache dump heap
Fixed a bug in dcache dump heap that prevented root from dumping the heap of non-root domains.
Changelog 1.9.12-20 to 1.9.12-21
- [r17380] info: don't log IllegalArgumentException (Merge of r16493)
- [r17381] scripts: check whether hostkey.pem is stored in PKCS#8 format (Merge of r16698)
- [r17382] libs: Upgrade to latest version of EMI voms and trustmanager
- [r17383] webdav: fix browsing and access of non-ASCII files and directories (Merge of r16724)
- [r17384] info: fix "cannot find StateChangeSet" error message (Merge of r17017, r17013)
- [r17385] webdav: work-around Java bug (Merge of r17037)
- [r17386] cells/gPlazma: fix race condition when killing a cell (Merge of r17062)
- [r17388] pool: Fix 'dcache pool ls' in case of missing setup file (Merge of r17278)
- [r17389] c3p0: add descriptions
- [r17390] xrootd: Allow simultaneous authn and authz plugins and remove bucket
- [r17400] SubjectsTests: made Java 7 compliant (Merge of r16438)
- [r17401] ConsistentStore: fix broken unit-test
- [r17450] ftp: Avoid zero-copy calls (Merge of r17407)
- [r17455] pool: Upgrade to Berkeley DB 4.1.21 (Merge of r17415)
- [r17459] pool: Fix xrootd NPE in case of missing UUID (Merge of r17431)
- [r17462] srm: fix copy-n-paste mistake in error message (Merge of r17436)
- [r17465] ftp: fix kerberos.batch file to define default value of poolManager parameter (Merge of r17438)
- [r17468] pool: Fixed IllegalStateException in case of xrootd mover timeout (Merge of r17446)
- [r17507] services: fix test for srmSpaceManagerEnabled (Merge of r17366)
- [r17519] cells: Fix ConcurrentModificationException in CellRoutingTable (Merge of r17481)
- [r17523] cells: Fix race condition and logging bug in RoutingManager (Merge of r17482)
- [r17528] scripts: fix pool ownership on create (Merge of r17486)
- [r17536] script: Make dcache dump heap user id aware (Merge of r17489)
- [r17542] xrootd: Do not require ALICE authorization token for directory listing (Merge of r17491)
- [r17543] xrootd: Fix race condition and vector read framing bugs (Merge of r17301)
- [r17545] ftp: add read-only as an option (Merge of r17061)
- [r17548] pool: Clear erroneous NDC for xrootd and http transfers (Merge of r17485)
- [r17552] ftp: fix issue with anonymous read-only access (Merge of r17512)
- [r17556] xrootd: fix unit-test from broken back-port
- [r17570] release 1.9.12-21
dCache 1.9.12-20
Java
Java 7 is allowed as runtime platform.
Chimera
A potential problem, where chimera can leak a database connection has been fixed. Affected sites will see "list full" in their log file.
Changelog 1.9.12-19 to 1.9.12-20
- [r17327] scripts: allow java7 for as runtime platform
- [r17347] Chimera: fix potential database connection leak
dCache 1.9.12-19
NFS
Fixed behaviour on bad RPC messages.
Chimera
Switched to EXISTS statement to avoid duplicate key error in Postgres log files.
Service: webdav
Fixes a problem in the WebDAV door in which proxied reads would pause 5 to 10 seconds after the transfer.
Code
Fixed compatibility with Java7
Changelog 1.9.12-18 to 1.9.12-19
- [r17269] xdr: do not process RPC messages which are not a CALL or REPLY
- [r17275] chimera: use exists statement to prevent constrain violations
- [r17276] chimera: fixed bad merge
- [r17280] common: fix java7 compilation
- [r17287] webdav: Disable persistent connection between door and pool
dCache 1.9.12-18
Service: xrootd
Fix incorrect fd access on READV.
Service: pool
Checksum scanner should survive orphan files.
Scripts: hsmcp
Fix storage class parsing and escape unsafe characters in URI.
Changelog 1.9.12-17 to 1.9.12-18
- [r17167] xroot: fix incorrect fd access on READV
- [r17208] hsmcp: Fix storage class parsing and escape unsafe characters in URI
- [r17221] pool: checksum scaner should survive orphan files
dCache 1.9.12-17
Chimera
Fixed problem where user was not able to rename a file into an existing one if the existing file had some data.
Service: ftp
Improved RFC 959 compliance w.r.t. handling of data channels and closing data channels.
Improved security by only listening for data channel connections on one address rather than on the wildcard address.
Improved GFD 21 (gridftp 1) compliance by no longer enforcing the concurrent stream limit (controlled by the gsiftpMaxStreamPerClient setting) on passive mode transfers. The setting now only applies to active mode transfers (which is the only case allowed by the specification).
Changed the interpretation of GFD 47 (gridftp 2) w.r.t. to the scope of parameters to GET and PUT commands. To our knowledge this change is supported by all known GridFTP 2 clients.
Service: gplazma
A bug in gPlazma 1, which was triggered when a user authenticates with Kerberos (NFS 4.1, Kerberised FTP, Kerberised dcap), has been fixed.
Changelog 1.9.12-16 to 1.9.12-17
- [r17024] chimera: fix move of existing file with non empty level
- [r17027] ftp: Don't close listener socket for passive mode
- [r17084] gPlazma1 - fix UnsupportedOperationException when using Kerberos
dCache 1.9.12-16
Service: webdav
Fix buffering issues in the door.
Service: xrootd
Allow read-only mode.
Service: broadcast
Messages are no longer sent to disabled SpaceManager.
Service: gPlazma
kpwd: Fix support for updating password files via the dcache script.
Suppress log4j initialization in XACML plugin.
Changelog 1.9.12-15 to 1.9.12-16
- [r16953] webdav: fix bad merge in r16679
- [r16970] xrootd: respect read-only mode for all type of auth plugins
- [r16979] broadcast: do not send messages to SpaceManager if it's disabled
- [r16985] kpwd: fix support for updating password files
- [r16988] gplazma: Suppress log4j initialization in XACML plugin
dCache 1.9.12-15
Service: srm
Fix problem with 3rd party transfers with SRM version 2.
Service: pool
Fix classpath for pool reconstruct command.
Service: nfsv41
Reactivated expired session.
Service: xrootd, webdav
Allow embedded gPlazma to be used and allow the gPlazma cell name to use to be configurable.
dCache Script
Fix dCache script for Solaris machines.
Changelog 1.9.12-14 to 1.9.12-15
- [r16763] srm: fix webservice path for 3rd party transfer
- [r16821] scripts: fix dcache script for Solaris machines
- [r16824] pool: Fix classpath for pool reconstruct command
- [r16888] nfsv41: reactived expired session
- [r16894] xrootd: resolve bad merge in r16729
dCache 1.9.12-14
Service: nfsv3
Reduce log level of client errors.
Service: nfsv41
Reduce memory consumption for directory cache.
Service: dcap
Fix harmless null pointer exception.
Service: webdav
Avoid creation of temporary buffer files in /tmp. These were created during directory listing or when redirect on read was enabled. Instead the door will now use HTTP chunked encoding.
Service: gridftp, kerberosftp
Fix regression in directory listing which resulted in the DN or Kerberos principal rather than the mapped user name being used. The regression was introduced in 1.9.12-8.
Service: httpd
Fix match feature of Pool Selection Configuration page.
Service: xrootd, webdav
Allow embedded gPlazma to be used and allow the gPlazma cell name to use to be configurable.
Init scripts
The return code of dcache status now reflects the status of the service.
The dcache status command now lists orphaned domains for which the wrapper process has died. Previously such domains were reported as stopped and the dCache script would happily create a second instance. Now such instances can be killed by dcache stop.
Logging
Fix time stamp formatting in some log files.
Changelog 1.9.12-13 to 1.9.12-14
- [r16666] nfsv3: do not log client errors with error level
- [r16678] dcap: work around raise condition in getIoDoorEntry
- [r16679] webdav: Fix buffering issues in the door
- [r16713] sysvinit: Block non-sysvinit actions
- [r16715] ftp: Fix user name output in directory list
- [r16717] sysvinit: Use correct exist status for status command
- [r16719] httpd: fix Pool SelectionUnit page
- [r16722] logback: Fix date pattern
- [r16729] webdav,xrootd: Make gPlazma name configurable and allow use of embedded
- [r16730] dcache: Report status for orphaned domains
- [r16736] nfs: use google's guava ComputingMap as direcotry listing cache
- [r16761] gplazma2: Fix leak of password hash
dCache 1.9.12-13
Service: pool
Fix another HTTP compliance issues related to range request. This time it is about using the correct reply code (416) when the client requests a range beyond the end of the file. This fix resolves a compatibility issue with ARC.
Fix 'dcache pool ls' to show the pool size stored in the pool's setup file, in case such a pool size exists.
Service: xrootd, webdav, ftp, gridftp, kerberosftp
Fix a harmless but noisy null pointer exception. The problem could affect FTP and WebDAV doors too, but was first observed in the xrootd door.
Service: webdav
Fix a null pointer exception being thrown while generating the error page.
Service: xrootd
Fix regression in Alice token authorization plugin. The plugin failed to load in 1.9.12-12.
Changelog 1.9.12-12 to 1.9.12-13
- [r16648] http: introduce HttpException with corresponding status code
- [r16651] pool: Include correct pool size in 'dcache pool ls'
- [r16653] doors: Fix NPE in periodic mover check
- [r16655] webdav: Fix NPE in error page
- [r16660] xroot: fix plugin loading
dCache 1.9.12-12
Service: pool
Fix several race conditions that could lead to stale hidden movers, TCP connection leaks and link count inconsistencies.
Service: admin
Add detection of terminal size. Some ssh clients do not transmit terminal resize messages. For those clients resizing the terminal after login to the admin service will break line editing.
Service: ftp, gridftp, kerberosftp
Make timeout of write mover creation configurable. The timeout is shared with read mover creation. The timeout can be configured by adjusting the gsiftpPoolTimeout property. A side effect is that the default timeout for write mover creation increases from 10 seconds to 10 minutes.
Service: xrootd
Refactored authorization plugin support to make plugins truly pluggable. These are now loaded through the Java ServiceLoader facility. This allows third parties to develop authorization and path mapping plugins the dCache xrootd implementation. Existing third party xrootd authorization plugins will have to be updated.
Miscellaneous
Fix well known cell name resolving when using JMS messaging.
Added a plugin directory to dCache. The following directories are scanned for plugins: /usr/share/dcache/plugins, /usr/local/share/dcache/plugins, /opt/d-cache/share/plugins. Plugins are represented as sub-directories to any of these directories. jar files are included on the Java classpath, properties files are parsed and added as default configuration values, and batch files are made recognized as service definition scripts.
Changelog 1.9.12-11 to 1.9.12-12
- [r16589] pool: fix several raise condition in IO queue
- [r16596] admin: Add terminal size detection
- [r16600] ftp: Make mover startup timeout configurable for write
- [r16603] cns: Fix JMS route registration
- [r16606] boot: Add plugin directory to dCache
- [r16607] boot: Clean up BootLoader configuration loading
- [r16608] boot: Let BootLoader load plugin defaults
- [r16609] xrootd: Make authorization handler a true plugin
- [r16629] boot: fix loading property values from dcache.conf
- [r16630] xrootd: fix r16609 merge conflict
dCache 1.9.12-11
Service: dcap, gsidcap, kerberosdcap
Resolve an issue when the dcap client is in active mode (ie when the -A option is used) and the server cannot resolve the host name sent by the client (eg when the client is behind NAT). Instead the door now uses the remote address of the control channel when selecting an appropriate pool.
Improve logging in case of errors during GSI or Kerberos sessions.
Fix race condition that could occur during shutdown.
Service: pool
Ensure forward compatibility of the HTTP mover.
Fix several HTTP compliance issues related to range request (ie partial reads).
Service: ftp, gridftp, kerberosftp
Add glob support to the NLST command. This allows mget to work in some clients.
Fix race condition that could occur during shutdown.
Service: webdav
Improve logging on fatal errors. This change may help debug the source of the errors.
Service: poolmanager
Fix a race condition.
Service: admin
Fix several bugs related to line editing. Add history search (press Ctrl-R).
An unfortunate side effect of these fixes is that line editing capabilities need to know the terminal size. The admin door currently does not support determining the terminal size and thus always assumes that the terminal is 80 characters wide. Line editing will misbehave with other terminal sizes.
Fix listing of admin service in dcache services.
Service: pnfsmanager, nfsv3, nvfsv41
Fix name space corruption when moving directories (chimera only).
Fix RFC 5661 compliance of ACL delete permission check. A file or directory can now be deleted if the subject has the DELETE_CHILD permission on the parent directory, or the DELETE permission on the entry being deleted, or in case neither are specified if the subject has the ADD_FILE permission on the parent directory.
Service: srm
Improve robustness against ill formed srmLs requests.
Service: info
Fix logging of several erroneous warning and error messages.
Info provider
Fix GLUE2 compliance. Be sure you have at least v2.0.8 of glue-schema RPM installed on the node running the info provider.
Miscellaneous
Improve documentation in configuration defaults.
Improve white space processing around property values.
Fix race condition in getcellinfos command.
Add service definition for hopping manager. The name of the new service is hopping.
Changelog 1.9.12-10 to 1.9.12-11
- [r16216] dcap: take client ip from the connected socket
- [r16227] http: mover: set SessionID to the HttpDoorUrlInfoMessage
- [r16246] dcap: cleanup in gss/gsi tunnel
- [r16252] ftp: added mget functionality by allowing pattern as NLST argument
- [r16299] http: do not send multi-range reply on single range request
- [r16304] http: fix partial response as well as return correct mutipart type
- [r16332] webdav: Log stack trace on RuntimeException
- [r16335] poolmanager: Fix synchronization and entry expiration in cost module
- [r16336] docu fix
- [r16339] http: make multipart reply rfc2046 compatible
- [r16373] shell: remove leading and trailing whitespace around a property value
- [r16386] cells: factor-out the name of the LoginBroker for the admin service
- [r16390] admin: Upgrade to JLine 1.0
- [r16403] http: send a correct file size on range request
- [r16425] chimera: move of a directory have to adjust reference to the parent
- [r16446] properties: tidy up obsolete and forbidden messages
- [r16461] srm: make srmLs more robust
- [r16467] cells: fix race condition in getcellinfos
- [r16468] FTP/dcap: fix race condition on shutdown
- [r16470] services: add hopping manager
- [r16518] info-provider: fix the GLUE2 we publish
- [r16522] info: fix xcm ls with zero updates error message
- [r16525] info: silence errors due to info restart
- [r16526] chimera: Make delete permission check RFC 5661 compliant
- [r16529] alice-token: Fix lack of path mapping for stat and statx
- [r16532] defaults: update description of obscure feature
dCache 1.9.12-10
Service: pool
Fixed bug in which failure to stage would leave behind a repository entry in state D. The only way to get rid of the entry was by restarting the poo.
Service: pinmanager
Improved robustness of pinmanager. In particular an out-of-memory problem has been fixed.
Changelog 1.9.12-9 to 1.9.12-10
- [r16199] pool: do not fail on removal of non existing files
- [r16206] pinmanager: Improve robustness
- [r16209] build: fix FSH rpm packaging
dCache 1.9.12-9
Service: poolmanager
Fixed a race condition that caused a null pointer exception in rebalancer.
Fixed persistence of the slope setting.
Updated format of PoolManager.conf such that settings are persistet using the partition manager commands introduced in dCache 1.7. The legacy commands are still supported, but will be removed in a future version. Starting with version 1.9.12-9 the pool manager save command save in a format that will be compatible with future versions.
Services: ftp,gridftp,kerberosftp,xrootd,webdav
Use correct transaction duration for billing records.
Services: pool
Fixed the 'dcache pool reconstruct' on Solaris. Without the fix the command would fail with the error "File name too long".
Fixed URI generation in hsmcp.rb. The pool now also logs incorrect URIs returned by the HSM script.
Fixed HTTP range support. Added support for multiple ranges. Fixed path check. These fixes resolve compatibility issues with ARC and ROOT.
Service: billing
Fixed a race condition related to formating of timestamps.
Service: pnfsmanager
Fixed a bug in the PNFS backend that prevented listing through SRM.
Service: xrootd
Improved robustness of transfer timeout logic.
Changelog 1.9.12-8 to 1.9.12-9
- [r16133] poolmanager: Fix NPE in rebalancer
- [r16136] doors: Use correct transaction time for billing record
- [r16140] dcache: Fix 'dcache pool reconstruct' command
- [r16144] billing: Fix race condition in SimpleDateFormat access
- [r16147] pool: Make HSM URI failure error message more verbose and fix hsmcp.rb
- [r16155] pnfsmanager: do not query location for directories
- [r16167] xrootd: Wrap ping task in FireAndForgetTask
- [r16170] poolmanager: Write slope to PoolManager.conf
- [r16173] poolmanager: Don't save setup using legacy format
- [r16181] http: fix support for multi-range request
dCache 1.9.12-8
Service: pnfsmanager
Fixed service instantiation with PNFS backend.
Service: pool
Avoid verbose logging of NoSuchElementException when listing movers. The fix possible solves similar problems for other operations and other services.
Avoid verbose logging of non-fatal DCAP client failures.
Service: ftp, gridftp, kerberosftp
Fixed a race condition that would cause the door to become unresponsive to client commands.
Fixed a potential denial of service problem.
Fixed a null pointer exception triggered by running the info command before the client was authenticated.
Avoid verbose logging when clients do not disconnect cleanly.
Service: spacemanager
Fixed pool selection cost model for systems with space manager enabled. Without the fix a system with space manager would perform pool selection with outdated cost information.
dcache script
Fixed file system size check for the dcache pool create command. It didn't actually check the file system size and in some cases it caused the operation to fail, leaving an incomplete pool behind.
Changelog 1.9.12-7 to 1.9.12-8
- [r16081] pnfsmanager: Fix batch file for PNFS backend
- [r16083] all: Let cells propagate declared exceptions back to client
- [r16085] dcache: Fix pool create
- [r16089] ftp door: add comment containing Globus Online bug-report reference
- [r16092] ftp: Fix race condition in command execution
- [r16095] ftp: Allow GSI handshake to be interrupted
- [r16100] pnfsmanager: Fix another instantiation problem with the PNFS backend
- [r16102] ftp: Avoid NPE in info command
- [r16103] ftp: Avoid error during unclean control channel shutdown
- [r16107] spacemanager: Ensure that deliver and accept messages are sent to PoolManager
- [r16110] pool: Suppress stack traces for non-fatal DCAP failures
dCache 1.9.12-7
Service: gplazma
The configuration property gPlazmaNumberOfSimutaneousRequests was deprecated and replaced by gPlazmaNumberOfSimultaneousRequests.
Service: webdav
Generate themable HTML for 404 errors.
Service: nfsv41
Improved compatibility with early versions of rhel5 and MacOS.
Service: transfermanagers
Fixed deletion of the target file in case of transfer failures. Previous releases would fail with a stack trace logged.
Service: dcap
Reduced logging noise triggerd by file upload.
Service: ftp, gridftp, kerberosftp
Added a workaround for a bug in Globus Online.
Service: srm, webdav, webadmin
Downgraded to Jetty 7.2-2. Jetty 7.4.2 introduced in dCache 1.9.12-6 was the source of a regression in socket error handling.
Init script
Ensure that dcache start starts all domains even when some are already running.
Changelog 1.9.12-6 to 1.9.12-7
- [r15909] gplazma: Fixed spelling of gPlazma property
- [r15912] gplazma: Fixed spelling of gPlazma property
- [r15928] webdav: remove obsolete template
- [r15932] webdav: generate 404 page via StringTemplates
- [r15949] nfsv40: fix setclientid reply
- [r15952] scripts: Don't fail start if a domain is already running
- [r15956] CopyManager: register that the destination of the copy is writing data
- [r15959] scripts: add additional info to the "cowardly refusing.." message
- [r15962] dcap: don't report failed dcap operations as errors
- [r16013] ftp: Workaround bugs in Globus Online
- [r16024] PoolManager: fix option max-copies
- [r16027] all: Mark broker properties as not-for-services
- [r16030] webdav: Clean up DcacheResponseHandler
- [r16033] srm,webdav,webadmin: Upgrade to Jetty 7.4.4
- [r16039] jetty: reverse to version 7.2.2
dCache 1.9.12-6
Service: nfsv3
Fixed a regression introduced in 1.9.12-5.
Fixed portmap registeration: Previous releases registered TCP even when only UDP was enabled.
Fixed attribute caching.
Service: acl
Make acladmin cell a well known cell (like it was in 1.9.5).
Service: dcap
Enable TCP_NODELAY on the control and data channels. Should reduce latency.
Service: ftp
Enable TCP_NODELAY on the control channel. Should reduce latency.
Service: admin
Enable TCP_NODELAY. Should reduce latency.
Service: webdav
Upgraded to Jetty 7.4.2.
Fixed error reporting. Previous versions reported all errors as "server error". Now the proper HTTP error code is reported.
Added HTML formatting and CSS styling of error messages. The template file has been replaced. The property webdav.templates.list is deprecated and has been replaced by webdav.templates.html.
Service: webadmin
Upgraded to Jetty 7.4.2.
Service: gplazma (version 2)
The kpwd plugin now uses defaults from /opt/d-cache/share/defaults/gplazma.properties.
Now supports ${...} substitution for properties defined in dcache.conf or the layout file.
Fixed reloading of the kpwd file.
Service: xrootd
Fixed recovery logic from internal message loss. Ensures that transfers are removed from the active transfers page of the httpd service.
Info provider
Fixed path to XSLT file in FHS packages.
1.9.5 migration script
Updated list of properties to rewrite during upgrade.
Changelog 1.9.12-5 to 1.9.12-6
- [r15807] info-provider: fix info-provider for FHS-compliant build
- [r15812] nfsv3: fix regression introduced by v3/v4 merge
- [r15819] rpc: publish correct netid
- [r15825] migration: update migration support to process additional obsolete
- [r15828] acl: Make acladmin a well known cell
- [r15831] dcap,ftp,admin: Enable TCP_NODELAY
- [r15834] migration: Fix typo in property translation
- [r15850] chimera: update parent directories ctime on new object creation
- [r15865] webdav: Fix exception handling
- [r15868] webdav,webadmin: Upgrade to Jetty 7.4.2
- [r15871] webdav,webadmin: Upgrade to Jetty 7.4.2
- [r15872] gplazma: Define defaults for kpwd plugin
- [r15875] gplazma: Implement placeholder expansion for gplazma arguments
- [r15878] xrootd: Fix transfer timeout
- [r15881] gplazma: Fix reload in case kpwd file does not exist
- [r15884] webdav: improving webdav style pages
- [r15887] gplazma: Remove reverseMap method
dCache 1.9.12-5
Service: nfsv41
Fix a null pointer exception.
Adds optional support for NFS 3 in the NFS 4 door. Eventually the nfsv3 and nfsv41 doors will be merged into one.
Service: ftp
Support UTF-8 encoded file names independent of the local of the host system.
Output permission information by default in MLSD and MLST commands.
Fixed encoding of the UNIX.mode fact introduced in 1.9.12-4. The mode field is now encoded as an octal. This ensure compatibility with UberFTP and other clients.
Added authorization checks for various attributes accessed with MLSD and MLST commands.
Surpress the size fact for MLSD and MLST commands with the entity is a directory. This improves RFC 3659 compliance.
Publish GridFTP protocol version to the "Active Transfers" monitoring page.
Service: admin
Fix null pointer exception.
Service: pool
Improve error handling in case file deletion fails.
Chimera client
Fix readtag and writetag commands and added writefile command.
Configuration
Flags the properties httpdEnablePoolCollector and removeUnexistingEntriesOnFlush as obsolete. The properties have not been used since dCache 1.9.5.
1.9.5 migration script
Fixed translation of the dCacheSetup file. In previous releases only the first line of dCacheSetup was translated.
Logging
dCache configuration properties are now available as ${...} placeholders in logback.xml.
The log files /tmp/events.out and /tmp/trace.out have been replaced by per domain log files. This solves a permission problem when domains are run as different users.
Changelog 1.9.12-4 to 1.9.12-5
- [r15777] nfsv4: try ro return layout on close only for minorverion > 0
- [r15779] nfsv4: optionally start v3 if required
- [r15780] chimera cli: fix and update readtag and writetag, and add writefile
- [r15781] configuration: add missing httpd property with it's obsolete annotation
- [r15782] ftp: Use UTF-8 for encrypted control channel
- [r15783] ftp: Add perm to the default MLST/MLSD fact set
- [r15784] configuration: fix annotations of scoped properties
- [r15785] configuration: add obsolete removeUnexistingEntriesOnFlush
- [r15786] configuration: fix XML Entity handling of scoped assigments
- [r15787] admin: fix NPE in SSH admin
- [r15790] pool: throw a runtime exception if we fail to remove a file
- [r15797] migration: fix migrate-from-1.9.5.sh so it processes all of dCacheSetup
- [r15798] ftp: Fix MLST/MLSD fact encoding
- [r15799] logging: Make configuration parameters available in logback.xml
- [r15800] ftp: Publish correct GridFTP version for active transfers
dCache 1.9.12-4
Service: nfsv41
Fixed attribute caching.
Service: xrootd
Kill mover if client or door timed out before the mover was started. Should reduce the number of mover timeouts.
Service: ftp
Implemented support for RFC 3659 UNIX facts (these are not specified by the RFC, but several clients implement these facts).
Improved compatibility with the UberFTP client (recursive directory listing now works).
Service: srm
Fixed a bug that would cause the message "can not find handler with id=nnnn" to be logged by the transfer manager after srmCopy requests successfully completed.
Service: webadmin
Fixed authorization.
Service: pnfsmanager
Fixed batch file for use with PNFS.
Service: gplazma
Fixed banning in gplazmalite-vorole-mapping plugin. Adding the entry:
"DN" -
will now ban the DN no matter whether it is used in combination with an FQAN or not. Note that banning in the gplazmalite-vorole-mapping plugin doesn't prevent other plugins from authorizing the request.
Fixed a bug which would cause a missing entry in storage-authzdb to authorize a user rather than reject the request. The problem would only occur if grid-vorolemap mapped a user to a user name that isn't mapped in storage-authzdb.
Service: pool
Fixed a conflict between the HTTP and xrootd mover that caused them to use the same TCP port when they shouldn't.
Fixed logging context handling. The bug could cause session IDs from transfers to erroneously be assigned other transfers and could even bleed through to other components.
The pool now no longer updates the last access time of a file during checksum scanning.
Fixed TCP buffer size handling in DCAP movers. The bug in some cases caused the requested buffer size to be incorrectly applied.
Enabled TCP keep alive in DCAP movers. This will ensure that DCAP movers eventually fail if the client abruptly disappears (eg due to a operating system crash or a network failure).
Service: pinmanager
Avoid unintended pool manager cost adjustment when pinning file. Also requires update of poolmanager.
Service: poolmanager
Avoid unintended pool manager cost adjustment when pinning file. Also requires update of pinmanager.
Service: info
Add context information to log messages.
Fixed bug that triggered "StateUpdate with zero updates encountered" log messages.
Removed the GlueSAAccessControlBaseRule work-around that publishes legacy values for broken clients. Any problems with lcg-infosites should be solved by upgrading to the latest version of the UI.
Service: admin
Fixed a couple of error messages.
Fixed an issue that could cause socket to be leaked.
Packaging
Fixed a number of RPM packaging errors in the EMI packages.
Other
Fixed System cell CLI help output.
Changelog 1.9.12-3 to 1.9.12-4
- [r15644] nfsv4: use ctime for inode change indication
- [r15648] gplazma: thwon an error if config file not exists
- [r15649] webadmin: take the changed principal needed into account
- [r15654] pnfsmanager: Fix the batch for for PnfsManager with PNFS backend
- [r15655] pool: Fix unintended interaction between HTTP and xrootd movers
- [r15656] gplazma: Fix most compiler warning in gPlazma 2
- [r15685] pinmanager: Suppress cost update when selecting read pool
- [r15692] info: add NDC describing the current data-gathering activity
- [r15693] cells: handle duplicate commands correctly
- [r15695] info: fix "StateUpdate with zero updates encountered" message
- [r15696] gplazma: Fix NPE
- [r15697] gplazma: Change user name + password encoding
- [r15698] Admin Interface: fix Error Messages
- [r15699] xrootd: Reduce the negative effect of busy pools
- [r15701] gplazma: Fail authorization if storage-authzdb is missing a record
- [r15708] gplazma: Fix DN banning of vorolemap gplazma 1 plugin
- [r15716] gplazma: Fix logging of VOMS plugin
- [r15717] pool: Fix logging context
- [r15719] srm: Don't cancel copy transfer if it already succeeded
- [r15722] build: fix invalid scope for maven dependecies
- [r15725] build: fix broken scope
- [r15737] rpm: Fix most rpmlint errors for FHS package
- [r15738] deb: Fix lintian warnings and streamline the build process
- [r15739] ftp: Improve RFC 3659 compliance
- [r15740] ftp: Implement UNIX specific facts
- [r15741] pool: Don't update atime on checksum calculation
- [r15744] gplazma: add identity plugin with map/reverseMap operations
- [r15749] gplazma: use properties for gPlazma2 plugin configuration
- [r15750] pool: Fix socket option handling in DCAP mover
- [r15751] deb: Change FHS package name to dcache
- [r15752] ssh: fix multiple problems with connections
- [r15753] gplazma2: Add new kpwd plugin
- [r15754] info-provider: remove GlueSA.ACBR legacy value work-around
dCache 1.9.12-3
Service: info
The info service lists the network interfaces of each door in dCache. The DNS name of the interface is used as a key in the list. This causes problems when several interfaces share the same name and would often result in the host name being absent in the data published by the info provider. This problem has now been fixed by giving each network interface a unique name.
Service: httpd
Fixed a number of image files that were corrupted in the 1.9.12-2 release.
Service: pool
The lifetime of sticky bits are now reported in human readable format. The output of the 'rep sticky ls' command is affected.
Service: dcap
Minor logging related cleanups.
Service: nfsv3
Improved server side readdir caching.
Service: srm
Fixed processing of escape codes in SURLs and TURLs.
Out of memory handling
dCache triggers an automatic restart when a domain runs out of memory. We have observed that he restart sometimes fails, presumably because further out of memory errors are triggered during shutdown. A number of changes have been made to make the shutdown sequence more robust.
Since 1.9.12-1 dCache automatically generates heap dump files upon out of memory errors. A check has now been added to the startup script to check for the existing of these files and issue a warning.
Libraries
Several third party libraries have been upgraded. The upgrades represent minor releases that fix bugs.
RPM packaging
Ownership of the billing directory is now preserved accross upgrades.
Changelog 1.9.12-2 to 1.9.12-3
- [r15553] info: fix listing door interfaces for multihomed nodes
- [r15562] source: fix the year in copyright
- [r15563] gplazma: Let gridmap plugin map to UserNamePrincipal
- [r15575] poolmanager: Make exception handling more robust
- [r15576] cells: Make shutdown sequence more robust
- [r15577] chimera: Fix infinite recursion in AbstractBlockingQueueDecorator
- [r15578] httpd: Fix comparator in pool manager view
- [r15579] poolmanager: Make exception handling more robust
- [r15586] SrmSpaceManager : use proper coding in DB access layer
- [r15587] build: Avoid corruption of image files
- [r15592] nfs: merge nfs code into dCache code base
- [r15602] cells: Allow cells and beans to be environment aware
- [r15603] ftp: Fix milliseconds/seconds documentation bug
- [r15604] config: Removed bogus deprecation flag from port parameters
- [r15609] nfs: prevent reverse dns lookup
- [r15610] scripts: Generate a warning if heap dump files exists
- [r15621] pool: fix how sticky record describes itself
- [r15622] pool: fail if admin attempts to set sticky with negative duration
- [r15623] deb: Change FHS package name to dcache
- [r15624] libs: Upgrade DataNucleus, Berkeley DB JE and Milton
- [r15625] dcache: Add kpwd command set
- [r15626] dcap: logging cleanup
- [r15628] nfsv3: fix hashCode calculation of cookieverf3
- [r15632] packaging: perserve billing directory ownership across RPM upgrade
- [r15633] srm: Properly encode non ASCII characters in URI
dCache 1.9.12-2
Service: srm
The following properties can now be specified separately for ls, put, get, bringonline and reserve requests: srmDatabaseEnabled, srmCleanPendingRequestsOnRestart, srmKeepRequestHistoryPeriod, srmExpiredRequestRemovalPeriod, srmRequestHistoryDatabaseEnabled, and srmStoreCompletedRequestsOnly. These properties replace the now deprecated properties srmJdbcMonitoringLogEnabled, srmNumberOfDaysInDatabaseHistory, srmOldRequestRemovalPeriodSeconds, srmJdbcSaveCompletedRequestsOnly, srmJdbcEnabled, and srmCleanPendingRequestsOnRestart.
Service: pool
Fix the processing of the waitForFiles property. The property did not have the inteded effect.
Migration script
Fix a typo that caused the script to fail in 1.9.12-1.
Added translation of SRM properties deprecated in 1.9.12-2.
Added support for moving admin authorization files to new location.
dCache now refuses to start if it detects a 1.9.5 configuration. This prevents that dCache is accidentally started before the configuration has been migrated.
Service: gplazma
Propagate the read-only flag back to doors. In recent versions the flag was ignored.
Several gPlazma 2 related changes were made, but gPlazma 2 is still under development and should not be used in this version. We expect gPlazma 2 to be production ready in 1.9.12-3.
Service: pnfsmanager
Reintroduce check that prevents existing checksums from being altered. The check was present in 1.9.5 but was lost in later versions.
Service: admin
SSH keys have moved from /opt/d-cache/config/ to /opt/d-cache/etc/. The migrate-from-1.9.5.sh script has been updated to move the files. However when upgrading from a version newer than 1.9.5 the SSH keys have to be moved by hand.
Location manager
Fixed a problem in which dCacheDomain would fail to start if /opt/d-cache/etc/lm.config could not be created. This would typically happen if dCacheDomain is started as a user different from root.
The location of lm.config is now configurable.
Service: httpd
Static content used by the monitoring service has been moved to /opt/d-cache/share/httpd/static/.
Service: webdav
WebDAV doors now report the DN to billing when files are deleted.
Changelog 1.9.12-1 to 1.9.12-2
- [r15403] loginBroker: add option to pull information from disabled doors
- [r15411] webdav: Upgrade to Milton 1.5.9
- [r15412] srm: Add per request type database parameters
- [r15413] pool: fix waitForFiles
- [r15416] migration: fix broken script
- [r15417] migration: Add translation of SRM properties deprecated in 1.9.12-2
- [r15418] migration: Fix typo in replacement patterns for deprecated properties
- [r15419] gplazma: Preserve read-only status
- [r15429] openmq: Fix cns and topo services
- [r15430] spacemanager: handle expected exceptions
- [r15431] scripts: make boot-loader more robust
- [r15432] pnfsmanager: Reintroduce check that prevents checksums to be altered
- [r15443] cns: Add ActiveMQ support for cell name service
- [r15444] gplazma: Fix parsing of quoted paths in authzdb
- [r15449] authorization: Minor updates to principals and AuthorizationRecord
- [r15450] gplazma: Renamed and cleaned up GridmapFileHandler
- [r15451] gplazma: convert checked exceptions into AuthorizationException
- [r15452] gplazma: Plugins for vorolemap, grid-mapfile, storage-authzdb, etc
- [r15453] openmq: Add missing OpenMQ CNS Spring file
- [r15454] openmq: Enable reconnect on failure
- [r15455] gplazma: Plugins for vorolemap, grid-mapfile, storage-authzdb, etc
- [r15470] admin: move default location of ssh keys to /opt/d-cache/etc
- [r15471] scripts: don't start if non-migrated config is detected
- [r15472] config: Get rid of JHRM.map
- [r15473] gplazma: Decouple gPlazma from SessionAttributes
- [r15474] pinmanager: Make logger static
- [r15475] conf: Get rid of pool_path.template
- [r15476] deb: Fix build of FHS packages
- [r15477] gplazma: Get rid of VerifiedUserPincipal
- [r15478] jms: Escape dashes in domain names
- [r15480] nfsv41: fix transfer information
- [r15483] login: Add map and reverse map methods
- [r15502] deb: Move files to proper locations
- [r15504] deb: Fix path to loadConfig.sh and bootLoader.sh
- [r15505] lm: Fix setup file creation problem for non-root installation
- [r15506] config: Make location of lm.config and PoolManager.conf configurable
- [r15507] deb: Move files and directories written by dCache to /var/lib/dcache
- [r15508] deb: Generate ssh keys during postinst
- [r15509] cns: Fix deployment with broker.scheme=none
- [r15510] files: move SQL files from etc into share
- [r15511] deb: Add dcache user to ssl-cert group
- [r15512] deb: More reshuffling of files
- [r15513] httpd: Move static content to share/httpd/static/
- [r15514] deb: Add logrotate configuration
- [r15515] deb: Add init script
- [r15516] packaging: move template files out of etc
- [r15517] gplazma: Replace fqan wildcard matching with longest prefix match
- [r15518] gplazma: introduce reasonable defaults for argus plugin
- [r15523] packaging: Make DEB specific FHS packaging generic
- [r15524] rpm: FHS compliant packaging
- [r15525] admin: use dcache.paths.ssh-keys property rather than keyBase property
- [r15528] cells: fix updating CNS if a domain is stops running a well-known cell
- [r15533] files: migrate admin interface authz files to new location
- [r15534] gplazma: Fix parsing of user names
- [r15535] scripts: fix location of node_config in test for non-migrated dCache
- [r15542] utils: don't use sun private class
- [r15543] webdav: billing logs onDelete
dCache 1.9.12-1
Service: pool
The /opt/d-cache/bin/dcache pool commands have been updated to provide an ls subcommand to list configured pools. The create subcommand has been extended to add the new pool directly to the layout file. The command no longer creates the pool setup file and instead adds the pool specific configuration to the layout file. The command thus provides the functionallity of both the create and add commands found in some earlier releases.
The migration module was extended to reduce interference between multiple migration jobs running on the same pool. The migration module now respects the pool mode on the target pool.
Service: srm
The SRM used to have the ability to periodically submit vacuum commands to the database. This command is PostgreSQL specific and thus tied the SRM to PostgreSQL. PostgreSQL has for quite a while supported auto vacuuming, and manually executing the vacuum command is no longer necessary. We have therfore removed the submission of vacuum commands from dCache. The configuration parameters srmVacuum and srmVacuumPeriod are obsolete.
Some SRM operations may be processed synchronously or asynchronously, at the server's discretion. Since 1.9.10 dCache can start to process such requests synchronously and, if this is taking too long, reply asynchronously and continue to work on the operation in the background. In dCache 1.9.12 this behaviour is enabled by default for get, put, bring online, and list operations. The default switch over time is 1 second. The old behaviour can be restored by setting the srm*SwitchToAsynchronousModeDelay parameters.
The RemoteHttpTransferManager cell has been removed. The functionality has been merged into the RemoteGsiftpTransferManager cell, which has in turn been renamed to RemoteTransferManager. The SRM was extended to support third party srmCopy transfers from HTTP resources. Third party upload to HTTP resources and support for HTTPS is not available yet in the SRM.
Service: dcap, gsidcap, kerberosdcap
DCAP supports prestaging from tape. The DCAP door used to rely on a stager service to perform the staging. In dCache 1.9.12 the DCAP door instead uses the PinManager to trigger staging. The dummy-prestager service is obsolete and can be deleted from the layout.
Service: pnfsmanager
The PNFS manager has been restructured. One consquence is that the output of the info command in the cell admin interface has changed slightly: It is now grouped into sections in line with how it is done in pool manager and pools. Third party scripts may need to be adjusted.
Service: webdav
WebDAV doors now submit remove entries to billing when files are deleted.
The HTML rendering of a directory listing now includes a column containing the file size. The HTML is customizable through a template file. As a consequence the properties webdav.images.logo, webdav.images.directory, webdav.images.file, webdav.style.css, webdavLogoPath, webdavDirIconPath, webdavFileIconPath, webdavCssPath, dirIconPath, fileIconPath, and cssPath are obsolete. Please read the document embedded in the default template file /opt/d-cache/share/webdav/templates/list.stg for details on customizing the look and feel.
The property webdavAnonymousListing was introduced to prevent unauthorized users from listing directories.
The property webdavProtocol was extended with a third option, https-jglobus, to enable HTTPS support based on the JGlobus library. The advantages compared to the https option are that host and CA certificates are read and refreshed directly from /etc/grid-security/ and that proxy certificates (including VOMS certificates) can be used for client authentication.
Info Provider
The info provider is an XSLT script that transforms the output of the info service into GLUE compliant LDIF. The XSLT script used to be stored in /opt/d-cache/etc/glue-1.3.xml and used to contain both the transformation logic and local configuration parameters.
In dCache 1.9.12 the info provider has been extended to support GLUE 2.0 in addition to GLUE 1.3, and it has been restructured to separate configuration parameters from the transformation logic.
The old info provider file, /opt/d-cache/etc/glue-1.3.xml, is obsolete. All site specific configuration is done within the file /opt/d-cache/etc/info-provider.xml. The configuration parameters httpHost, xsltProcessor, xylophoneConfigurationDir, xylophoneConfigurationFile, xylophoneXSLTDir, and saxonDir are deprecated and have been replaced by other parameters. The configuration parameter httpPort is obsolete. The parameter info-provider.publish specifies whether to publish GLUE 1.3, 2.0, or both. See /opt/d-cache/share/defaults/info-provider.properties for details.
Service: pinmanager
The pin manager is responsible for staging and pinning files on behalf of SRM and DCAP doors. In this release the pin manager has been reimplemented from scratch. The design has been significantly simplified to increase performance and robustness of this critical component.
The database schema is separate from the old implementation's schema. The new schema is created the first time the pin manager is started. Existing pins are automatically imported if the same database is used. We strongly recommend using the same database. The tables of the old pin manager are not deleted, however downgrading is not possible without loosing the pins that were created with the new pin manager.
Notice that the migration from the old pin manager requires at least PostgreSQL 8.3. The migration will fail when using older versions.
The command line interface of the pin manager has changed slightly and third party scripts may have to be updated.
Service: poolmanager
Pool manager has seen a fair amount of cleaning in this release, however user visible changes should be small. The most significant change is that pool manager no longer verifies the correctness of location information stored in the name space, that is, on which pools a file is stored. Instead pool manager expects doors to verify the information and deal with any stale locations. This change eliminates a round trip to the pool and thus reduces the latency of opening files.
Pool mode changes are now logged at warning level by pool manager.
Doors and DNS lookups
Most doors and movers have been updated to avoid DNS lookups and reverse lookups that slow down transfers. Instead IP addresses are passed around. This may break existing deployments on multi-homed hosts. When this happens doors may need to be bound explicitly to particular network interfaces.
DCAP, FTP, SSH and admin doors now support the listen property to bind the door to a particular IP address.
Service: gplazma
The get mapping cell command was added. The command was present in earlier releases, but was dropped when the gPlazma cell was reimplemented in 1.9.9.
Database Support
The /opt/d-cache/bin/dcache script has been updated with several new commands related to databases used by various services.
The subcommand database ls lists the databases used by services on this host and is supported by all services that use an external database.
Other database subcommands only apply to services using the LiquiBase schema management library. Currently pin manager is the only service that uses this library. Please consult the man page for details about these commands.
Scripts
Besides changes to the pool and database commands, /opt/d-cache/bin/dcache now supports the services command. The command is similar to the services command found in dCache 1.9.5: It lists all configured services and the domains they belong to. The status command was extended to provide additional information. The new check-config command provides diagnostic information about obsolete and deprecated properties used.
The migration script was moved and renamed to /opt/d-cache/libexec/migrate-from-1.9.5.sh. It was updated to follow best practice by using the host name as the name of the layout. The new version does its best to remove lines copied verbatime from the original template into dCacheSetup. Many deprecated properties are automatically renamed to the new names. At the end of the migration the script invokes the check-config command to report about any deprecated or obsolete properties being used.
The /opt/d-cache/install/install.sh script is no longer needed and has been removed. The remaining two scripts in /opt/d-cache/install/ have been moved to /opt/d-cache/libexec/.
Configuration Properties
Lots of minor issues have been fixed in the configuration system. Many properties that were already deprecated or obsoleted in earlier releases are now marked as such and generate errors or warnings. Other properties that were removed during the transition to the new configiration system in 1.9.7 have been reintroduced as deprecated properties. Default declarations of most services have been moved from batch files and into properties files in /opt/d-cache/share/defaults/.
Changelog 1.9.11-1 to 1.9.12-1
Crossed out entries have been merged into the 1.9.11 branch.
- [r14944] web-admin: fix NPE on shutdown
- [r14945] properties: factor out cleaner default properties
- [r14970] nfsv41: door have to respect ioQueue name specified in layout file
- [r14971] login: Add 'get mapping' command (regression)
- [r14975] bootloader: Fix compatibility with IBM JDK
- [r14976] boot: Refactored configuration file parsing
- [r14977] boot: Include service specific parameters in getProperty shell function
- [r14978] scripts: Add 'pool ls' and 'services' commands
- [r14979] Fixed tests to match changes in previous patch
- [r14989] source: remove unused SysTime.java with corresponding JNI part
- [r14990] gPlazma2 plugin for the ARGUS Authorization System
- [r14991] scripts: add user process is running as in "dcache status" output
- [r14992] scripts: fix column layout in dcache command
- [r14993] scripts: show configured user if domain isn't running
- [r14995] boot: refactor shell oracle generation
- [r14996] poolManager: remove dead code branch
- [r14998] cleaner: Add required parameters
- [r15000] cells: Protect cell startup from option parsing failures
- [r15001] boot: add ability to print configuration as a list of XML entities
- [r15007] pool manager: user NavigableMap instead of SortedMap
- [r15008] PoolManager: Log at a higher log-level when pool changes state
- [r15009] chimera: add migrated chimera configuration to global context
- [r15010] PinManager: Fix failure handling in 1.9.11 and trunk
- [r15014] poolManager: change Pool selection unit interface
- [r15015] gPlazma: changing the order of plugins while mapping
- [r15017] unmark space as being used regardless of
- [r15019] utils: Complete generic type in ReflectionUtils
- [r15020] libs: Fix log4j reference
- [r15022] script: Fail if boot loader fails
- [r15023] cells: Make MessageReply generic and implement Future
- [r15024] PinManager: Replace messages with those from the new PinManager
- [r15025] scripts: Add database management commands
- [r15029] batch: Make services using Chimera show up in 'database ls' output
- [r15032] nfsv41: door have to respect ioQueue name specified in layout file
- [r15033] White-space only fix
- [r15034] cells: don't fail if password file isn't present
- [r15035] White-space only fix
- [r15036] exceptions: Drop DirNotExistsCacheException
- [r15038] srm: fix bug in altering 3rd-party copy requests (srmAbortFile, srmExtendFileLifeTime, srmReleaseFiles)
- [r15039] Solaris: Fixed Solaris compatibility issue with an awk script
- [r15041] info: fix thread unsafe usage of SimpleDateFormat
- [r15042] info: fix removeStateWatcher method
- [r15043] core: use InetSocketAddress in protocolInfo
- [r15044] gPlazma: Fixing authorization without any role
- [r15046] properties: add some missing unit-tests
- [r15047] poolmanager: remove dead code
- [r15048] FTP: Avoid reverse lookups
- [r15049] PinManager: Fix stageConfigurationFilePath check
- [r15051] space-manager: remove deadcode
- [r15052] configuration: add comment about how dcache.home is specified
- [r15053] pool: set request state QUEUED prior sending to execution
- [r15055] pool: fix compatibility with FTP doors
- [r15057] findbugs: suppress infinite-recursion-loop warning in parboil parser rules
- [r15058] build: don't package billing plotting config
- [r15059] build: fix RPM packaging warning message
- [r15070] Stripped white space
- [r15071] cells: add command for checking whether a file exists or a cell is running
- [r15072] batch: don't generate meaningless error messages
- [r15073] nfsv41: avoid unnecessary byte array copy
- [r15075] New PinManager
- [r15076] PoolManager: Replace state constants with enum
- [r15077] logging: Fix CDC setup in cells
- [r15078] PnfsManager: Use Spring for instantiation and wiring
- [r15079] FTP: Make performance markers work with recent pools
- [r15080] pool: Fix NPE in xrootd mover
- [r15081] PoolManager: Remove file check
- [r15082] dcap: Use FileAttributes
- [r15083] pool: Let migration module respect pool disable mode
- [r15084] classpath: autodetect liquibase-core jar file
- [r15085] batch: add support for gPlazma2
- [r15086] Whitespace-only fix
- [r15087] srm: remove periodic invocation of "VACUUM ANALYZE" on DB backend
- [r15088] cells: make CellEvent immutable and thread-safe
- [r15089] pool: tidy up PoolV4
- [r15090] cells: fix synchronisation when shutting down private executors
- [r15091] Security: protect against possible NPE
- [r15092] removed empty files
- [r15093] httpd: remove reference to deleted class
- [r15094] PoolManager: Let PoolMgrGetPoolMsg rely on FileAttributes
- [r15095] PoolManager: Let RequestContainer use PnfsHandler
- [r15096] xrootd: Downgrade protocol version to avoid kXR_locate
- [r15097] services: Clean up batch files of replica, spacemanager and billing services
- [r15098] cells: Avoid reply loop
- [r15099] PinManager: Fix moving of stale pins
- [r15100] pool: Fix termination in case of failed transfers
- [r15114] gPlazma2: fix warning message from white-space
- [r15116] info: move almost all startup into the init method
- [r15118] info: log NoRouteToCell at info level
- [r15120] cells: make CellMessage serialised payload final
- [r15123] libs: Upgrade to DataNucleus 2.2.1
- [r15125] build: remove batik from commons dependencies
- [r15127] space-manager: don't try to create tables if they already exist
- [r15128] configuration: add possibility to mark properties not-for-services
- [r15129] PnfsManager: fix rare index-out-of-bound error when processing messages without PnfsId
- [r15130] chimera: add missing CLI
- [r15131] White-space only fix
- [r15132] auth: create usnix-like subject based on uid and gid
- [r15133] srm : correct typos in names of a private class and logging
- [r15134] cells: fix typo in SshKeyManager shutdown
- [r15135] http: don't flooding log file on shutdown
- [r15136] httpd: fix shutdown
- [r15137] cells: add diagnostics for cells reluctant to shutdown
- [r15138] chimera-nfsv3: fix shutdown
- [r15139] pool: don't warn about pool repository path
- [r15140] info: log NoRouteToCell exceptions at info level
- [r15141] nfsv4: shutdown dataserver in pools
- [r15144] Whitespace-only fix
- [r15145] Whitespace-only fix
- [r15146] ssh: silent error message when clients connect
- [r15147] srm: add orderly shutdown methods
- [r15148] srm: make DbJobStorage less noisy on shutdown
- [r15149] build: add possibility to steer location of maven's local repository
- [r15151] configuration: tweak error messages
- [r15152] info: don't make http engine a cell
- [r15153] pool-manager: tidy up request container
- [r15157] pool: Add a lock for files processed by migration module
- [r15158] ftp: Fix NPE on login failure
- [r15159] xrootd: Add logging hooks in xrootd door
- [r15160] xrootd: Add kXR_prepare command
- [r15161] xrootd: Implement toString in a few requests and response messages
- [r15162] PoolManager: Fix compatibility with pools older than 1.9.10
- [r15163] pool: Make FairQueueAllocation interrupt safe
- [r15164] logging: Direct AspectjJ logging towards SLF4J
- [r15179] source: remove unused ACL classes
- [r15180] pool-manager: remove PoolOperator
- [r15181] ssh: fix NPE if subject is unknown
- [r15182] admin: fail with detail error msg if admin doesn't have keys
- [r15183] install: remove install.sh script
- [r15184] ssh: don't trigger IOException when user disconnects
- [r15185] pool-manager: tidy up PoolManagerv5
- [r15186] httpd: low-level tidy up of pool-manager HTML generation
- [r15187] pool-manager: add missing annotation
- [r15188] info provider: Downgrade xrootd protocol version
- [r15189] configuration: take the value of an obsolete property as error message
- [r15190] pool: partial revert of r.15139
- [r15191] pool-manager: add missing annotations
- [r15192] httpd: fix problem if http has different cellname
- [r15193] gPlazma: remove commented-out code
- [r15194] commons: fix gauge so it doesn't throw an exception if there's a
- [r15203] TransferObserver: log interruption at info level
- [r15204] build: add smc target
- [r15205] dcap: Fix space manager integration and add configurable cell names
- [r15206] dcap: Use pin manager rather than the stager
- [r15207] spacemanager: Remove SelectPoolCompanion (dead code)
- [r15208] transfermanagers: Clean up CopyManager
- [r15209] doors: Minor cleaning of Transfer class
- [r15210] ftp: Mark pool proxy property as obsolete
- [r15211] transfermanagers: Clean up TransferManager class
- [r15212] http: don't warn when shutting down restore monitoring thread
- [r15213] poolManager: fix leftovers from PoolOperator removal
- [r15214] pool: propagate error codes in post-execution step
- [r15216] properties: better warning message for deprecated properties
- [r15217] webdav: fail with meaningful error if KeyStore or TrustStore is missing
- [r15219] PoolManager: Avoid inheriting from AbstractCellComponent in some classes
- [r15220] PoolManager: Make several classes serializable
- [r15222] PoolManager: Clean up interface of PoolMonitorV5
- [r15223] config: Added various deprecated, obsolete and forbidden properties
- [r15224] The info-provider depends on an external package, xylophone
- [r15225] poolmanager: Fix message queue overflow handling in RequestContainerV5
- [r15226] webdav: Add remove notification
- [r15227] info-provider: publish GLUE2 information
- [r15228] gPlazma: fix extracting list of FQANs
- [r15229] poolmanager: fix percentile cost calculation
- [r15230] PoolManager: Reverted r15225
- [r15231] pool-manager: fix NPE on PoolUp message
- [r15232] webdav: Add size column to listing and use template engine for rendering
- [r15233] webdav: Move template for directory listing to share directory
- [r15234] poolmanager: Fix message queue overflow handling in RequestContainerV5
- [r15235] SRM: Move configuration defaults to properties file
- [r15236] doors: Improve Transfer.selectPoolAndStartMover and use it in doors
- [r15237] doors: Fix IllegalStateException in Transfer class
- [r15238] configuration: activate support for not-for-services annotation
- [r15239] gPlazma2: added VORoleMap Plugin
- [r15240] info-provider: add GLUE2 StorageService.Name and StorageManager.Name
- [r15241] gPlazma2: Refactoring of Argus Plugin
- [r15242] gPlazma2: Plugin Configuration
- [r15243] CopyManager: Update CopyManager to use the Transfer facade
- [r15244] webdav: Refactor redirect and finish message processing
- [r15245] webdav: Use Transfer.selectPoolAndStartMover
- [r15246] cells: Fix missing timeout on message queue overflow
- [r15247] libs: Upgrade to patched version of DataNucleus RDBMS
- [r15250] transfermanagers: Let PnfsManager perform permission checks
- [r15251] transfermanagers: Generalize RemoteGsiftpTransferManager
- [r15252] pinmanager: Refactor bulk pinning
- [r15253] config: add check-config command, fix line numbering
- [r15254] properties: add missing not-for-services and deprecated annotations
- [r15255] srm: Alter defaults for asynchronous/synchronous switch over times
- [r15256] webdav: Add flag to disallow anonymous listing
- [r15257] pinmanager: Activate new pinmanager, delete old pinmanager
- [r15258] PoolManager: Clean up various fields in RequestContainer
- [r15259] libs: Upgrade to Netty 3.2.4
- [r15260] libs: Upgrade to logback 0.9.28
- [r15261] An earlier patch deleted the persistence JAR from dCache (because the included JPA2 supersedes it)
- [r15262] all: Refactor MessageCallback
- [r15263] config: fix check-config for single files
- [r15264] gPlazma2: Configuration passing for VORoleMap and ARGUS plugins
- [r15265] config: remove unused method
- [r15266] configuration: fix discovery of synonyms
- [r15267] migration: run "dcache check-config" after migration
- [r15268] info-provider: refactor
- [r15269] configuration: tidy up output from check-config
- [r15270] migration script: Generate `hostname`.conf rather than imported.conf
- [r15271] gPlazma2: Fix parameter handling by VORoleMap Mapping Plugin
- [r15272] Whitespaces and Warnings
- [r15273] libs: Upgrade to DataNucleus 2.2.2
- [r15274] libs: Upgrade to LiquiBase 2.0.1
- [r15275] pinmanager: Refactor LiquiBase Spring integration
- [r15276] webdav: Add https-jglobus protocol option
- [r15277] scripts: Move scripts from install/ to libexec/
- [r15278] documentation: update dcache man page with check-config
- [r15279] scripts: Move scripts from install/ to libexec/
- [r15280] services: Move defaults from batch files to default files
- [r15281] migration: filter out dCacheSetup.template from dCacheSetup
- [r15282] 1.9.12 branch
- [r15291] PinManager: Fix Liquibase 1.9.5 -> 2.0.1 migration
- [r15292] pool: Remove singleton thread pools
- [r15294] info-provider: Mark info-provider.xml as a configuration file on Debian
- [r15295] PoolManager: Fix compatibility with migration module
- [r15300] pool: save and restore slf4j context for a request
- [r15303] build: add missing maven repositories for srmclient build
- [r15310] migration: fix use of tail
- [r15311] migration: handle RPM removing or renaming chimera-config.xml
- [r15312] pool: fix way too short timeout for http transfer
- [r15317] utils: preserve principals order by extracting fqans from a Subject
- [r15324] httpd: Introduce cleaned up version of HttpServiceCell
- [r15330] httpd: fix html generated by PoolManager
- [r15337] junit: fix incorrect test
- [r15338] jms: Rename tunnel options to explicitly refer to amq
- [r15342] jms: Rename tunnel options to explicitly refer to amq
- [r15345] pinmanager: Make pin manager more resilient against DB failure
- [r15346] doors: Use a session ID compatible with InfoMessage.getTransaction
- [r15354] httpd: Set default mime type to text/html
- [r15360] httpd: rectrict http reqquests paths
- [r15366] build: use absolute when building a solaris package
- [r15376] srm: do not print a stacktrace if authentication failed
- [r15377] doors: add listen option
- [r15378] jms: Add OpenMQ support
- [r15379] migration-from-1.9.5.sh: Convert legacy options
- [r15383] jms: Clean up the code before replacing cell name lookup
- [r15384] cells: Introduce central cell name service for JMS brokers