release notes | Book: 1.9.5, 1.9.12 (opt, FHS), 2.11 (FHS), 2.12 (FHS), 2.13 (FHS), 2.14 (FHS), | Wiki | Q&A black_bg
Web: Multi-page, Single page | PDF: A4-size, Letter-size | eBook: epub black_bg

SpaceManager configuration for Explicit Space Reservations

[return to top]

SRM SpaceManager and Link Groups

SpaceManager is making reservations against free space available in link groups. The total free space in the given link group is the sum of available spaces in all links. The available space in each link is the sum of all sizes of available space in all pools assinged to a given link. Therefore for the space reservation to work correctly it is essential that each pool belongs to one and only one link, and each link belongs to only one link group. Link groups are assigned several parameters that determine what kind of space the link group corresponds to and who can make reservations against this space.

[return to top]

Making a Space Reservation

Now that the SRM SpaceManager is activated you can make a space reservation. As mentioned above you need link groups to make a space reservation.

[return to top]

Prerequisites for Space Reservations

Login to the admin interface and cd to the cell SrmSpaceManager.

[user] $ ssh -c blowfish -p 22223 -l admin headnode.example.org
(local) admin > cd SrmSpaceManager

Type ls to get information about reservations and link groups.

(SrmSpaceManager) admin > ls
Reservations:
total number of reservations: 0
total number of bytes reserved: 0

LinkGroups:
total number of linkGroups: 0
total number of bytes reservable: 0
total number of bytes reserved  : 0
last time all link groups were updated: Tue Sep 20 11:15:19 CEST 2011(1316510119634)

This output tells you that there are no reservations yet and no link groups. As there are no link groups no space can be reserved.

[return to top]

The Link Groups

For a general introduction about link groups see the section called “Link Groups”.

Example:

In this example we will create a link group for the VO desy. In order to do so we need to have a pool, a pool group and a link. Moreover, we define unit groups named any-store, world-net and any-protocol. (See the section called “Types of Units”.)

Define a pool in your layout file, add it to your pool directory and restart the poolDomain.

[poolDomain]
[poolDomain/pool]
path=/srv/dcache/spacemanager-pool
name=spacemanager-pool
[root] # mkdir -p /srv/dcache/spacemanager-pool
[root] # /opt/d-cache/bin/dcache restart

In the Admin Interface cd to the PoolManager and create a pool group, a link and a link group.

(SrmSpaceManager) admin > ..
(local) admin > cd PoolManager
(PoolManager) admin > psu create pgroup spacemanager_poolGroup
(PoolManager) admin > psu addto pgroup spacemanager_poolGroup spacemanager-pool
(PoolManager) admin > psu removefrom pgroup default spacemanager-pool
(PoolManager) admin > psu create link spacemanager_WriteLink any-store world-net any-protocol
(PoolManager) admin > psu set link spacemanager_WriteLink -readpref=10 -writepref=10 -cachepref=0 -p2ppref=-1
(PoolManager) admin > psu add link spacemanager_WriteLink  spacemanager_poolGroup
(PoolManager) admin > psu create linkGroup spacemanager_WriteLinkGroup
(PoolManager) admin > psu set linkGroup custodialAllowed spacemanager_WriteLinkGroup true
(PoolManager) admin > psu set linkGroup replicaAllowed spacemanager_WriteLinkGroup true
(PoolManager) admin > psu set linkGroup nearlineAllowed spacemanager_WriteLinkGroup true
(PoolManager) admin > psu set linkGroup onlineAllowed spacemanager_WriteLinkGroup true
(PoolManager) admin > psu addto linkGroup spacemanager_WriteLinkGroup spacemanager_WriteLink
(PoolManager) admin > save
(PoolManager) admin > ..
	

Check whether the link group is available.

(local) admin > cd SrmSpaceManager
(SrmSpaceManager) admin > ls
Reservations:
total number of reservations: 0
total number of bytes reserved: 0

LinkGroups:
0 Name:spacemanager_WriteLinkGroup FreeSpace:7278624768 ReservedSpace:0 AvailableSpace:7278624768 VOs: onlineAllowed:true nearlineAllowed:false replicaAllowed:true custodialAllowed:true UpdateTime:Mon Nov 28 12:12:51 CET 2011(1322478771030)
total number of linkGroups: 1
total number of bytes reservable: 7278624768
total number of bytes reserved  : 0
last time all link groups were updated: Mon Nov 28 12:12:51 CET 2011(1322478771030)

The link group spacemanager_WriteLinkGroup was created and has the id 0.

[return to top]

The SpaceManagerLinkGroupAuthorizationFile

Now you need to edit the LinkGroupAuthorization.conf file. This file contains a list of the link groups and all the VOs and the VO Roles that are permitted to make reservations in a given link group.

Specify the location of the LinkGroupAuthorization.conf file in the /opt/d-cache/etc/dcache.conf file.

SpaceManagerLinkGroupAuthorizationFileName=/path/to/LinkGroupAuthorization.conf

The file LinkGroupAuthorization.conf has following syntax:

LinkGroup <NameOfLinkGroup> followed by the list of the Fully Qualified Attribute Names (FQANs). Each FQAN on a separate line, followed by an empty line, which is used as a record separator, or by the end of the file.

FQAN is usually a string of the form <VO>/Role=<VORole>. Both <VO> and <VORole> could be set to *, in this case all VOs or VO Roles will be allowed to make reservations in this link group. Any line that starts with # is a comment and may appear anywhere.

#SpaceManagerLinkGroupAuthorizationFile

LinkGroup <NameOfLinkGroup>
/<VO>/Role=<VORole>

Note

You do not need to restart the srm or dCache after changing the LinkGroupAuthorization.conf file. The changes will be applied automatically after a few minutes.

Use update link groups to be sure that the LinkGroupAuthorization.conf file and the link groups have been updated.

(SrmSpaceManager) admin > update link groups
update started

Example:

In the example above you created the link group spacemanager_WriteLinkGroup. Now you want to allow members of the VO desy with the role production to make a space reservation in this link group.

#SpaceManagerLinkGroupAuthorizationFile
# this is comment and is ignored

LinkGroup spacemanager_WriteLinkGroup
#
/desy/Role=production

Example:

In this more general example for a SpaceManagerLinkGroupAuthorizationFile members of the VO desy with role test get the right to make a space reservation in a link group called desy-test-LinkGroup. Moreover, all members of the VO desy get the right to make a reservation in the link group called desy-anyone-LinkGroup and anyone will get the right to make a space reservation in the link group called default-LinkGroup.

#SpaceManagerLinkGroupAuthorizationFile
# this is comment and is ignored

LinkGroup desy-test-LinkGroup
/desy/Role=/test

LinkGroup desy-anyone-LinkGroup
/desy/Role=*

LinkGroup default-LinkGroup
# allow anyone :-)
*/Role=*

[return to top]

Making and Releasing a Space Reservation as dCache Administrator

[return to top]

Making a Space Reservation

Example:

Now you can make a space reservation for the VO desy.

(SrmSpaceManager) admin > reserve -vog=/desy -vor=production -desc=DESY_TEST 5000000 10000
110000 voGroup:/desy voRole:production retentionPolicy:CUSTODIAL accessLatency:NEARLINE linkGroupId:0 size:5000000 created:Fri Dec 09 12:43:48 CET 2011 lifetime:10000000ms expiration:Fri Dec 09 15:30:28 CET 2011 description:DESY_TEST state:RESERVED used:0 allocated:0

The id of the space token is 110000.

Check the status of the reservation by

(SrmSpaceManager) admin > ls
Reservations:
110000 voGroup:/desy voRole:production retentionPolicy:CUSTODIAL accessLatency:NEARLINE linkGroupId:0 size:5000000 created:Fri Dec 09 12:43:48 CET 2011 lifetime:10000000ms expiration:Fri Dec 09 15:30:28 CET 2011 description:DESY_TEST state:RESERVED used:0 allocated:0
total number of reservations: 1
total number of bytes reserved: 5000000

LinkGroups:
0 Name:spacemanager_WriteLinkGroup FreeSpace:23747563520 ReservedSpace:5000000 AvailableSpace:23742563520 VOs:{/desy:*} onlineAllowed:true nearlineAllowed:true replicaAllowed:true custodialAllowed:true UpdateTime:Fri Dec 09 12:49:29 CET 2011(1323431369046)
total number of linkGroups: 1
total number of bytes reservable: 23742563520
total number of bytes reserved  : 5000000
last time all link groups were updated: Fri Dec 09 12:49:29 CET 2011(1323431369046)

You can now copy a file into that space token.

[user] $ srmcp file:////bin/sh srm://<dcache.example.org>:8443/data/world-writable/space-token-test-file -space_token=110000

Now you can check via the Webadmin Interface or the Web Interface that the file has been copied to the pool spacemanager-pool.

There are several parameters to be specified for a space reservation.

(SrmSpaceManager) admin > reserve [-vog=voGroup] [-vor=voRole] [-acclat=AccessLatency] \
[-retpol=RetentionPolicy] [-desc=Description] [-lgid=LinkGroupId] [-lg=LinkGroupName] \
<sizeInBytes> <lifetimeInSecs>
[-vog=voGroup]

voGroup should match the VO you specified in the LinkGroupAuthorization.conf file. If you do not want to make a space reservation for a certain VO then the entry in the LinkGroupAuthorization.conf should read

LinkGroup NameOfLinkGroup
*/Role=*
[-vor=voRole]

voRole can be specified if it is used in the LinkGroupAuthorization.conf file.

[-acclat=AccessLatency]

AccessLatency needs to match one of the access latencies allowed for the link group.

[-retpol=RetentionPolicy]

RetentionPolicy needs to match one of the retention policies allowed for the link group.

[-desc=Description]

You can chose a value to describe your space reservation.

[-lgid=LinkGroupId]

You can either use the LinkGroupId to make a space reservation or

[-lg=LinkGroupName]

you use the LinkGroupName to make a space reservation.

<sizeInBytes>

The size of the space reservation should be specified in bytes.

<lifetimeInSecs>

The life time of the space reservation should be specified in seconds. Choose "-1" for a space reservation that will never expire (use quotes around the negative one).

[return to top]

Releasing a Space Reservation

If a space reservation is not needet anymore it can be released with

(SrmSpaceManager) admin > release <spaceTokenId>

Example:

(SrmSpaceManager) admin > reserve -vog=/desy -vor=production -desc=DESY_TEST 5000000 600
110042 voGroup:/desy voRole:production retentionPolicy:CUSTODIAL accessLatency:NEARLINE linkGroupId:0 size:5000000 created:Thu Dec 15 12:00:35 CET 2011 lifetime:600000ms expiration:Thu Dec 15 12:10:35 CET 2011 description:DESY_TEST state:RESERVED used:0 allocated:0
(SrmSpaceManager) admin > release 110042
110042 voGroup:/desy voRole:production retentionPolicy:CUSTODIAL accessLatency:NEARLINE linkGroupId:0 size:5000000 created:Thu Dec 15 12:00:35 CET 2011 lifetime:600000ms expiration:Thu Dec 15 12:10:35 CET 2011 description:DESY_TEST state:RELEASED used:0 allocated:0

You can see that the value for state has changed from RESERVED to RELEASED

[return to top]

Making and Releasing a Space Reservation as a User

A user who has been given the right to make a space reservation can make a space reservation. To achieve this the right entry in the LinkGroupAuthorization.conf file is required.

[return to top]

VO based Authorization Prerequisites

In order to be able to take advantage of the virtual organization (VO) infrastructure and VO based authorization and VO based access control to the space in dCache, certain things need to be in place:

Only if these 3 conditions are satisfied the VO based authorization of the SpaceManager will work.

[return to top]

VO based Access Control Configuration

As mentioned above dCache space reservation functionality access control is currently performed at the level of the link groups. Access to making reservations in each link group is controlled by the SpaceManagerLinkGroupAuthorizationFile.

This file contains a list of the link groups and all the VOs and the VO Roles that are permitted to make reservations in a given link group.

When a SRM Space Reservation request is executed, its parameters, such as reservation size, lifetime, AccessLatencyand RetentionPolicy as well as user’s VO membership information is forwarded to the SRM SpaceManager.

Once a space reservation is created, no access control is performed, any user can store the files in this space reservation, provided he or she knows the exact space token.

[return to top]

Making and Releasing a Space Reservation

A user who is given the rights in the SpaceManagerLinkGroupAuthorizationFile can make a space reservation by

[user] $ srm-reserve-space -retention_policy=<RetentionPolicy> -lifetime=<lifetimeInSecs> -desired_size=<sizeInBytes> -guaranteed_size=<sizeInBytes>  srm://<example.org>:8443
Space token =SpaceTokenId

and release it by

[user] $ srm-release-space srm://<example.org>:8443 -space_token=SpaceTokenId

Note

Please note that it is obligatory to specify the retention policy while it is optional to specify the access latency.

Example:

[user] $ srm-reserve-space -retention_policy=REPLICA -lifetime=300 -desired_size=5500000 -guaranteed_size=5500000  srm://srm.example.org:8443
Space token =110044

The space reservation can be released by:

[user] $ srm-release-space srm://srm.example.org:8443 -space_token=110044

[return to top]

Space Reservation without VOMS certificate

If a client uses a regular grid proxy, created with grid-proxy-init, and not a VO proxy, which is created with the voms-proxy-init, when it is communicating with SRM server in dCache, then the VO attributes can not be extracted from its credential. In this case the name of the user is extracted from the Distinguished Name (DN) to use name mapping. For the purposes of the space reservation the name of the user as mapped by gplazma is used as its VO Group name, and the VO Role is left empty. The entry in the SpaceManagerLinkGroupAuthorizationFile should be:

#LinkGroupAuthorizationFile
#
<userName>

[return to top]

Space Reservation for non SRM Transfers

Edit the file /opt/d-cache/etc/dcache.conf to enable space reservation for non SRM transfers.

SpaceManagerReserveSpaceForNonSRMTransfers=true

If the spacemanager is enabled, SpaceManagerReserveSpaceForNonSRMTransfers is set to true, and if the transfer request comes from a door, and there was no prior space reservation made for this file, the SpaceManager will try to reserve space before satisfying the request.

Possible values are true or false and the default value is false.

[return to top]

SRM configuration for experts

There are a few parameters in /opt/d-cache/share/defaults/*.properties that you might find useful for nontrivial SRM deployment.

[return to top]

srmSpaceManagerEnabled

srmSpaceManagerEnabled tells if the space management is activated in SRM.

Possible values are yes and no. Default is yes.

Usage example:

srmSpaceManagerEnabled=yes

[return to top]

srmImplicitSpaceManagerEnabled

srmImplicitSpaceManagerEnabled tells if the space should be reserved for SRM Version 1 transfers and for SRM Version 2 transfers that have no space token specified. Will have effect only if srmSpaceManagerEnabled.

Possible values are yes and no. This is enabled by default. It has no effect if srmSpaceManagerEnabled is set to no.

Usage example:

srmImplicitSpaceManagerEnabled=yes

[return to top]

overwriteEnabled

overwriteEnabled tells SRM and GridFTP servers if the overwrite is allowed. If enabled on the SRM node, should be enabled on all GridFTP nodes.

Possible values are yes and no. Default is no.

Usage example:

overwriteEnabled=yes

[return to top]

srmOverwriteByDefault

srmOverwriteByDefault Set this to true if you want overwrite to be enabled for SRM v1.1 interface as well as for SRM v2.2 interface when client does not specify desired overwrite mode. This option will be considered only if overwriteEnabled is set to yes.

Possible values are true and false. Default is false.

Usage example:

srmOverwriteByDefault=false 

[return to top]

srmDatabaseHost

srmDatabaseHost tells SRM which database host to connect to.

Default value is localhost.

Usage example:

srmDatabaseHost=database-host.example.org

[return to top]

spaceManagerDatabaseHost

spaceManagerDatabaseHost tells SpaceManager which database host to connect to.

Default value is localhost.

Usage example:

spaceManagerDatabaseHost=database-host.example.org

[return to top]

pinManagerDbHost

pinManagerDbHost tells PinManager which database host to connect to.

Default value is localhost.

Usage example:

pinManagerDbHost=database-host.example.org

[return to top]

srmDbName

srmDbName tells SRM which database to connect to.

Default value is dcache.

Usage example:

srmDbName=dcache

[return to top]

srmDbUser

srmDbUser tells SRM which database user name to use when connecting to database. Do not change unless you know what you are doing.

Default value is srmdcache.

Usage example:

srmDbUser=srmdcache

[return to top]

srmDbPassword

srmDbPassword tells SRM which database password to use when connecting to database. The default value is srmdcache.

Usage example:

srmDbPassword=NotVerySecret

[return to top]

srmPasswordFile

srmPasswordFile tells SRM which database password file to use when connecting to database. Do not change unless you know what you are doing. It is recommended that MD5 authentication method is used. To learn about file format please see http://www.postgresql.org/docs/8.1/static/libpq-pgpass.html. To learn more about authentication methods please visit http://www.postgresql.org/docs/8.1/static/encryption-options.html, Please read "Encrypting Passwords Across A Network" section.

This option is not set by default.

Usage example:

srmPasswordFile=/root/.pgpass

[return to top]

srmRequestHistoryDatabaseEnabled

srmRequestHistoryDatabaseEnabled enables logging of the transition history of the SRM request in the database. The request transitions can be examined through the command line interface. Activation of this option might lead to the increase of the database activity, so if the PostgreSQL load generated by SRM is excessive, disable it.

Possible values are true and false. Default is false.

Usage example:

srmRequestHistoryDatabaseEnabled=true

[return to top]

srmDbLogEnabled

srmDbLogEnabled tells SRM to store the information about the remote (copy, srmCopy) transfer details in the database. Activation of this option might lead to the increase of the database activity, so if the PostgreSQL load generated by SRM is excessive, disable it.

Possible values are true and false. Default is false.

Usage example:

srmDbLogEnabled=false

[return to top]

srmVersion

srmVersion is not used by SRM; it was mentioned that this value is used by some publishing scripts.

Default is version1.

[return to top]

pnfsSrmPath

pnfsSrmPath tells SRM what the root of all SRM paths is in pnfs. SRM will prepend path to all the local SURL paths passed to it by SRM client. So if the pnfsSrmPath is set to /pnfs/fnal.gov/THISISTHEPNFSSRMPATH and someone requests the read of srm://srm.example.org:8443/file1, SRM will translate the SURL path /file1 into /pnfs/fnal.gov/THISISTHEPNFSSRMPATH/file1. Setting this variable to something different from / is equivalent of performing Unix chroot for all SRM operations.

Default value is /.

Usage example:

pnfsSrmPath="/pnfs/fnal.gov/data/experiment"

[return to top]

parallelStreams

parallelStreams specifies the number of the parallel streams that SRM will use when performing third party transfers between this system and remote GSI-FTP servers, in response to SRM v1.1 copy or SRM V2.2 srmCopy function. This will have no effect on srmPrepareToPut and srmPrepareToGet command results and parameters of GridFTP transfers driven by the SRM clients.

Default value is 10.

Usage example:

parallelStreams=20

[return to top]

srmBufferSize

srmBufferSize specifies the number of bytes to use for the in memory buffers for performing third party transfers between this system and remote GSI-FTP servers, in response to SRM v1.1 copy or SRM V2.2 srmCopy function. This will have no effect on srmPrepareToPut and srmPrepareToGet command results and parameters of GridFTP transfers driven by the SRM clients.

Default value is 1048576.

Usage example:

srmBufferSize=1048576

[return to top]

srmTcpBufferSize

srmTcpBufferSize specifies the number of bytes to use for the tcp buffers for performing third party transfers between this system and remote GSI-FTP servers, in response to SRM v1.1 copy or SRM V2.2 srmCopy function. This will have no effect on srmPrepareToPut and srmPrepareToGet command results and parameters of GridFTP transfers driven by the SRM clients.

Default value is 1048576.

Usage example:

srmTcpBufferSize=1048576

[return to top]

srmAuthzCacheLifetime

srmAuthzCacheLifetime specifies the duration that authorizations will be cached. Caching decreases the volume of messages to the gPlazma cell or other authorization mechanism. To turn off caching, set the value to 0.

Default value is 120.

Usage example:

srmAuthzCacheLifetime=60

[return to top]

srmGetLifeTime, srmPutLifeTime and srmCopyLifeTime

srmGetLifeTime, srmPutLifeTime and srmCopyLifeTime specify the lifetimes of the srmPrepareToGet (srmBringOnline) srmPrepareToPut and srmCopy requests lifetimes in millisecond. If the system is unable to fulfill the requests before the request lifetimes expire, the requests are automatically garbage collected.

Default value is 14400000 (4 hours)

Usage example:

srmGetLifeTime=14400000
srmPutLifeTime=14400000
srmCopyLifeTime=14400000

[return to top]

srmGetReqMaxReadyRequests, srmPutReqMaxReadyRequests, srmGetReqReadyQueueSize and srmPutReqReadyQueueSize

srmGetReqMaxReadyRequests and srmPutReqMaxReadyRequests specify the maximum number of the files for which the transfer URLs will be computed and given to the users in response to SRM get (srmPrepareToGet) and put (srmPrepareToPut) requests. The rest of the files that are ready to be transfered are put on the Ready queues, the maximum length of these queues are controlled by srmGetReqReadyQueueSize and srmPutReqReadyQueueSize parameters. These parameters should be set according to the capacity of the system, and are usually greater than the maximum number of the GridFTP transfers that this dCache instance GridFTP doors can sustain.

Usage example:

srmGetReqReadyQueueSize=10000
srmGetReqMaxReadyRequests=2000
srmPutReqReadyQueueSize=10000
srmPutReqMaxReadyRequests=1000

[return to top]

srmCopyReqThreadPoolSize and remoteGsiftpMaxTransfers

srmCopyReqThreadPoolSize and remoteGsiftpMaxTransfers. srmCopyReqThreadPoolSize is used to specify how many parallel srmCopy file copies to execute simultaneously. Once the SRM contacted the remote SRM system, and obtained a Transfer URL (usually GSI-FTP URL), it contacts a Copy Manager module (usually RemoteGSIFTPTransferManager), and asks it to perform a GridFTP transfer between the remote GridFTP server and a dCache pool. The maximum number of simultaneous transfers that RemoteGSIFTPTransferManager will support is remoteGsiftpMaxTransfers, therefore it is important that remoteGsiftpMaxTransfers is greater than or equal to srmCopyReqThreadPoolSize.

Usage example:

srmCopyReqThreadPoolSize=250
remoteGsiftpMaxTransfers=260

[return to top]

srmCustomGetHostByAddr

srmCustomGetHostByAddr srmCustomGetHostByAddr enables using the BNL developed procedure for host by IP resolution if standard InetAddress method failed.

Usage example:

srmCustomGetHostByAddr=true

[return to top]

RecursiveDirectoryCreation

RecursiveDirectoryCreation allows or disallows automatic creation of directories via SRM, allow=true, disallow=false.

Automatic directory creation is allowed by default.

Usage example:

RecursiveDirectoryCreation=true

[return to top]

hostCertificateRefreshPeriod

This option allows you to control how often the SRM door will reload the server’s host certificate from the filesystem. For the specified period, the host certificate will be kept in memory. This speeds up the rate at which the door can handle requests, but also causes it to be unaware of changes to the host certificate (for instance in the case of renewal).

By changing this parameter you can control how long the host certificate is cached by the door and consequently how fast the door will be able to detect and reload a renewed host certificate.

Please note that the value of this parameter has to be specified in seconds.

Usage example:

hostCertificateRefreshPeriod=86400

[return to top]

trustAnchorRefreshPeriod

The trustAnchorRefreshPeriod option is similar to hostCertificateRefreshPeriod. It applies to the set of CA certificates trusted by the SRM door for signing end-entity certificates (along with some metadata, these form so called trust anchors). The trust anchors are needed to make a decision about the trustworthiness of a certificate in X.509 client authentication. The GSI security protocol used by SRM builds upon X.509 client authentication.

By changing this parameter you can control how long the set of trust anchors remains cached by the door. Conversely, it also influences how often the door reloads the set of trusted certificates.

Please note that the value of this parameter has to be specified in seconds.

Tip

Trust-anchors usually change more often than the host certificate. Thus, it might be sensible to set the refresh period of the trust anchors lower than the refresh period of the host certificate.

Usage example:

trustAnchorRefreshPeriod=3600