SpaceManager
is making reservations against free space
available in link
groups. The total free space in the given link
group is the sum of available spaces in all links. The
available space in each link is the sum of all sizes of
available space in all pools assinged to a given
link. Therefore for the space reservation to work correctly it
is essential that each pool belongs to one and only one link,
and each link belongs to only one link group. Link groups are
assigned several parameters that determine what kind of space
the link group corresponds to and who can make reservations
against this space.
Now that the SRM
SpaceManager
is activated you can make
a space reservation. As mentioned above you need link groups to
make a space reservation.
Login to the admin
interface and cd to the cell
SrmSpaceManager
.
[user] $
ssh -c blowfish -p 22223 -l admin headnode.example.org
(local) admin >
cd SrmSpaceManager
Type ls to get information about reservations and link groups.
(SrmSpaceManager) admin >
ls
Reservations: total number of reservations: 0 total number of bytes reserved: 0 LinkGroups: total number of linkGroups: 0 total number of bytes reservable: 0 total number of bytes reserved : 0 last time all link groups were updated: Tue Sep 20 11:15:19 CEST 2011(1316510119634)
This output tells you that there are no reservations yet and no link groups. As there are no link groups no space can be reserved.
For a general introduction about link groups see the section called “Link Groups”.
Example:
In this example we will create a link group for the VO
desy
. In order to do so we need to
have a pool, a pool group and a link. Moreover, we define
unit groups named any-store
,
world-net
and
any-protocol
. (See the section called “Types of Units”.)
Define a pool in your layout file, add it to your pool
directory and restart the poolDomain
.
[poolDomain] [poolDomain/pool] path=/srv/dcache/spacemanager-pool name=spacemanager-pool
[root] #
mkdir -p /srv/dcache/spacemanager-pool
[root] #
/usr/bin/dcache restart
In the Admin Interface cd to the
PoolManager
and create a pool group, a link and a link
group.
(SrmSpaceManager) admin >
..
(local) admin >
cd PoolManager
(PoolManager) admin >
psu create pgroup spacemanager_poolGroup
(PoolManager) admin >
psu addto pgroup spacemanager_poolGroup spacemanager-pool
(PoolManager) admin >
psu removefrom pgroup default spacemanager-pool
(PoolManager) admin >
psu create link spacemanager_WriteLink any-store world-net any-protocol
(PoolManager) admin >
psu set link spacemanager_WriteLink -readpref=10 -writepref=10 -cachepref=0 -p2ppref=-1
(PoolManager) admin >
psu add link spacemanager_WriteLink spacemanager_poolGroup
(PoolManager) admin >
psu create linkGroup spacemanager_WriteLinkGroup
(PoolManager) admin >
psu set linkGroup custodialAllowed spacemanager_WriteLinkGroup true
(PoolManager) admin >
psu set linkGroup replicaAllowed spacemanager_WriteLinkGroup true
(PoolManager) admin >
psu set linkGroup nearlineAllowed spacemanager_WriteLinkGroup true
(PoolManager) admin >
psu set linkGroup onlineAllowed spacemanager_WriteLinkGroup true
(PoolManager) admin >
psu addto linkGroup spacemanager_WriteLinkGroup spacemanager_WriteLink
(PoolManager) admin >
save
(PoolManager) admin >
..
Check whether the link group is available.
(local) admin >
cd SrmSpaceManager
(SrmSpaceManager) admin >
ls
Reservations: total number of reservations: 0 total number of bytes reserved: 0 LinkGroups: 0 Name:spacemanager_WriteLinkGroup FreeSpace:7278624768 ReservedSpace:0 AvailableSpace:7278624768 VOs: onlineAllowed:true nearlineAllowed:false replicaAllowed:true custodialAllowed:true UpdateTime:Mon Nov 28 12:12:51 CET 2011(1322478771030) total number of linkGroups: 1 total number of bytes reservable: 7278624768 total number of bytes reserved : 0 last time all link groups were updated: Mon Nov 28 12:12:51 CET 2011(1322478771030)
The link group
spacemanager_WriteLinkGroup
was created
and has the id 0
.
Now you need to edit the
LinkGroupAuthorization.conf
file. This
file contains a list of the link groups and all the VOs and
the VO Roles that are permitted to make reservations in a
given link group.
Specify the location of the
LinkGroupAuthorization.conf
file in the
/etc/dcache/dcache.conf
file.
spacemanager.authz.link-group-file-name=/path/to/LinkGroupAuthorization.conf
The file LinkGroupAuthorization.conf
has following syntax:
LinkGroup <NameOfLinkGroup> followed by the list of the Fully Qualified Attribute Names (FQANs). Each FQAN on a separate line, followed by an empty line, which is used as a record separator, or by the end of the file.
FQAN is usually a string of the form
<VO>/Role=<VORole>. Both
<VO> and
<VORole> could be set to
*
, in this case all VOs or VO Roles will
be allowed to make reservations in this link group. Any line
that starts with # is a comment and may appear anywhere.
#SpaceManagerLinkGroupAuthorizationFile LinkGroup <NameOfLinkGroup> /<VO>/Role=<VORole>
Note
You do not need to restart the srm
or dCache
after changing the
LinkGroupAuthorization.conf
file. The
changes will be applied automatically after a few minutes.
Use update link groups to be sure that
the LinkGroupAuthorization.conf
file
and the link groups have been updated.
(SrmSpaceManager) admin >
update link groups
update started
Example:
In the example above you created the link group
spacemanager_WriteLinkGroup
. Now you
want to allow members of the VO desy
with the role production
to make a
space reservation in this link group.
#SpaceManagerLinkGroupAuthorizationFile # this is comment and is ignored LinkGroup spacemanager_WriteLinkGroup # /desy/Role=production
Example:
In this more general example for a
SpaceManagerLinkGroupAuthorizationFile
members of the VO desy
with role
test
get the right to make a space
reservation in a link group called
desy-test-LinkGroup
. Moreover, all
members of the VO desy
get the right to
make a reservation in the link group called
desy-anyone-LinkGroup
and anyone will
get the right to make a space reservation in the link
group called default-LinkGroup
.
#SpaceManagerLinkGroupAuthorizationFile # this is comment and is ignored LinkGroup desy-test-LinkGroup /desy/Role=/test LinkGroup desy-anyone-LinkGroup /desy/Role=* LinkGroup default-LinkGroup # allow anyone :-) */Role=*
Example:
Now you can make a space reservation for the VO desy
.
(SrmSpaceManager) admin >
reserve -vog=/desy -vor=production -desc=DESY_TEST 5000000 10000
110000 voGroup:/desy voRole:production retentionPolicy:CUSTODIAL accessLatency:NEARLINE linkGroupId:0 size:5000000 created:Fri Dec 09 12:43:48 CET 2011 lifetime:10000000ms expiration:Fri Dec 09 15:30:28 CET 2011 description:DESY_TEST state:RESERVED used:0 allocated:0
The id of the space token is 110000
.
Check the status of the reservation by
(SrmSpaceManager) admin >
ls
Reservations: 110000 voGroup:/desy voRole:production retentionPolicy:CUSTODIAL accessLatency:NEARLINE linkGroupId:0 size:5000000 created:Fri Dec 09 12:43:48 CET 2011 lifetime:10000000ms expiration:Fri Dec 09 15:30:28 CET 2011 description:DESY_TEST state:RESERVED used:0 allocated:0 total number of reservations: 1 total number of bytes reserved: 5000000 LinkGroups: 0 Name:spacemanager_WriteLinkGroup FreeSpace:23747563520 ReservedSpace:5000000 AvailableSpace:23742563520 VOs:{/desy:*} onlineAllowed:true nearlineAllowed:true replicaAllowed:true custodialAllowed:true UpdateTime:Fri Dec 09 12:49:29 CET 2011(1323431369046) total number of linkGroups: 1 total number of bytes reservable: 23742563520 total number of bytes reserved : 5000000 last time all link groups were updated: Fri Dec 09 12:49:29 CET 2011(1323431369046)
You can now copy a file into that space token.
[user] $
srmcp file:////bin/sh srm://<dcache.example.org>:8443/data/world-writable/space-token-test-file -space_token=110000
Now you can check via the Webadmin Interface or the Web Interface that the
file has been copied to the pool
spacemanager-pool
.
There are several parameters to be specified for a space reservation.
(SrmSpaceManager) admin >
reserve [-vog=voGroup] [-vor=voRole] [-acclat=AccessLatency] \ [-retpol=RetentionPolicy] [-desc=Description] [-lgid=LinkGroupId] [-lg=LinkGroupName] \ <sizeInBytes> <lifetimeInSecs>
- [-vog=voGroup]
voGroup
should match the VO you specified in theLinkGroupAuthorization.conf
file. If you do not want to make a space reservation for a certain VO then the entry in theLinkGroupAuthorization.conf
should readLinkGroup NameOfLinkGroup */Role=*
- [-vor=voRole]
voRole
can be specified if it is used in theLinkGroupAuthorization.conf
file.- [-acclat=AccessLatency]
AccessLatency
needs to match one of the access latencies allowed for the link group.- [-retpol=RetentionPolicy]
RetentionPolicy
needs to match one of the retention policies allowed for the link group.- [-desc=Description]
You can chose a value to describe your space reservation.
- [-lgid=LinkGroupId]
You can either use the
LinkGroupId
to make a space reservation or- [-lg=LinkGroupName]
you use the
LinkGroupName
to make a space reservation.- <sizeInBytes>
The size of the space reservation should be specified in bytes.
- <lifetimeInSecs>
The life time of the space reservation should be specified in seconds. Choose
"-1"
for a space reservation that will never expire (use quotes around the negativeone
).
If a space reservation is not needet anymore it can be released with
(SrmSpaceManager) admin >
release <spaceTokenId>
Example:
(SrmSpaceManager) admin >
reserve -vog=/desy -vor=production -desc=DESY_TEST 5000000 600
110042 voGroup:/desy voRole:production retentionPolicy:CUSTODIAL accessLatency:NEARLINE linkGroupId:0 size:5000000 created:Thu Dec 15 12:00:35 CET 2011 lifetime:600000ms expiration:Thu Dec 15 12:10:35 CET 2011 description:DESY_TEST state:RESERVED used:0 allocated:0(SrmSpaceManager) admin >
release 110042
110042 voGroup:/desy voRole:production retentionPolicy:CUSTODIAL accessLatency:NEARLINE linkGroupId:0 size:5000000 created:Thu Dec 15 12:00:35 CET 2011 lifetime:600000ms expiration:Thu Dec 15 12:10:35 CET 2011 description:DESY_TEST state:RELEASED used:0 allocated:0
You can see that the value for state
has
changed from RESERVED
to
RELEASED
A user who has been given the right to make a space reservation
can make a space reservation. To achieve this the right entry in
the LinkGroupAuthorization.conf
file is
required.
In order to be able to take advantage of the virtual organization (VO) infrastructure and VO based authorization and VO based access control to the space in dCache, certain things need to be in place:
User needs to be registered with the VO.
User needs to use voms-proxy-init to create a VO proxy.
dCache needs to use
gPlazma
with modules that extract VO attributes from the user’s proxy. (See Chapter 10, Authorization in dCache, have a look atgplazmalite-vorole-mapping
plugin and see the section called “Authentication and Authorization in dCache” for an example withgplazmalite-vorole-mapping
.
Only if these 3 conditions are satisfied the VO based
authorization of the SpaceManager
will work.
As mentioned above dCache space
reservation functionality access control is currently
performed at the level of the link groups. Access to making
reservations in each link group is controlled by the
SpaceManagerLinkGroupAuthorizationFile
.
This file contains a list of the link groups and all the VOs and the VO Roles that are permitted to make reservations in a given link group.
When a SRM
Space Reservation request is executed, its
parameters, such as reservation size, lifetime,
AccessLatency
and
RetentionPolicy
as well as user’s VO
membership information is forwarded to the SRM
SpaceManager
.
Once a space reservation is created, no access control is performed, any user can store the files in this space reservation, provided he or she knows the exact space token.
A user who is given the rights in the
SpaceManagerLinkGroupAuthorizationFile
can
make a space reservation by
[user] $
srm-reserve-space -retention_policy=<RetentionPolicy> -lifetime=<lifetimeInSecs> -desired_size=<sizeInBytes> -guaranteed_size=<sizeInBytes> srm://<example.org>:8443
Space token =SpaceTokenId
and release it by
[user] $
srm-release-space srm://<example.org>:8443 -space_token=SpaceTokenId
Note
Please note that it is obligatory to specify the retention policy while it is optional to specify the access latency.
Example:
[user] $
srm-reserve-space -retention_policy=REPLICA -lifetime=300 -desired_size=5500000 -guaranteed_size=5500000 srm://srm.example.org:8443
Space token =110044
The space reservation can be released by:
[user] $
srm-release-space srm://srm.example.org:8443 -space_token=110044
If a client uses a regular grid proxy, created with
grid-proxy-init, and not a VO proxy, which
is created with the voms-proxy-init, when
it is communicating with SRM
server in dCache, then the VO
attributes can not be extracted from its credential. In this
case the name of the user is extracted from the Distinguished
Name (DN) to use name mapping. For the purposes of the space
reservation the name of the user as mapped by gplazma
is used as its VO Group name, and the VO Role is left
empty. The entry in the
SpaceManagerLinkGroupAuthorizationFile
should be:
#LinkGroupAuthorizationFile # <userName>
Edit the file /etc/dcache/dcache.conf
to enable space
reservation for non SRM
transfers.
spacemanager.enable.reserve-space-for-non-srm-tranfers=true
If the spacemanager
is enabled,
spacemanager.enable.reserve-space-for-non-srm-tranfers
is set to true
, and if the transfer
request comes from a door, and there was no prior space
reservation made for this file, the SpaceManager
will
try to reserve space before satisfying the request.
Possible values are true
or
false
and the default value is
false
.
There are a few parameters in
/usr/share/dcache/defaults/*.properties
that you might find useful for nontrivial SRM
deployment.
dcache.enable.space-reservation
tells if the space
management is activated in SRM
.
Possible values are true
and
false
. Default is true
.
Usage example:
dcache.enable.space-reservation=true
srm.enable.space-reservation.implicit
tells if
the space should be reserved for SRM
Version 1 transfers and
for SRM
Version 2 transfers that have no space token
specified.
Possible values are true
and
false
. This is enabled by default. It has no
effect if dcache.enable.space-reservation
is set to
true
.
Usage example:
srm.enable.space-reservation.implicit=true
dcache.enable.overwrite
tells SRM
and
GridFTP
servers if the overwrite is allowed. If enabled on
the SRM
node, should be enabled on all GridFTP
nodes.
Possible values are true
and
false
. Default is false
.
Usage example:
dcache.enable.overwrite=true
srm.enable.overwrite-by-default
Set this to
true
if you want overwrite to be enabled
for SRM
v1.1 interface as well as for SRM
v2.2 interface
when client does not specify desired overwrite mode. This
option will be considered only if
dcache.enable.overwrite
is set to
true
.
Possible values are true
and
false
. Default is false
.
Usage example:
srm.enable.overwrite-by-default=false
srm.db.host
tells SRM
which
database host to connect to.
Default value is localhost
.
Usage example:
srm.db.host=database-host.example.org
spaceManagerDatabaseHost
tells SpaceManager
which database host to connect to.
Default value is localhost
.
Usage example:
spaceManagerDatabaseHost=database-host.example.org
pinmanager.db.host
tells PinManager
which database host to connect to.
Default value is localhost
.
Usage example:
pinmanager.db.host=database-host.example.org
srm.db.name
tells SRM
which database
to connect to.
Default value is dcache
.
Usage example:
srm.db.name=dcache
srm.db.user
tells SRM
which database
user name to use when connecting to database. Do not change
unless you know what you are doing.
Default value is srmdcache
.
Usage example:
srm.db.user=srmdcache
srm.db.password
tells SRM
which
database password to use when connecting to database. The
default value is srmdcache
.
Usage example:
srm.db.password=NotVerySecret
srm.db.password.file
tells SRM
which
database password file to use when connecting to database. Do
not change unless you know what you are doing. It is
recommended that MD5 authentication method is used. To learn
about file format please see http://www.postgresql.org/docs/8.1/static/libpq-pgpass.html. To
learn more about authentication methods please visit http://www.postgresql.org/docs/8.1/static/encryption-options.html,
Please read "Encrypting Passwords Across A Network" section.
This option is not set by default.
Usage example:
srm.db.password.file=/root/.pgpass
srm.request.enable.history-database
enables
logging of the transition history of the SRM
request in the
database. The request transitions can be examined through the
command line interface. Activation of this option might lead
to the increase of the database activity, so if the PostgreSQL
load generated by SRM
is excessive, disable it.
Possible values are true
and
false
. Default is false
.
Usage example:
srm.request.enable.history-database=true
transfermanagers.enable.log-to-database
tells SRM
to store
the information about the remote (copy, srmCopy) transfer
details in the database. Activation of this option might lead to
the increase of the database activity, so if the PostgreSQL load
generated by SRM
is excessive, disable it.
Possible values are true
and
false
. Default is false
.
Usage example:
transfermanagers.enable.log-to-database=false
srmVersion
is not used by SRM
; it was
mentioned that this value is used by some publishing scripts.
Default is version1
.
srm.root
tells SRM
what the root
of all SRM
paths is in pnfs. SRM
will prepend path to all
the local SURL paths passed to it by SRM
client. So if the
srm.root
is set to
/pnfs/fnal.gov/THISISTHEPNFSSRMPATH
and
someone requests the read of
srm://srm.example.org:8443/file1
, SRM
will
translate the SURL path /file1
into
/pnfs/fnal.gov/THISISTHEPNFSSRMPATH/file1
. Setting
this variable to something different from /
is equivalent of performing Unix chroot for
all SRM
operations.
Default value is /
.
Usage example:
srm.root="/pnfs/fnal.gov/data/experiment"
srm.limits.parallel-streams
specifies the number of the
parallel streams that SRM
will use when performing third
party transfers between this system and remote GSI-FTP
servers, in response to SRM
v1.1 copy or SRM
V2.2 srmCopy
function. This will have no effect on srmPrepareToPut and
srmPrepareToGet command results and parameters of GridFTP
transfers driven by the SRM
clients.
Default value is 10
.
Usage example:
srm.limits.parallel-streams=20
srm.limits.transfer-buffer.size
specifies the number of bytes
to use for the in memory buffers for performing third party
transfers between this system and remote GSI-FTP
servers, in
response to SRM
v1.1 copy or SRM
V2.2 srmCopy
function. This will have no effect on srmPrepareToPut and
srmPrepareToGet command results and parameters of GridFTP
transfers driven by the SRM
clients.
Default value is 1048576
.
Usage example:
srm.limits.transfer-buffer.size=1048576
srm.limits.transfer-tcp-buffer.size
specifies the number of
bytes to use for the tcp buffers for performing third party
transfers between this system and remote GSI-FTP
servers, in
response to SRM
v1.1 copy or SRM
V2.2 srmCopy
function. This will have no effect on srmPrepareToPut and
srmPrepareToGet command results and parameters of GridFTP
transfers driven by the SRM
clients.
Default value is 1048576
.
Usage example:
srm.limits.transfer-tcp-buffer.size=1048576
srm.service.gplazma.cache.timeout
specifies the
duration that authorizations will be cached. Caching decreases
the volume of messages to the gPlazma
cell or other
authorization mechanism. To turn off caching, set the value
to 0
.
Default value is 120
.
Usage example:
srm.service.gplazma.cache.timeout=60
srm.limits.request.bring-online.lifetime
,
srm.limits.request.put.lifetime
and
srm.limits.request.copy.lifetime
specify the lifetimes of
the srmPrepareToGet (srmBringOnline) srmPrepareToPut and
srmCopy requests lifetimes in millisecond. If the system is
unable to fulfill the requests before the request lifetimes
expire, the requests are automatically garbage collected.
Default value is 14400000
(4 hours)
Usage example:
srm.limits.request.bring-online.lifetime=14400000 srm.limits.request.put.lifetime=14400000 srm.limits.request.copy.lifetime=14400000
srm.limits.request.scheduler.ready.max
and
srm.limits.request.put.scheduler.ready.max
specify the
maximum number of the files for which the transfer URLs will
be computed and given to the users in response to SRM
get
(srmPrepareToGet) and put (srmPrepareToPut) requests. The rest
of the files that are ready to be transfered are put on the
Ready
queues, the maximum length of these
queues are controlled by
srm.limits.request.scheduler.ready-queue.size
and
srm.limits.request.put.scheduler.ready-queue.size
parameters. These
parameters should be set according to the capacity of the
system, and are usually greater than the maximum number of the
GridFTP
transfers that this dCache instance GridFTP
doors can sustain.
Usage example:
srm.limits.request.scheduler.ready-queue.size=10000 srm.limits.request.scheduler.ready.max=2000 srm.limits.request.put.scheduler.ready-queue.size=10000 srm.limits.request.put.scheduler.ready.max=1000
srm.limits.request.copy.scheduler.thread.pool.size
and
transfermanagers.limits.external-transfers
.
srm.limits.request.copy.scheduler.thread.pool.size
is used to specify how many parallel
srmCopy file copies to execute simultaneously. Once the
SRM
contacted the remote SRM
system, and obtained a Transfer
URL (usually GSI-FTP
URL), it contacts a Copy Manager
module (usually RemoteGSIFTPTransferManager
), and asks it to
perform a GridFTP
transfer between the remote GridFTP
server
and a dCache pool. The maximum number of simultaneous
transfers that RemoteGSIFTPTransferManager
will support is
transfermanagers.limits.external-transfers
, therefore it is important that
transfermanagers.limits.external-transfers
is greater than or equal to
srm.limits.request.copy.scheduler.thread.pool.size
.
Usage example:
srm.limits.request.copy.scheduler.thread.pool.size=250 transfermanagers.limits.external-transfers=260
srm.enable.custom-get-host-by-address
srm.enable.custom-get-host-by-address
enables using the BNL developed
procedure for host by IP resolution if standard InetAddress
method failed.
Usage example:
srm.enable.custom-get-host-by-address=true
srm.enable.recursive-directory-creation
allows or
disallows automatic creation of directories via SRM
. Set this to true
or false
.
Automatic directory creation is allowed by default.
Usage example:
srm.enable.recursive-directory-creation=true
This option allows you to control how often the SRM
door will reload the
server’s host certificate from the filesystem. For the specified period,
the host certificate will be kept in memory. This speeds up the rate at
which the door can handle requests, but also causes it to be unaware of
changes to the host certificate (for instance in the case of renewal).
By changing this parameter you can control how long the host certificate is cached by the door and consequently how fast the door will be able to detect and reload a renewed host certificate.
Please note that the value of this parameter has to be specified in seconds.
Usage example:
hostCertificateRefreshPeriod=86400
The trustAnchorRefreshPeriod
option is similar to
hostCertificateRefreshPeriod
. It applies to the set
of CA certificates trusted by the SRM
door for signing end-entity
certificates (along with some metadata, these form so called
trust anchors).
The trust anchors are needed to make a decision about
the trustworthiness of a certificate in X.509 client
authentication. The GSI
security protocol used by SRM
builds upon
X.509 client authentication.
By changing this parameter you can control how long the set of trust anchors remains cached by the door. Conversely, it also influences how often the door reloads the set of trusted certificates.
Please note that the value of this parameter has to be specified in seconds.
Tip
Trust-anchors usually change more often than the host certificate. Thus, it might be sensible to set the refresh period of the trust anchors lower than the refresh period of the host certificate.
Usage example:
trustAnchorRefreshPeriod=3600